№ 03 · Entry III

Teaching a network the family card game.

Zheng Shang You is a 4-player climbing game — shed your hand first, beat the last combination or pass, bombs outrank everything. The rules took a week. Getting a gradient to point anywhere took months, and the training log keeps every wrong turn.

pytorchppo + bcleague self-playrust engine

§ I — The dead end

Self-play with zero signal

The obvious approach — DouZero-style deep Monte Carlo self-play — produced a result worth framing: every network scored ~33% regardless of quality. In symmetric multiplayer self-play, when all seats improve together, relative win rate is flat and the gradient carries no information. The training log records it, dates it, and moves on.

What replaced it is a recipe, and each step is measurable: clone the behaviour of a scripted strategist to warm-start; fine-tune with PPO regularized back toward the clone; give the critic perfect information (it sees all hands during training — the policy never does) for a lower-variance baseline; shape rewards with a dynamic-programming oracle for minimum-steps-to-shed, 2.6ms per game, 90% cache hit rate.

§ II — Figure I

The recipe, one fix at a time

Win rate in a 4-player ring against three scripted strategists. Seat parity is 0.25 — anything above it is real skill.

FIG. I — From cloning to search

Fig. I — every step is one dated fix from the training log. The last one — determinized search on top of the policy — is worth ten points on its own.

The search deserves its own footnote. Rolling out D=6 determinized worlds and scoring win/lossgave zero lift. The fix wasn’t depth: score rollouts by final rank (1.0 / 0.45 / 0.225 / 0) and D=32 is worth +10 points. The estimator’s variance was the problem all along — a very card-game lesson, where second place and fourth place look identical to a win/loss signal.

§ III — Figure II

Generations, honestly

Training win rate by iteration across the league runs — including the ones that went nowhere.

FIG. II — League self-play runs

Fig. II — gen-1 reached 0.471 (1,000 games, CI [0.44, 0.50]); gen-2 beats three gen-1s at 0.409 over 450 games; gen-3 is currently flat, and the log says so.

Each generation trains against a mixture of the frozen previous champion, past snapshots, and itself — pure fixed-opponent training plateaus, pure self-play collapses, the league keeps the gradient pointed at the actual target. Gen-3 is the live negative result: five checkpoints of flat proxy matchups against a same-class anchor, and an open diagnosis — the 512×4 policy class may simply be saturated.

Supporting cast: a PyO3 Rust engine that simulates a game in 0.15ms (14× Python, but only once whole workloads were batched across the boundary), and a behaviour-cloning pipeline that OOM-killed the desktop — wildcard hands can have thousands of legal moves — until cross-entropy against 24 sampled distractors bounded the memory at equal quality.

§ IV — The rough edges