Syndicated from the original on lkforge.com. The engines are live in your browser at lkforge.com/games; the full harness is in this gist.
My browser board-game opponents — Connect 4, Checkers, Othello and Chess — all think the same way: negamax with alpha-beta pruning, looking a fixed number of moves ahead. The single knob that makes them stronger is search depth. So how much is one more ply actually worth? I ran 2,400 headless self-play games to find out, and the honest answer is: it depends enormously on which ply.
The counterintuitive result
I expected a tidy law — "each ply is worth some steady fraction of the last." I got nothing of the sort.
The very first extra ply is worth a fortune — up to +953 Elo in Connect 4 — because a depth-1 engine barely beats random. After that the gains fall off a cliff and stop behaving: each further ply adds a small, game-specific, non-monotonic amount. There is no single clean curve.
| Engine | Depths tested | First ply (1→2) | Each later ply |
|---|---|---|---|
| Connect 4 | 1–7 | +953 | +9 to +55 |
| Othello | 1–6 | +610 | +107 to +229 |
| Checkers | 1–7 | +546 | +49 to +100 |
| Chess | 1–4 | +321 | +225 to +315 |
In 3 of the 4 games, the first ply is the single biggest gain — by a mile.
Why depth, not milliseconds
The live game gives its AI a time budget so the page stays responsive. But a time budget is the wrong unit for a benchmark: "half a second of thinking" buys a fast laptop far more nodes than a phone, so a strength number tied to milliseconds says more about the reader's hardware than the engine.
A fixed search depth is the hardware-independent knob. Pin the depth and the same seed produces the same games on any machine — which is the whole point of publishing a number someone else can check.
Why the first ply does all the work
A depth-1 search only avoids one-move blunders and otherwise plays close to random. The jump to depth 2 turns a coin-flip opponent into a real one — so that ply captures most of the available strength in one step. After that, each additional ply is chasing rarer and rarer mistakes while costing exponentially more nodes, and how much it finds depends entirely on the game.
Chess is the exception that proves the point. Its first two plies are almost equal (+321 then +315) rather than front-loaded, because its branching factor is so large that even a depth-2 search is still missing a great deal — there's plenty left for the next ply to find.
Connect 4 is the opposite extreme: a small, near-solved game where, once past the blunder-avoiding first ply, extra depth barely moves the result (+9 to +55) until the search can see a whole forced line. Its later plies also zig-zag — a real parity effect, because whether the search horizon ends on your move or the opponent's changes what a shallow search sees, so odd and even depths behave differently.
The method
- Each engine's shipped
search()is deterministic, so every game starts from a random opening (a handful of random legal moves), then the two depth-limited engines play it out. - Every opening is played twice with the sides swapped, so any imbalance in the random start cancels exactly. Draws score half a point.
- 120 colour-balanced games per adjacent depth pair. Depth — not milliseconds — is the knob, so numbers are hardware-independent and reproduce from a seed (mulberry32, master seed 20260819).
- Elo per step =
400 · log10(p / (1 − p))from the colour-balanced scorep.
(Draw rates run high in Checkers and Chess, so read those Elo gaps as approximate. The figures are internal — each engine against weaker copies of itself, not against human ranks or a program like Stockfish.)
The contrast with MCTS
The companion study benchmarks my Go engine, which uses Monte Carlo Tree Search instead of negamax. There, more search bought a smooth, even growing return — each doubling of playouts added more Elo than the last, no diminishing returns out to 1,600 playouts.
Depth-limited negamax has no such curve: it's front-loaded and game-specific. Same goal — pick the strongest move — two search paradigms, two completely different shapes of return on compute. That's the fun of measuring instead of assuming.
Full harness, four vendored engine snapshots, and reproduce steps: gist · original writeup with charts: lkforge.com/blog/search-depth-scaling · the Go/MCTS companion: Does thinking twice as long make a Go AI twice as good? · play the engines: lkforge.com/games.
Top comments (0)