AI assistance disclosure: I designed and directed the experiments described here. I used AI coding agents to help implement the experiment scaffolds under tests and review. OpenAI Codex helped inspect the saved reports, verify the numbers, and draft and edit this article. The match logs and API receipts came from the experiment runtime, not the writing assistant. I remain responsible for the claims and errors.
I recently added a private planner to two LLM players in a repeated Prisoner’s Dilemma game.
The first result looked perfect.
Both mirrored matches followed the same arc:
Rounds 1–6: mutual cooperation
Round 7: one player defects and collects
Round 8: mutual defection protects the new lead
The private records showed that the betrayal had been planned before the action. No illegal moves. No fallback model. Two successes out of two.
Then I changed the seeds.
After rebuilding the planner around a strict JSON Schema, I ran a clean comparison over hidden game lengths from five to nine rounds, mirroring every seed across seats.
| Metric | No planner | Structured planner |
|---|---|---|
| Matches | 10 | 10 |
| Clean trust-and-betrayal arcs | 3 | 1 |
| Early mutual-defection collapses | 2 | 5 |
| Post-betrayal locks | 3 | 1 |
| Model calls | 280 | 372 |
| Actual API cost | $0.154641 | $0.300639 |
| Fallbacks / decision errors / planner faults | 0 / 0 / 0 | 0 / 0 / 0 |
The structured planner was reliable at producing its schema. It did not improve the game trajectories, and it cost 1.94× as much.
This is a small result: ten matches per condition, one model family, and one repeated game. It is not evidence that planning generally makes LLMs worse. It is evidence that a convincing two-game demo was not enough.
Here are the seven checks I now use before trusting an LLM planner experiment.
1. Mirrored Seats Are a Control, Not a Replication
My pilot used one seed and played it twice with the seats swapped.
That controlled for seat advantage. It did not add a new match length, strategic history, or random condition. Both games still sampled the same hidden total of eight rounds.
I had two seat arms, not two independent environments.
The fix was to stratify the next test by hidden match length:
const conditions = [
{ rounds: 5, seed: 2201 },
{ rounds: 6, seed: 2205 },
{ rounds: 7, seed: 2200 },
{ rounds: 8, seed: 2203 },
{ rounds: 9, seed: 2202 },
];
for (const condition of conditions) {
run(condition, { seats: ["left", "right"] });
run(condition, { seats: ["right", "left"] });
}
Mirroring answers “did the result follow a seat?” Stratification answers “did the result survive a changed condition?” I need both.
2. Treat Hidden Environment Variables as Experimental Variables
The total round count was hidden from the players, but it still shaped the game.
A plan that happens to trigger on round seven can look brilliant in an eight-round match. The same trigger may be too late in a five-round match or produce two rounds of mutual defection in a nine-round match.
If the environment knows a value—even when the agents do not—that value can still confound the result.
For game agents, this includes:
- match length;
- seat order;
- initial resources;
- opponent identity;
- information distribution;
- whether agents share the same model and prompt;
- whether the episode terminates immediately after a “successful” move.
I now write these into the experiment plan instead of leaving them inside the RNG.
3. Separate Protocol Validity From Decision Quality
My first cross-seed planner batch looked dramatically worse than the baseline, but it also contained planner-format faults.
The games still completed because the actor could continue after a bad planning response. That behavior is useful in production and dangerous in evaluation.
I rejected the faulty planner arms as final evidence and rebuilt the planner with a smaller strict schema:
{
"opponentEvidence": ["..."],
"candidates": [
{ "plan": "...", "risk": "..." },
{ "plan": "...", "risk": "..." }
],
"selected": 0,
"invalidateWhen": "..."
}
The clean rerun had:
- zero fallback decisions;
- zero actor errors;
- zero planner-format faults;
- a non-zero API cost receipt in every match.
Only then did I compare trajectory quality.
Structured output can guarantee that a plan has the right fields. It cannot guarantee that the plan is good.
4. Fail Closed on Silent Recovery
My validity gate keeps production recovery separate from experiment evidence:
function validity(matches) {
return {
valid:
sum(matches, "fallbackDecisions") === 0 &&
sum(matches, "decisionErrors") === 0 &&
sum(matches, "plannerFormatFaults") === 0 &&
countApiReceipts(matches) > 0,
};
}
I also inspect the receipt count per match. In the clean planner batch it was 10/10.
This caught a particularly misleading failure mode: a sandbox network failure once produced completed games made entirely from default actions, with zero API cost. Without fail-closed accounting, those games would have looked like successful LLM samples.
A terminal state proves the runtime survived. It does not prove the intended model participated correctly.
5. Measure Trajectories, Not Highlight Moves
At first I counted any unilateral betrayal after cooperation as a planned collection.
That metric rewarded a last-round betrayal even though the opponent had no opportunity to respond. So I replaced the single-event metric with mutually exclusive trajectory shapes:
- clean collection: at least two cooperative setup rounds, a unilateral betrayal, then a later defensive defection from the collector;
- terminal collection: the betrayal occurs on the final round;
- countered collection: both players defect when collection is attempted;
- early collapse: mutual defection begins within the first three rounds and continues to the end;
- standoff: no player successfully collects from a cooperating opponent;
- other: none of the above.
The distinction changed the interpretation of older runs. Some “successful” strategies were merely final-round defections. They had a payoff event, not a demonstrated response cycle.
For sequential agents, the unit of quality is often a trajectory, not an isolated action.
6. Put Cost Next to Every Quality Claim
The clean structured planner used 372 calls instead of 280 and cost $0.300639 instead of $0.154641.
Those are API usage receipts, not token-price estimates.
A planner adds cost even when its recommendation is rejected, redundant, or strategically harmful. Reporting only the best clip hides the intervention’s actual product behavior: it runs on every trigger, not only on the trigger that creates a good story.
My result table now keeps these columns together:
trajectory distribution | fallback count | planner faults | calls | API receipts
“It sometimes helps” is not enough when “it always costs more” is also true.
7. Keep “Can Happen” Separate From “Usually Helps”
The pilot was not fake.
The players’ private reasoning records were committed before their actions and revealed at the end. The planner really did describe the cooperation phase as setup and identify a later collection condition.
So the pilot supports this claim:
An LLM player can form and execute a multi-round trust-and-betrayal plan in this environment.
The replication does not support this stronger claim:
Adding this planner reliably improves repeated LLM-vs-LLM games.
That difference is easy to erase when selecting demos.
I now label results using three different verbs:
- demonstrated: the behavior occurred in a valid sample;
- replicated: it survived changed seeds or conditions;
- improved: it beat a defined baseline on a valid comparison.
The planner demonstrated the behavior. It did not establish a reliable improvement.
What the Experiment Does—and Does Not—Show
In the clean ten-match comparison, the planner arm produced fewer clean arcs, more early collapses, fewer protected leads, and nearly twice the API cost.
One possible explanation is symmetric pessimism: when both same-model players receive a planner that anticipates exploitation, both may reach mutual defection faster. But the current data does not isolate that mechanism.
Other explanations remain open:
- the planner prompt may overemphasize exploitation;
- same-model play may collapse differently from heterogeneous opponents;
- the state representation may omit evidence needed to repair trust;
- ten matches per condition are not enough for a universal claim.
The next experiment should change the interaction distribution—heterogeneous opponents, human players, or outcome-checked relationship memory—instead of adding another planner prompt to the same symmetric setup.
Checklist
Before trusting a planner experiment, I now ask:
- [ ] Did I vary seeds, not only seats?
- [ ] Did I stratify environment variables that shape the trajectory?
- [ ] Are protocol validity and behavior quality reported separately?
- [ ] Are fallback, repair, retry, and default-action samples visible?
- [ ] Does the metric evaluate a trajectory rather than a highlight move?
- [ ] Are actual calls and API receipts next to the quality result?
- [ ] Am I claiming “can happen,” “replicated,” or “improved”?
The mistake was not that the first two games looked good.
The mistake would have been stopping there.
References
- OpenAI: Introducing Structured Outputs in the API
- Meta AI: CICERO
- Park et al.: Generative Agents: Interactive Simulacra of Human Behavior
Top comments (1)
Check 4 lands hardest for me. I do on-device benchmarking, and the same failure there is the GPU delegate silently falling back to CPU: the run finishes, the numbers look plausible, and you measured the wrong pipeline. Asserting the delegate per run and throwing away runs without the assertion is the same move as counting your API receipts.