Every week someone posts "I switched from Claude Code to Codex" (or the reverse), and the evidence is always the same: vibes.
We run a small AI team — a personal AI secretary system with several agents — and we got tired of vibes. So we started running same-task battles: both agents get the identical task, the identical prompt, and the identical input. Then we score them against an answer key we wrote before the battle started.
Here is exactly how it works, with a worked example.
The four fairness rules
- One prompt. Both agents receive the same instruction text, character for character. No tuning the prompt to each model's strengths.
- Golden answer first. Before either agent runs, we write the ideal output (or the checklist it must satisfy). If you write the answer key afterwards, you will unconsciously grade toward whoever went first.
- Mechanical scoring. A 100-point rubric (below). Anything that can't be scored mechanically gets cut from the rubric.
- Rework counts. If an agent's output needs fixing, the fix is part of its score — not a footnote. "It got there eventually" is a cost, and costs belong in the result.
The rubric
| Category | Points | What we check |
|---|---|---|
| Accuracy | 40 | Matches the golden answer: facts, figures, required items |
| Rework | 20 | Deductions for every correction a human had to make |
| Speed | 15 | Wall-clock time, scored relative to the faster agent |
| Quality | 15 | Structure, readability, formatting as specified |
| Instruction-following | 10 | Did exactly what the prompt said — no more, no less |
A worked example (sample data)
Task: "Turn these meeting notes into a decision summary: decisions, owners, deadlines. Nothing else."
The (fictional) notes mention a launch date of March 3, two owners, one deadline — plus a long digression about office snacks. The golden answer lists exactly the three decisions, both owners, and the deadline.
What typically happens:
- Agent A returns a clean summary, correct date, both owners — but adds a "Risks" section nobody asked for. Accuracy 40, instruction-following −4.
- Agent B returns exactly the requested format, but writes the deadline as March 13. Accuracy −10 for the wrong date, no matter how pretty the formatting is.
Final scores might land at 96 vs 89 — and crucially, you can point at why. The score sheet survives the argument.
Our series so far
Real battles, same rules:
- Battle 1 — meeting-notes cleanup (writing task): Claude Code 99, Codex 92. Nearly perfect on accuracy; the gap came from rework.
- Battle 2 — CSV to monthly aggregation (implementation task): Codex 96, Claude Code 93. The rematch flipped the winner. Task type matters more than brand loyalty.
Series record: 1–1. The honest conclusion after two battles is that "which agent is better" is the wrong question. "Which agent is better at this task type, under this rubric" is one you can actually answer — and route work with.
Try it yourself
We packaged the whole method — the shared prompt template, the 100-point rubric with the scoring rules for speed and rework, and a how-to for building golden answer keys — as a small kit:
Same-Task Battle Kit on Gumroad ($5)
It's the exact files we use for the series, not a summary of them. Run one battle with your own task, and you'll know more about your agents than a month of vibes will tell you.
We write about running a personal AI secretary and its agent team — the illustrated series is on note (English edition: kajuiwa_en) and Medium.
Top comments (0)