"Which AI coding agent is better?" usually gets answered with vibes. I wanted numbers, so I gave Claude Code and OpenAI Codex the same starter repo, the same 396-word prompt and the same 20 automated checks, and recorded everything.
The app: TaskPilot, a full-stack task manager with a FastAPI backend, a React frontend, login, per-user data, search, filters, a dashboard and a mobile layout.
The setup
| Claude Code | OpenAI Codex | |
|---|---|---|
| Version | Claude Code 2.1.266 | Codex CLI 0.147.0 |
| Model | Claude Opus 5 | GPT-5.6-Sol |
| Mode | headless -p, effort high, no approval prompts |
exec, effort medium, no approval prompts |
The rules, fixed before either run:
- Same starter commit (26 files), same prompt (396 words, hash recorded before the runs).
- 45 minutes maximum for the first build.
- Up to 2 repair rounds, 10 minutes each. A repair prompt only lists which checks failed.
- No human code edits. The only human action was committing the working tree after each session ended.
- A coin flip decided who went first (Codex).
- The evaluator, 20 browser and API checks, was frozen before either agent ran.
The 20 checks
These are the things that break in real apps, not "does it compile":
- Auth (C01–C04): valid login, a useful error on bad credentials, protected API routes deny anonymous requests, and logout really invalidates the session (replaying the old cookies must fail).
- Data isolation (C05–C06): user A can't list, read, update or delete user B's tasks, even with direct API calls.
- Tasks (C07–C11): create, edit, mark done, and delete with confirmation. Whitespace-only and over-length titles are rejected in both the UI and the API.
- Search and filters (C12–C13): case-insensitive search across title and description, and filters that combine and reset.
- Dashboard (C14–C15): counts match the database, including overdue edge cases (yesterday, today, future, no date).
- Persistence and errors (C16–C17): data survives a refresh and a backend restart, and empty states and API failures make sense.
- Mobile (C18): sign in, create a task, search and sign out at 390 × 844 with no horizontal scrolling.
- Keyboard and validation (C19–C20): the core flow works with the keyboard only, and invalid dates, status or priority values are rejected.
Results
| Claude Code | OpenAI Codex | |
|---|---|---|
| First build | 20/20 | 19/20 |
| Build time | 11m 46s | 9m 36s |
| Failed check | none | C18 (mobile, 390 × 844) |
| Repair rounds | 0 | 1 (took 41 seconds) |
| Final score | 20/20 | 20/20 |
| Human follow-up messages | 0 | 1 (the repair prompt) |
| Human code edits | 0 | 0 |
| Tokens in / cached / out | 2.38M / 2.26M / 61.6K | 1.82M / 1.75M / 27.2K |
| Estimated API-equivalent cost | $3.79 | $1.54 |
Both runs used subscriptions, so neither cost was actually charged. The dollar figures are what the same tokens would cost at each model's list API price on the test date.
What actually separated them
Claude Code was right the first time. It passed all 20 checks on its first build with no repair round. It also checked its own work: at about 4 minutes, its backend tests had 2 failures out of 39. It said it was fixing a test-data bug and empty filter parameters, and all 39 passed at 4:26. Near the end it took desktop and mobile screenshots and made one CSS fix to the mobile layout before finishing.
Codex was faster and cheaper, and it missed mobile. Its first build failed only C18, where the core flow at 390 px wide didn't fit without horizontal scrolling. Given the failed check, it fixed it in 41 seconds. It also caught one of its own bugs mid-run: a backend test expected "game" to match "gaming", and it fixed the test.
The usability review (scored out of 5 after both builds were frozen):
| Claude Code | Codex | |
|---|---|---|
| Navigation | 5: every row action is a visible text button | 4: edit, delete and done are icon-only (screen-reader labelled) |
| Task entry | 5 | 5 |
| Readability | 5 | 5 |
| Feedback | 5 | 5 |
| Mobile | 5: all dashboard counts visible at once | 4: counts become a swipe carousel, header buttons go icon-only |
So which should you use?
From this one test:
- Use Claude Code when you want the first build to be right, mobile and accessibility details included, and you'd rather pay more tokens than review a repair.
- Use Codex when speed and cost matter and you have good tests. A failing check plus a 41-second repair got it to the same 20/20 at less than half the estimated cost.
Either way, the 20 checks did the real work. Without them, "it looks done" would have hidden a mobile layout bug in one app, and you'd never see the auth and data-isolation checks pass or fail. Write the checks before you hand the task to an agent.
Fine print
This is one controlled run per tool, not proof that one agent always wins. AI agents vary between runs. The Claude Code result is from a fresh retry, chosen after two usage-limit interruptions in earlier attempts. Same starter, same prompt, same rules. Every session is recorded in the video, with timestamps, logs and the final apps.
Have you run your own head-to-head? I'd like to compare notes in the comments.
Drafted with AI assistance. All numbers come from the recorded test logs and the frozen evaluator results.
Top comments (3)
This is an exceptional, production-grade benchmark. Evaluating coding agents through a rigorous, frozen test suite—specifically checking operational requirements like data isolation, session invalidation, and strict API-level validation bounds rather than simple code compilation—is the only way to measure actual agent reliability.
The comparison highlights two distinct behavioral profiles under test:
Thanks Oleksandr, really appreciate the careful read. C16–C17 check that data survives a backend restart and that API failures show sensible errors, but nothing injects faults during a write. A round with dropped DB connections mid-transaction and random latency spikes would show whether each agent actually rolls back cleanly. Good idea for the next comparison.
Dropping DB connections mid-transaction is exactly where most AI agents break down in production. I’ll definitely build out a chaos engineering round for the next benchmark to see how they handle clean rollbacks!