Current coding agents are good enough that "just describe the ticket" usually works. So is writing a spec still worth the time? We stopped arguing about it and recorded runs instead.
We gave Claude Code three ordinary tickets on small fixture repos and ran each one several times on four models: Opus 5.5, Sonnet 5.5, Haiku 4.5 and Fable 5.1. Every ticket was run from a one-line prompt and from a spec. Two of them were also run with the repo's written rules deleted. That came to 78 runs and about $14.84 of API-equivalent cost.
The short version: with a spec, 28 of 30 runs passed every hidden check. Without one, 12 of 48 did.
The setup
-
Agent: Claude Code 2.1.284, headless (
claude -p), one prompt per run, no follow-up messages. -
Isolation: a fresh copy of the fixture repo for every run, with no user settings, hooks, plugins or MCP servers (
--setting-sources project --strict-mcp-config). - Scoring: each case has a hidden acceptance suite, written and frozen before the first run and never shown to the agent. The suites only call interfaces that existed before the change, and they accept any reasonable way of reporting an error, so a different API design doesn't count as a failure.
- No cleanup: nobody touched the agent's output before it was scored.
The three tickets:
- Checkout coupon codes: "Add coupon codes to checkout."
-
API error envelope: "Clean up our API errors." The repo has a mobile client in the field that reads
body.erroras a string. -
Split-name migration: "Split name into first_name and last_name." A billing sync outside the repo still reads
users.name.
Results
Each cell counts runs that passed every hidden check, out of runs recorded. "No docs" and "no rules" mean the repo's written constraints were removed. Fable 5.1 got one run per variant and skipped the no-docs variants, so read its column as a spot check.
| Case and variant | Opus 5.5 | Sonnet 5.5 | Haiku 4.5 | Fable 5.1 |
|---|---|---|---|---|
| Coupon, vague | 2/3 | 3/3 | 0/3 | 1/1 |
| Coupon, spec | 3/3 | 3/3 | 3/3 | 1/1 |
| API errors, no docs | 0/3 | 0/3 | 0/3 | not run |
| API errors, docs | 0/3 | 0/3 | 0/3 | 0/1 |
| API errors, spec | 3/3 | 3/3 | 3/3 | 1/1 |
| Migration, no rules | 2/3 | 0/3 | 0/3 | not run |
| Migration, rules | 3/3 | 1/3 | 0/3 | 0/1 |
| Migration, spec | 3/3 | 3/3 | 1/3 | 1/1 |
| All spec runs | 9/9 | 9/9 | 7/9 | 3/3 |
| All non-spec runs | 7/15 | 4/15 | 0/15 | 1/3 |
"Passed every check" is a strict bar. Most of the API "docs" runs failed exactly one check: they didn't use a single error shape across every endpoint.
What actually went wrong without a spec
The failures weren't crashes. The code ran, and the agent's own tests passed. What went wrong was a decision nobody asked for:
- Coupons: one Opus run decided coupons have no usage limit, and said so in its summary. The three vague runs split three different ways on how to charge a lapsed coupon.
-
API errors: without the contract doc, two of three Opus runs changed
body.errorfrom a string to an object. It's a well-designed envelope, and it shows every user on mobile v3.x a blank error banner. A 13-line contract doc was enough to stop that change. -
Migration: one Opus run wrote a correct backfill and then ran
DROP COLUMN name, with a warning in the summary that nobody would act on. All six of its non-spec runs split names at the first space.
Four things we didn't expect
1. A spec makes the models nearly interchangeable. With a spec, the smallest and largest models finished within two runs of each other. Without one, results ranged from 7 of 15 down to 0 of 15. On a risky ticket, the spec moved the result more than the choice of model did.
2. Benchmark order isn't safety order. Anthropic reports Sonnet 5.5 at 70.6% on Terminal-Bench 4.0, above Opus 5.5 at 66.4%. On the migration with no written rules, Sonnet 5.5 dropped users.name in all three runs, and Opus 5.5 did it once. Benchmarks reward finishing the task. This failure is what finishing it too thoroughly looks like.
3. Small models skip the README. With a README line saying the billing sync reads users.name, Opus and Sonnet kept the column in 6 of 6 runs. Haiku 4.5 dropped it in 3 of 3. Rules in the repo only help if the model goes and reads them. Rules in the prompt reached all four models.
4. A cheap model with a spec isn't cheap. Haiku 4.5 with a spec averaged $0.20 and 129 seconds a run, over 15 to 40 turns. Opus 5.5 with a spec averaged $0.22 and 63 seconds. Sonnet 5.5 was the cheapest route to a passing run: $0.11 and 33 seconds with a spec.
The two spec runs that failed
Both were Haiku 4.5 on the migration, and both passed their own tests:
- One stored "Mary Ann Smith" as first name "Ann Mary", last name "Smith". Every other name in the data was right, so a spot check would miss it. The hidden "no characters lost" check caught it.
- The other put the backfill in application code inside
open()instead of in the migration. Any deploy that runs migrations separately, or any other service reading the database, would see empty columns.
The lesson there is about tests, not specs. The checks that caught these ran the migration the way production would.
What this doesn't show
Three runs per variant is enough to show that something can happen, not how often it does. We kept the repos small and clean on purpose, so each outcome could be traced to one prompt or one file. Larger codebases give an agent more to misread. We also wrote the specs, the docs and the suites ourselves, which is how specs work in practice, but it means the specs cover exactly what we test. The one gap a spec left open (in the coupon case) is reported on the case page. Only Claude models were tested, through one agent.
The data
Every prompt, fixture repo, hidden suite and runner script is downloadable, along with each run's diff, score and final message on every model (CC BY 4.0). One correction we found while preparing this post: seven Haiku 4.5 runs have empty diffs, because our recording script diffed against HEAD and those runs most likely committed their own work (one says so in its final message). Their scores stand, since the suite ran on the working tree, but the diffs are gone. We fixed the runner and replayed the other 71 diffs through it, and every hidden-suite result matched.
If you rerun these with another agent (Codex, Cursor, Gemini CLI, Aider, anything), we'd like to publish your results next to ours. The runner takes an AGENT_CMD variable, so pointing it at a different agent is one line:
AGENT_CMD='codex exec "$PROMPT"' ./run.sh coupon vague 1 codex
Top comments (1)
The gap between repo rules and prompt specs in point 3 matches what happens in tool harnesses. Instructions injected at the prompt boundary are present in every attention pass, whereas repository READMEs rely on the model choosing to issue a read call before it starts writing code.
The cost reversal on Haiku in point 4 is also worth highlighting. A smaller model burning 30 to 40 turns of tool call overhead quickly wipes out its per-token pricing discount against Sonnet finishing cleanly in a handful of turns.