Why “Spec-Driven Development” Needs a Deletion Test

Most of what we write for coding agents has never been asked to prove it does anything. Here is the test I’d apply, and where I think its author oversells it.

There is a file in almost every repository I touch now. Sometimes it’s called AGENTS.md, sometimes CLAUDE.md, sometimes it hides in a dot-folder. It starts small. Then the agent does something dumb, someone adds a paragraph, and the paragraph never leaves. Six months later the file is a museum of past irritations, and nobody can say which of its lines is still doing work.

Over the last few weeks I read four pieces by Jaroslaw Wasowski that go straight at this problem, plus two pieces from other authors that, read together with his, change how I think about it. This is my attempt to pull them into one argument, and to say plainly where I’m not convinced.

The short version: I think the deletion test is the most useful idea in the whole spec-driven development conversation. I also think the series around it carries more confidence than its evidence can hold, and that one of its own tools quietly runs into a problem another author named better.

A rules file is not a spec

Wasowski’s starting point is a distinction I wish I’d had words for earlier. A rules file tells the agent how to work in this repository: which command runs the tests, what not to touch, which convention we follow instead of the default. A spec says what must be true after one particular change.

When you mix them, every task pays for lines that matter to one task. The agent reads “a cancelled order returns 409” while it’s renaming a CSS class. That’s not free. It costs context, attention, and sometimes it pulls the agent toward work nobody asked for.

He leans on two 2026 studies that look like they contradict each other. One (Gloaguen et al., as he cites it) found that context files generally don’t improve task success and raise inference cost by more than 20 percent on average. The other (Lulla et al.) found that an AGENTS.md cut median agent run time by about 29 percent on small pull requests. His reading is that both can be true because they measure different things: one measures whether the task passed and what it cost, the other measures wall-clock time and output tokens. I agree with that reading. It’s also a good habit in general: before you quote a number about agents, ask what it actually counted.

The surveys he cites also say something uncomfortable about what teams put in these files. Testing procedures and architecture show up in most of them. Security and performance show up in roughly one file in seven. So we spend the most words on what the agent can discover by reading the code, and the fewest on what it can’t.

The deletion test, in my own words

Here’s how I’d explain it to someone on my team who hasn’t read any of this.

Every line in the rules file is a claim: “if this line weren’t here, the agent would do something worse.” The deletion test is just checking that claim. You take the line out, run the same small set of tasks you ran before, and look at what changed. Not only whether the tasks passed, but how the agent got there: did it run the full test suite blindly, did it touch a folder it shouldn’t, did it spend twice the tokens wandering around.

If nothing changes, and the line isn’t a security or compliance rule, the line is a candidate to go. If something breaks, you put it back, and now you know why it’s there.

I think of it like the sticky notes on my fridge. Some of them I’d miss the next morning. Most of them I stopped reading months ago, and the only reason they’re still there is that removing one feels like a decision, while leaving it feels like nothing. The deletion test turns “leaving it” into a decision too.

Wasowski’s companion piece on spec size adds the mirror image of this, and I think the two belong together. For every clause in a spec, you should be able to name a plausible wrong implementation that the clause would stop. If you can’t, the clause forbids nothing. A rule that changes nothing when deleted, and a spec clause that rejects no wrong code, are the same disease in two places: text that looks like control and isn’t.

Rules that survive, rules that don’t

Plain text, because this is about wording, not tooling.

Rules I’d expect to survive deletion:

1. "Run unit tests with the project wrapper, not the bare test runner.
    The bare runner skips the database fixtures and every test looks green."
Why it survives: the agent can't guess this from the repo, and its absence
   produces a visible, repeatable mistake.
2. "The full integration suite takes about 25 minutes. Run it once,
    at the end of the task, not after every edit."
   Why it survives: it may not change pass/fail at all, but it changes the
   course of the work. Wasowski cites an ablation where a warning like this
   was linked to fewer blind full-suite runs (one agent, exploratory).
3. "Never edit anything in the generated folder. Change the schema and
    regenerate instead."
   Why it survives: without it, the agent fixes the symptom in a file that
   gets overwritten on the next build.

Rules I’d expect to die:

1. "The src folder contains the API, the background workers and shared
    utilities."
Why it dies: the agent reads the folder anyway. A tour of the code is
   the weakest thing you can pay for on every task.
2. "Write clean, maintainable code and handle edge cases gracefully."
   Why it dies: no implementation fails this sentence. Every diff can be
   called graceful. It stops nothing, so deleting it stops nothing.
3. "Cancelling an order that has already shipped returns 409."
   Why it dies here: it might be a perfectly good rule, but it belongs to
   one change. It should live in that change's spec, next to its tests,
   not in a file every task reads.

And one that should stay even if the test says it does nothing:

"Never write access tokens or card numbers to logs."

A benchmark built on functional tests won’t notice this line is missing, because no hidden test checks for a leak. Wasowski makes this point himself, and it’s the right call: don’t delete security rules on the strength of a pass/fail result that was never designed to price them. But I’d go one step further than “keep it.” A rule like this shouldn’t live only as prose. It should be enforced by something that doesn’t read prose.

Failing examples before tests

The third piece is, for me, the most practical. When you’re fixing existing behavior, the best entry point into a spec isn’t a document. It’s a single wrong answer: this input, this expected result, this actual result. An order worth exactly 100.00 should get a 10 percent discount and gets nothing.

From that example you derive boundary values (just below, at, just above the threshold), then counterexamples (the same order with a promo code, a refund), and only then the rule in words. Then he asks the agent for a contract (preconditions, postconditions, cases the spec doesn’t cover) before it writes any test, so you can see what it understood in two minutes instead of in a diff.

The part I found most honest is his own small demo. He wrote two test suites for the same discount function. One recorded what the buggy code currently does. The other came from the contract. The first was green on the buggy code and red after the fix. The second did the opposite. That’s exactly the trap of letting an agent write tests from code that is wrong today, and it lines up with something Shibui Yusuke says in a very different article: ask an AI to write tests and you tend to get snapshots of current behavior that enshrine the bug as the spec. Two authors, different starting points, same warning. That convinces me more than either one alone.

He also ran a property-based test against the buggy version, expecting the framework to find the threshold bug on its own. It mostly didn’t. Across 30 runs per setting, a wide input range with 100 examples found the bug in 0 of 30 runs, a narrow range around the threshold found it in 14 of 30, and only adding the failing example explicitly found it every time. I want to be clear about what that is: his own demo, on one deterministic function, with a contract he wrote by hand. It’s a good illustration of why the failing example should stay in the spec even when you add properties. It is not a measurement of how property-based testing behaves in general.

The one-year-later look

The fourth piece looks back at a year of spec-driven development frameworks. Fifteen products a year ago, four pattern families today, and no winner. The single most useful number in it, for me, isn’t about any framework. It’s that in a sample of public repositories, 81.5 percent of spec files have exactly one commit. He’s careful to say that some of these are archived change records, so it isn’t the share of stale specs. But it tells you how rarely anyone goes back.

His conclusion is that the cost nobody has measured is keeping a spec true after the second requirement change. I think that’s correct, and I think it’s the real reason the deletion test matters. A rules file and a spec both rot by addition. The test is one of the few habits that pushes in the other direction.

Where I disagree

I like this series. I’m going to keep using parts of it. But I’d be doing my readers a disservice if I passed it on as settled.

Much of it is one author’s conclusion, not evidence. To his credit, Wasowski labels this himself, often in the same paragraph. The claim that mixing rules with a spec raises cost without improving accuracy is, in his words, his synthesis, not a measured result. The year-later piece says upfront that it contains no runs of his own with the frameworks. The deletion protocol is his synthesis of three studies plus his own practice. The risk with a series like this isn’t any single article. It’s that when you read five of them in a row, each with its own matrix, template and decision tree, the system starts to feel proven because it’s coherent. Coherence isn’t evidence.

The flagship article ships without its result. In the AGENTS.md piece, the spot where his own ablation on a real repository should be is a placeholder marked “to be completed before publication.” The matrix in the summary is labeled as example values. So the article’s central tool, the deletion test, is recommended without its author showing one run of it. I don’t think that makes the idea wrong. I do think a reader should know it before adopting the template.

The numbers that exist are small, and he knows it. The best framework measurement he found is a benchmark from a commercial publisher, with 10 tickets per cell and a noise band of plus or minus 10 points, not independently reproduced. The contract-before-tests result from Google’s team was significant at five agent runs and inside the noise at one, and it cost about 38 percent more tokens. Most small teams run the agent once.

The deletion test can’t see what it’s supposed to see. This is the one that bothers me most, and the evidence comes from inside his own article. He cites a preprint (Khatri) showing that with 17 tasks and three repetitions, even a large 30-point effect is detected only 57 percent of the time, and that detecting a 10-point effect needs something like 120 to 200 tasks. His recommended set is about ten. So on pass/fail, the test is mostly blind. He handles this by saying the decision should rest on compliance and process metrics instead. That’s fair, but it narrows the test a lot: it works best for rules a machine can detect in the output. And for a rule a machine can detect, I’d rather the machine enforce it than keep the line in prose and audit it by deletion.

Anthony Alcaraz’s piece on graders makes the same point from the other side. One of his four failures is a score that moves when nothing changed. His fix is to rerun the unchanged system first and treat that spread as the smallest change worth believing. Hex, in the case he describes, needed 10 tries per case before reruns stopped wobbling. If you run a deletion test without first measuring how much your ten tasks move on their own, you’ll delete lines because of noise and keep lines because of luck.

Lines aren’t independent. He mentions this briefly: instructions can substitute for each other, so you should re-check the whole file after pruning. I’d put it higher. Two lines that both nudge the agent toward running unit tests will each pass the deletion test alone and fail together. One-at-a-time deletion is a decent heuristic, not a clean experiment.

The counterexample prompt has an independence problem. In the spec-size article, he checks a spec’s discriminating power by asking an agent, one that sees only the spec, for plausible wrong implementations, then marking which clause stops each. I like the move. But this is where Shibui’s chapter on independence lands hard. He goes back to Knight and Leveson’s 1986 experiment, where 27 independently written programs failed on the same inputs far more often than independence would predict. If people trained differently share blind spots, models trained on overlapping data share more. Shibui calls AI review of AI code closer to looking twice with the same eyes than to defense in depth.

Alcaraz’s data backs that up uncomfortably well. In the ICML ranking study he describes, four models from four organizations agreed with each other at about 0.62, and with human reviewers at only 0.07 to 0.10. Agreement between models told you almost nothing about whether either was right.

So when an agent generates counterexamples for a spec an agent helped write, the gaps it can’t imagine are probably the gaps the spec has. Wasowski does say to repeat on a different model for risky changes. Shibui’s point is that a different vendor gives less independence than it appears. The counterexample prompt is a good way to find dead clauses. I don’t trust it to find missing ones. That part still needs a person who knows what the business fears.

The grader believes the agent. Alcaraz’s first failure is a grader that reads the agent’s own account of its work. His most striking case is a research agent that wrote its own completion checker from the launcher’s list instead of its plan, found every scheduled run complete, and never noticed four planned runs had never been scheduled. In the audit he cites, lenient self-verification appeared in 58 of 64 sampled submissions. This is the strongest argument I know for Wasowski’s “protected tests” idea in the failing-example piece: the acceptance scenarios come from a human, and the agent has no right to edit them. Where Wasowski and Alcaraz agree, I’m fully on board. The thing the agent is judged by has to be out of its reach.

What a small team should actually put in the file

Here’s where I’ve landed. No templates, no frameworks, just what I’d tell a team of four or five people with one rules file and one agent.

Put in:

• Commands the agent can’t guess, with the trap spelled out. Not “run the tests,” but which command, and what goes wrong with the obvious one.
• Cost warnings that change how the agent works. The slow suite. The expensive build. The external call that bills per request.
• Conventions that differ from the default, with the direction of travel. “We’re moving off the old client. If you touch a module that uses it, move it.”
• Boundaries in three buckets: always, ask first, never. The folders that are generated, the migrations a human reviews, the public API nobody changes casually.
• Security and compliance rules, even when no test shows they matter. Next to each one, a short note on what enforces it, so the line is a pointer to a guardrail, not the guardrail itself.

Leave out:

• A tour of the repository. The agent reads the code. If your docs are good, it reads those too.
• Adjectives. Clean, robust, graceful, maintainable. If you can’t name the wrong code a sentence would stop, the sentence doesn’t stop anything.
• Anything about one change. That goes in that change’s spec, next to its failing example and its tests.
• Lines you added after one bad run and never looked at again. Those are the first ones to try deleting.
• Any rule a machine can check. Move it to a linter, a hook or CI. Prose is for what a machine can’t verify.

And two habits around the file:

• When you prune, measure the noise first. Rerun your small task set unchanged a few times and see how much it moves. Then remove one line at a time, and treat a removal as reversible, not as a verdict.
• Keep the judge out of the agent’s reach. Acceptance scenarios and failing examples come from a person, and the agent doesn’t edit them. If you use a model to grade the agent’s work, check it against a person on a sample of what it passed, not just what it failed.

The deletion test isn’t proof of anything. With ten tasks, it can’t be. What it does is force every line in the file to have a reason, and make “nobody remembers why this is here” an answer you’re no longer allowed to give. That alone is worth more than most of what the frameworks are selling.

Sources

• Jaroslaw Wasowski, “An Agent Rules File (AGENTS.md, CLAUDE.md) Is Not a Spec: The Deletion Test and a Minimal File” (member-only; the studies it cites are as reported there, and its own ablation result is not yet published): https://medium.com/@wasowski.jarek/spec-driven-development-an-agent-rules-file-agents-md-e47a650450a6
• Jaroslaw Wasowski, “The Smallest Spec That Rejects Wrong Code: Discriminating Power, Counterexamples, and Spec Weight” (member-only): https://levelup.gitconnected.com/the-smallest-spec-that-rejects-wrong-code-discriminating-power-counterexamples-and-spec-weight-d948e0909afc
• Jaroslaw Wasowski, “Failing Example as the Start of a Spec: Boundary Values, Contract Before Tests, Protected Tests” (member-only; the test-suite and property-based demos are the author’s own runs on one function, with a hand-written contract): https://levelup.gitconnected.com/failing-example-as-the-start-of-a-spec-boundary-values-contract-before-tests-protected-tests-9c77bb4d91d7
• Jaroslaw Wasowski, “Spec-Driven Development Frameworks a Year Later: Pattern Families, Measurements and a Choice Matrix” (member-only; no runs by the author, main benchmark is from a commercial publisher): https://medium.com/@wasowski.jarek/spec-driven-development-frameworks-a-year-later-pattern-families-measurements-and-a-choice-matrix-125bcddf1e64
• Shibui Yusuke, “Software Development Where AI Is Useless” (Chapter 5, the independence boundary): https://shibuiyusuke.medium.com/software-development-where-ai-is-useless-80b713a5e54e
• Anthony Alcaraz, “How To Evaluate Agentic Systems: Four Ways The Grader Fails, And The Fix For Each” (member-only; figures are from the studies and company write-ups it cites): https://medium.com/data-science-collective/how-to-evaluate-agentic-systems-four-ways-the-grader-fails-and-the-fix-for-each-a4a7ca415c86

Tags: AI Agents, Spec Driven Development, AGENTS.md, Claude Code, AI Coding, Software Engineering, LLM Evaluation, Code Review