When a coding agent writes the feature and the tests, a green test suite only tells you that its code agrees with its tests. Neither of those is your spec.
I wrote a guide to spec-driven development that covers how to write a spec for an agent and how to work with one in Claude Code, Spec Kit, OpenSpec and Kiro. Near the end I say the agent can still ignore the spec, so you still need tests and a review. This post picks up from there, with two checks that run in CI:
- A small script that proves every acceptance criterion has a test that really ran and passed.
- Mutation testing, which proves those tests fail when the code is wrong.
Everything below comes from a real run with Node 24, Vitest 5.0.5 and Stryker 10.1.1. You can copy the code as it is.
The setup: a small spec and a weak implementation
The spec is a shorter version of the CSV export example from the guide. The only thing I added is a stable ID on each criterion, so that a machine can follow it:
# Spec: Export transactions to CSV
## Acceptance criteria
- AC-1: WHEN the user exports with no filters
THEN all their transactions are included, newest first.
- AC-2: WHEN a description contains a comma or quotes
THEN the field is quoted and escaped (RFC 4180).
- AC-3: WHEN the filtered list is empty
THEN the file has only the header row.
If you use Spec Kit, you already have IDs: its template numbers requirements as FR-001. You'd only change one regex in the script.
Then I wrote the code badly on purpose. It has a real bug, because escapeField handles commas and forgets about quotes:
// src/csv.js
const COLUMNS = ['date', 'description', 'category', 'account', 'amount']
function escapeField(value) {
const text = String(value)
if (text.includes(',')) return `"${text}"`
return text
}
export function buildCsv(rows) {
const sorted = [...rows].sort((a, b) => b.date.localeCompare(a.date))
const lines = sorted.map((row) =>
COLUMNS.map((column) => escapeField(row[column])).join(','),
)
return [COLUMNS.join(','), ...lines].join('\n')
}
The tests are weak on purpose too. Each one names its criterion in the title, and one of them is skipped. In the demo they sit inside a describe('buildCsv', ...) block, which is where the buildCsv > prefix in the reports below comes from:
// src/csv.test.js
import { expect, it } from 'vitest'
import { buildCsv } from './csv.js'
const row = (over = {}) => ({
date: '2026-10-01',
description: 'Coffee',
category: 'Food',
account: 'Checking',
amount: 4.5,
...over,
})
it('AC-1: includes all transactions', () => {
const csv = buildCsv([row(), row({ date: '2026-10-02' })])
expect(csv.split('\n')).toHaveLength(3)
})
it('AC-2: escapes commas and quotes', () => {
const csv = buildCsv([row({ description: 'Coffee, "large"' })])
expect(csv).toContain('Coffee')
})
it.skip('AC-3: empty list gives only the header', () => {
expect(buildCsv([])).toBe('date,description,category,account,amount')
})
Vitest is happy with this:
Tests 2 passed | 1 skipped (3)
A grep for the IDs is happy as well, since all three appear in the test file.
Check 1: does each criterion have a test that ran and passed?
The usual way to link specs to tests is to search the test files for the IDs. That search can't see .skip, so it would miss AC-3 here. The test runner knows what happened, so I read its report.
Have Vitest write JUnit XML:
// vitest.config.js
import { defineConfig } from 'vitest/config'
export default defineConfig({
test: {
reporters: ['default', 'junit'],
outputFile: { junit: 'reports/junit.xml' },
},
})
Then compare the report with the spec:
// scripts/check-spec.mjs
import { readFileSync } from 'node:fs'
const [specPath = 'SPEC.md', junitPath = 'reports/junit.xml'] =
process.argv.slice(2)
const ID = /\bAC-\d+\b/g
const specIds = new Set(readFileSync(specPath, 'utf8').match(ID) ?? [])
// id -> list of test outcomes: 'passed', 'failed' or 'skipped'
const results = new Map()
const junit = readFileSync(junitPath, 'utf8')
const testcases = junit.matchAll(
/<testcase\b([^>]*?)(\/>|>([\s\S]*?)<\/testcase>)/g,
)
for (const [, attrs, , body = ''] of testcases) {
const name = attrs.match(/\bname="([^"]*)"/)?.[1] ?? ''
const outcome = body.includes('<skipped')
? 'skipped'
: /<(failure|error)\b/.test(body)
? 'failed'
: 'passed'
for (const id of name.match(ID) ?? []) {
results.set(id, [...(results.get(id) ?? []), outcome])
}
}
const problems = []
for (const id of specIds) {
const outcomes = results.get(id) ?? []
if (outcomes.length === 0) problems.push(`${id}: no test`)
else if (outcomes.includes('failed')) problems.push(`${id}: failing`)
else if (!outcomes.includes('passed')) problems.push(`${id}: only skipped tests`)
}
for (const id of results.keys()) {
if (!specIds.has(id)) problems.push(`${id}: test exists, not in spec`)
}
if (problems.length > 0) {
console.error(`Spec check failed:\n ${problems.join('\n ')}`)
process.exit(1)
}
console.log(`Spec check passed: ${specIds.size} criteria covered.`)
Run it against the code above:
$ node scripts/check-spec.mjs
Spec check failed:
AC-3: only skipped tests
The last loop runs in the other direction and catches drift. When I renamed AC-3 to AC-4 in the spec, leaving the tests alone, both sides were reported:
Spec check failed:
AC-4: no test
AC-3: test exists, not in spec
The same script also worked unchanged on the JUnit output of Node's built-in runner (node --test --test-reporter=junit). Jest and Playwright can write JUnit too, but I didn't test those, so check how they mark skipped tests.
I removed the .skip, and the script went green:
Tests 3 passed (3)
Spec check passed: 3 criteria covered.
The feature is still wrong, though. The code doesn't escape quotes, and nothing tests the sort order. This script can only tell you that a test exists and passes. It has no idea whether expect(csv).toContain('Coffee') proves anything.
Check 2: would the tests notice if the code were wrong?
Mutation testing changes your code in small ways (it flips a condition, removes a method call, empties a string) and runs the tests against each changed version, called a mutant. If the tests still pass, the mutant survived, and that line of code isn't really being tested.
This is the question to ask about tests written by an agent, and you don't have to read them all to get an answer. Install Stryker:
npm i -D @stryker-mutator/core @stryker-mutator/vitest-runner
Add a config:
{
"testRunner": "vitest",
"mutate": ["src/csv.js"],
"reporters": ["clear-text", "progress"],
"coverageAnalysis": "perTest"
}
Then npx stryker run. It took 3 seconds on this one file and found 6 survivors out of 21 mutants, for a score of 71.43%. These two are the interesting ones:
[Survived] ConditionalExpression
src/csv.js:5:7
- if (text.includes(',')) return `"${text}"`
+ if (false) return `"${text}"`
Tests ran:
buildCsv > AC-1: includes all transactions
buildCsv > AC-2: escapes commas and quotes
[Survived] MethodExpression
src/csv.js:10:18
- const sorted = [...rows].sort((a, b) => b.date.localeCompare(a.date))
+ const sorted = [...rows]
Tests ran:
buildCsv > AC-1: includes all transactions
buildCsv > AC-2: escapes commas and quotes
buildCsv > AC-3: empty list gives only the header
The first mutant turns off the escaping entirely, and the AC-2 test still passes. The second deletes the sort, and the AC-1 test still passes. Those are the two features the spec asked for, and because the tests are named after criteria, the report points straight at the criteria with fake coverage.
Fixing what the mutants found
No mutant mentions the quotes bug. Stryker changes the code that exists, and the quote handling was never written. The mutant on line 5 only tells you the AC-2 test is too weak, and you find the bug when you fix the test:
const HEADER = 'date,description,category,account,amount'
it('AC-1: includes all transactions, newest first', () => {
const csv = buildCsv([
row({ date: '2026-10-01' }),
row({ date: '2026-10-03' }),
row({ date: '2026-10-02' }),
])
expect(csv.split('\n')).toEqual([
HEADER,
'2026-10-03,Coffee,Food,Checking,4.5',
'2026-10-02,Coffee,Food,Checking,4.5',
'2026-10-01,Coffee,Food,Checking,4.5',
])
})
it('AC-2: quotes fields with commas', () => {
const csv = buildCsv([row({ description: 'Coffee, large' })])
expect(csv).toContain(',"Coffee, large",')
})
it('AC-2: escapes quotes by doubling them', () => {
const csv = buildCsv([row({ description: 'The "large" one' })])
expect(csv).toContain(',"The ""large"" one",')
})
it('AC-3: empty list gives only the header', () => {
expect(buildCsv([])).toBe(HEADER)
})
Now the bug shows up:
FAIL src/csv.test.js > buildCsv > AC-2: escapes quotes by doubling them
Tests 1 failed | 3 passed (4)
Spec check failed:
AC-2: failing
The fix is one line:
if (/[",]/.test(text)) return `"${text.replaceAll('"', '""')}"`
After that, the spec check passes and Stryker kills all 23 mutants, for a score of 100%. Don't expect that number on real code. This is one small function, and some mutants in a real project can't be killed because they don't change the behavior at all.
Running both in CI
The order matters: the tests write the report, the script reads it, and Stryker runs last because it's the slowest step.
- run: npx vitest run
- run: node scripts/check-spec.mjs
- run: npx stryker run
To make Stryker fail the build, add a threshold to its config. I checked this with a score of 95.65% against a break of 100, and it exits with code 1:
"thresholds": { "break": 80 }
The 80 is a starting point I picked for this post, so adjust it to what your codebase can hold. On a large project, point mutate at the files in the spec's "Files and interfaces" section, and use npx stryker run --incremental, which saves its results to a file and reuses them on the next run.
You can also give the survivors to the agent. Use a fresh session, and make it clear what it can't touch, because deleting the assertion or editing the spec is the fastest way to turn a check green:
These mutants survived Stryker. For each one, find the acceptance
criterion in SPEC.md that should have caught it, and strengthen that
test. Do not edit src/ or SPEC.md. If a stronger test fails because the
code is wrong, stop and tell me which criterion the code doesn't meet.
With Claude Code, you can run the same checks in a Stop hook. A hook blocks only on exit code 2 and the script exits with 1, so the hook command needs to convert it: npx vitest run; node scripts/check-spec.mjs || exit 2. I didn't run this hook in the demo.
What these checks can't tell you
Both checks work from the IDs in the test titles, and nothing verifies that a test labeled AC-2 is about AC-2. A strong test with the wrong label passes both. You still have to read the tests for the criteria that matter most, but the report tells you where to start.
The spec itself can also be wrong or incomplete. If nobody wrote the criterion for 100,000 rows, no script will ask about it. That part happens earlier, when you write and review the spec, and the guide has a full example, the agent interview prompt, and a comparison of Spec Kit, OpenSpec and Kiro.
I've only run this on a small example, so if you've used a mutation score as a CI check on code written by an agent, I'd like to know what threshold held up on a real codebase.
Top comments (2)
Feeding the survivor report back to an agent is where this gets practical, but the failure mode I run into on larger repos is equivalent mutants. An agent will spend five tool turns trying to kill a mutation on an unreachable fallback or an internal ordering detail that cannot be observed from the public API. What worked for me is constraining the mutation scope strictly to git diff lines rather than the whole file, and locking the test config so the model cannot casually bump timeouts or ignore uncovered branches to force a green run. A 70 to 75 percent mutation threshold on the diff catches the fake assertions without turning CI into an infinite loop on untestable edge cases.
Prompting is the easy layer. The interesting work is in what happens after the model answers.