I Stopped Writing Prompts and Started Writing Fixtures

For the first two weeks I treated agent quality as a prompt problem. Reword the instruction, add an example, add a rule. It moved nothing.

The thing that moved the numbers was fixtures: a fixed starting state plus a fixed expected end state, checked automatically.

What a fixture looks like for an agent task

Not a unit test. A small environment:

• a directory with the exact files the task expects, at exact versions
• the services it needs, already running, with seeded data
• the broken thing that makes the task necessary, reproduced deterministically
• an assertion on the end state: file contents, exit codes, rows in a table

Then: run the task, assert, throw the environment away.

The assertion is the whole point. Without an end state I was grading the agent's summary of its own work, which is exactly the thing I should not trust.

Prompts are context; fixtures are truth

A prompt says what I want. A fixture proves whether I got it. They fail differently:

• a bad prompt produces plausible wrong work, and I only notice at review time
• a bad fixture produces a red assertion immediately, before I have paid for review

Same hour of work, wildly different cost of being wrong.

The three fixtures that paid for themselves

1. Rename a column and update everything referencing it. Deterministic, boring, and it caught every case where the agent edited the migration but not the fixtures.
2. Bring up a service that fails on a port collision. The task is not "start it", it is "notice the collision, find the other process, decide". Reproducible collisions are easy to build and the failure modes are exactly the ones that used to eat an hour.
3. Apply a config change that must be rejected. Negative tests are the ones I skipped, and the ones that found the real bugs - an agent that never gets told "no" will happily push a bad config.

What this is not

Fixtures do not make the model better. They make the failures legible and cheap. My agent still fails; the difference is that it fails in 40 seconds, on a machine, with a red line telling me which promise it broke, instead of failing in front of a person at review time.

If you run agents on long tasks, what is in your fixture? Mine is consistently thinner than I expect when I write it down.