Most prompt suites are laid out the way the application is laid out: one directory per feature, one file per prompt. That mirrors the code and it is exactly wrong, because a prompt change does not respect feature boundaries and a failure organised by feature tells you nothing about what kind of failure you have.
Why organising by feature collapses
Two things go wrong, and they compound.
The first is duplication. Every feature directory needs the same handful of checks — the output parses, no forbidden field appears, the required tool fires — so the same three assertions are written six times with six slightly different helper functions. When you tighten one of them you tighten one of six, and the other five quietly stop matching. Suites reach the point of being unmaintainable through this route far more often than through sheer size.
The second is that the failure carries no information. A red build that says checkout_summary.test.ts failed tells you which prompt to open. It does not tell you whether the model returned malformed JSON, refused a perfectly reasonable request, ignored the instruction to answer in the user’s language, or invented an order number. Those four have completely different causes and completely different responses, and you find out which one you have by reading the diff. At three red cases that is fine. At thirty it is an afternoon.
Organise by failure mode and both problems dissolve. Each mode owns one assertion, written once. Each mode is a directory whose name is the answer to the first triage question.
The objection is that a feature layout mirrors ownership, and a mode layout does not. That objection is real and it is the wrong trade. Prompt failures are not owned by feature teams in practice — the person who edits the shared system preamble affects every feature at once — so a layout that pretends otherwise puts each of them in six places.
Six modes with distinct assertions
A mode earns a directory when it has an assertion shape nothing else shares. Six do.
- Format. The output does not satisfy the contract: not valid JSON, a missing required field, a string where a number was declared, an enum value outside the set. The assertion is a schema validation — a Zod
safeParse, a Pydantic model constructor, a JSON Schema validator — and it is binary, which makes this the cheapest mode to test and the one that belongs on the gating tier in full. - Refusal. The model declines an input it should handle. The bad assertion here is a substring search for “I’m sorry”, which breaks the first time the refusal is phrased differently. The good one requires cooperation from the prompt: have it emit a
statusfield with a closed set of values and assertstatus !== "declined". You have converted a prose judgement into a format check, which is the move this whole cluster is about. - Instruction drop. A constraint stated in the prompt is silently not applied: answer in the input’s language, stay under 200 words, never mention a competitor, always cite a source id. Each of these is mechanically checkable against the input — script detection, a token count, a denylist, a set membership test — without any opinion about quality.
- Fabrication. The output asserts something absent from the context you supplied. Untestable in general; entirely testable when you control the context, which in a regression suite you always do. Extract every identifier-shaped token from the output — order numbers, SKUs, dates, prices, cited document ids — and assert each appears verbatim in the input. This catches invented references without needing to know whether the prose is true.
- Tool behaviour. Four distinct failures live here: no tool call when one was required, the wrong tool, the right tool with arguments that do not satisfy its schema, and the same tool called repeatedly in a loop. All four are assertions on structured fields, not on text: the tool name, the parsed arguments, the call count.
- Leakage. Something from the context that should never be echoed appears in the output — a system-prompt fragment, another tenant’s record, an API key that got into a retrieved document, a raw email address. Assert by pattern and by exact-value denylist seeded from the fixture itself, so the test knows precisely which strings were secret.
The mode that has no assertion
Tone is the seventh thing people want to organise by, and it does not belong with the others. There is no invariant for “too curt”. Every check available to you is a rubric applied by a human or by another model, which means it is a judgement, which means it has a spurious failure rate an order of magnitude above the six above.
That does not make it untestable. It makes it a different tier. Tone cases belong on the scheduled run, scored against a written rubric, reported as a trend rather than as a gate, and never allowed to block a merge. If you put a judged tone case on the gating tier you will spend the entire triage budget derived in the sizing arithmetic on it, and the five deterministic modes will be starved out. The existing page on evaluation rubrics is the right treatment for the scoring itself.
What it looks like on disk
The features become fixture data. The directories become modes.
tests/regression/
fixtures/
checkout.json # inputs, grouped by feature
support.json
onboarding.json
format_test.py # one assertion, every fixture
refusal_test.py
instruction_test.py
fabrication_test.py
tools_test.py
leakage_test.py
nightly/
tone_test.py # judged; not on the gate
Each file loads every fixture and applies one assertion to all of them. Adding a feature means adding a JSON file, and it inherits all six checks for free — which is the opposite of the by-feature layout, where adding a feature means writing six assertions again.
If you would rather select by mode than by path, pytest markers do it without moving files. Register them so a typo is an error rather than a silently empty run:
# pyproject.toml
[tool.pytest.ini_options]
markers = [
"format: output contract violations",
"refusal: declines a valid request",
"instruction: a stated constraint was dropped",
"fabrication: asserts something absent from context",
"tools: wrong tool, args, or call count",
"leakage: context content echoed into output",
]
# then
# pytest -m "format or tools" gating tier
# pytest -m "not (format or tools)" everything else
What a red build tells you now
The layout pays off at the moment of failure, and the payoff is a decision you can make from the summary line without opening anything.
Fourteen format failures spread across every fixture is not fourteen bugs. It is one output-contract break, almost certainly caused by the last prompt edit touching the section that describes the response shape, and the response is to revert that edit. One fabrication failure in one fixture is the opposite: a specific, local thing worth reading carefully. A cluster of tool failures alongside zero format failures points at the tool descriptions rather than the prompt body, which is a different file and often a different author.
The layout also makes absence legible, which is the part people underrate. An empty leakage_test.py is a mode you have never tested, sitting in the tree where you can see it. Under a by-feature layout that same gap is invisible — every directory looks populated, and you discover the missing mode when it reaches production. That is the mechanism behind most green suites that miss a regression, and a directory listing is a cheap defence against it.
Top comments (0)