Do your agent system prompts do anything? I measured 19 of mine

The body of an agent system prompt is text. It goes to the same model that would have answered without it, and by itself it changes no weights and adds no tools. The skeptical reading follows on its own: a specialist prompt is a checklist, the model reads the checklist, and whatever the checklist buys is too small to justify maintaining hundreds of them.

That position has a strong advocate. Boris Cherny, who built Claude Code, says in a talk given after the Opus 5 release that they deleted 80% of its system prompt for that release. He then describes deleting the rest as an experiment, and what they find: "the model is actually a little bit more intelligent without these prompts."

I had written the same argument myself, in a six-month audit of my own agent harness, as the case for deleting the delegation mandate: delegating to a specialist buys context and not competence, because it is the same model reading a different checklist. Then I ran the experiment, and the same document records what happened to my argument. Refuted by local measurement.

Fifteen of nineteen configurations scored lower without their prompt. Thirteen of those survive the noise floor I established afterwards. The gap between those two numbers is worth reading. So is the gap between my result and his, which comes down to scope.

Here is the number, the procedure that produced it, and where it is weaker than the headline.

The procedure

The audit these numbers live in runs someone else's protocol. It is the Six-Month Audit by Charlie Hills, a published checklist for finding the instructions in a Claude setup that newer models have outgrown: seven delete checks, five add checks, run against live guidance and never a hardcoded copy. What it asked me to do was judge which instructions had gone stale. The number is the part I added.

That measurement also has a real name I did not use in my own code. Cherny gives it in that talk: "we call in research, you call this ablation... you delete the entire system prompt and then you bring it back line by line to figure out what is the impact of each individual line." He compresses it a sentence later: an ablation is an eval where you delete things to figure out the impact. My harness calls its version a mutation gate, which is the name in my code and not a better one. What it runs is an ablation with a single fixed lesion.

Nineteen of my agent configurations carry golden tasks: a small set of inputs with graded outputs, run through promptfoo, which reports how many cases pass. That corpus already existed, for catching regressions when a prompt gets edited. The ablation reuses it to ask a different question.

Swap the agent's system prompt for You are a helpful assistant. Re-run the same golden tasks, same subject model, same grader. Compare the pass count against the recorded baseline.

Anthropic runs its own version of that arm, and Cherny names the switch: an undocumented CLAUDE_CODE_SIMPLE environment variable that, he says, means "it'll delete all the system prompts including from the tools", used "as a sort of ablation to figure out is the prompt useful". Take the behavior as his description: in the version I have installed, 2.1.222, the binary carries both that name and a separate CLAUDE_CODE_SIMPLE_SYSTEM_PROMPT, and I did not trace which one reaches the prompt. His cut is bigger than mine, every prompt gone including the tools', where I swap one agent's prompt for a trivial one. Same instrument at a different depth. Nothing below is exotic; it is the vendor's own diagnostic pointed at my agents.

A configuration whose score drops is discriminating: the prompt was producing the result, not the base model. A configuration whose score holds is telling you that its tasks measure the base model. That is a statement about the tests before it is a statement about the prompt, and conflating the two is the main way this experiment gets misread.

It needs no new corpus: if you have regression tests for your agents, you already own the instrument. And the null arm is fixed and boring, so a drop cannot be read as one prompt beating another.

One property it does not have, which I claimed in the first draft and had to take back: this is not a controlled within-run comparison. The with-prompt column is a committed baseline recorded on 2026-07-27; the null run happened on 2026-08-04. Eight days separate the arms, and only 2 of the 19 baseline entries were recorded as a median of three runs. So the table is an observation measured against a reference. Calling it a controlled experiment would be a stretch.

The result

Agent	With prompt	Null prompt
devops-engineer	3/3	0/3	discriminating
system-design-interviewer	5/5	0/5	discriminating
backend-architect	3/3	1/3	discriminating
devils-advocate	4/4	1/4	discriminating
frontend-developer	3/3	1/3	discriminating
humanizer-skill	3/3	1/3	discriminating
observability-engineer	3/3	1/3	discriminating
prompt-engineer	3/3	1/3	discriminating
typescript-agent	3/3	1/3	discriminating
code-reviewer	3/3	2/3	discriminating
deep-researcher	3/3	2/3	discriminating
nextjs-developer	3/3	2/3	discriminating
python-agent	3/3	2/3	discriminating
pragmatic-engineering-advisor	4/4	3/4	discriminating
security-auditor	4/4	3/4	discriminating
docs-architect	2/2	2/2	non-discriminating
git-workflow-manager	2/2	2/2	non-discriminating
sales-automator	2/2	2/2	non-discriminating
technical-writer	2/2	2/2	non-discriminating

The two zeros collapse for different reasons

The quotable rows are the two that go to zero, and the honest reading of them splits in half.

system-design-interviewer drops 5/5 to 0/5, and that zero was decided before the model answered. All five of its cases assert a literal turn marker the agent is required to emit, and a generic assistant has no way to know the string exists. grep -l for deterministic assertions across the 19 configs returns exactly two files, and this is one of them. That row measures the instrument. It says nothing about behavior. The same mechanism explains the row below it: devils-advocate scores exactly 1/4, and the case it passes is the only one of its four with no signature assertion.

devops-engineer is the clean result. Its three cases are graded by rubric on content, with no format contract anywhere, and the null arm failed all three on substance. Two of them turn on knowledge of one machine: that persistent data here belongs on /mnt/storage and never the root disk, and that analytics and uptime are already covered by services running on this VPS. Neither fact is in the input, so there is nothing to reason from and a textbook-correct answer fails. The third is a Cloudflare TLS trap, and there the input does state the private part, so a careful generic assistant could pass it. One of the three also carries a second, entirely generic rubric about healthchecks, and the run does not record which assertion sank the case.

So that row is weaker than a bare 3/3 to 0/3 suggests, and it is still the only row in the table where the grading is about behavior at all. That narrows what generalizes. Some of what these prompts buy is judgment, which your agents could carry too. Some of it is knowledge of one machine, which is decisive here and worth nothing on yours.

Anthropic runs this experiment and reports the opposite direction

The quote at the top is not a stray line. Cherny is describing something they repeat on a schedule and treat as an eval: every model release, delete the prompt, bring it back line by line, keep only what the model demonstrably still needs. He also says what a lot of that text had been doing, which was correcting for behaviors the model should have known but did not, until it did.

So the person who built Claude Code deletes prompt text and reports a slightly smarter model, while I delete prompt text and watch 13 of 19 configurations get worse. The asymmetry matters before any reconciling: his side is a direction with no number attached, offered as "something that we've been finding", and mine is a table. The reconciliation is scope, and he supplies it himself, two sentences after the line everyone will quote: "when you use [Claude Code] as a product, you do actually want some of these prompts because it helps you use the product..." (The talk's auto-captions mangle "Claude Code" several ways, "cloud code" and "quad code" among them. The brackets are mine; nothing else in the quotes is.)

Claude Code's system prompt is generic by construction. It ships to every user and every repository there is, and much of what it accumulates is correction for one model's specific failures. That text has a half-life. Deleting it on the cadence of model releases is just maintenance.

The prompts I measured are not generic, and the two devops-engineer cases above are why. /mnt/storage and the services already running on this VPS are facts about my infrastructure. No model release supplies them. A better model produces a better textbook answer and still fails. Correction text is the opposite case: a new model is precisely what makes it redundant.

I demonstrated that mechanism on exactly one row, and I should not stretch it further than that. I separated corrective text from knowledge-bearing text for devops-engineer by reading its rubrics. For the other twelve I have a drop and no such reading, so calling all thirteen knowledge-bearing would be the theory eating the evidence. The ablation does not draw that line, and neither does anything else in the table.

Prompt text that corrects the model expires, roughly on the cadence of releases. Prompt text carrying knowledge the model has no way to have does not expire at all. An ablation cannot tell you which kind you are holding. It tells you the text is load-bearing today. Reading why is still your job.

The noise floor, and what it does to my own headline

Every number above comes out of a non-deterministic system with an LLM grader. Run the same configuration twice and it does not always agree with itself.

I have a number for that. On the same day as this run, a routine baseline reported a regression on a configuration nobody had touched: 4/4 down to 3/4. The same configuration had scored 4/4 twenty minutes earlier, with git status confirming no prompt, skill or eval config changed in between. The instrument has run-to-run variance of at least one test case, which is the same magnitude as the effect it is supposed to detect.

That cuts into my own headline, so I would rather do it here than have a reader do it for me. Two of the fifteen, pragmatic-engineering-advisor and security-auditor, drop by exactly one case. That is the noise magnitude, measured on the same instrument, the same week. Thirteen of nineteen survive the floor. The other two are unresolved, not confirmed.

So the protocol has an order, and the measurement comes second:

1. Run the unchanged baseline three times. Record the spread.
2. Treat any delta smaller than that spread as nothing. Not a weak signal, nothing.
3. Where your harness reports assertion counts as well as case counts, compare assertions. On one of my configurations that is 10 numbers instead of 4, and a finer counter is harder for noise to dominate.

A 3/3 to 0/3 collapse survives this easily. A one-case drop does not survive it at all.

The four that held are a finding about my tests

Four configurations scored identically with and without their prompt. That is not four useless prompts. All four have exactly two test cases, the smallest corpus in the set, and both already pass at baseline.

A two-case corpus has a resolution of fifty points. The smallest drop it can record is half the corpus, and two cases at the ceiling cannot separate a prompt that contributes nothing from a prompt whose contribution those two cases never probe. The likelier explanation is weak test power rather than a useless prompt, and the correct response is to write harder cases before drawing any conclusion about those four.

There is a rival explanation for those rows, and it comes from the same talk. Cherny puts the working life of an eval at one to three model generations and names what usually ends it: "very often we just saturate the eval and then we have to throw it away". Four suites sitting at 2/2 with and without the prompt are what a saturated eval looks like from the inside.

It is not what happened here, and git log is why. Those four configs were added on 2026-07-03, a month before the run, in the same commit series as the rest. devops-engineer, which collapsed 3/3 to 0/3, is five days older still. Same age, opposite verdicts, so age is not the variable; corpus size is. The response is the same either way: throw the eval away and write a harder one.

This is why the gate only warns in my harness. A non-discriminating result is a work order for the test corpus, and wired as a blocking check it would delete prompts on the strength of thin tests.

The mistake that invalidated my first attempt

An earlier version of this experiment reported a 43-point gain for agent prompts. That number was wrong, and the way it was wrong is worth more than the number.

The harness had quietly stopped delivering agent prompts to the model. A credential name was changed in the config files and never changed in the harness that injects it, so the subject provider was never built. The resulting error went to a log file nobody reads while the run scored anyway. Measured prompt token counts came back at 60 to 112, against roughly 10,000 for a real agent body. The model was answering the golden-task question with no system prompt at all.

Both arms were promptless. The experiment compared nothing against nothing and reported a large win for prompts, which is the shape a broken measurement takes when it happens to agree with what you expected. A second defect found in the same investigation, a module renamed across an extension boundary, turned out to be latent rather than the cause. Naming the wrong one would have left the real one live.

So there is a step that belongs before all three above: prove your baseline arm actually loads the prompt. Prompt tokens per request is the cheapest proof there is. If it is not carrying thousands of tokens of agent body, you are measuring your plumbing.

What the number does not cover

It tests prompts, not delegation. The comparison is prompt against null prompt inside the same call. It says nothing about whether spawning a subagent beats running the same prompt inline in the main thread. Those are separate questions and only one of them is answered here. Anyone using this result to defend an orchestration architecture is using it for something it does not support.

The sample is not the roster. Nineteen configurations, and they are the ones I invoke most, which is the opposite of a random sample. The finding is that the prompts of my most-used agents do measurable work. Everything outside that set is unmeasured, and until it is measured, archiving any of it is a decision about frequency rather than merit.

You cannot inspect my harness, so I extracted the instrument instead

The repository holding these agents is private and stays private. It carries credentials, client context and personal data, so opening it is not on the table.

Which leaves me holding a number nobody can check, and a number nobody can check is a claim. So I pulled the mutation mode out of the harness and published it on its own: prompt-mutation-gate, MIT. It knows nothing about my directory layout: point it at a promptfoo config and a prompt file and it runs on your agents:

$ npm install && npm run example

strict-reviewer  DISCRIMINATING
  cases 6   baseline 6/6 [6 6 5]   null-prompt 0/6 [0 1 0]   drop 6   noise floor 1
    Removing the prompt cost 6 of 6 cases, against a run-to-run spread of 1.

tiny-suite       INCONCLUSIVE  (low-test-power; not conclusive)
  cases 2   baseline 2/2 [2 2 2]   null-prompt 2/2 [2 2 2]   drop 0   noise floor 0
    2 test cases is below the 5-case minimum. A suite this small, sitting at its
    ceiling, has no room to show a drop.

That runs with no API key and no account: the example parses recorded promptfoo result files and classifies them the same way a live run would, so the verdicts are derived rather than printed.

Two things in it matter more than the verdict, and both are answers to something in this post. It prints the noise floor beside every row, from the spread of the baseline runs, because I read fifteen off my own table before I had a floor to read it against. And below a five-case minimum it returns INCONCLUSIVE instead of non-discriminating, because four of my rows held at exactly that size, and non-discriminating lets a statement about the corpus wear the costume of a statement about the prompt.

The procedure it automates is short enough to run by hand if you would rather not take a dependency:

1. Pick every agent that already has graded tests. Three is enough to start.
2. Verify the baseline arm delivers the prompt. Check tokens per request.
3. Run the baseline three times unchanged and record the spread. Run the null arm three times too — it varies as well, and a single null run can be the outlier just as easily as a single baseline run.
4. Replace each system prompt with You are a helpful assistant. and re-run.
5. Before reading any row, check what its assertions grade. A case asserting a literal output format is a floor check on your harness, not evidence about behavior.
6. Classify: dropped beyond the spread is discriminating; held is a note about your tests.
7. For anything that held, write harder cases before concluding anything about that prompt.

That is roughly an afternoon on an existing test corpus, and it produces your own version of the table above.

If your numbers come out flat where mine dropped, there are three readings and only one of them is about me. Your corpus may be too thin to record the drop, mine may be measuring something local I have mistaken for general, or your prompts may genuinely be carrying less than you thought — which is a real result and the one worth knowing. That disagreement is more useful to me than agreement, and it is the only scrutiny a private repository is still open to.