An edit to an agent's instruction file is a deploy, and I stopped letting mine ship on a hunch.
The daily tuning job that maintains my assistant's CLAUDE.md now requires a statistically
significant improvement — Welch's t-test, p < 0.10, at least a 5% lift — measured against a
14-day rolling baseline, before an instruction change is allowed to stay.
Then I ran the power calculation on my own gate and found the 5% threshold is decorative. At
14 days per window the test can only detect a shift of roughly 0.97 standard deviations.
Anything smaller is invisible regardless of what the lift threshold claims.
Both halves of that are this post. The mechanism is worth copying. The window size is the part
I got wrong, and the arithmetic is short enough to check.
The prompt is the least-tested code in most agent systems
Every other artifact in an agent pipeline has a gate. Application code gets a test suite and a
review. Infrastructure gets a plan and a diff — I have written enough Terraform modules and
managed enough remote state to know nobody merges those blind.
The instruction file gets none of that. Someone notices the agent did something annoying, adds
a line telling it not to, and ships. There is no baseline, no holdout, and no record of whether
the previous eleven lines are still earning their tokens.
That is how instruction files rot. They accumulate rules that were true about one bad afternoon
and have been costing context ever since.
You cannot test what you do not score
The gate needs a dependent variable, so the first build was a rubric, not a test. Five
dimensions, each scored 0–10 per conversation: goal clarity, rework rate, context hit rate,
scope discipline, and response density. The daily score is the average across that day's
conversations, weighted by conversation length.
The rubric is frozen. Changing it invalidates every historical comparison, so a rubric change is
itself a tracked meta-edit that resets all baselines.
Scores are anchored to observable friction rather than self-assessment. The scorer parses
transcripts for five pattern classes — corrections ("no, that", "actually", "don't"), rework
("redo", "start over"), scope drift ("I didn't ask", "just do"), missed context ("I told you
before", "check memory"), and praise ("exactly", "nailed it"). Friction pulls the dimension
score down, praise pulls it up.
This is the load-bearing decision. An agent grading its own transcripts will drift toward
generosity. Regex over the human's actual words will not.
Welch, not Student, and why that is not pedantry
Each edit gets a 14-day pre-window and a 14-day post-window of daily scores, compared with
Welch's t-test. Three outcomes: p < 0.10 and lift ≥ 5% marks the edit KEPT; p < 0.10 and
lift ≤ −5% triggers an automatic revert; everything else is INCONCLUSIVE and the edit
stays on probation.
Welch rather than Student's t because the two windows should not have equal variance. A good
instruction usually works by removing a failure mode, which compresses the bad tail — the
variance drops as much as the mean rises.
Student's t assumes equal variance and over-rejects when the smaller-variance group is the
larger sample. That is precisely the case you care about, so the pooled test would hand you your
most confident false positives on your best edits. Welch costs a few degrees of freedom and
removes the assumption.
The arithmetic that broke the design
A two-sample test at α = 0.10 and 80% power detects a minimum effect of roughly
(t_α/2 + t_β) × √(2/n) standard deviations. For a 14-day window that is:
| Days per window | Minimum detectable effect |
|---|---|
| 7 | 1.42 SD |
| 14 | 0.97 SD |
| 21 | 0.78 SD |
| 28 | 0.67 SD |
| 42 | 0.55 SD |
| 60 | 0.46 SD |
The gate can only see effects above the yellow line. The 5% threshold sits inside the shaded band — under it the whole time, doing no work.
Now convert to the units the gate actually uses. Assume a 7.0 baseline on the 0–10 rubric and a
day-to-day standard deviation of 0.6 points, which is unremarkable for a metric averaged over a
handful of conversations.
A 5% lift is 0.35 points, or 0.58 SD. The 14-day window detects 0.97 SD, which is 0.58 points —
8.3% of baseline. The statistical test is roughly 1.7× stricter than the lift threshold
sitting next to it.
So the 5% number never binds. It is doing no work. Every edit that clears the t-test has already
cleared 5% by a wide margin, and every edit that fails does so on power, not on effect size.
Detecting a genuine 5% lift at that variance takes 38 days per window, not 14. And the
requirement moves fast with variance:
| Daily score SD | 5% lift, in SD | Days/window needed | What 14 days actually detects |
|---|---|---|---|
| 0.4 | 0.88 | 17 | 5.5% |
| 0.5 | 0.70 | 27 | 6.9% |
| 0.6 | 0.58 | 38 | 8.3% |
| 0.8 | 0.44 | 66 | 11.1% |
| 1.0 | 0.35 | 102 | 13.8% |
The honest label on the gate. Every bar right of the dashed line is a threshold the config never mentions.
Read the last column as the honest label on the gate. At SD 1.0 a "5% threshold" is really a 14%
threshold, and the difference is entirely hidden from whoever reads the config file.
The window has to be sized to the variance, not to a calendar. Fourteen days was chosen because
two weeks is a tidy number, which is not a reason.
There are two ways out and only one of them is cheap. Lengthen the window, and you wait longer
per lesson. Or shrink the variance by scoring more conversations per day — the daily mean's
standard error falls with √n, so tripling daily volume cuts the SD by about 42% and pulls the
required window from 38 days down to roughly 17.
Put the same arithmetic the other way around. If an edit truly delivers a 5% lift, a 14-day
window calls it significant 44% of the time. That is a coin flip on your own best changes.
At 38 days it is 81%.
Same arithmetic, run forward. Each frame is a window length; the shaded tail is how often a
genuinely good edit gets called significant.
What the gate cannot do, stated plainly
It does not control for multiple comparisons. At the configured cadence of one low-risk edit
every three days, that is about 122 evaluations a year. At α = 0.10 two-sided, noise alone
produces roughly six spurious KEPT verdicts and six spurious auto-reverts per year. The
auto-revert side is the one that stings: the system will occasionally roll back a good edit with
statistical confidence.
It is also not a randomized experiment. Pre and post windows are consecutive calendar time, so a
model version change, a vacation, or a month of unusually messy work lands entirely in one window
and gets attributed to the edit.
That makes this a noise filter, not a causal claim. It stops the obviously-worse edits and the
obviously-imaginary wins. It will not tell you why anything moved.
The safeguards do more work than the test
Three rules keep the loop from eating itself, and they matter more than the p-value.
Cooldowns. One low-risk auto-edit per three days, one high-risk proposal per week. Without a
cooldown the windows overlap so badly that no edit is ever cleanly attributable.
Plateau detection. If the 14-day rolling score has not improved 2% over the prior 14 days and
no friction is firing, the day's edit is skipped. The easy wins arrive early; after that, editing
is mostly a way to add variance.
A risk split with a human in it. Wording and formatting changes auto-apply. Anything that
adds a section or changes how the agent decides gets written to a proposals folder and waits.
Every edit backs up the prior file first and lands in a revertible log, tagged, so a bad call is
a one-line rollback rather than an archaeology project.
The config file has a section titled "when NOT to edit," and the last line of it is the most
useful thing in the whole system: doing nothing is always a valid action.
The cost
Latency. That is the whole bill, and it is larger than I estimated when I built this.
A gated instruction file learns on a 14-day clock at best and, if the variance numbers above are
right, a 38-day clock in practice. An ungated one learns in an afternoon and is wrong in ways
nobody measures. I would still take the slow version, but I would not pretend the tradeoff is
free, and I would not build this at all for a system I was still prototyping.
I have no KEPT verdicts to report yet, because the first honest one cannot exist until a full
post-window closes. Publishing the design before the results is the point — the design is the
part that is checkable, and the arithmetic above is the part I would want someone to argue with.
If you are gating prompt or instruction changes statistically at your shop, I want to know what
window you landed on and what your daily variance looks like. That number is the whole ballgame
and almost nobody publishes it.
Originally published at michael-kaminski.io. I write field notes on agent infrastructure — evals, MCP servers, and what it costs to run agents in production.
Top comments (0)