You have a rule you want your AI agent to follow. You write it in AGENTS.md. It gets broken. You write it in more detail. Still broken. You add another clause.
After 48 days of doing that, my resident instruction files went from 90KB to 147KB.
Violations did not drop.
Here are the numbers. All from one machine, one operator (n=1).
1. 94.4% of my tokens were re-reading the same text
First, what happens to cost when you grow the instruction file. 48-day totals:
| Metric | Measured |
|---|---|
| Cumulative tokens | 62,250M |
| Of which cache reads | 58,751M (94.4%) |
| Actual generated output | 185.9M (0.30%) |
| Messages | 268,766 |
| Re-read per message | ~219k tokens |
Resident instructions are four files totalling 204,309 bytes. That gets re-injected every single turn.
So "cumulative tokens" is not a measure of how much work the agent did. It is a function of instruction length times turn count. If you use it as a proxy for how heavily you operate, the person who wrote the longest instructions looks like the heaviest user.
Cache hit rate of 99.3% is the same trap. It is not evidence of efficiency. It is evidence that I keep throwing the same context.
2. Violations still did not drop
I run a small script that counts, per rule, how many times it has been broken. Nothing clever. It just increments a counter when I get corrected.
The counter kept going after I added clauses. At one point a clause added the previous day was broken the next day.
Adding clauses was not the fix. It was the symptom.
3. Splitting 24 rules into two groups
The 24 tracked rules happened to fall into two groups:
- Group A (8 rules): violations blocked in code. PreToolUse hooks, checker scripts, a guard right before send.
- Group B (16 rules): written as a clause in a document, nothing more.
Same machine, same agent, same period.
| Group | Rules | Rules that recurred | Total recurrences | Per rule |
|---|---|---|---|---|
| A: code gate | 8 | 0 (0%) | 0 | 0.00 |
| B: document only | 16 | 16 (100%) | 19 | 1.19 |
Group B was a wipeout. All 16 recurred at least once.
Group A has gone 2,500 to 4,053 turns without a recurrence since the gate went in.
4. Is this actually the gate working?
Three objections worth taking seriously.
Objection 1: selection bias. You gated the worst rules.
Correct. My operating rule is that anything violated more than three times gets moved into code. So Group A is the 8 rules that recurred the most. They are now at zero. The bias makes the result stronger, not weaker.
Objection 2: reverse causation. Learning had already happened by the time you gated.
I cannot rule this out. "Gate it on the third time" correlates with elapsed time by construction. But three rules in Group B are sitting at two recurrences and have not stopped, so time alone does not explain it.
Objection 3: the agent just is not reading the document.
It is reading it. All 204KB is in context every turn. The point of this measurement is that it is being read and still not followed.
5. Half my learning queue was thrown away
I also run a queue: every correction gets logged and tracked until it becomes a rule. Cumulative:
| State | Count |
|---|---|
| Promoted to a clause | 231 |
| Verified | 178 |
| Rejected | 229 |
| Unprocessed | 12 |
| Waiting on a human | 63 |
Rejection rate 49.8%. Half the learning signal I captured never became a rule. And 63 items are stuck waiting on a human decision. The automation I built ended up bottlenecked on me.
6. What I changed
Rules now live in three tiers:
- First violation: log only. Do not write a clause.
- Second: write the clause. But inject only the top 3 most-violated rules at the head of every turn, not the full document.
- Third: writing is no longer allowed to close it. Write a code gate, and do not close the item until a deliberately broken input is caught by that gate.
Tier three is the one that matters. As long as "I updated the clause" can close an item, the same violation comes back a fourth time.
Injecting only the top 3 is deliberate too. Add more and they stop being read. Adding unread clauses is what caused this in the first place.
7. What I did not measure
Being honest about the holes:
- n=1. One machine, one operator. No idea if this reproduces.
- I have not measured gate accuracy. My regression judge itself has a measured 66% false-positive rate at one sample (2 of 3 "confirmed" regressions passed on 3-sample re-measurement). Verifying the gates is still ahead of me.
- "Followed" is weakly defined. Recurrence counts start from a human noticing. Violations nobody caught are not counted. The real rate is higher.
Summary
- Growing instructions from 90KB to 147KB did not reduce violations
- 94.4% of tokens were re-reads of the same instructions, so token volume is not a work metric
- Across 24 rules: 8 moved into code recurred 0%, 16 left as documents recurred 100%
- Adding clauses was the symptom, not the fix
If anyone has run the same measurement on their own machine, I want to see the numbers. Specifically whether the Group B recurrence rate moves across environments.
Measured while building untactit, a control plane for the skills, rules and memory AI agents run on, currently in early access. The drift detection piece is open source as agent-drift, MIT, no dependencies, single file.
Top comments (0)