This is a follow-up to part four, not a new part. Part four measured my stated routing policy against what actually ran: 425 decisions, 96 deviations, $1,248.13, none of them on the main thread.
Then a reader asked what happens when no rule applies. I went to measure it, and the answer was that my checker had been inventing verdicts. In both directions.
The question
Brian Jin, in the comments:
Have you considered making "no applicable rule" a first-class unresolved state rather than allowing it to fall through to a default? That seems useful for separating real compliance from policy coverage gaps.
I had not. My policy file has a default_tier, and any decision that matched no rule quietly took it. That felt safe when I wrote it. Every decision gets a verdict, no gaps in the table, nothing to explain.
What the default was doing
Frozen corpus, the same 425 decisions from part four: 424 matched an explicit rule. One did not.
That one is a product-manager component. No rule in my file mentions it. It took the default frontier expectation, ran Opus, and scored compliant. $0.91.
Replaying the same policy on the live window, 784 decisions, the fall-through population has doubled. A claude-code-guide component first appeared on July 30, matched nothing, took the same default frontier expectation, ran Haiku, and got flagged as a deviation. $0.08.
Two rows, two verdicts, opposite signs, and neither one has a rule behind it. The default fabricated compliance in the first case and fabricated a deviation in the second.
The fabricated deviation is the worse half
A fake compliant row is a missed catch. Annoying, but the audit was never going to catch what it does not know about.
A fake deviation is worse, because it puts noise into the one table the whole exercise asks you to trust. And in this case the routing was right: Haiku for a docs-lookup subagent is exactly what I would choose. What is wrong there is not the model. It is my policy's silence.
If I had acted on that row I would have gone and "fixed" a correct routing decision.
An unresolved state catches both rows for the same reason. Neither verdict has a rule behind it, so the honest output is not a verdict at all.
The dual: rules no decision ever reached
Measuring the fall-throughs turned up their mirror image, which I had not gone looking for.
My file has seven rules. One of them, research-explore outside the main thread to the cheap tier, has matched zero decisions in both windows.
Not because the case is rare. The frozen corpus alone has 83 research-explore decisions. Every component that does research has its own component-level rule sitting earlier in the file, and the specificity tie-break shadows the task rule every time. The rule is dead and the file gives no sign of it.
So coverage has two halves, and they fail differently:
| direction | what it means |
|---|---|
| decision → no rule | uncovered behavior |
| rule → no decision | dead, shadowed, or unexercised policy |
Those two labels are Brian's, not mine. I had the two halves and called them a coverage report; he named them after the change shipped, and the names are better than what I was using.
A policy file that grows for six months accumulates both, and reading it will not tell you which rules are load-bearing.
The same hole on the actual side
The expected side of the comparison is not the only one that can fail to resolve.
When claude-opus-5 entered my corpus with no entry in the tier table, 5,992 traces sat unscoreable for two weeks. Not compliant, not deviant, silently absent from the table. That is part four's unpriced-model incident wearing a different field.
Expected side or actual side, whenever a verdict cannot be resolved the honest output is a third state, and the size of that state is the coverage metric.
What shipped
Policy.match now returns no match instead of falling through, and a verdict column carries four values: compliant, deviation, unresolved:no_rule, unresolved:unknown_model. The default_tier key still exists, but it only applies to a rule that matched and omitted its expected tier. Nine new tests, 437 pass.
The summary carries the two coverage counts next to the verdict tallies: decisions out of coverage, and rules that never matched.
Current corpus, 789 decisions:
| verdict | count |
|---|---|
| compliant | 627 |
| deviation | 160 |
| unresolved:no_rule | 2 |
| unresolved:unknown_model | 0 |
That zero turned out to be the interesting entry. unknown_model is empty because I added the claude-opus-5 tier two days before the verdict column went in, and those 5,992 traces would all have landed there.
So the two unresolved classes decay differently, which is why they are counted apart. unknown_model is an operational gap: fill in the tier table and the count returns to zero by itself. no_rule does not. Those two rows sit until somebody writes a rule or decides the case belongs outside policy, and the count stays honest precisely because nothing clears it automatically.
Brian's reply when I showed him the result put it better than I had:
That makes unresolved much more than an error bucket. It becomes an observable property of the policy surface itself.
What I still get wrong
Tae Kim, in the same comment thread, described his orchestrator defaulting to Opus on every retry. He had only specified the model for the main agent, and it ran about three weeks before he caught it by accident.
My method would miss that. The grain is (unit, component), and a retry is not a separate component. It inherits inside the same one, so the deviation and the compliant call collapse into a single verdict. Same defect as mine, invisible to my checker.
I have not fixed it, because I have exactly one corpus and it is my own, which is a poor basis for a schema change. If your orchestrator writes the model per attempt rather than per step, I would like to see a sample.
If you want to run this
The policy file is 30 lines of YAML and the checker reads Claude Code's own session transcripts. Nothing leaves your machine.
python -m traceguard.routing_audit.agent_lint
That is the one-minute version: it reads .claude/agents/**/*.md frontmatter and tells you which agents never pinned a model. The full per-decision scoring needs the trace store, and it is in traceguard.
The part worth stealing even if you never run any of it: if your compliance checker has a fallback default, it is producing verdicts nobody wrote a rule for, and you cannot tell those apart from the real ones by looking at the output. Count them separately. Count the dead rules too.
Questions welcome, and if you have found a way to make the retry case visible I would rather hear it than work it out myself.
Top comments (0)