Open-source maintainer building Judgment Pack, a testable specification and runtime for AI-agent decisions.
Former enterprise data and software architect.
The finding that uncovered behavior can actually make compliance look better is especially interesting. Have you considered making “no applicable rule” a first-class unresolved state rather than allowing it to fall through to a default? That seems useful for separating real compliance from policy coverage gaps.
It shipped. Policy.match now returns no match instead of falling through to a default tier, and a verdict column carries compliant / deviation / unresolved:no_rule / unresolved:unknown_model. Nine tests, 437 pass. Your comment is named in the commit message.
The zero turned out to be the interesting half. unknown_model is empty because I added the claude-opus-5 tier two days before the verdict column went in, and those 5,992 unscoreable traces would all have landed there. So the two unresolved classes decay differently. unknown_model is an operational gap: fill in the tier table and the count returns to zero by itself. no_rule does not. Those 2 rows sit until somebody writes a rule or decides not to, and the count stays honest precisely because nothing clears it automatically.
Both coverage counts are in the summary now, decisions out of coverage and rules that never matched.
Open-source maintainer building Judgment Pack, a testable specification and runtime for AI-agent decisions.
Former enterprise data and software architect.
@lizhuojunx86 - this is exactly the distinction I was hoping the experiment would expose, and I like that you carried it all the way into the implementation.
The separation between unresolved:no_rule and unresolved:unknown_model is especially useful.
unknown_model is an operational knowledge gap. Add the missing model-to-tier mapping and the same historical records can become scoreable.
no_rule is different. It is a policy coverage gap. Nothing should automatically clear it because there is no judgment to recover - someone has to add a rule or explicitly decide the case should remain outside policy.
I also really like the dual coverage metrics you found:
decisions no rule reached
rules no decision reached
Those catch two different failure modes:
decision -> no rule
= uncovered behavior
rule -> no decision
= dead, shadowed, or unexercised policy
That makes unresolved much more than an error bucket. It becomes an observable property of the policy surface itself.
And your claude-code-guide example is a good demonstration of why this matters: the default didn't merely fabricate compliance, it could fabricate a deviation too. The honest answer was neither.
Thanks for actually measuring this, shipping the change, and closing the loop on the comment. This is a much stronger result than simply adding another fallback rule.
I hadn't, so I went and measured what that state would have held. More than I expected, and not in the direction I expected.
Frozen corpus, 425 decisions: 424 matched an explicit rule. The one that fell through is the product-manager row from the post — no rule mentions it, default frontier, ran Opus, scored compliant. $0.91.
Same replay on the live window, 784 decisions: the fall-through population has doubled. A claude-code-guide component first appeared on Jul 30, matched nothing, took the default frontier expectation, ran Haiku — and got flagged as a deviation. $0.08.
So a default fabricates verdicts in both directions. product-manager scored compliant on a rule nobody wrote; claude-code-guide scored deviant on a rule nobody wrote. The second is the worse failure: a fabricated deviation is noise in the one table the audit asks you to trust, and Haiku for a docs-lookup subagent is the right routing. What's wrong there is the policy's silence, not the model. An unresolved state catches both rows for the same reason — neither verdict has a rule behind it.
Two things the measurement added to your framing.
Fall-throughs have a dual: rules no decision ever reached. My file has seven rules, and one of them (research-explore outside main → cheap) has matched zero decisions in both windows. Not because the case is rare (the frozen corpus alone has 83 research-explore decisions), but because every component that does research has its own component rule sitting earlier in the file, and the specificity tie-break shadows the task rule. Dead rules and fall-throughs are the two halves of one coverage report: decisions no rule reached, rules no decision reached.
And the actual side of the comparison has the same hole. When claude-opus-5 entered the corpus with no tier entry, 5,992 traces sat unscoreable for two weeks — neither compliant nor deviant, silently. That's the post's unpriced-model incident wearing a different field. Expected side or actual side, whenever a verdict cannot be resolved the honest output is a third state, and the size of that state is your coverage metric.
So yes: unresolved goes in as a first-class verdict, with two counts next to it in the summary — decisions out of coverage, and rules that never matched.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The finding that uncovered behavior can actually make compliance look better is especially interesting. Have you considered making “no applicable rule” a first-class unresolved state rather than allowing it to fall through to a default? That seems useful for separating real compliance from policy coverage gaps.
It shipped. Policy.match now returns no match instead of falling through to a default tier, and a verdict column carries compliant / deviation / unresolved:no_rule / unresolved:unknown_model. Nine tests, 437 pass. Your comment is named in the commit message.
Current corpus, 789 decisions: 627 compliant, 160 deviation, 2 unresolved:no_rule, 0 unresolved:unknown_model.
The zero turned out to be the interesting half. unknown_model is empty because I added the claude-opus-5 tier two days before the verdict column went in, and those 5,992 unscoreable traces would all have landed there. So the two unresolved classes decay differently. unknown_model is an operational gap: fill in the tier table and the count returns to zero by itself. no_rule does not. Those 2 rows sit until somebody writes a rule or decides not to, and the count stays honest precisely because nothing clears it automatically.
Both coverage counts are in the summary now, decisions out of coverage and rules that never matched.
@lizhuojunx86 - this is exactly the distinction I was hoping the experiment would expose, and I like that you carried it all the way into the implementation.
The separation between
unresolved:no_ruleandunresolved:unknown_modelis especially useful.unknown_modelis an operational knowledge gap. Add the missing model-to-tier mapping and the same historical records can become scoreable.no_ruleis different. It is a policy coverage gap. Nothing should automatically clear it because there is no judgment to recover - someone has to add a rule or explicitly decide the case should remain outside policy.I also really like the dual coverage metrics you found:
Those catch two different failure modes:
That makes
unresolvedmuch more than an error bucket. It becomes an observable property of the policy surface itself.And your
claude-code-guideexample is a good demonstration of why this matters: the default didn't merely fabricate compliance, it could fabricate a deviation too. The honest answer was neither.Thanks for actually measuring this, shipping the change, and closing the loop on the comment. This is a much stronger result than simply adding another fallback rule.
I hadn't, so I went and measured what that state would have held. More than I expected, and not in the direction I expected.
Frozen corpus, 425 decisions: 424 matched an explicit rule. The one that fell through is the product-manager row from the post — no rule mentions it, default frontier, ran Opus, scored compliant. $0.91.
Same replay on the live window, 784 decisions: the fall-through population has doubled. A claude-code-guide component first appeared on Jul 30, matched nothing, took the default frontier expectation, ran Haiku — and got flagged as a deviation. $0.08.
So a default fabricates verdicts in both directions. product-manager scored compliant on a rule nobody wrote; claude-code-guide scored deviant on a rule nobody wrote. The second is the worse failure: a fabricated deviation is noise in the one table the audit asks you to trust, and Haiku for a docs-lookup subagent is the right routing. What's wrong there is the policy's silence, not the model. An unresolved state catches both rows for the same reason — neither verdict has a rule behind it.
Two things the measurement added to your framing.
Fall-throughs have a dual: rules no decision ever reached. My file has seven rules, and one of them (research-explore outside main → cheap) has matched zero decisions in both windows. Not because the case is rare (the frozen corpus alone has 83 research-explore decisions), but because every component that does research has its own component rule sitting earlier in the file, and the specificity tie-break shadows the task rule. Dead rules and fall-throughs are the two halves of one coverage report: decisions no rule reached, rules no decision reached.
And the actual side of the comparison has the same hole. When claude-opus-5 entered the corpus with no tier entry, 5,992 traces sat unscoreable for two weeks — neither compliant nor deviant, silently. That's the post's unpriced-model incident wearing a different field. Expected side or actual side, whenever a verdict cannot be resolved the honest output is a third state, and the size of that state is your coverage metric.
So yes: unresolved goes in as a first-class verdict, with two counts next to it in the summary — decisions out of coverage, and rules that never matched.