It shipped. Policy.match now returns no match instead of falling through to a default tier, and a verdict column carries compliant / deviation / unresolved:no_rule / unresolved:unknown_model. Nine tests, 437 pass. Your comment is named in the commit message.
The zero turned out to be the interesting half. unknown_model is empty because I added the claude-opus-5 tier two days before the verdict column went in, and those 5,992 unscoreable traces would all have landed there. So the two unresolved classes decay differently. unknown_model is an operational gap: fill in the tier table and the count returns to zero by itself. no_rule does not. Those 2 rows sit until somebody writes a rule or decides not to, and the count stays honest precisely because nothing clears it automatically.
Both coverage counts are in the summary now, decisions out of coverage and rules that never matched.
Open-source maintainer building Judgment Pack, a testable specification and runtime for AI-agent decisions.
Former enterprise data and software architect.
@lizhuojunx86 - this is exactly the distinction I was hoping the experiment would expose, and I like that you carried it all the way into the implementation.
The separation between unresolved:no_rule and unresolved:unknown_model is especially useful.
unknown_model is an operational knowledge gap. Add the missing model-to-tier mapping and the same historical records can become scoreable.
no_rule is different. It is a policy coverage gap. Nothing should automatically clear it because there is no judgment to recover - someone has to add a rule or explicitly decide the case should remain outside policy.
I also really like the dual coverage metrics you found:
decisions no rule reached
rules no decision reached
Those catch two different failure modes:
decision -> no rule
= uncovered behavior
rule -> no decision
= dead, shadowed, or unexercised policy
That makes unresolved much more than an error bucket. It becomes an observable property of the policy surface itself.
And your claude-code-guide example is a good demonstration of why this matters: the default didn't merely fabricate compliance, it could fabricate a deviation too. The honest answer was neither.
Thanks for actually measuring this, shipping the change, and closing the loop on the comment. This is a much stronger result than simply adding another fallback rule.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
It shipped. Policy.match now returns no match instead of falling through to a default tier, and a verdict column carries compliant / deviation / unresolved:no_rule / unresolved:unknown_model. Nine tests, 437 pass. Your comment is named in the commit message.
Current corpus, 789 decisions: 627 compliant, 160 deviation, 2 unresolved:no_rule, 0 unresolved:unknown_model.
The zero turned out to be the interesting half. unknown_model is empty because I added the claude-opus-5 tier two days before the verdict column went in, and those 5,992 unscoreable traces would all have landed there. So the two unresolved classes decay differently. unknown_model is an operational gap: fill in the tier table and the count returns to zero by itself. no_rule does not. Those 2 rows sit until somebody writes a rule or decides not to, and the count stays honest precisely because nothing clears it automatically.
Both coverage counts are in the summary now, decisions out of coverage and rules that never matched.
@lizhuojunx86 - this is exactly the distinction I was hoping the experiment would expose, and I like that you carried it all the way into the implementation.
The separation between
unresolved:no_ruleandunresolved:unknown_modelis especially useful.unknown_modelis an operational knowledge gap. Add the missing model-to-tier mapping and the same historical records can become scoreable.no_ruleis different. It is a policy coverage gap. Nothing should automatically clear it because there is no judgment to recover - someone has to add a rule or explicitly decide the case should remain outside policy.I also really like the dual coverage metrics you found:
Those catch two different failure modes:
That makes
unresolvedmuch more than an error bucket. It becomes an observable property of the policy surface itself.And your
claude-code-guideexample is a good demonstration of why this matters: the default didn't merely fabricate compliance, it could fabricate a deviation too. The honest answer was neither.Thanks for actually measuring this, shipping the change, and closing the loop on the comment. This is a much stronger result than simply adding another fallback rule.