Discussion on: My routing policy and my traces disagreed 96 times. Never once on the main thread.

[Brian Jin]

@lizhuojunx86 - this is exactly the distinction I was hoping the experiment would expose, and I like that you carried it all the way into the implementation.

The separation between unresolved:no_rule and unresolved:unknown_model is especially useful.

unknown_model is an operational knowledge gap. Add the missing model-to-tier mapping and the same historical records can become scoreable.

no_rule is different. It is a policy coverage gap. Nothing should automatically clear it because there is no judgment to recover - someone has to add a rule or explicitly decide the case should remain outside policy.

I also really like the dual coverage metrics you found:

decisions no rule reached
rules no decision reached

Those catch two different failure modes:

decision -> no rule
= uncovered behavior

rule -> no decision
= dead, shadowed, or unexercised policy

That makes unresolved much more than an error bucket. It becomes an observable property of the policy surface itself.

And your claude-code-guide example is a good demonstration of why this matters: the default didn't merely fabricate compliance, it could fabricate a deviation too. The honest answer was neither.

Thanks for actually measuring this, shipping the change, and closing the loop on the comment. This is a much stronger result than simply adding another fallback rule.