In the last article, I argued that the safety contract should move out of the LLM critic and into deterministic gates. The critic can vary. The gates should hold.
This one is about the failure mode I did not see coming: what happens when a gate stops working and nobody notices because the metrics look better.
The Problem That Was Hiding Inside the Solution
By v0.2.2, the blocker counts were dominated by structural families:
-
unsafe_sequencing: 226 blockers -
unverified_dependencies: 185 blockers -
weak_rollback: 86 blockers
Those numbers felt like evidence that the deterministic layer was doing its job. A high blocker count on structural categories meant the gates were firing.
Then a community reader named Artjoms Stukans pointed out the failure mode I had not modeled:
"If one blocker class stops firing after some refactor, your numbers only look better. 226 becomes 40 and that reads like plans got safer."
He described a Kubernetes incident where four releases in a row never actually ran. An old ReplicaSet kept one pod Running. Every health check passed. Every smoke test was green. Nothing said a word.
That is exactly the shape of this problem.
A deterministic gate is more reliable than an LLM critic. That is true. But it is also quieter. A non-deterministic critic advertises its own unreliability through label flips and evidence drift. A gate that silently stops firing fails the same way on every trial. Repeated runs agree. The agreement reads as confidence. You lose the variance signal exactly where the contract now lives.
Why the Existing Defenses Were Not Enough
Before Artjoms's comment, I would have said the system already had protection against this.
It did. Just not enough.
| Mechanism | What it detects | Why it misses silent gate death |
|---|---|---|
underclaim_approvals metric |
LLM critic failed to block a known-bad plan | Tests the LLM critic, not the deterministic gates |
| Deterministic gate unit tests | Each gate fires on its fixture | CI tests pass locally; the concern is production refactor |
| Regression detector | New blockers appearing between revisions | Detects blockers appearing, not blockers disappearing |
| Drift alert | Z-score spike in severity | Tracks severity changes, not total disappearance |
The gap was architectural: no mechanism asserted that each gate class still fires on a known-bad plan as a continuous health check. The gates were tested at build time. They were not monitored at runtime.
The Gate Canary
The fix is a gate canary — a deterministic, zero-LLM-cost health check that runs before every eval sweep and asserts each gate still fires on its canary bad plan.
Each gate class gets a (good_plan, bad_plan) pair:
| Gate | Good plan | Bad plan |
|---|---|---|
ordering |
t1 → t2 | t2 → t1 |
rollback_credible |
high risk + reachable rollback | high risk + unreachable rollback |
verification_ordering |
verify before consume | consume before verified |
preconditions |
precondition references established fact | precondition references nothing |
The canary runs as:
-
Pre-commit gate:
plancritic gates canary --check— fails CI if any gate stopped firing - Eval sweep integration: canary results appear in every field-test output
- Dashboard metric: one green/red per gate, visible at a glance
If a refactor or a bug silences the ordering gate, the canary turns red immediately. The blocker count dropping from 226 to 40 no longer reads as improvement. It reads as a regression that must be explained.
The Harder Lesson
The gate canary fixes the immediate gap. The harder lesson is about trust models.
I had been thinking about safety in two layers:
- Layer 1: the LM critic — unstable but flexible
- Layer 2: the deterministic gates — stable and authoritative
That framing is not wrong, but it is incomplete. The gates are stable only if someone watches them. Trusting a deterministic layer because it is deterministic is the same mistake as trusting a test suite because the tests pass. The question is not whether the gates are reliable by construction. The question is whether you know the moment they stop being reliable.
A deterministic safety layer does not remove the monitoring problem. It moves it.
Before the fix, the monitoring was on the critic's non-determinism (label_flip_rate, evidence_drift). After the fix, the monitoring has to be on the gates themselves — not on their outputs, but on their continued ability to fire.
That is what the gate canary does. It does not check whether the gate approved or blocked a particular plan. It checks whether the gate is still alive.
The Principle That Generalizes
If you build deterministic safety layers for LLM systems, ask yourself:
What is your equivalent of the Kubernetes ReplicaSet that kept one pod Running?
What component, if it stopped working, would produce better-looking metrics?
The answer is rarely obvious at design time. I only found mine because a reader described an incident from an unrelated system and the shape matched.
The general principle is:
- Every deterministic safety gate needs a canary that fires independently of the gate's normal operation.
- The canary must use a known-bad input so failure = silence is impossible to misinterpret.
- The canary status must be visible in the same dashboard as the gate metrics — not in a separate CI log that nobody reads.
That is the difference between a safety layer you trust and a safety layer you have not yet watched break.
Previous PlannerCritic articles
- A Reader Audited My OSS Release in Public
- My LLM Critic Disagreed With Itself on Every Trial
- My Agent Refused 96 Times. That Was the Right Output
- I Ran 157 Agent Plans Against a Real LLM
Links
- Gate Canary issue: #278
- F-20 deterministic-corruption blind spot: #296
- Source (Artjoms Stukans): dev.to comment
- V0.2.2 field-test report: https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/field-test/v0.2.2/field-test-results-0.2.2.md
Next in the sequence: Two Projects, One Problem — What PlannerCritic and AdversarialDebate Each Got Wrong
Top comments (0)