This is article 2 in a series about building PlannerCritic, an open-source engine where one LLM writes a plan and a second LLM reviews it. Article...
For further actions, you may consider blocking this person and/or reporting abuse
This is a good separation between probabilistic classification and deterministic consequence. The model can propose a finding, but code decides whether that finding is allowed to stop execution.
One failure mode I’d test next is label migration. The allowlist prevents
missing_stepsfrom becoming a blocker, but the model may describe the same concern asunsafe_sequencing, which is blocker-eligible. The guardrail still trusts the model to choose the family correctly.I’d keep the raw finding and normalized family in the evaluation data, then build boundary cases that differ by one fact: optional step versus required dependency, possible latency versus unsafe ordering, rollback improvement versus no viable rollback. A confusion matrix around those boundaries would show whether the guard removed the bad decision or merely moved it into classification.
For the highest-impact cases, a deterministic invariant such as “all irreversible steps have a verified predecessor and rollback condition” is stronger than accepting any model-selected blocker label.
This is a valuable extension, and it names exactly the next weakest seam in the design. The reason I kept model findings in the first place was to avoid pretending the model is deterministic; but if the label it chooses is what decides consequence, then I've just loaded the door decision back into the model. Separating the raw finding from the normalized decision that is allowed to hold authority is exactly the honesty I need.
On the boundary-case idea you lay out, differing by one fact, that's a really productive test to structure. If the guard only changes the classification while the decision still maps to the model, then nothing has been gained. I'll adopt the confusion-matrix framing in the next pass, and I'll keep both the raw finding and the normalized family in the eval data so the migration shows up as data instead of silently.
Counter-question: in your experience, has the label migration shown up mostly around a single fact boundary, or as a general drift? I'd genuinely like to know where you'd put the first boundary case. Thanks for the read, this is exactly the kind of review that moves the design forward.
I’d expect both, but I’d look for single-fact boundary failures first because they are easier to diagnose. General drift often appears later as the prompt, model or surrounding context changes.
Rollback is probably the cleanest first fixture. Give the critic two otherwise identical plans:
The first is a completeness warning. The second is a concrete execution-safety defect. If both become
weak_rollback, or the first migrates intounsafe_sequencing, the classifier is absorbing the consequence decision.I’d run each pair repeatedly and retain the explanation alongside both labels. That reveals not only label variance, but whether the model is changing the underlying fact it claims to have observed. The latter is more dangerous because a normalization layer cannot repair invented evidence.
Thank you — the rollback boundary twin you described already exists: generate_boundary_cases() in label_migration.py, shipped with #171 in v0.2.0, builds exactly the executable-but-imperfect versus named-but-nonexistent pair. What we had not built is your repeated-trial protocol — running each pair against a live critic multiple times while retaining explanations to separate label variance from invented evidence. That is now tracked: Live-critic boundary-case runner. Your point that a normalization layer cannot repair fabricated facts became an explicit evidence-drift metric in the design. Appreciate the precision here.
That is a solid implementation. One detail I’d preserve in the runner is the complete decision context for every trial: model version, temperature, system prompt hash, tool-schema hash, normalized finding, and the evidence cited in the explanation.
Otherwise a changed label can look like stochastic variance when the real cause was a prompt or tool-definition change between runs. I’d also score evidence validity separately from label stability. A critic that chooses the same label ten times while repeatedly citing a nonexistent function is stable, but still unsafe.
The useful output is probably two rates: decision disagreement across identical trials, and unsupported-evidence frequency. They expose different failure modes.
Both gaps conceded — they are real in the runner as shipped. Trial records today store verdicts and explanation text only, so a label shift caused by a prompt or tool-definition change between runs is indistinguishable from stochastic variance; and the evidence metric measures cross-trial consistency, which means a critic citing the same nonexistent function ten times scores perfectly stable and perfectly undetected. Your stable-but-still-unsafe case is the sharpest formulation of the blind spot.
Filed as decision-context capture plus unsupported-evidence frequency (#242): every trial records model id and version, temperature, system-prompt hash, tool-schema hash; explanations get their claimed facts extracted and validated against the boundary plans, whose ground truth is fully known; and the report leads with your two rates — decision disagreement across identical trials, and unsupported-evidence frequency — as separate axes. Thank you again; both rounds of this thread improved the harness materially.
This is a good bug because the code fix admits the prompt will drift. I like the severity whitelist more than the wording change. Any reviewer prompt can learn to sound stricter over time; the downgrade rule gives you one boring place where the product contract wins.
I think you've nailed the real point, which is to stop fighting prompt drift by trying to freeze the wording, and instead to invert it. The system admits that whatever the critic prompt becomes over time, there's going to be a fixed place where the product contract wins: a list that says which finding becomes a stop and which one is only advisory. That's a more durable separation than trying to get the model never to sound over-strict.
Where I'd keep digging: keep the raw finding and the normalized severity in the event data, always, so drift is visible instead of silent. That way the missed mapping shows up as data instead of as a surprise. Thanks for the read, it's validating, and "one boring place where the product contract wins" is a line I'm going to steal. Appreciate the time.
The six-word prompt is a perfect case of a spec that seems obvious but isn't. Completeness objections vs safety objections look the same inside the critic - you never told it which one counts as a blocker. I've hit similar drift with a "strict" flag that ended up meaning something different in each pipeline stage. Did you consider a warning-mode path where completeness concerns surface but don't block, so the human reviewer still sees them?
Tae, yes — and the warning-mode path is exactly how the critic layer works today after the fix. The _BLOCKER_ELIGIBLE_FAMILIES frozenset means every critic finding lands as a warning by default; only findings in the concrete defect families (unsafe_sequencing, weak_rollback, unverified_dependencies, feasibility) can escalate to blocker. Completeness concerns stay visible as warnings in the escalation payload — the human still sees them — but they can never block the pipeline alone.
The six-word prompt was a good lesson in how a persona instruction is not a severity taxonomy. We've since made the severity an API contract enforced in code, not a sentiment the prompt negotiates (github.com/deghosal-2026/planner-c... tracks the red-team harness for this boundary).
On the drift problem — the notable part of this design is that the blocker/warning split lives in code, not in a "strict" flag. The six gates are hard blockers (deterministic, injection-immune — the LLM never votes on them), and the critic's findings are advisory by default: only families in _BLOCKER_ELIGIBLE_FAMILIES can escalate to blocker. So a warning-mode path isn't an add-on here — it's the critic layer's default lane, and the taxonomy can't drift between stages because it's a frozenset at the boundary, not prompt text.
The seam I'd watch is the warning lane's coverage: the gates are a fixed list, so any under-claim defect class they don't enumerate (e.g. a rollback that's present but unreachable — runs after the failure point, or depends on the resource it destroys) lands in the advisory lane no matter how strict the taxonomy is. That's where a fixture pair pinning "verifies-before-mutate vs verifies-after-mutate" would earn its keep.
Worth noting what the frozenset guardrail does not do: it is one-directional. It can only downgrade an over-claim (blocker -> warning); it can never upgrade an under-claim. A real structural defect that the critic happens to label 'risk' or 'missing_steps' passes straight through as advisory and the plan proceeds. In safety terms the under-claim is the worse failure: an over-claim costs a human a false escalation, an under-claim lets an unsafe plan run. Pairing the frozenset with a small structural pre-pass that runs before the critic would cover that direction - deterministic checks like 'does the plan contain a rollback section at all', 'do irreversible steps name a rollback condition'. If one fails, that is a blocker the severity taxonomy never gets to vote on. Then the frozenset owns the over-claim direction, the pre-pass owns the under-claim direction, and both stay code rather than prompt.
Great observation — and you're absolutely right, the guardrail is one-directional by design. But the good news is: the engine already has exactly the pre-pass you're describing.
The six deterministic gates (schema_valid, no_dep_cycles, ordering_sane, verification_present, rollback_present, preconditions_referenced (github.com/deghosal-2026/planner-c...) run before the LLM critic and are injection-immune — the LLM never gets to vote on them. If rollback_present finds a high-blast-radius step with no rollback, that's a blocker regardless of what the critic labels it. The frozenset guardrail only applies to the critic's own findings; the deterministic gates own the under-claim direction independently.
So the architecture is exactly what you described:
Both stay code, not prompt. The field test (github.com/deghosal-2026/planner-c...) verified this across 65 goals — zero gate false negatives, zero blocker noise from the critic. The injection-immunity regression corpus (github.com/deghosal-2026/planner-c...) (coming in v0.2.0) will back this with real-CVE-grounded test cases.
Good to know the six gates are exactly that pre-pass - the two directions do split cleanly. One seam remains: the gates are a fixed list, so the under-claim direction is only as strong as its coverage. rollback_present catches 'no rollback section', but does it catch a rollback that is unreachable (runs after the failure point, or depends on the resource the plan destroys)? And does ordering_sane cover verification-before-irreversible, or only dependency ordering? If verification can pass yet run after the mutate step, a critic mislabeling that defect as 'risk' reopens the under-claim direction at the seam between gate list and taxonomy. A fixture pair - verifies-before-mutate vs verifies-after-mutate - would pin that down.
Both seams confirmed — I checked the gates directly before responding. rollback_present fires only when a rollback section is absent, and ordering_sane only validates hard-dependency positions, so an unreachable rollback and a verification that runs after its result was consumed both pass clean today. Filed two issues: Rollback credibility gate covering unreachable, self-dependent, inconsistent-state, and post-consumed rollback patterns, and Verification-before-mutate ordering gate including the verifies-before-mutate versus verifies-after-mutate fixture pair you specified, wired into the label-migration harness. Together they close the under-claim direction at the seam between the fixed gate list and the severity taxonomy. Thanks for naming the exact fixtures — they went straight into the acceptance criteria.
The fixture pair went in — one addition that will save #219 from its own false-positive class: position alone can't be the predicate, because legitimate verification exists on both sides of the mutation. A check that reads pre-mutation state or constraints (the rollback precondition, the invariant before the write) must run before the mutate; a check that reads the mutation's output must run after it, or it verifies nothing. If the gate flags "after" unconditionally, that second class becomes a permanent false positive, and the natural fix under pressure is to move verification earlier — which silently turns output checks into pre-state checks. That's the same vacuous-verification drift #219 is meant to kill, just wearing the gate as its cover.
Keying the gate on the verification's data subject (what it reads) instead of its position handles both directions: subject = pre-state/constraint → require before; subject = mutation output → require after, and require it to actually consume the output (a reading-the-output assertion, same spirit as the rollback gate's reachability check). The fixture pair is really a triplet: verifies-before-mutate (must pass), verifies-after-mutate-on-pre-state (must fail), verifies-after-mutate-on-output (must pass). Third member is the one that stops the false-positive class from training the gate into submission.
Shipped exactly as you specified. The verification-ordering gate now keys on data subject rather than position: checks reading pre-mutation state or constraints must run before the mutate and are legitimate there; checks reading the mutation output must run after it and must actually consume that output. Position alone flags nothing (#230, completing #219). The fixture is your triplet — verifies-before-mutate passes, verifies-after-mutate-on-pre-state fails, verifies-after-mutate-on-output passes — with the third member doing exactly the job you named: stopping the false-positive class from training the gate into moving every check early. Interim subject derivation uses a prose prefix convention; a typed optional subject field replaces it in the 0.3.0 schema evolution. Thanks for refusing to let the fixture ship as a pair.
This is a great example of why AI systems need boundaries beyond the prompt itself. An LLM can be excellent at identifying potential issues, but deciding which issues should actually block a workflow is a different problem. In the kind of AI automation work we explore at IT Path Solutions, this distinction is especially important, reasoning can stay probabilistic while critical enforcement remains deterministic.
Glen, that is the exact design principle the engine now follows. The LLM critic is excellent at surfacing issues — it found 346 findings across the field test. But the decision about which issues are allowed to stop execution belongs to code, not the model. The deterministic gates (schema, cycles, ordering, rollback, preconditions, parallel safety) run before the critic and are injection-immune — the model never votes on them. The frozenset guardrail then constrains which critic families can escalate to blocker. Probabilistic identification, deterministic consequence.
The next step is making the under-claim direction observable too — tracking when the critic mislabels a structural defect as an advisory family, so that direction is visible as data rather than silent (github.com/deghosal-2026/planner-c..., planned for 0.2.0).
Hi Debashish, I've run into this same problem. I have created a quality control and a claude.md checker subagent to make sure that the plan is up to my standards and follows my Claude.md file. I found that when I tell claude to review a plan, code review, etc... it will always find at least one problem. Like you said, it's usually a completeness concern rather than a concrete example.
How did you come up with the _BLOCKER_ELIGIBLE_FAMILIES set, and is it possible to extend the set to include business logic cases? The blockers that my agent finds is usually related to business logic that is not possible.
Hi Shayan, thanks for reading and sharing — great to hear someone else hit the same wall!
The _BLOCKER_ELIGIBLE_FAMILIES set came directly from the 0.1.0 field test. I classified 346 findings by heuristic family and noticed: families like unsafe_sequencing, weak_rollback, unverified_dependencies, and feasibility produce structurally verifiable defects — you can check them in code. Families like risk and missing_steps produce advisory noise — "this seems risky" isn't actionable.
So the guardrail is simple: if the critic says "blocker" but the family is advisory, downgrade to warning. Only code-verifiable families earn blocker status. This cut blocker noise to 0% across all findings (field test results (github.com/deghosal-2026/planner-c...) · finding-quality audit test (github.com/deghosal-2026/planner-c...).
Yes, you can extend it for business logic — and that's exactly what the v0.2.0 Domain Pack framework (#139 (github.com/deghosal-2026/planner-c...) is for. The principle: add a new family to the blocker-eligible set only if you can write a deterministic gate for it. Your claudemd checker sounds like a perfect candidate — "does the plan reference every mandatory section?" is structural, not a judgment call.
If the business rule can be expressed as code (or an OPA/Rego policy — #129 (github.com/deghosal-2026/planner-c...), it earns blocker status. If it can't, keep it as a warning — the critic's opinion, not a hard gate. Full details in the field test plan (github.com/deghosal-2026/planner-c...) and test suite (github.com/deghosal-2026/planner-c...).
The frozenset downgrade is the durable part of this fix. Prompt severity rules will drift as models update, the code guardrail won't. I ran into the same pattern with an adversarial reviewer on infra changes: left unconstrained it optimizes for having an objection, and 'could also consider X' is the cheapest objection to generate. One thing I'd add: log every blocker-to-warning downgrade as a metric. If that rate climbs after a model swap, it's an early signal of judge drift before escalation counts blow up. Did you end up tracking downgrades, or just the post-fix blocker counts?
Great instinct — and yes, downgrades are tracked as of v0.2.0. drift.py records raw_severity versus normalized_severity for every critic finding, computes downgrade_rate with a per-family breakdown, and check_drift_alert() raises a z-score alert when a family's drift rate spikes over a trailing window — precisely the early judge-drift signal you describe after a model swap. It tracks critical_underclaims separately too, since a blocker demoted inside risk or missing_steps is the direction that actually hurts. You're right that post-fix blocker counts alone would never have caught it. Thanks for pushing the metric framing.
The deterministic guardrail is the right instinct and I'd generalize it past severity labels. We had a version of this with a hallucination judge: the prompt said something like 'flag anything unsupported' and it started flagging paraphrase and citation style as unsupported, not just factual errors, so our false positive rate sat around 30 percent for weeks before customer complaints made us look. The judge prompt was vague, so we rewrote it with concrete examples of what counted as unsupported versus what didn't. That took false positives from 30 percent down to 4. Your frozenset check is a stronger version of that because it doesn't rely on the model reading the examples correctly every time, it just refuses to let a disallowed severity out the door. One thing I'd want to see on your 132 post-fix blockers: did you sample any of them to check the code guardrail never downgraded something that actually deserved to block? A deterministic filter on top of a judge label is only as safe as the taxonomy underneath it, and that's usually where these things quietly go wrong.