DEV Community

Cover image for My LLM Critic Disagreed With Itself on Every Trial. The Safe Part Was the Code I Didn’t Trust It to Touch.
Debashish Ghosal
Debashish Ghosal

Posted on AI-assisted

My LLM Critic Disagreed With Itself on Every Trial. The Safe Part Was the Code I Didn’t Trust It to Touch.

#ai #healthydebate #llm #testing

In the My Agent Refused 96 Times. That Was the Right Output., I argued that the most valuable output from an agent planner is often a well-structured refusal. This one is about the harder engineering question: what makes the refusal trustworthy when the model providing it is fundamentally unstable?

The short answer is: the model should not be the final authority.

The Metric That Should Have Scared Me

The live boundary evaluator (#218) is a small benchmark. It runs only 6 cases across 5 trials per plan and 2 plans per case — 60 audits. That is tiny compared to the full 183-goal sweep.

But it produces the most uncomfortable number in the whole project.

On identical input, the critic produced:

  • label_flip_rate = 1.000
  • evidence_drift_rate = 1.000

Different verdict on every trial. Different explanation every time. The model is not just slightly inconsistent. It is maximally non-deterministic on identical input.

If your safety story depends on the model being consistent, that number should terrify you.

Why It Didn't Terrify Me by the End

The reason it did not destroy confidence is simpler than I expected: the safety contract was not sitting on the critic alone.

The architecture had already moved the critical path into code.

That was not the first version of the story. I had to earn that position.

The First Version of the Story Was Too Clean

In v0.2.1, the boundary run looked reassuring:

  • family_migration_rate = 0.000
  • underclaim_approvals = 0

That made for a clean narrative: the critic is non-deterministic, but the system is still safe. I wrote that story in the v0.2.1 field-test results and it was directionally correct.

Then v0.2.2 made the story harder. The first boundary run regressed:

  • family_migration_rate = 0.033
  • underclaim_approvals = 1

A seeded defective plan got zero blockers. That is not explanation drift. That is a real under-claim.

The good news is not that the first run failed. The good news is that the benchmark caught it, and the rerun told us something useful about the architecture.

The problematic case was verifies-before-consume-vs-consumes-before-verified, plan b, trial 4. The boundary harness had been framing the synthetic goal as balanced rather than strict. The benchmark was partially measuring a posture problem, not a pure critic problem.

We changed the framing to strict and reran. The final result returned to:

  • family_migration_rate = 0.000
  • underclaim_approvals = 0

That is the story I trust, because it survived a failure first.

The Safety Boundary Moved Into Code in Three Places

1. The deterministic gates own structural under-claim

The gates do not read persuasive prose. They parse the plan structure. That means ordering, preconditions, rollback, and traceability can fail regardless of what the critic says.

By v0.2.2, the blocker counts across the full 183-goal sweep were still dominated by structural families:

  • unsafe_sequencing: 226 blockers
  • unverified_dependencies: 185 blockers
  • weak_rollback: 86 blockers
  • feasibility: 34 blockers

That is what a real safety floor looks like: failure modes the model cannot talk its way around.

2. The severity taxonomy moved out of the model's control

One of the earliest hard lessons in PlannerCritic was that an "adversarial" critic will happily block plans for being incomplete rather than unsafe. That became the severity bug from v0.1.0.

The fix was not better prompt engineering. It was a code-enforced allowlist:

_BLOCKER_ELIGIBLE_FAMILIES = frozenset({
    "unsafe_sequencing",
    "weak_rollback",
    "unverified_dependencies",
    "feasibility",
})
Enter fullscreen mode Exit fullscreen mode

Even if the LLM labels a finding as blocker, the system downgrades it when the family is not blocker-eligible. The model's label is decorative. The family is load-bearing.

3. The benchmark infrastructure had to become deterministic too

The biggest surprise of v0.2.2 was that the harness itself needed hardening:

  • wrong output roots
  • stale benchmark script paths
  • provider-dir resolution mismatches
  • redaction corrupting numeric JSON
  • missing malicious fixtures despite a closed issue

If the evaluator is loose, the architecture can look safer or weaker than it really is.

What the Final Numbers Actually Say

By the end of v0.2.2, the system story was:

  • inherited corpus stable at the top level
  • 73/73 balanced approved
  • 96/97 strict escalated
  • 8/8 inherited adversarial blocked
  • boundary rerun restored underclaim_approvals = 0

At the same time, the critic was still fully non-deterministic.

That combination is the lesson. The model is not what made the system safe. The model was useful. The code was authoritative.

The Principle That Generalizes

If an LLM is on your critical path, ask two separate questions:

  1. What is it allowed to observe?
  2. What is it allowed to decide?

PlannerCritic got safer when the answer to question two became: not very much.

The critic can suggest. The critic can surface evidence. The critic can vary.

But the deterministic layers own the contract.

That is why a critic that disagrees with itself on every trial can still sit inside a safe system.

The Learning I Did Not Expect

I thought the uncomfortable result would be that the critic was too unstable to trust.

The actual learning was sharper:

You do not fix critic instability by demanding a more stable critic. You fix it by shrinking how much the critic is allowed to decide.

That is a much more useful design rule.

Previous PlannerCritic articles

Links

Next in the sequence: My Safety Benchmark Failed on the First Run. That's the Only Reason I Trust the Final Result.

Top comments (0)