DEV Community

Cover image for Two Projects, One Problem — What PlannerCritic and AdversarialDebate Each Got Wrong
Debashish Ghosal
Debashish Ghosal

Posted on AI-assisted

Two Projects, One Problem — What PlannerCritic and AdversarialDebate Each Got Wrong

#ai #healthydebate #dogfood #testing

I built two systems this year that try to solve the same problem from opposite directions.

PlannerCritic puts an LLM critic in a loop with a deterministic gate layer. The gates own the safety contract. The critic is advisory. A planner decomposes a goal into a typed plan. A critic audits every subtask. The plan is revised until approval — or escalated to a human.

AdversarialDebate puts two LLMs in a structured debate. The protocol owns the independence. The models are the evidence. Two reviewers analyze the same artifact without seeing each other's answers, debate their conclusions point by point, and produce either a converged decision — or a structured disagreement report that preserves the dissent.

Both systems are about the same question: how do you know when an LLM's judgment is wrong?

I thought the answer was architecture. Build the right structure, put code where the model can't be trusted, and the system becomes reliable. That was the thesis of both projects. PlannerCritic bet on code. AdversarialDebate bet on structure.

I was half right. The architecture worked. The systems are reliable. But both had the same blind spot, and it took me months to see it — because the blind spot made the metrics look better, not worse.

What Each System Bet On

PlannerCritic's bet was code over model. The decision path is a frozenset of blocker-eligible families — unsafe_sequencing, weak_rollback, unverified_dependencies, feasibility. The LLM critic can label a finding as blocker, but if the family isn't in the frozenset, the system downgrades it to a warning. The model's severity label is decorative. The family is load-bearing.

AdversarialDebate's bet was structure over consensus. A revelation gate ensures neither reviewer sees the other's output before committing. The debate protocol requires point-by-point responses — CONCEDED, REBUTTED, or CARRIED. The convergence score measures movement, not agreement. Two models that agree without engaging get a low score. Two models that genuinely challenge each other get a high one.

On paper, those bets are compatible. Code guards against structural under-claim. Debate guards against shared hallucination. Together, they cover more ground than either alone.

But each bet had a failure mode that the other system's design would have caught immediately. And neither system caught its own — because the failure made the dashboard look like things were improving.

PlannerCritic's Failure: The Gate That Stopped Blocking

In PlannerCritic, the deterministic gates own safety. They parse the plan's structure — ordering, preconditions, rollback, verification — and they block anything that doesn't meet the contract. By v0.2.2, the blocker counts across the full 183-goal sweep were dominated by structural families: unsafe_sequencing at 226 blockers, unverified_dependencies at 185, weak_rollback at 86.

Those numbers felt like evidence that the gates were doing their job.

Then a community reader named Artjoms Stukans left a comment that reframed how I thought about the entire system:

"If one blocker class stops firing after some refactor, your numbers only look better. 226 becomes 40 and that reads like plans got safer."

He described a Kubernetes incident where four releases in a row never actually ran. An old ReplicaSet kept one pod Running. Every health check passed. Every smoke test was green. Nothing said a word — because the pod was still in a "healthy" state, even though the deployment was broken.

That is exactly the shape of PlannerCritic's problem. A gate that silently stops firing produces better metrics. The blocker count drops. The dashboard reads as improved safety. The variance signal — label_flip_rate, evidence_drift_rate — the very metrics the project uses to surface critic problems — vanishes exactly where the safety contract moved. The critic is watched. The gates are trusted.

Trusted layers don't advertise their own failures. That's the trap.

AdversarialDebate already had the pattern for catching this. Each debate produces a transcript, and the capitulation detector reads that transcript for the shape of failure instead of trusting the aggregate convergence score. It doesn't ask "did the debate converge?" It asks "did one side concede everything in round 1 without a single rebuttal?" Those are different questions, and only the second one catches the failure.

PlannerCritic had no equivalent. It tested the gates at build time. It did not monitor them at runtime.

The fix was the Gate Canary — ten fixture pairs, one per gate class, each with a known-good plan and a known-bad plan. The CLI command plancritic gates canary --check runs them in under a second at zero LLM cost and exits 1 if any gate stopped firing on its bad plan. It's a dumb check. It's a cheap check. It catches exactly the failure mode that would have been invisible in every other metric the system produces.

It shipped in v0.2.3.

AdversarialDebate's Failure: The Pair That Surrendered

In AdversarialDebate, the strongest pair by convergence score was DeepSeek + Mistral: 0.982 average convergence, 97% verdict rate, 2,352 concessions across 41 debates. Every metric in the field test report pointed to this pair as the best production default.

Then I read the transcripts.

65% of those debates were capitulation cascades — one side conceding every claim in round 1 with zero rebuttals. The pair converged beautifully. It was almost useless for adversarial review.

The metrics saw convergence. The transcripts saw surrender.

PlannerCritic would have caught this. Its deterministic gates cannot be talked into approving a plan. A capitulation cascade that passes every aggregate check looks structurally different to a gate that inspects individual preconditions and rollback steps. The debate metrics measured whether the models agreed. PlannerCritic's gates would have measured whether the plan was safe. Those are not the same question, and only the second one catches the failure.

AdversarialDebate had no deterministic floor. The protocol was fine; the pair was bad; the metrics could not distinguish the two.

The fix was the capitulation detector — transcript-level inspection independent of convergence scores. But it was a detection mechanism added after the first surprise, not a health check designed before it.

Then v0.2.2 went further. The noise-floor baseline gave every metric a known confidence interval via bootstrap resampling — the GPT+Mistral vs DeepSeek+Mistral gap (0.536 vs 0.572) is 1.8 sigma, real but narrow. The permutation control shuffled claim-to-ground-truth pairings 500 times to build a null distribution — the LLM judge's 87.4% match rate sits 77.8 standard deviations above the vocabulary floor. The shared RLHF priors design note documented that the Mistral effect might be driven as much by non-Mistral models rubber-stamping each other as by Mistral's unique training.

Each fix was a separate measurement layer, added because the previous layer couldn't see the failure.

The Same Root Problem

Both failures have the same anatomy:

A layer you trusted failed silently, and the system interpreted the silence as improvement.

  • In PlannerCritic, a broken gate looks like fewer blockers — improved safety
  • In AdversarialDebate, a capitulating pair looks like higher convergence — improved debate

The fix in both cases was the same pattern: a secondary check that inspects the shape of the output, not just its aggregate value. PlannerCritic needed the gate canary. AdversarialDebate needed the capitulation detector. Neither check is expensive. Neither is clever. Both are the kind of thing you add after the first time the metrics lie to you.

Here's what I find most uncomfortable about this: I built both systems specifically because I didn't trust LLM judgment. I moved the safety contract into deterministic code. I built isolation protocols. I measured non-determinism. And then I trusted the deterministic layer so completely that I stopped watching it. The monitoring I built was for the LLM — label_flip_rate, evidence_drift_rate, capitulation_cascade. The layer that actually held the contract — the gates, the debate protocol — had no monitoring at all.

A deterministic safety layer does not remove the monitoring problem. It moves it.

Before the gate canary, PlannerCritic monitored the critic's non-determinism. After the fix, monitoring also had to cover the gates themselves — not their aggregate output, but their continued ability to fire.

Before the capitulation detector, AdversarialDebate monitored protocol compliance — rounds completed, claims resolved. After the fix, monitoring also had to cover whether the protocol was producing genuine engagement or procedural surrender.

The monitoring target shifts when you trust the deterministic layer. That is the blind spot both projects hit. And it's the blind spot I think most agent infrastructure has right now — we're all building deterministic guardrails and then walking away.

What v0.2.3 and v0.2.2 Actually Fixed

Both projects shipped their fixes in late August 2026. The fixes are real, tested, and in production.

PlannerCritic v0.2.3 (release notes, field test report):

  • Gate Canary: 10/10 gate fixtures passing in dev and Docker
  • Transit-integrity check: 0 corruption events on the boundary run — numeric JSON fields survive redaction
  • DecisionContext populated: trial records carry model_id, version, temperature, timestamp from the provider spec, not from the model's self-report
  • Approving_authority enforcement: PermissionError now fires from CLI, HTTP, and MCP — not just tests
  • 183 goals re-run: 0 errors, same approval/escalation distribution as v0.2.2

AdversarialDebate v0.2.2 (release notes, field test report):

  • Noise-floor baseline: bootstrap 10,000 resamples per pair, 95% CIs on every metric
  • Permutation control: 500 shuffles, null distribution with z-scores
  • Shared RLHF priors: documented competing causal mechanisms for the Mistral effect
  • 21 new tests, zero new LLM calls, $0.00 cost

Both projects now have measurement infrastructure that catches the failure class this article describes. But the infrastructure was built after the failure was found — by reading transcripts, by community comments, by running the benchmark and getting a number I didn't want.

What I Would Build First in the Next System

If I were building a third system — or advising someone building their first — I would start with what both projects missed.

A cross-system invariant checker. Not a feature. Not a gate. A dumb, cheap, separate utility that runs after every eval and asserts that the output was produced under honest conditions. It doesn't need to understand the domain. It needs to know the shape of the output and check whether that shape is still the right shape.

For PlannerCritic, that means: each gate class still fires on its canary fixture. The redaction layer didn't corrupt a number. The decision context wasn't "unknown."

For AdversarialDebate, that means: the noise floor was measured and the reported difference exceeds it. The permutation control null distribution is stable. No debate exceeded the capitulation threshold.

The same checker. Different project configs. A shared commitment to not trusting your own outputs.

The two projects were built months apart, on different architectures, for different use cases. PlannerCritic is about planning safety — does the plan survive structural inspection? AdversarialDebate is about review quality — does the second opinion actually challenge the first?

But they converged on the same engineering lesson, and it's the one I'd put on a wall:

The layer you trust the most is the layer you are most likely to stop watching.

The fix is to watch it anyway — with a separate, cheaper, dumber check that exists only to tell you whether the expensive layer is still alive. Not because the expensive layer is unreliable. Because you are unreliable when the metrics are green and the dashboard says "improved."

That is what the gate canary does for PlannerCritic. That is what the capitulation detector does for AdversarialDebate. And that is what I would build first, not last, in the next system — before the first field test, before the first release, before the first time the metrics lie to me and I don't notice.

Because the first time is always the one that ships.

Previous AdversarialDebate articles

Links

Top comments (0)