DEV Community

Cover image for I Thought My Multi-Agent Debate Engine Was Broken. The Real Bug Was the Prompt.
Debashish Ghosal
Debashish Ghosal

Posted on AI-assisted

I Thought My Multi-Agent Debate Engine Was Broken. The Real Bug Was the Prompt.

#ai #promptengineering #agents #testing

v0.2.1 RELEASED — Aug 28, 2026. Release notes · Field test report · PyPI

v0.2.1 Update: The 2,333->359 join collapse described in this article is now structurally prevented. v0.2.1 adds row-count invariant assertions at all 5 pipeline seams — the pipeline fails fast if any rows are silently lost. Also new: false-negative measurement (1.7-3.4% missed-issue rate, first recall data ever reported) and 55 new unit tests.

Previously: Most AI Second Opinions Are Fake

AdversarialDebate v0.2.0 is now released — v0.2.1 shipped Aug 28, 2026.

I built the hard parts first.

I built isolated reviewer sessions so the second model could not peek at the first model's answer. I built a revelation gate so debate only started after both reviews were committed. I built claim tracking, concession tracking, convergence scoring, transcript logging, and a full SQLite audit trail.

Then I ran the first real debates and almost nothing happened.

The models responded. The JSON parsed. The transcripts were written. The reports rendered.

But the debates were dead.

That was the uncomfortable moment in this project: the system looked like it worked, and the core interaction was still useless.

I was debugging infrastructure when the real bug was incentive design.

This post is about the most important engineering lesson in the entire AdversarialDebate build: the architecture gave the system structure, but the prompt decided whether the models were allowed to do the lazy thing.

And for the first version, I absolutely left the lazy thing open.

The Short Version

Here is the before-and-after that changed the project:

Metric Before prompt fix After prompt fix
Small-run theater rate 89% 0/9
Full-corpus theater rate not run 1/411 (0.2%)
Small-run average convergence score 0.02 0.445
Full-corpus average convergence score not run 0.65
Debates with at least one concession 1/9 410/411
Small-run verdicts 0/9 2/9

At full scale, the result held:

  • 411 debates run
  • 1 theater case total
  • 152 verdicts
  • 259 disputed outcomes
  • 8,894 concessions

One prompt rewrite moved the engine from looking fake to behaving like a real adversarial system.

That is not a cute prompt-engineering anecdote. That is the difference between shipping a product and shipping a demo-shaped illusion.

The v0.1.0 Bug And The v0.2.0 Bug

This is the cleanest way to understand the project now.

In v0.1.0, the real bug was the prompt

The system architecture was mostly fine.

The protocol was not.

I had given the models a safe low-effort move:

  • acknowledge the objection
  • keep the original position
  • avoid the cost of conceding
  • avoid the work of producing a real rebuttal

That produced debate theater.

The core v0.1.0 lesson was:

If your debate protocol leaves an easy escape hatch open, the model will use it.

Relevant v0.1.0 sources:

In v0.2.0, the real bug was the evaluation plumbing

Once the prompt was fixed, the next class of failures moved down a layer.

The biggest v0.2.0 issues were not about debate incentives. They were about whether the field-test pipeline was faithfully measuring what the system actually did.

The main bugs were:

  1. The whole pipeline still assumed PR-only identifiers
    Scripts were still built around pr_id even after the corpus became mixed-domain and moved to artifact_id.

  2. Non-PR artifacts were being ingested as raw HTML and dashboard noise
    Some incident and change-management artifacts were index pages, not pinned source documents. That created garbage reviewer inputs and bad-request failures.

  3. The LLM-as-judge merge step was collapsing evidence
    The workers judged 2333 rows correctly, but the merge logic keyed on the wrong identifier and collapsed the output to 359 rows.

The core v0.2.0 lesson was:

Once the prompt is fixed, the next bug is whether your evaluation pipeline is telling the truth.

Relevant v0.2.0 sources:

That is why this article is still relevant after v0.2.0.

It started as a prompt-engineering lesson.

Now it is also a systems lesson:

  • v0.1.0: the prompt was the real bug
  • v0.2.0: the data integrity layer became the real bug

What Debate Theater Actually Looks Like

When I say theater, I do not mean the models crashed or refused to answer.

I mean they looked busy while doing almost nothing useful.

Both sides would respond to the other review. Both sides would emit structured outputs. Both sides would keep their original positions. The transcript looked active, but the argument state barely moved.

In the first small run, 8 of 9 debates had zero concessions. The average convergence score was 0.0. That is not debate. That is two models politely maintaining their positions until the round cap ends the conversation for them.

At first glance, this kind of output is deceptive because it feels serious:

  • there are claims
  • there are objections
  • there are rebuttals
  • there is structured output
  • there is a final report

But none of that matters if the system makes it too easy for both sides to preserve their positions without paying any cost.

That was exactly the trap I had built.

The Bug Was Not In The Engine

My first instinct was to blame the architecture.

That was rational. The whole project depends on architecture-level invariants:

  • independent passes
  • delayed revelation
  • append-only review commits
  • bounded rounds
  • transcript lineage

If any of those were broken, the whole thesis would collapse.

So I debugged the engine first.

I checked whether the reviewer sessions were actually isolated.

I checked whether the revelation gate was opening too early.

I checked whether the wrong review text was being replayed.

I checked whether my evidence tracker was incorrectly preserving old state.

I checked whether the convergence score was miscomputed.

The engine was fine.

That is the part that took me a little too long to accept. The models were following instructions. The problem was that the instructions gave them an easy escape hatch.

The Escape Hatch Was CARRIED

The original debate protocol allowed three response types:

CONCEDED: you accept the objection
REBUTTED: you reject the objection with counter-evidence
CARRIED: you acknowledge the objection but maintain your position
Enter fullscreen mode Exit fullscreen mode

At first glance, that looked reasonable. In fact, it looked well-structured.

But the third option was under-specified.

CARRIED had no cost.

The prompt did not require the model to provide evidence for carrying a claim forward. It did not force the model to concede when the other side's evidence was stronger. It did not treat unsupported CARRIED as invalid.

So the models learned the obvious behavior immediately:

"I can acknowledge the objection, keep my original position, and avoid the risk of admitting I was wrong."

That is the safest move in the whole protocol.

It requires less reasoning than a real rebuttal.
It requires less humility than a concession.
It preserves face.
It keeps the transcript moving.

And most importantly, it makes the debate look engaged while preserving the original claim set almost unchanged.

In other words, it is the perfect move for a bad debate system.

Why The Models Behaved Rationally

This is the part I think many people miss when they talk about prompt failures.

The model was not being dumb.

The model was exploiting the protocol exactly the way a capable participant would exploit a weak process.

Imagine a meeting where every objection can be answered with, "I hear your concern, but I still disagree," and no one is required to provide new evidence, revise the argument, or explicitly concede. That meeting will run forever while changing nothing.

The original prompt created exactly that environment.

The LLM equivalent of "noted, no change" is not a weird emergent behavior. It is a completely predictable result of bad incentives.

Once I saw it that way, the fix became obvious.

The Three Sentences That Changed Everything

I did not rewrite the engine. I rewrote the behavior contract.

The key changes were simple.

1. CARRIED now requires evidence 

CARRIED: you acknowledge the objection but maintain your position.
You MUST provide a specific technical reason for maintaining your position.
CARRIED without a technical reason is invalid.
Enter fullscreen mode Exit fullscreen mode

2. Concession is mandatory when outmatched 

If the other reviewer's evidence is stronger than yours, you MUST CONCEDE.
Do not stubbornly CARRY. If you cannot rebut with specific evidence, CONCEDE.
Enter fullscreen mode Exit fullscreen mode

3. CARRIED is no longer the default safe option 

CARRIED is not a default. It requires justification.
If you have no technical reason to maintain your position, CONCEDE.
Enter fullscreen mode Exit fullscreen mode

That is it.

No new architecture.
No new orchestration framework.
No retraining.
No clever judge model.

Three behavioral constraints.

And the system changed dramatically.

The Numbers After The Fix

The small run changed first:

  • before: 8/9 debates had zero concessions
  • after: 0/9 theater
  • after: 2/9 verdicts
  • after: 0.445 average score on the small run

Then v0.1.0 validated the prompt change at full PR-only scale:

  • 1/411 theater
  • 0 true engine failures after retries
  • 8,894 total concessions
  • 37% verdicts, 63% disputed

But v0.2.0 taught me something more useful: the prompt fix was real, but the pipeline around it still had bugs that could easily have made me tell the wrong story.

At v0.2.0 scale, the corrected mixed-domain field test produced:

  • 150 artifacts across 4 domains
  • 217 debates
  • 0 theater
  • $0.42 total cost
  • 2070 MATCH / 263 PARTIAL / 0 NO_MATCH in PR-domain ground-truth judging

Reports:

And the pair-level results were exactly the kind of signal I wanted:

  • GPT + Mistral on the full corpus: 0.536 average convergence, 2/150 verdicts, 2,927 concessions
  • DeepSeek + Mistral on the validation subset: 0.572 average convergence, 1/36 verdicts, 936 concessions
  • GPT + Gemini on the negative-control subset: 0.033 average convergence, 0/24 verdicts, 34 concessions

That distribution matters even more than the original prompt win.

The prompt fix did not turn the system into a convergence machine. It turned it into a real debate machine. Most debates still ended with disagreement. That is correct. The goal was never agreement. The goal was getting the models to genuinely engage instead of perform engagement.

That is why I trust the post-fix numbers. They are not suspiciously neat.

The Second Bug: My Theater Detector Was Wrong Too

Fixing the prompt uncovered a second problem.

My original theater detector was too crude. It effectively treated zero concessions as theater.

That sounds reasonable until you read the transcripts.

A debate can have zero concessions and still be real. Two sides may genuinely rebut each other without yielding. That is not theater. That is stubborn disagreement.

So the original detector was overcounting the wrong failure mode. It was punishing debates where both sides engaged but neither side moved.

The corrected rule was better:

A debate is theater only when there are zero defense events.

In plain English: if nobody actually responded meaningfully, that is theater. If both sides addressed objections, even stubbornly, that is debate.

That distinction matters because theater was one of the release gates. If you define the metric poorly, you can convince yourself the engine is broken when the real problem is just disagreement.

After the detector fix, the field test landed at 1 theater case out of 411.

That is a believable number. Not zero. Not dozens. One edge case.

The Third Bug: I Was Replaying Reviews Instead Of Running Debate

The prompt was the biggest problem. It was not the only problem.

Early in the pipeline I also found a more embarrassing issue: the debate rounds were using a StoredProvider that replayed prewritten review text instead of invoking the model live during debate rounds.

That meant the controller expected responses with markers like:

  • CONCEDED
  • REBUTTED
  • CARRIED

But the provider was feeding it plain review prose.

So even when the rest of the pipeline looked healthy, I was not actually running debate. I was stapling two reviews together and asking the controller to pretend they were debate moves.

That became Issue 1 in the learnings log, and it was fixed by replacing the stored replay path with a live debate provider.

This was the most useful kind of bug: the kind you only catch when you stop trusting surface success.

The files existed. The reports existed. The JSON existed. The product still was not doing the thing it claimed to do.

Valid JSON is not proof of valid behavior.

That is exactly why transcript-level inspection matters.

v0.2.0 Was About Fixing The Data Around The Prompt

If v0.1.0 taught me that the prompt was the highest-leverage variable, v0.2.0 taught me that evaluation plumbing can still lie to you after the prompt is fixed.

The most important v0.2.0 fixes were not glamorous:

1. The whole field-test pipeline had to move from pr_id to artifact_id

v0.1.0 was PR-only. Every script assumed flat files, GitHub PR URLs, and a single identifier shape.

v0.2.0 moved to a mixed corpus:

  • PR review
  • incident response
  • change management
  • security incidents

That meant every stage had to be corrected:

  • corpus downloader
  • reviewer runner
  • pair combiner
  • debate runner
  • analysis scripts
  • ground-truth exporter
  • flakiness runner

This was not optional cleanup. If the pipeline still assumed pr_id everywhere, the field test would silently mis-merge rows, skip artifacts, or route pairs incorrectly.

2. Raw HTML was poisoning non-PR evaluation

This one was brutal because it looked like a model issue at first.

Some non-PR artifacts were downloaded from:

  • status dashboards
  • repo landing pages
  • index pages

The reviewer was then sending that raw HTML straight into the model.

The result was not a clean product insight. It was garbage-in behavior and HTTP 400 failures.

The fix was straightforward once I admitted the problem was ingestion, not reasoning:

  • strip HTML and script/style chrome
  • collapse whitespace
  • compact repeated dashboard noise
  • hard-cap non-PR prompt size

That bug matters because it is exactly the kind of thing that can make you blame the model for a broken runtime contract.

3. My LLM-as-judge merge logic was wrong

This was the most dangerous v0.2.0 bug because it hit the reporting layer.

The judge workers processed 2333 rows successfully.

But the merged output collapsed to 359 rows.

Why? Because the merge code was still keying on pr_id, while the corrected exporter had moved to artifact_id.

So the system did the expensive part correctly and then mangled the evidence at the last step.

That is the kind of bug that can poison a release narrative if you trust the final CSV more than the worker-level facts.

Once fixed, the real result was visible:

  • 2070 MATCH
  • 263 PARTIAL
  • 0 NO_MATCH

That is one of the strongest signals in the whole project.

What The Transcript Looked Like After The Fix

This is the part that finally convinced me the system had crossed from fake structure to real behavior.

After the fix, the transcripts contained moves like:

CONCEDED

CONCEDED obj_initial_b_0: The other reviewer's claim about the severity being high is consistent with my assessment, so I concede this point.

REBUTTED

REBUTTED cl_A_3: The evidence provided in staging/src/k8s.io/client-go/tools/metrics/metrics.go (Line 235) clearly shows the logical error where...

CARRIED

CARRIED: The severity of the issue remains high due to the potential for significant impact on functionality and metrics tracking...

Those are very different from the earlier transcripts.

After the fix:

  • concession references objection IDs
  • rebuttal cites file-line evidence
  • carried positions include a technical reason

That is the behavior contract I wanted from the beginning. Not because the model suddenly became wiser, but because the protocol made the lazy move invalid.

The Prompt Fix Revealed Something Else

It also made one pair look worse.

On rails#52531, the GPT + Gemini pair went from 4 concessions before the fix to 0 concessions after the fix. That might look like the prompt made the system worse.

I do not think it did.

I think it revealed the pair's real behavior.

Under the permissive prompt, the pair could drift through the debate with weak CARRIED behavior. Under the stricter prompt, both sides were forced to either concede or rebut with evidence. They chose rebuttal.

That is not failure. That is signal.

And it lines up with both field tests, where GPT + Gemini turned out to be the least productive pair overall.

In v0.1.0:

  • 0% capitulation
  • 4% verdict rate
  • 0.357 average score
  • 2.0 average rounds, meaning it nearly always exhausted the limit

In v0.2.0 negative-control revalidation:

  • 0/24 verdicts
  • 0 theater
  • 0 capitulation
  • 0.033 average score

The stricter prompt did not break a good pair. It exposed a stubborn one.

That is one of the most useful things a field test can do.

What Could Have Gone Better

This was one of those releases where the field test taught me more about my process than my code.

Three things could have gone better.

1. I should have tested behavior earlier

I invested heavily in architecture before running enough real debates to validate the protocol. The architecture was necessary, but the sequencing was backwards. One day of transcript-driven behavior testing earlier would have revealed the prompt problem much sooner.

2. I should have distrusted clean-looking outputs faster

If a system emits valid JSON and nicely formatted reports, it is very easy to believe the hard part is done. I needed to get to transcript inspection faster and ask the blunt question: "did anyone actually change their mind?"

3. I should have separated prompt debugging from product validation more explicitly

The small run was doing both jobs at once: proving the pipeline worked and proving the debate design worked. Those are not the same thing.

v0.2.0 partly fixed that by separating:

  • the full-corpus default pair
  • the validation subset
  • the negative control

That was the right move. But it also made it obvious that evaluation infrastructure deserves the same rigor as the model protocol.

What I Learned From The Field Test

This release left me with six strong opinions.

1. The prompt was the highest-leverage variable in the system

Not the framework. Not the storage layer. Not the debate controller. Not the CLI.

The single highest-leverage variable was the behavioral contract around concession, rebuttal, and carry.

2. Models will take the lowest-cost valid action every time

If you leave a safe path open, expect the model to take it.

That is not a flaw in the model. That is a flaw in the protocol.

3. Metrics are only as good as the semantics behind them

"Zero concessions" sounded like a useful proxy for theater until I looked at the transcripts. It was not. The detector had to be grounded in actual debate behavior, not a shallow surface count.

4. A field test should invalidate part of your story

This one did. It exposed the prompt problem, the replay-provider problem, and the detector problem. It also showed that some pairs were fundamentally stubborn even after the protocol improved. That is exactly what I wanted from a real test.

5. Prompt engineering is not separate from systems engineering

I think a lot of developers still talk about prompts as if they are a soft layer above the "real" system. That framing breaks down fast in agent products.

When the model is part of the runtime, prompt constraints are part of the system contract.

The prompt is not copy. The prompt is behavior.

6. Data integrity bugs are product bugs when your product is evaluation

If your pitch depends on:

  • debate transcripts
  • convergence metrics
  • ground-truth comparisons
  • preserved dissent

then a merge bug in the judge pipeline is not a back-office annoyance.

It is a product bug.

That was the deepest v0.2.0 lesson for me.

What I would Change Next (Update: v0.2.1)

The project is good enough to ship at v0.2.1. It is not the end state.

The next version still needs at least five things:

  1. Prompt A/B testing instead of stopping once one version works
  2. Better verdict-quality metrics so low-quality convergence is easier to separate from good debate
  3. LLM-generated would_resolve_if text instead of template-heavy resolution hints
  4. Pinned artifact URLs instead of index/dashboard-heavy candidate sources for non-PR domains
  5. A tighter protocol evaluation harness for testing debate behavior before large corpus runs

I also want to keep re-testing the weak and strong pairs deliberately.

One of the most valuable v0.2.0 outcomes is not that everything improved. It is that the system now makes pair quality visible instead of hiding it behind nice-looking output.

That is a much more useful result than a universal improvement story.

Questions I Want Developers To Push On

If you build agent systems, these are the questions I think are worth arguing about:

  1. What is the equivalent of CARRIED in your system, the low-cost valid action that keeps outputs moving without real reasoning?
  2. Are your evaluation metrics measuring actual behavior, or just structured output shape?
  3. How early in your build process do you inspect transcripts instead of summaries?
  4. If a protocol lets the model avoid intellectual risk, why would you expect good debate from it?

I would especially love to hear from people building:

  • judge models
  • planner/critic loops
  • verifier chains
  • review agents
  • multi-model approval systems

Because I think a lot of us are still underestimating how often a well-structured system is quietly allowing low-effort behavior.

v0.2.1 Update: The Pipeline Bugs Are Now Structurally Prevented

This article described two pipeline bugs from v0.2.0: the pr_id to artifact_id migration and the LLM-as-judge merge collapse (2,333 rows to 359). The v0.2.1 release addresses the underlying failure class.

Row-count invariants at all 5 pipeline seams. Every join now asserts count_post against count_pre. If rows vanish, the pipeline fails fast with the specific missing artifact IDs. The 2,333->359 collapse cannot recur — the pipeline stops at the seam where it happens, not after publishing wrong numbers.

False-negative measurement. The v0.2.1 release adds 09_missed_issues.py — a new script that feeds pre-fix artifacts from 59 known-bad PRs through each reviewer and measures how often the reviewer misses the issue that was eventually found. Result: 1.7-3.4% missed-issue rate across all four reviewers. This is the first recall data the project has ever reported.

55 new unit tests. 15 for pair configuration, 19 for seam assertion logic, 21 for false-negative detection. All deterministic, zero LLM calls.

What v0.2.0 had What v0.2.1 adds
One join bug fixed (2,333->359) 5 seam assertions preventing the entire class
Precision only (binary match rate) + Recall (1.7-3.4% missed-issue rate)
3 model pairs 5 pairs (+ DeepSeek+GPT separating experiment)
217 debates 367 debates
No pipeline integrity evidence Pipeline integrity table in every report

Full details in the v0.2.1 field test report.

AdversarialDebate is live here:

Next in the series: the strongest pair in the field test also had the most dangerous failure mode.

AdversarialDebate v0.2.0 is now released.

I built the hard parts first.

I built isolated reviewer sessions so the second model could not peek at the first model's answer. I built a revelation gate so debate only started after both reviews were committed. I built claim tracking, concession tracking, convergence scoring, transcript logging, and a full SQLite audit trail.

Then I ran the first real debates and almost nothing happened.

The models responded. The JSON parsed. The transcripts were written. The reports rendered.

But the debates were dead.

That was the uncomfortable moment in this project: the system looked like it worked, and the core interaction was still useless.

I was debugging infrastructure when the real bug was incentive design.

This post is about the most important engineering lesson in the entire AdversarialDebate build: the architecture gave the system structure, but the prompt decided whether the models were allowed to do the lazy thing.

And for the first version, I absolutely left the lazy thing open.

The Short Version

Here is the before-and-after that changed the project:

Metric Before prompt fix After prompt fix
Small-run theater rate 89% 0/9
Full-corpus theater rate not run 1/411 (0.2%)
Small-run average convergence score 0.02 0.445
Full-corpus average convergence score not run 0.65
Debates with at least one concession 1/9 410/411
Small-run verdicts 0/9 2/9

At full scale, the result held:

  • 411 debates run
  • 1 theater case total
  • 152 verdicts
  • 259 disputed outcomes
  • 8,894 concessions

One prompt rewrite moved the engine from looking fake to behaving like a real adversarial system.

That is not a cute prompt-engineering anecdote. That is the difference between shipping a product and shipping a demo-shaped illusion.

The v0.1.0 Bug And The v0.2.0 Bug

This is the cleanest way to understand the project now.

In v0.1.0, the real bug was the prompt

The system architecture was mostly fine.

The protocol was not.

I had given the models a safe low-effort move:

  • acknowledge the objection
  • keep the original position
  • avoid the cost of conceding
  • avoid the work of producing a real rebuttal

That produced debate theater.

The core v0.1.0 lesson was:

If your debate protocol leaves an easy escape hatch open, the model will use it.

Relevant v0.1.0 sources:

In v0.2.0, the real bug was the evaluation plumbing

Once the prompt was fixed, the next class of failures moved down a layer.

The biggest v0.2.0 issues were not about debate incentives. They were about whether the field-test pipeline was faithfully measuring what the system actually did.

The main bugs were:

  1. The whole pipeline still assumed PR-only identifiers
    Scripts were still built around pr_id even after the corpus became mixed-domain and moved to artifact_id.

  2. Non-PR artifacts were being ingested as raw HTML and dashboard noise
    Some incident and change-management artifacts were index pages, not pinned source documents. That created garbage reviewer inputs and bad-request failures.

  3. The LLM-as-judge merge step was collapsing evidence
    The workers judged 2333 rows correctly, but the merge logic keyed on the wrong identifier and collapsed the output to 359 rows.

The core v0.2.0 lesson was:

Once the prompt is fixed, the next bug is whether your evaluation pipeline is telling the truth.

Relevant v0.2.0 sources:

That is why this article is still relevant after v0.2.0.

It started as a prompt-engineering lesson.

Now it is also a systems lesson:

  • v0.1.0: the prompt was the real bug
  • v0.2.0: the data integrity layer became the real bug

What Debate Theater Actually Looks Like

When I say theater, I do not mean the models crashed or refused to answer.

I mean they looked busy while doing almost nothing useful.

Both sides would respond to the other review. Both sides would emit structured outputs. Both sides would keep their original positions. The transcript looked active, but the argument state barely moved.

In the first small run, 8 of 9 debates had zero concessions. The average convergence score was 0.0. That is not debate. That is two models politely maintaining their positions until the round cap ends the conversation for them.

At first glance, this kind of output is deceptive because it feels serious:

  • there are claims
  • there are objections
  • there are rebuttals
  • there is structured output
  • there is a final report

But none of that matters if the system makes it too easy for both sides to preserve their positions without paying any cost.

That was exactly the trap I had built.

The Bug Was Not In The Engine

My first instinct was to blame the architecture.

That was rational. The whole project depends on architecture-level invariants:

  • independent passes
  • delayed revelation
  • append-only review commits
  • bounded rounds
  • transcript lineage

If any of those were broken, the whole thesis would collapse.

So I debugged the engine first.

I checked whether the reviewer sessions were actually isolated.

I checked whether the revelation gate was opening too early.

I checked whether the wrong review text was being replayed.

I checked whether my evidence tracker was incorrectly preserving old state.

I checked whether the convergence score was miscomputed.

The engine was fine.

That is the part that took me a little too long to accept. The models were following instructions. The problem was that the instructions gave them an easy escape hatch.

The Escape Hatch Was CARRIED

The original debate protocol allowed three response types:

CONCEDED: you accept the objection
REBUTTED: you reject the objection with counter-evidence
CARRIED: you acknowledge the objection but maintain your position
Enter fullscreen mode Exit fullscreen mode

At first glance, that looked reasonable. In fact, it looked well-structured.

But the third option was under-specified.

CARRIED had no cost.

The prompt did not require the model to provide evidence for carrying a claim forward. It did not force the model to concede when the other side's evidence was stronger. It did not treat unsupported CARRIED as invalid.

So the models learned the obvious behavior immediately:

"I can acknowledge the objection, keep my original position, and avoid the risk of admitting I was wrong."

That is the safest move in the whole protocol.

It requires less reasoning than a real rebuttal.
It requires less humility than a concession.
It preserves face.
It keeps the transcript moving.

And most importantly, it makes the debate look engaged while preserving the original claim set almost unchanged.

In other words, it is the perfect move for a bad debate system.

Why The Models Behaved Rationally

This is the part I think many people miss when they talk about prompt failures.

The model was not being dumb.

The model was exploiting the protocol exactly the way a capable participant would exploit a weak process.

Imagine a meeting where every objection can be answered with, "I hear your concern, but I still disagree," and no one is required to provide new evidence, revise the argument, or explicitly concede. That meeting will run forever while changing nothing.

The original prompt created exactly that environment.

The LLM equivalent of "noted, no change" is not a weird emergent behavior. It is a completely predictable result of bad incentives.

Once I saw it that way, the fix became obvious.

The Three Sentences That Changed Everything

I did not rewrite the engine. I rewrote the behavior contract.

The key changes were simple.

1. CARRIED now requires evidence 

CARRIED: you acknowledge the objection but maintain your position.
You MUST provide a specific technical reason for maintaining your position.
CARRIED without a technical reason is invalid.
Enter fullscreen mode Exit fullscreen mode

2. Concession is mandatory when outmatched 

If the other reviewer's evidence is stronger than yours, you MUST CONCEDE.
Do not stubbornly CARRY. If you cannot rebut with specific evidence, CONCEDE.
Enter fullscreen mode Exit fullscreen mode

3. CARRIED is no longer the default safe option 

CARRIED is not a default. It requires justification.
If you have no technical reason to maintain your position, CONCEDE.
Enter fullscreen mode Exit fullscreen mode

That is it.

No new architecture.
No new orchestration framework.
No retraining.
No clever judge model.

Three behavioral constraints.

And the system changed dramatically.

The Numbers After The Fix

The small run changed first:

  • before: 8/9 debates had zero concessions
  • after: 0/9 theater
  • after: 2/9 verdicts
  • after: 0.445 average score on the small run

Then v0.1.0 validated the prompt change at full PR-only scale:

  • 1/411 theater
  • 0 true engine failures after retries
  • 8,894 total concessions
  • 37% verdicts, 63% disputed

But v0.2.0 taught me something more useful: the prompt fix was real, but the pipeline around it still had bugs that could easily have made me tell the wrong story.

At v0.2.0 scale, the corrected mixed-domain field test produced:

  • 150 artifacts across 4 domains
  • 217 debates
  • 0 theater
  • $0.42 total cost
  • 2070 MATCH / 263 PARTIAL / 0 NO_MATCH in PR-domain ground-truth judging

Reports:

And the pair-level results were exactly the kind of signal I wanted:

  • GPT + Mistral on the full corpus: 0.536 average convergence, 2/150 verdicts, 2,927 concessions
  • DeepSeek + Mistral on the validation subset: 0.572 average convergence, 1/36 verdicts, 936 concessions
  • GPT + Gemini on the negative-control subset: 0.033 average convergence, 0/24 verdicts, 34 concessions

That distribution matters even more than the original prompt win.

The prompt fix did not turn the system into a convergence machine. It turned it into a real debate machine. Most debates still ended with disagreement. That is correct. The goal was never agreement. The goal was getting the models to genuinely engage instead of perform engagement.

That is why I trust the post-fix numbers. They are not suspiciously neat.

The Second Bug: My Theater Detector Was Wrong Too

Fixing the prompt uncovered a second problem.

My original theater detector was too crude. It effectively treated zero concessions as theater.

That sounds reasonable until you read the transcripts.

A debate can have zero concessions and still be real. Two sides may genuinely rebut each other without yielding. That is not theater. That is stubborn disagreement.

So the original detector was overcounting the wrong failure mode. It was punishing debates where both sides engaged but neither side moved.

The corrected rule was better:

A debate is theater only when there are zero defense events.

In plain English: if nobody actually responded meaningfully, that is theater. If both sides addressed objections, even stubbornly, that is debate.

That distinction matters because theater was one of the release gates. If you define the metric poorly, you can convince yourself the engine is broken when the real problem is just disagreement.

After the detector fix, the field test landed at 1 theater case out of 411.

That is a believable number. Not zero. Not dozens. One edge case.

The Third Bug: I Was Replaying Reviews Instead Of Running Debate

The prompt was the biggest problem. It was not the only problem.

Early in the pipeline I also found a more embarrassing issue: the debate rounds were using a StoredProvider that replayed prewritten review text instead of invoking the model live during debate rounds.

That meant the controller expected responses with markers like:

  • CONCEDED
  • REBUTTED
  • CARRIED

But the provider was feeding it plain review prose.

So even when the rest of the pipeline looked healthy, I was not actually running debate. I was stapling two reviews together and asking the controller to pretend they were debate moves.

That became Issue 1 in the learnings log, and it was fixed by replacing the stored replay path with a live debate provider.

This was the most useful kind of bug: the kind you only catch when you stop trusting surface success.

The files existed. The reports existed. The JSON existed. The product still was not doing the thing it claimed to do.

Valid JSON is not proof of valid behavior.

That is exactly why transcript-level inspection matters.

v0.2.0 Was About Fixing The Data Around The Prompt

If v0.1.0 taught me that the prompt was the highest-leverage variable, v0.2.0 taught me that evaluation plumbing can still lie to you after the prompt is fixed.

The most important v0.2.0 fixes were not glamorous:

1. The whole field-test pipeline had to move from pr_id to artifact_id

v0.1.0 was PR-only. Every script assumed flat files, GitHub PR URLs, and a single identifier shape.

v0.2.0 moved to a mixed corpus:

  • PR review
  • incident response
  • change management
  • security incidents

That meant every stage had to be corrected:

  • corpus downloader
  • reviewer runner
  • pair combiner
  • debate runner
  • analysis scripts
  • ground-truth exporter
  • flakiness runner

This was not optional cleanup. If the pipeline still assumed pr_id everywhere, the field test would silently mis-merge rows, skip artifacts, or route pairs incorrectly.

2. Raw HTML was poisoning non-PR evaluation

This one was brutal because it looked like a model issue at first.

Some non-PR artifacts were downloaded from:

  • status dashboards
  • repo landing pages
  • index pages

The reviewer was then sending that raw HTML straight into the model.

The result was not a clean product insight. It was garbage-in behavior and HTTP 400 failures.

The fix was straightforward once I admitted the problem was ingestion, not reasoning:

  • strip HTML and script/style chrome
  • collapse whitespace
  • compact repeated dashboard noise
  • hard-cap non-PR prompt size

That bug matters because it is exactly the kind of thing that can make you blame the model for a broken runtime contract.

3. My LLM-as-judge merge logic was wrong

This was the most dangerous v0.2.0 bug because it hit the reporting layer.

The judge workers processed 2333 rows successfully.

But the merged output collapsed to 359 rows.

Why? Because the merge code was still keying on pr_id, while the corrected exporter had moved to artifact_id.

So the system did the expensive part correctly and then mangled the evidence at the last step.

That is the kind of bug that can poison a release narrative if you trust the final CSV more than the worker-level facts.

Once fixed, the real result was visible:

  • 2070 MATCH
  • 263 PARTIAL
  • 0 NO_MATCH

That is one of the strongest signals in the whole project.

What The Transcript Looked Like After The Fix

This is the part that finally convinced me the system had crossed from fake structure to real behavior.

After the fix, the transcripts contained moves like:

CONCEDED

CONCEDED obj_initial_b_0: The other reviewer's claim about the severity being high is consistent with my assessment, so I concede this point.

REBUTTED

REBUTTED cl_A_3: The evidence provided in staging/src/k8s.io/client-go/tools/metrics/metrics.go (Line 235) clearly shows the logical error where...

CARRIED

CARRIED: The severity of the issue remains high due to the potential for significant impact on functionality and metrics tracking...

Those are very different from the earlier transcripts.

After the fix:

  • concession references objection IDs
  • rebuttal cites file-line evidence
  • carried positions include a technical reason

That is the behavior contract I wanted from the beginning. Not because the model suddenly became wiser, but because the protocol made the lazy move invalid.

The Prompt Fix Revealed Something Else

It also made one pair look worse.

On rails#52531, the GPT + Gemini pair went from 4 concessions before the fix to 0 concessions after the fix. That might look like the prompt made the system worse.

I do not think it did.

I think it revealed the pair's real behavior.

Under the permissive prompt, the pair could drift through the debate with weak CARRIED behavior. Under the stricter prompt, both sides were forced to either concede or rebut with evidence. They chose rebuttal.

That is not failure. That is signal.

And it lines up with both field tests, where GPT + Gemini turned out to be the least productive pair overall.

In v0.1.0:

  • 0% capitulation
  • 4% verdict rate
  • 0.357 average score
  • 2.0 average rounds, meaning it nearly always exhausted the limit

In v0.2.0 negative-control revalidation:

  • 0/24 verdicts
  • 0 theater
  • 0 capitulation
  • 0.033 average score

The stricter prompt did not break a good pair. It exposed a stubborn one.

That is one of the most useful things a field test can do.

What Could Have Gone Better

This was one of those releases where the field test taught me more about my process than my code.

Three things could have gone better.

1. I should have tested behavior earlier

I invested heavily in architecture before running enough real debates to validate the protocol. The architecture was necessary, but the sequencing was backwards. One day of transcript-driven behavior testing earlier would have revealed the prompt problem much sooner.

2. I should have distrusted clean-looking outputs faster

If a system emits valid JSON and nicely formatted reports, it is very easy to believe the hard part is done. I needed to get to transcript inspection faster and ask the blunt question: "did anyone actually change their mind?"

3. I should have separated prompt debugging from product validation more explicitly

The small run was doing both jobs at once: proving the pipeline worked and proving the debate design worked. Those are not the same thing.

v0.2.0 partly fixed that by separating:

  • the full-corpus default pair
  • the validation subset
  • the negative control

That was the right move. But it also made it obvious that evaluation infrastructure deserves the same rigor as the model protocol.

What I Learned From The Field Test

This release left me with six strong opinions.

1. The prompt was the highest-leverage variable in the system

Not the framework. Not the storage layer. Not the debate controller. Not the CLI.

The single highest-leverage variable was the behavioral contract around concession, rebuttal, and carry.

2. Models will take the lowest-cost valid action every time

If you leave a safe path open, expect the model to take it.

That is not a flaw in the model. That is a flaw in the protocol.

3. Metrics are only as good as the semantics behind them

"Zero concessions" sounded like a useful proxy for theater until I looked at the transcripts. It was not. The detector had to be grounded in actual debate behavior, not a shallow surface count.

4. A field test should invalidate part of your story

This one did. It exposed the prompt problem, the replay-provider problem, and the detector problem. It also showed that some pairs were fundamentally stubborn even after the protocol improved. That is exactly what I wanted from a real test.

5. Prompt engineering is not separate from systems engineering

I think a lot of developers still talk about prompts as if they are a soft layer above the "real" system. That framing breaks down fast in agent products.

When the model is part of the runtime, prompt constraints are part of the system contract.

The prompt is not copy. The prompt is behavior.

6. Data integrity bugs are product bugs when your product is evaluation

If your pitch depends on:

  • debate transcripts
  • convergence metrics
  • ground-truth comparisons
  • preserved dissent

then a merge bug in the judge pipeline is not a back-office annoyance.

It is a product bug.

That was the deepest v0.2.0 lesson for me.

What I’d Change Next

The project is good enough to ship at v0.2.0. It is not the end state.

The next version still needs at least five things:

  1. Prompt A/B testing instead of stopping once one version works
  2. Better verdict-quality metrics so low-quality convergence is easier to separate from good debate
  3. LLM-generated would_resolve_if text instead of template-heavy resolution hints
  4. Pinned artifact URLs instead of index/dashboard-heavy candidate sources for non-PR domains
  5. A tighter protocol evaluation harness for testing debate behavior before large corpus runs

I also want to keep re-testing the weak and strong pairs deliberately.

One of the most valuable v0.2.0 outcomes is not that everything improved. It is that the system now makes pair quality visible instead of hiding it behind nice-looking output.

That is a much more useful result than a universal improvement story.

Questions I Want Developers To Push On

If you build agent systems, these are the questions I think are worth arguing about:

  1. What is the equivalent of CARRIED in your system, the low-cost valid action that keeps outputs moving without real reasoning?
  2. Are your evaluation metrics measuring actual behavior, or just structured output shape?
  3. How early in your build process do you inspect transcripts instead of summaries?
  4. If a protocol lets the model avoid intellectual risk, why would you expect good debate from it?

I would especially love to hear from people building:

  • judge models
  • planner/critic loops
  • verifier chains
  • review agents
  • multi-model approval systems

Because I think a lot of us are still underestimating how often a well-structured system is quietly allowing low-effort behavior.

AdversarialDebate is live here:

Next in the series: the strongest pair in the field test also had the most dangerous failure mode.

Top comments (0)