DEV Community

Cover image for The Best Model Pair in My Field Test Was Also the Least Trustworthy
Debashish Ghosal
Debashish Ghosal

Posted on AI-assisted

The Best Model Pair in My Field Test Was Also the Least Trustworthy

#ai #trustworthy #healthydebate #llm

v0.2.1 RELEASED — Aug 28, 2026. Release notes · Field test report

v0.2.1 Key Finding: The Mistral effect is confirmed. DeepSeek+GPT (two different labs, no Mistral) converged at 0.246 — same as the homogeneous GPT+GPT control (0.273). Mistral, not lab diversity, drives productive debate. The recommendation changes from "pick from different labs" to "always include Mistral."

Also new in v0.2.1: Row-count invariants at all 5 pipeline seams (the 2,333->359 collapse is eliminated), false-negative measurement (1.7-3.4% missed-issue rate, first recall data ever), 55 new unit tests.

AdversarialDebate v0.2.0 is released — v0.2.1 shipped Aug 28.

The strongest pair in my field test almost fooled me.

It had the best average convergence score.
It had the best verdict rate.
It was the cheapest to run efficiently.
It looked like the clean proof of the whole project thesis.

Then I read the transcripts.

What looked like the best debate pair was often doing something worse than disagreement: it was surrendering.

This post is about the most dangerous lesson from AdversarialDebate v0.1.0: a multi-agent system can produce excellent aggregate metrics for the wrong reason.

v0.2.0 did not invalidate that lesson. It refined it.

In v0.1.0, the strongest pair by raw metrics was DeepSeek + Mistral, and the danger was capitulation cascade.

In v0.2.0, DeepSeek + Mistral moved into a validation role while GPT + Mistral became the best full-corpus default. That means the exact pair ranking changed, but the core warning stayed the same:

Strong-looking multi-agent metrics are not automatically trustworthy multi-agent metrics.

If you only look at verdict counts, convergence scores, or API efficiency, you can mistake low-quality reasoning for success.

That is exactly what almost happened here.

The v0.1.0 Result And The v0.2.0 Update

In v0.1.0, the strongest pair looked almost too good

In the full field test, the strongest pair was DeepSeek + Mistral.

The top-line metrics were hard to argue with:

Pair Avg Score Verdict Rate Concessions
DeepSeek + Mistral 0.982 97% 2,352
GPT + Mistral 0.754 48% 1,728
GPT + GPT 0.688 57% 1,444
Gemini + DeepSeek 0.622 10% 1,470
Gemini + Mistral 0.512 4% 1,073
GPT + Gemini 0.357 4% 727

If I had stopped there, this article would have been simple.

I would have written:

  • diversity wins
  • cross-region pairs are best
  • the most diverse pair is the best production default

That would have been clean.
It also would have been incomplete.

Because the same pair that hit 97% verdict rate also had a 65% capitulation rate.

Relevant v0.1.0 sources:

That changes the story.

In v0.2.0, the same warning stayed true, but the shape changed

v0.2.0 gave me a cleaner pair story:

  • GPT + Mistral became the best full-corpus default: 0.536 average convergence, 2/150 verdicts, 2,927 concessions
  • DeepSeek + Mistral stayed stronger than the negative control on its validation subset: 0.572 average convergence, 1/36 verdicts, 936 concessions
  • GPT + Gemini remained the weak negative control: 0.033 average convergence, 0/24 verdicts

So the exact v0.1.0 headline, “the best pair was the least trustworthy,” is most true for v0.1.0 itself.

The broader lesson still holds in v0.2.0:

You cannot trust pair-level success metrics unless you also inspect how that success was produced.

Relevant v0.2.0 sources:

The Problem Hidden Inside A Great Score

Here is the actual failure mode:

capitulation cascade

That is what I call a debate where one side concedes almost everything immediately, usually in round 1, with zero real rebuttal pressure.

In the field test, the detection rule was:

  • at least 80% round-1 concessions
  • zero rebuttals

When that happens, the debate technically resolves. The engine can record it as converged. The score can land at 1.0. The report can look successful.

But it is not the kind of success you want.

It is not "both sides examined the evidence and converged."

It is "one side gave up."

That distinction matters a lot if you are building review systems people are supposed to trust.

The Short Version In Numbers

This is the compact version of the problem:

Metric Result
Total debates 411
Total capitulation cascades 80
Share of all debates 19%
pair5 verdict rate 97%
pair5 capitulation rate 65%
pair1 verdict rate 4%
pair1 capitulation rate 0%

That is the inversion that forced me to rethink the whole scorecard.

The pair with the best convergence was often the worst at producing trustworthy convergence.

The pair with terrible convergence was at least producing honest disagreement.

That sentence is the cleanest summary of v0.1.0.

For v0.2.0, the better summary is slightly narrower:

  • the negative control is still weak and honest
  • the stronger validation pair is still stronger
  • but the worst capitulation story was not reproduced at the same intensity

What A Capitulation Cascade Looks Like

Here is the simplified shape of one of these debates:

Round 1:
  Side A objects
  Side B: CONCEDED
  Side B: CONCEDED
  Side B: CONCEDED
  Side B: CONCEDED

Termination: all_resolved
Score: 1.0
Enter fullscreen mode Exit fullscreen mode

Perfect score. Fast resolution. Minimal API cost.

And almost no real adversarial value.

This is why I no longer trust convergence numbers by themselves in multi-agent systems. A 1.0 score can mean:

  1. both sides genuinely converged after evidence exchange
  2. one side folded immediately

If your metrics treat those as the same outcome, your dashboard is lying to you.

Where The Failure Concentrated

The distribution across pairs was not subtle.

Pair Capitulations % of Pair Debates
DeepSeek + Mistral 44 65%
GPT + GPT 21 30%
GPT + Mistral 14 21%
Gemini + Mistral 1 1.5%
Gemini + DeepSeek 1 1.4%
GPT + Gemini 0 0%

The strongest pair by convergence quality was also the strongest pair by capitulation risk.

That is not a side note. That is the main story.

It means diversity is not a monotonic good.

It helps.
Then it hurts.

At some point, the gap between the two models stops producing productive disagreement and starts producing evaluation failure, where one model cannot meaningfully assess the other model's reasoning and treats that as a cue to concede.

That was loud in v0.1.0 and much softer in v0.2.0.

That difference is important. It means the project did not just repeat the same story. It clarified it.

Same Artifact, Totally Different Kind Of Failure

What made this result convincing was that I could compare the strongest and weakest pairs on the same PRs.

For example:

PR DeepSeek + Mistral GPT + Gemini
django#18333 verdict, 1.000 disputed, 0.444
golang#54390 verdict, 1.000 disputed, 0.000
kubernetes#140860 verdict, 1.000 disputed, 0.463
kubernetes#140866 verdict, 1.000 disputed, 0.000

Same artifacts. Same engine. Same debate protocol.

But the failure modes were different.

GPT + Gemini failed by refusing to converge.
DeepSeek + Mistral often succeeded by converging too cheaply.

Once I saw that, it became impossible to keep calling all verdicts equal.

Why This Happens

My current hypothesis is simple.

When two models are moderately different, they can still evaluate each other's claims. That is where the best debate lives.

When they are too similar, they rebut forever because both sides produce strong arguments in the same reasoning neighborhood.

When they are too different, one side may not know how to evaluate the other side's argument well enough to rebut it. Under a prompt that says "concede when outmatched," that can collapse into mass concession.

That is why maximum diversity can look so strong in the aggregate. It is not necessarily producing better reasoning. It is sometimes producing quicker surrender.

This also explains why GPT + Gemini had:

  • 0% capitulation
  • 4% verdict rate
  • 2.0 average rounds

That pair is frustrating, but intellectually honest. It fights. It does not fold.

v0.2.0 reinforced that role exactly:

  • 0/24 verdicts
  • 0 theater
  • 0 capitulation
  • 0.033 average convergence score

That is terrible default-pair behavior and excellent negative-control behavior.

The Round Distribution Changed My Mind

This was the signal I almost missed.

At first, the round distribution looked like good news:

Rounds Count %
1 115 28.0%
2 295 71.8%

My first reading was naive: great, a quarter of the debates resolve early.

Then I cross-referenced round-1 resolutions with capitulation.

That produced a much more useful breakdown:

Outcome Count %
Genuine early convergence 35 8.5%
Capitulation cascade 80 19.5%
Full two-round debate 295 71.8%

That table changed the meaning of the result.

Round-1 resolution was not mostly evidence that the debate engine was efficient.

It was often evidence that one side had stopped being useful.

That is exactly the kind of thing a field test is supposed to expose before you hard-code the wrong success metric into the product.

Why The Metrics Were Misleading

Capitulation pollutes three metrics at once.

1. It inflates convergence

If one side concedes everything, the score trends toward 1.0 by construction.

2. It inflates claim resolution

pair5 resolved 34.6 of 35.5 average claims. pair1 resolved 10.5 of 30.7. On paper, that makes pair5 look three times better.

But a concession storm resolves claims without proving those claims survived real adversarial pressure.

3. It makes the pair look more efficient than it really is

API call data told the same story:

Pair Avg API Calls Avg Rounds
DeepSeek + Mistral 2.4 1.2
GPT + GPT 3.2 1.6
GPT + Mistral 3.3 1.6
Gemini + Mistral 3.8 1.9
Gemini + DeepSeek 3.9 2.0
GPT + Gemini 4.0 2.0

If you only care about cost and throughput, pair5 looks amazing.

If you care about whether the debate was real, pair5 is much harder to celebrate.

That is the trap.

What Saved The Release

The good news is that the engine did not hide this from me forever.

And the v0.2.0 report now makes the pair roles explicit instead of letting all pair outcomes blur together:

  • pair3_gpt_mistral = primary / positive pair
  • pair5_deepseek_mistral = validation pair
  • pair1_gpt_gemini = negative control
  • homogeneous_gpt = partial homogeneous control

v0.1.0 already had a failure-mode detector for capitulation cascade. It flagged debates where one side conceded almost everything without rebuttal. So the product did not silently ship fake confidence. The warning existed.

But the scorecards still counted those outcomes as verdicts.

That is the design tension I do not want to gloss over:

  • the engine detected the problem
  • the aggregate metrics still flattered the wrong pair

Detection is not the same as interpretation.

And this is exactly where a lot of developer tooling gets into trouble. We add a warning and keep the headline metric unchanged, which means the dashboard still pushes users toward the wrong conclusion.

What Could Have Gone Better

Three things should have happened earlier.

1. I should have defined verdict quality separately from verdict existence

Not every verdict is equally trustworthy. That seems obvious now, but the first scorecard did not encode it strongly enough.

2. I should have made pair guidance a product feature, not a post-test lesson

By the end of the run, it was obvious that users need guidance on pair selection. The wrong pair can make the product look broken. The superficially best pair can make the product look stronger than it is.

3. I should have expected aggregate metrics to lie

This is probably the biggest process lesson in the whole project. Any multi-agent metric that compresses many interaction types into one clean score is dangerous until you inspect the transcripts behind it.

What I Learned

This field test left me with five convictions.

1. Maximum diversity has a dark side

The most diverse pair produced the most convergence, but a large share of that convergence came from capitulation.

2. Honest disagreement is often better than cheap agreement

GPT + Gemini was a frustrating pair. But its failure mode was at least real disagreement, not surrender disguised as success.

3. A high score can hide low-quality reasoning

This is the part I want more agent builders to internalize. Aggregate metrics can absolutely point in the wrong direction if they are not paired with transcript-level failure analysis.

4. Failure-mode detection is necessary, but not sufficient

Flagging capitulation was good. Scoring it like normal convergence was not good enough.

5. The best production pair may not be the highest-scoring pair

If I had to recommend one pair after this test, I would not blindly recommend the pair with the highest verdict rate. I would recommend the pair that best balances diversity, evaluability, and willingness to concede honestly.

Right now that looks much closer to GPT + Mistral than DeepSeek + Mistral.

What I’d Change In v0.2.0

This is the part I care about most going forward.

The next version needs to treat capitulation as a first-class product concern.

The shortlist:

  1. Split verdicts into two classes: genuine verdicts vs capitulation verdicts
  2. Expose pair guidance in the product: not all pairs should be treated as equally safe defaults
  3. Warn users at configuration time when they choose highly capitulation-prone pairings
  4. Explore a forced rebuttal round after round-1 capitulation to test whether the surrender is real or lazy
  5. Add concession-quality scoring so not all concessions are treated equally

I do not yet know which of those will survive real testing.

But I do know this: shipping the current scorecards unchanged into a more public product would be a mistake.

Questions I Want Developers To Argue With Me About

This is the article in the series where I most expect strong comments, because it cuts against the clean version of the story.

Here are the questions I think are actually worth debating:

  1. Should a verdict reached through capitulation count as a verdict at all?
  2. If a weaker model cannot evaluate the other model's reasoning, is that a model-quality problem or a protocol-design problem?
  3. What is the right objective for multi-agent review systems: issue-finding, convergence, or trustworthy disagreement?
  4. How many current agent dashboards are overstating quality because they compress too many interaction patterns into one score?

If you have built multi-agent evaluators, judge systems, planner/critic flows, or review loops, I would genuinely like to know how you separate "resolved" from "actually trustworthy."

Because this field test convinced me those are not remotely the same thing.

AdversarialDebate is live here:

Next in the series: the strangest control result in the whole run, where the same model debating itself outperformed two supposedly more diverse pairs.

v0.2.1 Update: The Mistral Effect Is Confirmed

This article presented the v0.1.0 and v0.2.0 data showing that pairs containing Mistral (DeepSeek+Mistral, GPT+Mistral) outperformed pairs without it (GPT+Gemini, GPT+GPT). I interpreted this as evidence for diversity of training objective.

A reader (Heinrich Neb) pointed out that the data was equally consistent with a simpler explanation: "Mistral is the one that won't fold." Both stories predict every result I had.

The v0.2.1 separating experiment settled it. I ran DeepSeek+GPT-4o-mini — two different labs, no Mistral — on the same 150-artifact corpus:

Pair Convergence Contains Mistral?
GPT+Mistral 0.536 Yes
DeepSeek+Mistral 0.572 Yes
DeepSeek+GPT 0.246 No
GPT+GPT 0.273 No
GPT+Gemini 0.033 No

DeepSeek+GPT converged at 0.246 — statistically indistinguishable from the homogeneous GPT+GPT control (0.273). Without Mistral, two models from different labs performed like two copies of the same model.

The Mistral effect is confirmed. The recommendation changes from "pick from different labs" to "always include Mistral."

Full details in the v0.2.1 field test report.

Top comments (1)

Collapse
 
fatt profile image
Fatt Coder •

Spot on. I've noticed the exact same 'trustworthiness' issue when evaluating high-performing LLMs for real-world apps. Great read!