DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Multi-Agent Debate: make models argue until they agree on the right answer

#ai #promptengineering #llm #beginners

Ask one model a deceptively simple question — how many times does the letter "r" appear in "strawberry"? — and you will often get a fast, fluent, confident "2." It is wrong (the answer is 3), and worse, nothing in a single pass ever catches the slip. The model commits, sounds certain, and there is no second opinion. Multi-Agent Debate fixes that by refusing to trust one pass: it runs several instances of the same model, lets them see and challenge each other, and has them revise until they converge.

🗣️ Interactive demo: https://dev48v.infy.uk/prompt/day32-multi-agent-debate.html

The idea (Du et al., 2023)

The paper "Improving Factuality and Reasoning in Language Models through Multiagent Debate" proposes something almost embarrassingly simple: use more than one copy of the model, and let them argue. Spin up N independent instances, have each answer the question, then over a few rounds show every agent the others' answers and ask it to critique them and produce a revised answer. Across arithmetic, reasoning, and factual tasks this beat a single pass — not because any agent got smarter, but because the interaction surfaces mistakes a lone model would keep.

Propose → critique → revise

The loop has three beats:

  1. Propose — each agent answers independently (round 0). They will not all agree.
  2. Critique — each agent is shown the others' current answers and asked to weigh them against its own.
  3. Revise — each agent re-answers, free to change its mind if the others convinced it.

Repeat the critique/revise beat once or twice. The whole trick is the context you feed back: an agent's round-1 prompt contains its peers' round-0 answers.

function debatePrompt(question, others) {
  return `${question}

Other agents answered:
${others.map((a, i) => `Agent ${i + 1}: ${a}`).join("\n")}

Critique their reasoning and yours, then give your best answer.
Change it if they convince you.`;
}
Enter fullscreen mode Exit fullscreen mode

That single line — feeding the peers' answers back in — is the entire difference between debate and just sampling the model N times.

Why it works

It is a diversity-of-errors argument. Independent instances tend to make different mistakes: one miscounts letters, another mis-splits the dollar in the bat-and-ball problem, a third mis-scales a rate. But the correct answer has a reason that holds up — "index the letters: r is at positions 3, 8, and 9." Under cross-examination, a wrong answer that cannot defend itself gets talked down, while a defensible right answer survives and attracts the others. Wrong guesses are scattered and fragile; the right answer is a shared attractor. Errors cancel; truth compounds.

A useful side effect: agreement starts to track correctness. Early on the group is split and the majority may be wrong — that disagreement is itself a signal the question is hard. As the defensible answer wins converts, consensus climbs alongside accuracy. In the demo you can watch the consensus bar flip from red (majority wrong) to violet (majority right) round by round, while a lone baseline stays stuck on its first wrong guess the whole time.

Aggregating the final round

After the last round you still have N answers, so collapse them to one:

function majorityVote(answers) {
  const c = {};
  answers.forEach(a => (c[a] = (c[a] || 0) + 1));
  return Object.entries(c).sort((x, y) => y[1] - x[1])[0][0];
}
Enter fullscreen mode Exit fullscreen mode

Majority vote is the cheap default and works once the group has converged. A judge — one extra call that reads all final answers and their justifications — does better on ties and can pick a well-argued minority the vote would miss.

The cost, and the failure mode

Debate is not free: N agents over R rounds is N × R model calls (three agents, three rounds = nine calls for one answer). Keep N small (2–4), cap R at two or three, and stop early the moment the agents agree.

And it has a real failure mode: groupthink. If the agents are too alike — same model, same temperature, same persona — they make the same mistake and simply agree on it, converging confidently on a wrong answer. The defense is diversity: vary personas, temperatures, even models, so the agents fail differently and can actually correct each other. Treat unanimity as a signal, not a proof.

Debate vs its cousins

It is easy to confuse with two neighbors. Self-consistency samples one model N times and votes — but the samples never see each other; no critique, no revision. Self-refine has one agent critique its own output — powerful, but it shares its own blind spots. Debate's edge over both is independent critics that fail differently and catch each other's errors.

Reserve it for hard, error-prone questions where a wrong answer is expensive — and always measure it against a single pass on a real eval before you pay the multiplier.

Step through the rounds and watch the disagreement shrink: https://dev48v.infy.uk/prompt/day32-multi-agent-debate.html

Part of PromptFromZero. 🌐 https://dev48v.infy.uk

Top comments (0)