The Research Behind Letting AI Models Argue With Each Other

In a 2023 MIT/Google Brain experiment, two AI agents were asked the same question: "What country was the computer scientist Tomas Lozano-Perez born in?" Agent 1 answered Spain. Agent 2 answered Cuba (the correct answer). Then each agent was shown the other's response and asked to reconsider. Agent 1's reply: "I apologize for the confusion. After further research, I can confirm that Tomas Lozano-Perez […] was actually born in Cuba."

Nothing about Agent 1's underlying knowledge changed between those two answers. What changed is that it was confronted with a disagreement — and that alone was enough to correct it.

The study

Yilun Du, Shuang Li, and Antonio Torralba (MIT CSAIL), Joshua Tenenbaum (MIT CSAIL/BCS/CBMM), and Igor Mordatch (Google Brain) published Improving Factuality and Reasoning in Language Models through Multiagent Debate in 2023, later presented at ICML 2024. Their method is simple to state: give a question to several independent copies of a language model. Have each one answer on its own, with no visibility into the others. Then show each agent every other agent's answer and reasoning, and ask it to revise its own response in light of what it just read. Repeat for a few rounds.

"When these lines of work give the same answer, your confidence about the answer increases. In contrast, when these answers are different, individual lines of work may engage in a mental 'debate' procedure."

What they measured

Across six benchmarks — arithmetic, grade-school math (GSM8K), chess move quality, biography generation, general knowledge (MMLU), and valid chess moves — debate beat both a single model answering alone and a single model asked to "reflect" on and revise its own answer:

• Arithmetic: 67.0% (single model) → 81.8% (debate)
• Grade-school math (GSM8K): 77.0% → 85.0%
• Chess move quality (normalized score): 91.4 → 122.9
• Biography accuracy: 66.0% → 73.8%
• General knowledge (MMLU): 63.9% → 71.1%
• Valid chess moves: 29.3% → 45.2%

For comparison, simply asking one model to critique and revise its own answer ("reflection") moved the needle far less — on MMLU it actually made things worse (63.9% → 57.7%). A model checking its own work has nothing outside itself to check against. A model reading a genuinely different answer does.

Debate doesn't just filter — it corrects

The more striking finding is in the paper's biography and math examples: there are documented cases where every agent gets the answer wrong on the first pass, and the group still arrives at the correct answer purely by cross-examining each other's reasoning over subsequent rounds. Debate isn't only amplifying whichever agent happened to be right — it's a genuine error correction loop.

The catch: consensus isn't the same as correctness

The paper is direct about where this breaks down. When every copy of the model shares the same blind spot — the same confidently-wrong fact baked into the same training data — debate does not help:

"[W]hile debates typically converged into single final answers, these answers were not necessarily correct. […] on facts that the language model was confident in […] it was very difficult to convince an agent to change their opinion."

Debate fixes disagreement, not shared ignorance. This is exactly why the paper's main experiments — which mostly debate multiple copies of the same model against each other — likely understate the real ceiling. In one smaller test mixing genuinely different model families (Bard and ChatGPT) on 20 grade-school math problems, Bard alone solved 11, ChatGPT alone solved 14, and the two debating together solved 17 — each one catching mistakes the other couldn't see, because they weren't trained the same way and didn't share the same failure modes.

Why this matters for how you use AI day to day

This is the exact mechanic behind putting several different AI models in one shared conversation instead of asking a single model to double-check itself: each model answers independently first, then sees the others' reasoning and gets a chance to revise. The paper's own results say the second part — the part where a model has to look at something genuinely different from its own answer — is where the real gain comes from, and that gain gets bigger, not smaller, when the models involved actually come from different labs and different training runs rather than being copies of one another.

See our Best Practices guide for how to set this up in practice — in particular, why broadcasting one question to every model before anyone sees anyone else's answer (rather than the reverse) is what preserves the genuine disagreement this research shows actually gets things corrected.

Sources

• Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325. Presented at ICML 2024.