More AI Answers Can Beat a Bigger AI Model, Research Shows

Asked once, Llama2-13B answers grade-school math problems (GSM8K) correctly 35% of the time. Llama2-70B — a model roughly 5.4 times larger — gets 54%. That gap looks like it's about size. It mostly isn't. Ask the small model the same question 40 times, take the most common answer across those 40 tries, and it scores 59% — beating the big model's single-shot answer, using the model that's a fraction of the size.

The study

Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye (Tencent) published More Agents Is All You Need — a title that winks at the original Transformer paper, "Attention Is All You Need" — in 2024 (Transactions on Machine Learning Research). Their method, Agent Forest, is close to the simplest thing you could try: query the same model N times with randomness turned on, so each run can land on a slightly different answer, then let the answers vote. For open-ended tasks like code, "most common" is measured by textual similarity (BLEU score) between candidates; for multiple-choice-style tasks, it's literally the most frequent answer.

"A brute-force ensemble of smaller LLMs can achieve comparable or superior performance to larger LLMs […] without the need for additional handcraft prompt design or complex collaboration frameworks."

What they found

Tested across five benchmarks (math, reasoning, chess-state tracking, general knowledge, and code generation) and three model sizes, going from 1 query to an ensemble of up to 40:

• Llama2-13B on GSM8K: 35% (single) → 59% (40-way ensemble) — beats Llama2-70B's single-shot 54%
• Llama2-70B on GSM8K: 54% (single) → 74% (ensemble)
• GPT-3.5-Turbo on GSM8K: 73% (single) → 85% (ensemble)
• GPT-3.5-Turbo on MMLU (general knowledge): 59% (single) → 70% (ensemble)
• GPT-3.5-Turbo on HumanEval (code generation): 67% (single) → 73% (ensemble)

For scale: GPT-4, queried once, scored 88% on GSM8K in the same evaluation. GPT-3.5-Turbo's ensemble of 40 (85%) gets close to that with a materially cheaper model — not by getting smarter, just by not settling for its first answer.

Diminishing returns, and where they kick in

The gains aren't unlimited. The paper's own figures show accuracy climbing steeply up to roughly 15–20 samples, then flattening — most of the benefit arrives well before 40. They also found the effect is uneven by task difficulty: the harder the problem, the more an ensemble helps. On the harder MATH dataset, the smaller Llama2-13B model gained 28%–200% relative improvement from ensembling; on the easier GSM8K, only 16%–69%. Difficult problems apparently give a model more distinct ways to be right or wrong on different tries, which is exactly the raw material majority voting needs.

The honest limitation: this is one model, many rolls of the dice

It's worth being precise about what Agent Forest actually is: every "agent" in this paper is the same model, sampled repeatedly with randomness. That's a genuinely different setup from putting several different models — trained by different labs, on different data — in conversation with each other. Cost scales linearly with ensemble size (40 samples means roughly 40x the tokens of one query), and the paper found the method doesn't compose with everything: combined with a debate-style method on code generation specifically, performance got worse, which the authors attribute to noise from other samples disrupting otherwise-coherent code logic.

The connection to our earlier piece on multi-agent debate is worth drawing out directly: that research found debate between genuinely different models (Bard and ChatGPT) caught more errors than debate between copies of the same model, because different models don't share the same blind spots. If simply re-rolling the same model — the weakest form of diversity available — already produces gains this large, it's a reasonable expectation (not something this specific paper tested) that querying models that are actually independent of each other compounds the effect rather than diluting it.

What this means in practice

The headline finding here isn't "use a bigger model." It's that a single query to a single model — regardless of size — is leaving real accuracy on the table, especially on harder problems. Getting more than one independent attempt at an answer before you settle on one is doing genuine work, not just adding noise.

See our Best Practices guide for how to apply this directly — broadcasting one question to several models at once and comparing answers is the same underlying principle this paper demonstrates with one model rolled 40 times, just with more diversity built in from the start.

Sources

• Li, J., Zhang, Q., Yu, Y., Fu, Q., & Ye, D. (2024). More Agents Is All You Need. Transactions on Machine Learning Research. arXiv:2402.05120.