I Built a Tool to Test Whether Multiple LLMs Working Together Can Beat a Single Model

Richard Simmons — Wed, 08 Apr 2026 19:58:08 +0000

The Question

Can you get a better answer by having multiple LLMs collaborate than by just asking one directly?

That's the thesis behind Occursus Benchmark — an open-source benchmarking platform that systematically tests multi-model LLM synthesis pipelines against single-model baselines across 4 providers and 29 orchestration strategies.

What It Does

Occursus Benchmark runs the same task through 29 different orchestration strategies — from a simple single-model call to a 13-call graph-mesh collaboration — and scores every output using dual blind judging (two frontier models score independently on a 0-100 scale, averaged). This tells you whether adding pipeline complexity actually improves quality, or just burns tokens and money.

The tool supports 4 LLM providers: Ollama (local/free), OpenAI (GPT-5.4), Anthropic (Claude Opus 4.6), and Google Gemini 2.5 Pro. You toggle models on and off; the tool auto-assigns them to pipeline roles (generator, critic, synthesizer, reviewer).

The 29 Pipelines

Tier	Pipelines	Strategy
1 — Baseline	Single Model, Best of 3, Sample & Vote	Direct call and simple selection
2 — Synthesis	Full Merge, Critique Then Merge, Ranked Merge	Multi-persona generation + synthesis
3 — Adversarial	2-Way Debate, Dissent Merge, Red Team/Blue Team, Expert Routing, Constraint Checker	Models challenge each other's work
4 — Deep	Chain of Verification, Iterative Refinement, Mixture of Agents, Self-MoA, Adaptive Debate, Reflexion	Multi-round reasoning loops
5 — Experimental	Persona Council, Adversarial Decomposition, Reverse Engineer, Tournament, Graph-Mesh, Mesh+Verify, Mesh+Ranked, GSV, Mesh+Ranked+Verify, Adaptive Cascade, Managed Team, Corp Hierarchy	Heavy orchestration and combination pipelines (2-17 calls)

Several pipelines implement architectures from recent research papers:

Self-MoA (Princeton 2025): Same-model sampling outperforms multi-model mixing by 6.6%
Adaptive Debate / A-HMAD (2025): Specialist debaters achieved +13.2% over baselines on GSM8K
Reflexion (2023+): Verbal self-reflection memory produces >18% accuracy gains
Graph-Mesh (MultiAgentBench ACL 2025): All-to-all topology outperforms star/chain/tree

Three Ways to Call LLMs

Mode	Speed (29 pipelines x 8 tasks)	Cost	Best for
API	10-14 hours	~$50-80	Fastest, full parameter control
Hybrid	12-18 hours	~$15-25	Best balance — CLI primary, API for parallelism
Subscription CLI	30-45 hours	$0	Cheapest, overnight/weekend runs

API Mode

Standard REST API calls. Full control over temperature, token limits, and concurrency. Uses flagship models: Claude Opus 4.6, GPT-5.4, Gemini 2.5 Pro.

Hybrid Mode (Recommended)

CLI primary ($0), API as parallel fallback. When a subscription CLI call would be blocked waiting for another call to the same provider, the system automatically uses API instead. If API credits run out mid-run, it gracefully falls back to CLI-only — no data loss, just slower. ~2x faster than pure CLI at ~30% of pure API cost.

Subscription CLI Mode

Routes calls through your existing paid subscriptions at $0 extra cost:

Claude via claude -p --model opus (Anthropic Pro/Max subscription)
ChatGPT via codex exec (OpenAI subscription)
Gemini via gemini -p --model gemini-2.5-pro (Google subscription)

How It Works

1. Toggle models on/off — 6 preset models across 4 providers, simple checkboxes

2. Select pipelines and tasks — Choose which strategies to benchmark against which problems

3. Click Run — The tool auto-assigns models to roles:

Claude or GPT as the primary generator and synthesizer
The other as critic and alternative generator
Gemini for diversity in multi-model pipelines
Ollama for speed

4. Watch results stream in — Real-time Server-Sent Events update a score matrix, bar charts, and statistics as each cell completes

5. Dual blind judge — Both Claude and GPT score every output independently. Scores are averaged into a single 0-100 result. Neither judge knows which pipeline produced the output.

4 Task Suites

Suite	Tasks	Difficulty	Purpose
Smoke	5	Easy	Quick validation
Core	12	Easy-Medium	Standard benchmark
Stress	8	Hard	Complex reasoning and planning
Thesis	8	Very Hard	Designed to break single-model ceilings

The thesis tasks specifically target areas where research suggests multi-model approaches should excel:

Cross-domain synthesis: Design silicon-based biology with chemistry equations (requires deep knowledge from two unrelated fields)
Multi-file code refactoring: Refactor Flask to FastAPI with 6 simultaneous requirements (SQL injection fix, OAuth2, async, Pydantic, bcrypt, preserve JSON schema)
Constraint satisfaction: Write a debate without using the letter 'z', with exactly 3 rhetorical questions and a 10-word final sentence
Needle-in-haystack: Find contradictions across 5 quarterly financial reports and calculate EBITDA

These are problems where a single LLM routinely drops constraints, misses cross-domain connections, or loses track of conflicting information.

Enhancement Toggles

Beyond pipeline selection, the tool offers toggles that modify how every pipeline behaves:

Chain-of-Thought — Forces step-by-step reasoning before final answers
Token Budget Management — Reserves 60% of the token budget for the synthesis step (prevents verbose intermediate steps from starving the final answer)
Adaptive Temperature — Auto-classifies tasks (factual/code/analytical/creative) and sets optimal temperature
Repeat Runs — Run each cell 1/3/5 times, report mean ± std dev for statistical significance
Cost Tracking — Display estimated $ per pipeline using published per-token pricing

Benchmark Results

With flagship models and strict dual judging (Opus 4.6 + GPT-5.4)

On hard thesis tasks (cross-domain synthesis, code refactoring, constraint satisfaction, needle-in-haystack):

Rank	Pipeline	Avg Score	vs Baseline	Calls
1	Graph-Mesh Collab	93.3	+10.8	10
2	Chain of Verification	91.3	+8.8	4
3	Ranked Merge	91.2	+8.7	5
4	Tournament	90.0	+7.5	12
5	Reverse Engineer	89.7	+7.2	4
—	Single (baseline)	82.5	—	1

Multi-model pipelines beat single-model by up to 10.8 points on hard tasks. This is not the 1-2% margin we saw on easy tasks — it's a genuine, substantial improvement.

Key findings

Selection pressure (Sample & Vote, Ranked Merge) consistently outperforms synthesis — generating multiple candidates and picking the best beats trying to merge them
Graph-mesh topology (all-to-all agent communication) is the strongest collaboration pattern
Structured verification catches errors that single models miss, especially on constraint-heavy tasks
Debate-style pipelines underperform — forcing adversarial opposition on complex tasks destroys quality
Flagship judges are stricter — Opus 4.6 + GPT-5.4 produce more discriminating scores than mid-tier models, making the differentiation between pipelines more meaningful
2-2-2 provider balance — distributing roles evenly (2 per provider) maximises parallelism

What doesn't work

Generic debate (57.1) — forcing opposition on settled facts
Dissent-then-merge (68.9) — harsh critique without structure loses good content
Self-MoA on hard tasks (77.0) — same-model sampling lacks diverse perspectives

29 pipelines including 5 combination variants (fusing the top-3 architectures) and 2 corporate hierarchy variants are being benchmarked.

Tech Stack

Backend: Python, FastAPI, fully async
Providers: Ollama, OpenAI, Anthropic, Gemini — with auto-routing by model name and retry with exponential backoff
Frontend: Vanilla HTML/JS/CSS, Chart.js, Server-Sent Events
Storage: SQLite (WAL mode), CSV/JSON export
Pipelines: 10 module files implementing 22 strategies, all sharing a common BasePipeline interface

Try It

git clone https://github.com/rich1398/Multi-Model-Benchmarking.git
cd Multi-Model-Benchmarking
pip install -r requirements.txt
python app.py

Open http://localhost:8000, configure your API keys (or just use Ollama for free local testing), and run your first benchmark.

GitHub: github.com/rich1398/Multi-Model-Benchmarking

This is an active research project. The next benchmark run will test all 22 pipelines against the thesis task suite with enhancement toggles enabled. If you have ideas for pipeline architectures that might beat single-model baselines, open an issue or PR.

DEV Community: Richard Simmons