"Just add more agents" sounds great until a weaker model in the aggregator seat throws away a correct answer from a stronger one.
We run Metis — a verification layer over any LLM: Understanding Council → confidence gate → mixture-of-agents → verifier. It ships with the AI-Factory ecosystem as an optional cognition tier (pip install aimarket-metis).
In July 2026 we ran live HTTP benchmarks (no mocks) across reasoning sets. Three results stood out — including one where a council scored 30 points below its best member.
All raw JSON lives in the repo: metis/docs/benchmarks/.
What Metis actually does (30 seconds)
Metis is not "GPT but with more chatbots." It's a route you can call when you want an answer and a machine-readable confidence score.
On the council route, several LLM roles run in sequence. Proposers don't see each other's outputs (by design — reduces sycophantic pile-on). A verifier emits verify_score (0–1) and a verified flag your app can gate on.
That's the product: catch confidently-wrong tails + give callers something to retry/escalate on. Not magic accuracy on easy work.
Case 1 — The council got dumber than Qwen alone
Setup: 10 olympiad-style integer-answer problems (AIME-flavoured counting / modular arithmetic). Two small open models as proposers: Qwen-2.5-7B and Llama-3.1-8B, with a weak aggregator on the same tier.
| System | Accuracy | Avg latency |
|---|---|---|
| Qwen-2.5-7B (solo) | 90% (9/10) | 6.9 s |
| Llama-3.1-8B (solo) | 60% (6/10) | 9.5 s |
| Metis council (Llama + Qwen) | 60% (6/10) | 215 s |
Qwen got nine right alone. The council got six — tied with the worse model, not the better one. On three problems Qwen was correct solo; the weak aggregator corrupted the synthesis.
Takeaway: multi-agent is not a free lunch. Quality concentrates in the aggregator and verifier, not in headcount. A dumb synthesizer is a liability.
We went further: weak proposers + strong DeepSeek aggregator still scored 50% — garbage in, garbage synthesized. A strong seat can't rescue weak proposals.
Architectural fix: Metis now has a capability gate (on by default): aggregator / verifier / synthesizer = strongest configured model; proposers below a floor lose their vote. Details in capability.py.
Case 2 — Lifting a mid-tier model with the same base engine
Setup: 24 curated reasoning questions — multi-step math, logic, science, deduction, plus 6 classic traps (questions where a fluent wrong answer is likely).
Same flagship class, single call vs Metis council on DeepSeek-V4-Pro as base:
| System | Overall | Traps (6) | Median latency |
|---|---|---|---|
| DeepSeek-V4-Pro (solo) | 96% | 5/6 | 0.3 s |
| Metis (V4-Pro base) | 100% | 6/6 | 89.6 s |
| MiniMax-M3 (solo) | 100% | 6/6 | 6.6 s |
Easy categories (math, logic, science) saturated at 100% for everyone. The gap was one trap:
"How many months of the year have exactly 28 days?"
Correct: 12 (every month has at least 28 days).
DeepSeek, Kimi, Qwen3-Max, and GLM-5.2 each answered 1 solo. Metis on the same V4-Pro base answered 12 withverify_score: 1.0.
Honest caveat: MiniMax-M3 also hit 100% solo in ~6.6 s. Metis didn't beat every frontier model — it lifted its own base to match the strongest single model we tested, at ~90 s latency. The durable extras: verify scores on every item and catching tails your single call won't flag.
On a separate 10-case simple harness: DeepSeek direct 80% → 90% with Metis (~11× latency). Directional, not a leaderboard.
Case 3 — Five strong agents in council beat every solo model
When every seat is capable, diversity can add accuracy — not just cost.
Setup: same 10 olympiad problems. Config D — all-star council: five strong families as proposers — DeepSeek, Kimi, Qwen-Max, GLM, MiniMax — with strong aggregator + verifier.
| Config | Accuracy | Avg latency |
|---|---|---|
| Best single model (any of the strong solos) | 90% (9/10) | ~7–15 s |
| D: all-star council (5 families) | 100% (10/10) | 240 s |
Every solo model missed problem #1. The diverse strong council got it. Diversity paid — but only among the capable.
Contrast with Case 1: five weak voices didn't help; five strong voices did. The policy implication is the same: don't crowd the room — curate it.
On a strong base + strong aggregator, a bake-off on 8 hard/trap items showed 100% for every mix — heterogeneity added latency, not accuracy (Self-MoA regime). Diversity matters most when the base has blind spots checkable verification can still catch.
When to use Metis (cheat sheet)
| ✅ Good fit | ❌ Bad fit |
|---|---|
| Confidence gate on high-stakes agent steps (factory architect, methodologist) | Wrapping an already-strong model on easy checkable tasks — same accuracy, ~15× latency |
| Lifting mid-tier engine on traps / ambiguous specs | Expecting weak models + weak aggregator to beat a strong solo |
| Cheap diverse proposers under a strong aggregator + verifier | Naive debate with small models (literature: can drop below solo) |
Rule of thumb: put your best model in aggregator + verifier. Proposer diversity is optional — it helps on weaker bases or open-ended work, not always on saturated hard-checkable sets.
Try it
- Live demo: metis.modelmarket.dev — 3D cognition panel, no login
- Source: github.com/alexar76/metis (MIT)
-
Full write-up + JSON:
docs/benchmarks/HEAD-TO-HEAD-2026-07-11.md -
Reproduce:
metis calibrate+ benchmark harness undermetis/benchmarks/
If you're building agents that pay, deploy, or invoke other agents — verification is a handoff problem, not a vibes problem. These numbers are one snapshot; the architecture lessons held across every config we tried.
⭐ If this was useful, the ecosystem map lives at github.com/alexar76/aicom — factory, hub, ARGUS, oracles, and Metis as one stack.
Benchmarks run 2026-07-11 against live provider APIs. Sample sizes are small (10–24 items per suite) — treat as directional engineering evidence, not a vendor leaderboard.
Top comments (0)