The obvious version of a multi-model tool is a black box: send it a question, it asks GPT and Gemini and Claude and Grok, and hands you back one blended answer. That was the first thing I cut.
I built mcp-second-opinion, a small MCP stdio server that lets an agent like Claude Code or Cursor consult rival models mid-task. It ships two tools. ask_other_model hits one named model. ask_the_panel convenes every model you have a key for. The interesting decision is what the panel does not do: it never reconciles the answers. It hands back four labeled responses and lets the caller sort it out.
This post is about why that's the right v0, and the parts of the fan-out that were harder than they looked.
The problem with a consensus layer
A synthesizer sounds helpful until you ask what it's actually doing. To merge four answers into one, something has to decide who's right. That something is either a fifth model call (now you're trusting a judge that has the same failure modes as the panelists) or a heuristic like majority vote (which is meaningless when the question isn't multiple-choice).
The whole reason you convene a panel is the disagreement. If GPT says the migration is safe and Claude flags a race condition, the gap is the signal. Averaging that into "the migration is probably fine, but watch for edge cases" throws away the one thing you called four models to get.
So I wrote it into the design spec as an explicit non-goal and shipped the panel as raw labeled output. A summary tool can come later if it earns its place. The v0 surface is honest: here is what each model said, you decide.
The fan-out
ask_the_panel runs all enabled providers concurrently and collects them. The core is one asyncio.gather:
start = time.monotonic()
if coros:
results = await asyncio.gather(*coros.values())
for provider, result in zip(coros.keys(), results):
if result.error is not None:
responses[provider] = {
"answer": None,
"model": result.model,
"error": result.error,
"latency_ms": result.latency_ms,
"cost_usd": None,
}
else:
responses[provider] = {
"answer": result.answer,
"model": result.model,
"error": None,
"latency_ms": result.latency_ms,
"tokens": result.tokens,
"cost_usd": result.cost_usd,
}
total_latency_ms = int((time.monotonic() - start) * 1000)
Two details in there that I got wrong the first time.
First, total_latency_ms is wall-clock, measured once around the whole gather. It is not the sum of per-provider latencies. If Grok takes 4s and everyone else takes 1s, the panel took 4s, not 7s. Reporting the sum would be a lie about how long the user actually waited. The per-provider latency_ms is still there in each slot if you want to see who was slow.
Second, gather has no return_exceptions=True. That looks like a bug until you see the provider layer. provider.ask never raises. It catches everything and returns a ProviderResponse with an error field set:
try:
response = await asyncio.wait_for(
litellm.acompletion(model=model, messages=messages, max_tokens=max_tokens),
timeout=timeout,
)
except asyncio.TimeoutError:
return ProviderResponse(model=model, answer=None,
error=f"timeout after {timeout}s", ...)
except Exception as e:
return ProviderResponse(model=model, answer=None, error=str(e), ...)
This is the part that makes partial failure boring. One provider timing out doesn't take down the panel. Its slot comes back with answer: None and error: "timeout after 30s", and the other three answers are untouched. If I'd let exceptions propagate into gather, a single flaky provider would blow up the whole call. Pushing error handling down to the provider boundary means the panel is failure-isolated by construction, not by a try/except wrapped around the gather.
Graceful degradation over hard config
A four-provider tool that demands four API keys is a tool nobody runs. I don't have a Grok key half the time. So enabled-ness is derived from the environment, not declared:
PROVIDER_KEYS = {
"openai": "OPENAI_API_KEY",
"gemini": "GEMINI_API_KEY",
"anthropic": "ANTHROPIC_API_KEY",
"grok": "XAI_API_KEY",
}
enabled = frozenset(
provider
for provider, env_key in PROVIDER_KEYS.items()
if os.environ.get(env_key)
)
Set one key, you get a one-model panel. Set three, you get three. The missing ones still appear in the response with error: "XAI_API_KEY not set" instead of vanishing, so the caller can see the panel is incomplete rather than wondering why Grok never spoke. The server never crashes on missing config. It just tells you what it can and can't do.
LiteLLM does the provider-routing work under this. Model names get a prefix so LiteLLM knows where to send them:
def resolve_model(name: str) -> str:
if name.startswith("claude-"):
return f"anthropic/{name}"
if name.startswith("gemini-"):
return f"gemini/{name}"
if name.startswith("grok-"):
return f"xai/{name}"
return name
That's the whole abstraction. One litellm.acompletion call works for all four vendors, and pricing comes back through litellm.completion_cost, which is where the per-slot cost_usd and the panel total come from.
The self-skip footgun I built on purpose
Here's one that only shows up when the host is itself a panelist. If Claude Code calls this server and the panel includes Anthropic, you're asking Claude to grade a room that Claude is sitting in. Sometimes that's fine. Sometimes you specifically want outside opinions.
So there's MCP_SECOND_OPINION_SELF_SKIP. Set it to a provider key and that provider drops out of the panel:
if provider == config.self_skip:
responses[provider] = {
"answer": None,
"model": resolved,
"error": "skipped (self)",
}
continue
It's off by default, and an unknown value logs a warning and skips nothing rather than failing. Small feature, but it's the difference between "second opinion" and "echo."
What I'd do differently
The profiles (flagship, balanced, cheap) pin one model per provider per profile. That was the right call for a v0.1, but it means you can't run gpt-4o and claude-haiku in the same panel without editing the profile map. A real config would let you compose the panel model-by-model. I deferred it because nobody had asked for it yet, and shipping the constrained version taught me what people actually reach for.
I'd also revisit the flat 30s timeout. It's per-provider and uniform, which punishes fast models by making the slow one set the floor for wall-clock latency. A smarter version might cut the panel off once N of 4 have answered. But that reopens the consensus question I spent this whole post avoiding, so I left it alone.
Takeaway
If you're building a multi-model tool, the hard part isn't calling four APIs, it's resisting the urge to blend the answers, because the disagreement is the product.
Source: mcp-second-opinion on PyPI, MIT licensed.
Top comments (0)