Switchboard: building a tool router so your AI agent stops drowning in MCP tools

Shashank Shah — Sun, 09 Aug 2026 07:26:05 +0000

Keyword search picks the right tool for an AI agent 21% of the time. The router
we built picks it 88% of the time, while cutting the tokens spent describing
those tools by 99.6%. This is how we got there, including the parts we got
wrong first.
Architecture: Link

The problem nobody talks about until the bill shows up

Model Context Protocol (MCP) solved a real problem. It gave every AI agent a
standard way to talk to external tools: CRMs, observability stacks, email,
vector databases, internal APIs. It created a quieter problem in the process.

Once you connect more than a handful of MCP servers to a single agent, you are
no longer describing a tool catalog to the model. You are dumping an entire
warehouse inventory into every prompt. In our own environment, three backends
exposed 142 tools. Every one of those schemas, with its names, descriptions,
and parameter shapes, gets tokenized and sent to the LLM on every turn,
whether the user's question needs one of them or none of them.

That has three costs, and only one of them shows up on an invoice.

The first is token cost. You are paying to re-describe N tools you will
never call, every single time.

The second is selection accuracy. The more tools an LLM has to choose from in
a single context, the more often it picks the wrong one, hallucinates
parameters, or gets confused by two similarly-named tools from different
servers.

The third is operational fragility. Every new MCP server you connect makes the
prompt bigger and the agent's job harder, so scaling tool count and scaling
reliability end up pulling in opposite directions.

We built Switchboard to decouple those three curves. The name is the metaphor:
a telephone switchboard operator connects your call to the right line so you
never need to know the number. Switchboard does that for tools. The host asks
for what it wants in plain language, and the router works out which of N tool to patch it through to.

Two tools instead of two hundred

Switchboard sits between your AI host (Claude Code, a custom chat agent, an
IDE, anything that speaks MCP) and every backend MCP server you own. From the
host's point of view Switchboard is an MCP server, but instead of exposing
your full tool catalog it exposes exactly two meta-tools.

find_tools(request) takes the user's request in plain language and returns a
small, dynamically-sized set of relevant tools: zero, one, or a handful, never
a fixed top-k and never the whole catalog. invoke(tool_id, args) then calls
the selected tool against whichever backend actually owns it.

Everything else is the router's problem rather than the host's: which backend
hosts which tool, how many servers are connected, how the catalog changes over
time.

How find_tools actually decides

This is the part that took the most iteration, because search alone is not
enough. A naive nearest-neighbor lookup either returns too much, which defeats
the purpose, or misses the right tool because of vocabulary mismatch between
how a user asks and how a tool is described.

The retrieval pipeline runs in four stages.

It starts with concurrent dense and sparse search against a Pinecone vector
registry. Dense embeddings catch semantic similarity, so "send the report to
finance" finds send_email_with_attachment. Sparse keyword-style search
catches the exact-term matches dense embeddings sometimes miss: API names,
service identifiers, error codes. Both index round-trips are independent I/O,
so they run in parallel and cost one round-trip of wall-clock rather than two.

Next, a cosine-similarity gate filters out anything too far from the query's
intent before it ever reaches the LLM. It is cheap and deterministic, and it
keeps obviously irrelevant tools from wasting judge tokens.

Then an LLM judge does the expensive part properly: capability-fit selection,
de-duplication across near-identical tools from different backends, ordering
by execution sequence, and deciding to ask a clarifying question instead of
guessing when the request is genuinely ambiguous.

Finally, the result count is fully dynamic. Most routing systems force a fixed
top-k. We don't. Some queries need zero tools, some need exactly one, and a
few legitimately need several, so the judge decides rather than a hardcoded
number.

If find_tools comes back empty there is a fallback, find_more_tools, that
relaxes the gate before giving up and clarifying. It is a second chance before
the decision gets punted back to the user.

How we measured accuracy, and why the grader is strict

The headline is 85 to 90% accuracy on a held-out suite of 70 realistic
multi-tool queries, against a roughly 21% naive keyword-search baseline on the
same suite. The baseline matters more than the headline, because it is the
difference between "the pipeline works" and "an embedding lookup would have
done fine."

The grader is deliberately unforgiving. A case passes only if every required
tool appears in the result and no forbidden tool appears, where forbidden: ["*"] means the correct answer is no tools at all. For allow_any_of groups,
which are sets of interchangeable tools, exactly one member must be present.
Returning two valid alternatives is a failure rather than a hedge, because it
pushes the choice back onto the model we are trying to protect. When an order
is specified, the returned tools have to contain it as a subsequence, with
relative order preserved and unrelated tools allowed to interleave. And when a
case is marked should_clarify, the router must return zero tools and set the
clarify flag, which alone decides the case.

That last rule is the one I would defend hardest. An accuracy metric that does
not reward asking instead of guessing quietly incentivizes overconfident wrong
answers, because the model learns that any answer beats admitting ambiguity.
Ours treats correct abstention as a pass, which means the 88% includes the
system knowing what it does not know.

One honest limitation: cases requiring the same tool to be called multiple
times are structurally unsatisfiable, because route() emits each tool id
once. We count those as real misses rather than excluding them, so the
reported number is a floor rather than a flattered figure.

The decisions that mattered, and the alternatives we rejected

Most of the interesting engineering here is not in what the pipeline does. It
is in the four or five places where we deliberately chose the harder option.

Dynamic-K over fixed top-k

The obvious design is "return the top 5 matches." We rejected it because a
fixed k is wrong in both directions at once. For a query that needs one tool,
top-5 injects four irrelevant schemas and reintroduces exactly the
selection-confusion problem the router exists to solve. For a genuine
multi-step request, top-5 might truncate a plan that needed six. And for an
out-of-scope question, top-5 confidently returns five wrong tools.

Making k dynamic means the judge has to answer "how many?" as well as
"which?", which is a harder prompt and a harder thing to evaluate. It was
still the right trade. Our measured average is 1.2 tools per call out of 142,
which no fixed k would have produced.

One hard-coded rule, and only one

There is a real temptation to encode catalog-specific heuristics, something
like "queries mentioning 'log' should prefer the observability server." We
kept exactly one code-side rule: an absolute cosine floor for out-of-scope
detection.

The reasoning is a division of labor. That floor is the one judgment the LLM
cannot make cheaply. To know that nothing in the catalog fits, a judge would
have to see the entire catalog, which is precisely the cost we are
eliminating. A cosine threshold answers it in one vector op. Everything else
(which tools, dedup, ambiguity, ordering) is semantic work and gets delegated
to the judge. Rules that encode catalog specifics would need rewriting every
time someone connects a new backend, which defeats the pluggability goal.

Judge order, not score order

Early on we sorted the returned tools by cosine score. That is wrong for
multi-step requests. "Pull last week's errors and open a ticket for the worst
one" has an inherent execution order that has nothing to do with which tool
embeds closer to the query. The judge reasons about sequence, so the judge's
output order is the router's output order, and we explicitly do not re-sort.

Filter unhealthy backends before the judge, not after

The intuitive design for backend health is retry-on-failure: select a tool,
call it, and handle the error if the backend is down. We invert it. A backend
marked down has its tools excluded from the candidate set before the judge
sees them, re-evaluated on every call.

This matters because of the specific failure it prevents. The judge picks the
perfect tool, explains its reasoning, and then invoke fails, so the user
gets a wrong-looking answer for a right-looking decision. Filtering early
means the judge selects the best reachable tool instead, possibly a
second-choice tool on a healthy backend, which is the correct behavior.

We also chose reactive health detection over a heartbeat loop. A backend gets
marked down on an actual failed call and retried on next use. A separate
polling process is one more thing to keep in sync with reality, and health
checks that themselves flake produce false "down" states.

Deprecated tools stay fetchable

Our first filter dropped every tool marked deprecated. We removed that clause
deliberately, because a request like "export it in the legacy format"
specifically needs the deprecated version. Governance filtering still applies,
with PII-touching tools gated behind an explicit flag and destructive tools
behind another, but deprecation is metadata for the judge to weigh rather than
a hard exclusion for code to enforce.

Keeping the registry honest

A tool router is only as good as its index, and tool catalogs are not static.
Backends add tools, deprecate others, and change descriptions. Switchboard
runs a background ingestion pipeline, independent of request-time traffic,
which polls every registered backend on an interval and also reacts
immediately to admin-triggered registration or removal. Same pipeline, two
triggers.

Each tool runs through enrichment: deterministic structural and length checks,
LLM-based description-quality and off-topic detection, TF-IDF duplicate and
outlier detection, and incremental LLM clustering against a persisted
taxonomy.

There is a circuit breaker on the LLM enrichment step. If fewer than half the
tools survive validation, the pass is treated as an outage and the registry is
left untouched. This was a deliberate answer to a real failure mode, since a
flaky upstream model silently emptying your tool catalog is worse than having
no enrichment at all.

Before touching Pinecone the pipeline diffs against a Redis hash cache, so a
full re-poll of a backend only costs an embedding call for what actually
changed rather than the whole catalog every time. Each tool is then embedded
as three views (name, description, parameters) into dense and sparse Pinecone
indexes.

That three-view embedding deserves a note. A single embedding of a
concatenated tool blob dilutes each signal, and a distinctive parameter name
gets averaged into prose. Embedding name, description, and parameters
separately, then collapsing to the best-scoring view per tool at query time,
means a query that matches strongly on one dimension still surfaces the tool
instead of being averaged into mediocrity.

Plugging in a new backend without touching a running system

This was a hard requirement from day one. Adding or removing a backend MCP
server should never mean a restart, a redeploy, or a config-file hand-edit
under load.

curl -X POST http://<host>:<port>/admin/backends \
  -H "X-Admin-Token: $ROUTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"server_id":"firecrawl","transport":"http",
       "url":"https://my-mcp-server.example.com/mcp",
       "auth_header":"Authorization","auth_value":"Bearer <token>"}'

One authenticated call hot-connects the backend. Every tool it exposes is
validated and embedded through the same enrichment pipeline the interval
process uses, backends.json is updated, and the moment the call returns
those tools are live and discoverable through find_tools. No restart, no
downtime for existing traffic. Removal is the mirror image: one DELETE call
clears Pinecone, Redis, and the config in one step, so you never leave
orphaned vectors behind.

Registration is serialized behind a lock so two concurrent calls cannot race
on shared state. It is an unglamorous detail that matters the first time two
people onboard backends simultaneously.

On the host side, plugging in is equally uneventful. Switchboard speaks
standard MCP over HTTP, so it is just another entry in mcp.json. We wired it
into Claude Code with zero special prompting, no "please use the router"
instruction needed. A plain question like "show me the error logs for
checkout-api in the last hour" triggers find_tools automatically, gets back
exactly the one tool that matters, and resolves. That was the actual bar we
were aiming for. The router should be invisible when it is working.

What this actually saves

We instrumented every routing decision end to end. Every find_tools call
writes a telemetry row recording the routed-versus-full-catalog token
estimate, search latency, judge latency, and outcome. Those writes are
fire-and-forget, so they never block the response even if the DB write fails.
A Streamlit dashboard reads that table live, and nothing on it is hardcoded.

Measured against the 142-tool catalog, routing cuts input tokens per call by
about 99.6% versus sending the full catalog every time, which works out to
roughly 28,700 tokens saved per call (about 124 tokens routed against about
28,860 for the full catalog dump). The average call selects 1.2 tools out of
142 available, which is the dynamic-k judge working as intended rather than a
fixed top-k masquerading as precision.

End-to-end latency averages about 6.5 seconds, with roughly 4.9 seconds of
that sitting in the LLM judge step. That is the honest cost of doing selection
properly instead of guessing. The clarify rate is about 12%, meaning genuinely
ambiguous requests get kicked back to the user instead of the router guessing
and invoking the wrong tool.

Projected from those per-call savings, at 1,000 calls a day (one active team's
traffic) and Sonnet-class input pricing, that comes to roughly 10.5 billion
tokens and about $31,000 a year in avoided token spend. The per-call saving is
measured. The annual figure is that measurement multiplied by an assumed call
volume, and it scales linearly with whatever volume your deployment actually
sees.

Why a tool router matters more than it looks like it should

It is tempting to file this under prompt optimization. It isn't. As agentic
systems connect to more of an organization's real infrastructure, three things
become true at once.

Tool catalogs only grow, because nobody removes MCP servers once they are
useful, they just add more. Context windows are not free even when they are
technically large enough, since every token spent on tool schemas is a token
not spent on reasoning, retrieved context, or conversation history. And
selection errors compound, because a wrong tool call against a real backend
(sending an email, querying production data, triggering a workflow) is not a
cosmetic bug. It is an action with consequences.

A router answers all three at once. It decouples how many tools exist from how
many tools the model has to reason about, and it puts a validating,
explainable decision layer between "the user asked for something" and "a real
backend got called."

Handling the edges

A few decisions exist specifically because the happy path is not where systems
like this actually fail.

If the judge is unavailable or its call fails, routing falls back to dense
top-5 rather than returning nothing, because a degraded answer beats an
outage. The admin API 404s entirely unless an admin token is configured, so
there is no insecure-by-default state to accidentally ship. When find_tools
cannot confidently resolve intent the system asks rather than picking the
closest-sounding tool and hoping. Instrumentation writes are fire-and-forget,
so a Postgres hiccup degrades your dashboard and never your response latency.
And ingestion is idempotent and incremental, so backends can be re-polled
constantly without re-embedding a catalog that has not changed.

What's next

The current bottleneck is not accuracy. It is the 4.9-second judge step, plus
the fact that the system learns nothing from its own traffic. Both are
addressable.

Draining admin-triggered ingestion asynchronously. Registering a new
backend currently blocks the API call until every tool is validated and
embedded. That is honest but not ideal at scale. The target is to enqueue the
job and let the background worker drain it so registration returns
immediately. The work is already queue-shaped; the admin path just drains its
own jobs synchronously today.

Cutting judge latency with a cascade. Roughly 75% of end-to-end latency is
one LLM call, and most queries are not close calls. When the top candidate
leads by a wide cosine margin and no near-twins exist, a small fast model or
even a deterministic path could resolve it, escalating to the full judge only
for genuinely contested cases. The measurement to run first is what fraction
of traffic is actually contested. If it turns out to be 20%, a cascade cuts
average latency substantially at close to zero accuracy cost.

Query rewriting before retrieval. Terse or jargon-heavy requests like
"502s on checkout" embed poorly against prose tool descriptions. Expanding the
query before it hits Pinecone closes the vocabulary gap between how users ask
and how tools are documented. The trade-off is another model call in the hot
path, which argues for doing it only when the first retrieval scores weakly.
Rewrite as a fallback, not a default.

Hard negative mining from real clarify and miss cases. Every clarify is a
labeled example of "these tools looked similar but the query was ambiguous,"
and every user retry after a bad selection is a labeled negative. Mining those
into a fine-tune or a re-ranker trains on our own traffic distribution instead
of generic semantic similarity. This is the highest-leverage accuracy lever we
have, because it compounds. More traffic means better signal.

Feedback-loop learning from invoke outcomes. We currently log which tools
were selected, but not whether the invocation succeeded or whether the user
immediately tried something else. Closing that loop, from selection through
invocation to outcome, turns the telemetry table from a cost dashboard into a
training set. It is mostly a schema change and a follow-up write.

Per-tenant re-ranking. Usage patterns differ sharply by deployment. A team
that lives in observability tooling should see different ranking than one that
lives in CRM tools. A lightweight per-tenant prior over the global model
captures most of that gain without maintaining separate indexes.

Tightening the enrichment quality bar. Bad retrieval is often a symptom of
bad tool descriptions rather than a bad retriever. Flagging ambiguous or
poorly-described tools at ingestion time, and reporting them back to the
backend owner, fixes the problem at its source instead of compensating for it
at query time.

Cross-system observability. The dashboard reports router cost, latency,
and outcome. Unifying it with orchestrator-side telemetry, whether
OTel/SigNoz/Langfuse or a shared events table, would let cost and accuracy be
viewed end to end rather than only at the router boundary.

The takeaway

The interesting problem in agentic AI right now is not whether the model can
call a tool. It is whether the model can call the right tool, reliably, as the
number of tools you connect grows past what any single prompt should ever have
to describe. That is an infrastructure problem rather than a prompting
problem, and infrastructure problems deserve infrastructure solutions: a
registry, a retrieval pipeline, a validation layer, and telemetry that tells
you the truth about whether it is working.

That is Switchboard. One operator, any number of lines, and a host that never
has to know the number.

DEV Community: Shashank Shah