How many tools should an MCP server expose? A 5-to-100 benchmark

Every MCP server I've built followed the same curve: one tool, then three, then a dozen, and at some point the model quietly started calling the wrong one. The standard advice is "expose fewer tools", which is useless until you know what fewer means in practice. So I turned the tool count into an environment variable and measured what actually changes between five tools and a hundred.

Two things move at once when the list grows: the input tokens you pay for on every single request, and the difficulty of picking the right tool. This walkthrough keeps them apart instead of hand-waving about them. It takes about an hour, assumes intermediate Python, and produces a results.csv with your numbers in it — not mine, and not anyone else's.

What an MCP tool list actually costs

MCP (Model Context Protocol) is an open protocol that lets an LLM client call functions living in a separate process. The server publishes a list of tools; each one travels to the model as a name, a description, and a JSON Schema describing its arguments.

Because that whole list is part of every request, the tool count sets two costs simultaneously:

• Input tokens — money and context-window space, reported back as usage.prompt_tokens.
• Selection difficulty — the probability that the model calls the wrong tool.

Byte size of the tool definitions is only a proxy for the first cost. The benchmark below measures the real numbers instead:

questions.jsonl ──► bench.py ──► MCP server (env TOOL_COUNT=N)
                        │                 │
                        │           tools/list → name + description + inputSchema
                        ▼                 ▼
              usage.prompt_tokens + chosen tool name ──► results.csv

Before you start

You need:

• Linux, macOS, or WSL2 on Windows 10/11
• Python 3.10 or newer with venv
• An OpenAI-compatible endpoint: LLM_BASE_URL, LLM_API_KEY, LLM_MODEL (the exact model ID from your provider's list)
• About 200 MB of free disk for the virtualenv and results
• Quota for roughly 150 short requests (5 runs × 30 questions)

Quick sanity pass first:

• python3 --version prints 3.10+
• pip show mcp openai returns both packages
• echo $LLM_BASE_URL ends in /v1
• echo $LLM_MODEL matches a model that actually exists for your provider

Then set up the environment:

python3 -m venv .venv && source .venv/bin/activate
pip install --upgrade pip
pip install mcp openai
python -c "import mcp, openai; print('ok')"

If that prints ok, you're good. If the import fails, the packages landed outside the active virtualenv — rerun with .venv/bin/pip install mcp openai.

One more thing worth doing before you spend any tokens: look up your model's context window and any documented cap on tools per request. Those values differ per provider, and I'm deliberately not assuming any of them here.

Step 1: build a server whose tool count is a variable

A fixed server can't be swept from 5 to 100. Making the count configurable means one code path and one process shape, run five times — so a difference in the results is caused by the count, and not by different code.

The trick is that tools are registered with generated names and descriptions, each hinting that it should only be called when the question names that specific shard:

# server.py
import os
from mcp.server.fastmcp import FastMCP

TOOL_COUNT = int(os.environ.get("TOOL_COUNT", "5"))
mcp = FastMCP("tool-count-bench")

def build(i: int):
    def handler(query: str) -> str:
        return f"shard-{i:03d}: no records for '{query}'"
    handler.__name__ = f"search_shard_{i:03d}"
    handler.__doc__ = (
        f"Search documentation shard {i:03d}. "
        f"Call this only when the question names shard {i:03d}."
    )
    return handler

for i in range(TOOL_COUNT):
    mcp.tool()(build(i))

if __name__ == "__main__":
    mcp.run(transport="stdio")   # stdio = JSON-RPC over stdin/stdout

TOOL_COUNT is read at import time, so every subprocess registers a different number of handlers. The descriptions are intentionally distinguishable — otherwise a low accuracy score at N=100 tells you nothing about whether the model is bad at choosing or the tools are simply indistinguishable from each other.

Step 2: check what tools/list really returns

Before benchmarking, confirm that the count you asked for is the count you got, and see how big the definitions are on the wire:

# inspect_tools.py
import asyncio, json, os, sys, time
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def inspect(n: int):
    params = StdioServerParameters(
        command=sys.executable,
        args=["server.py"],
        env={**os.environ, "TOOL_COUNT": str(n)},
    )
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            t0 = time.perf_counter()
            tools = (await session.list_tools()).tools
            dt = (time.perf_counter() - t0) * 1000
            payload = json.dumps([t.model_dump(exclude_none=True) for t in tools])
            print(f"n={n:>3} tools={len(tools):>3} bytes={len(payload):>6} list_ms={dt:.1f}")

async def main():
    for n in (5, 10, 25, 50, 100):
        await inspect(n)

asyncio.run(main())

python inspect_tools.py

You should get one line per configuration, with tools= matching the requested N. The output below shows the shape of the result only — the byte counts and timings are placeholders, not measurements, and yours will differ:

n=  5 tools=  5 bytes=  1900 list_ms=  40.0
n= 10 tools= 10 bytes=  3800 list_ms=  42.0
n=100 tools=100 bytes= 38000 list_ms=  90.0

The byte count climbs because each extra tool drags in a name, a description, and an argument schema. list_ms measures the round trip to the local server process, which is a different thing from the model's latency — keep that distinction in mind when you look at the CSV later.

Step 3: measure tokens and correct tool choice

The benchmark pulls tool definitions over MCP, converts them into the OpenAI tools format, and asks the model one question per tool with tool_choice="auto" and temperature=0. For each answer it records which tool was called and how many prompt tokens the response reports.

The conversion is mechanical:

# bench.py (excerpt)
from openai import OpenAI

client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])
MODEL = os.environ["LLM_MODEL"]

def to_openai(tools):
    return [{"type": "function",
             "function": {"name": t["name"],
                          "description": t.get("description", ""),
                          "parameters": t["inputSchema"]}} for t in tools]

def pick(tools, question: str):
    resp = client.chat.completions.create(
        model=MODEL, tools=tools, tool_choice="auto", temperature=0,
        messages=[{"role": "user", "content": question}],
    )
    calls = resp.choices[0].message.tool_calls or []
    return (calls[0].function.name if calls else None), resp.usage.prompt_tokens

Around that, run(n) fetches the tool list for a given N, derives a question per available tool ("What does shard 007 contain?" and so on), calls pick() for each, tallies correct against the tool name it expected, and appends one row per N to results.csv with the tool count, prompt_tokens, correct, and total.

Two details matter for reproducibility:

• temperature=0 keeps selection as stable as the endpoint allows.
• usage.prompt_tokens from the response, not a locally estimated count — that is the number that ends up on your invoice and inside your context window.

Sanity check when it finishes: every accuracy value should land between 0 and 100, and each N should contribute its own row.

Reading the CSV without fooling yourself

Once the file exists, look at three curves rather than one:

1. Definition bytes and prompt_tokens. They usually move together, but the token figure is the one that costs money. If bytes double and tokens barely budge, your tokenizer is doing something you should know about.
2. tools/list latency. It typically grows with the count, but this is a local stdio round trip — it is not the reason your users are waiting. Don't confuse it with model latency.
3. Accuracy as N grows. This is the interesting one. A drop between 25 and 50 tools means selection is the bottleneck, and adding more context budget will not fix it.

I won't quote per-model figures here, because I don't have measurement data for them and neither should you trust numbers someone else measured on a different endpoint, prompt format, and version of mcp. Treat any published model-specific table — including the one you might expect from a post like this — as "go check the documentation and rerun this benchmark."

What I'd actually do with the result

The output of this exercise isn't a magic number, it's a decision boundary for your own server. A few things that follow naturally:

• If accuracy holds flat and tokens are acceptable, your tool count is fine and you can stop optimizing.
• If accuracy collapses while tokens are still cheap, the problem is naming and description overlap, not budget. Sharpen the descriptions, or collapse similar tools into one with an argument.
• If tokens dominate, the same collapse trick reduces the list length directly.
• Rerun the sweep whenever you add a batch of tools or change the model — the boundary moves.

That's the whole loop: one parameterised server, five configurations, one CSV. It costs about an hour and roughly 150 short requests, and it replaces a lot of guessing.

Takeaway

The number of tools an MCP server should expose isn't a universal constant — it's whatever keeps selection accurate at a token cost you're willing to pay on every request. Measure both. The server in this post is deliberately dumb: its tools return no records strings, because the only thing under test is how the model chooses among them.

This article was originally published on AI Release — https://ai-release.org — where the full walkthrough and setup notes live. More practical AI guides land regularly on the channel: https://t.me/ai_release1.