DEV Community: MCPulse

A server that gets better makes fewer calls

MCPulse — Thu, 22 Oct 2026 11:39:10 +0000

"We did 4,000 calls yesterday" sounds like a measure of how many people use your MCP server.

It isn't, and the gap between what it sounds like and what it is turns out to be the most interesting thing about the chart.

What it actually counts

Every tool call that reached the server and was recorded, bucketed by the hour it started in UTC. A call is one tool, invoked once. That's all.

Why calls aren't people

Calls per conversation vary enormously. One user asking one question can produce one call or fifteen, depending on how many tools the model chains and how many times it has to reword and try again.

Which produces a genuinely perverse result: a server that gets better — clearer descriptions, fewer retries, fewer round trips — makes fewer calls for the same number of users. On this chart, improvement looks like decline.

If you've just rewritten an ambiguous tool description and your call count fell, that might be the fix working.

One heavy user looks like many light ones. An agent running a loop overnight can out-call every human user you have, combined.

Your own testing is in there. In the first weeks of a server, a large share of calls are its author.

The deliberate absence

The obvious response is to count users instead. We don't, and the reason is a design constraint rather than an oversight.

Nothing on the wire identifies a person. A call carries the tool name, the client name, a session ID, the timing, the outcome, the response size, and a hash of the arguments. That's the whole payload. No user ID, no IP stored against a call, no account name.

The session ID is generated by the SDK and means nothing outside your own data.

Adding a user field would make a nicer chart and a worse product. A user identifier is personal data, and MCP servers sit on email, calendars, CRM records — exactly the places where a security review stops being a short conversation.

The principle is the same one that keeps arguments out: if there's no field for it, there's nothing to leak and nothing to configure. A privacy guarantee you can switch off isn't one.

What to read instead

Sessions. Closer to a working period than a call is. Still not a person — one user opens many — but it moves with usage rather than with chattiness.

One implementation note if you're building this: count sessions from their own rows, not by grouping calls by day. A session that crosses midnight is one session, and a naive daily grouping counts it twice.

The client split. How many distinct clients connect, and what share each makes, says more about who your audience is than any total. It also tends to be the thing that actually surprises people.

Calls per tool. The total going up is less informative than which tools it's going up on. Growth concentrated in one tool is a workflow someone has found.

Read the shape, not the height

The height is ambiguous. The shape isn't:

A cliff — calls dropping to near zero across every client at once — is an outage, a broken deploy, or a key that stopped working. Worth an alert.
A cliff in one client only usually means that client changed something. A model, a config, how it lists your tools. Not your bug, and you'd never have attributed it correctly from a blended total.
A step up after a directory listing, a post or a release tells you which of those actually moved anything. This is the cheapest marketing attribution you'll get.
A weekly rhythm tells you whether people use the server for work.

The useful framing

Treat it as a pulse. It tells you the server is alive and roughly how hard it's working.

Everything else people want from it — how many users, whether they're happy, whether the tools are understood — is a different measurement, and reading it off this chart produces confident wrong answers.

The specific trap is the one at the top: a server that improves makes fewer calls. Any metric where the number going down can mean the thing is working needs a second number beside it before you act on either.

I build MCPulse — two lines inside your server, never your arguments or your results.

Originally published at getmcpulse.com.

A server-wide average hides which client is struggling

MCPulse — Thu, 15 Oct 2026 10:47:28 +0000

Say a tool's first-call success is 62%. What do you change?

You can't tell. 62% might be a tool every model finds mildly confusing, in which case the description needs work. Or it might be a tool one client reads perfectly and another reads badly, which is a completely different problem with a completely different fix.

The number is identical either way.

An MCP server has several readers, not one

Your server isn't called by "the model". It's called by whatever clients your users connect from — Claude Desktop, Cursor, Codex, and frequently something you didn't expect — each putting a different model, a different system prompt and a different tool-selection strategy in front of the same descriptions.

Those readers disagree with each other. A description that's unambiguous to one can be read two ways by another. A required parameter one client fills from context, another has to guess at.

And all of it lands in the same column of your metrics, averaged together.

What the average costs you

Here's the shape, from a deliberately-constructed sample server — treat the numbers as an illustration rather than a measurement:

Client	First-call success on one tool
`claude-desktop`	76%
`cursor`	21%
All clients	62%

Same tool. Same description. Same day.

Three readings of that:

From the server-wide figure, the tool is a bit weak. You'd reword the description in general terms and hope.

From the split, the tool is fine in one client and badly broken in another. The handler isn't the problem. The description isn't generally the problem either — something specific in it is being read differently by one reader.

And the larger client hides the smaller. The good client's traffic pays for the bad one's, so the average looks survivable while one group of your users is having a terrible time.

The gap between those two rows is the finding. A single number cannot contain it.

62% is also the worst place for a metric to sit: too high to look urgent, too low to look healthy. You glance at it, decide it's not today's problem, and move on. Which it isn't — unless you're one of the Cursor users.

It's not just first-call success

That's the sharpest case, but nothing is immune:

Bad arguments concentrated in one client point at a parameter that client's model fills differently.
Empty answers in one client usually mean it's passing narrower filters.
Latency can differ by client, when one sends larger requests or calls tools in a different order.
Dead tools are sometimes dead in one client only, because its model always reaches for a sibling instead.

Where the client name comes from

You don't have to guess it. Every MCP connection opens with an initialize request, and the client names itself in clientInfo.name. Read it from there, attach it to every call in the session.

The cost of keeping it is a column — but which column matters, and this is worth getting right the first time.

Counters stored one row per hour, per tool, per client let you ask "how is search_orders doing in Cursor only". Tool counts and client counts in two separate tables cannot answer that question at all — you can filter by tool, or filter by client, and never both. That was a real fix on our side rather than a hypothetical, and retrofitting it meant a migration.

If you're building this yourself, key by both from the start.

What to do when you find a gap

When one client is far below the others on one tool:

1. Read the description as that client would. Look for the word with two meanings, the parameter whose format is implied rather than stated, the constraint that lives only in the schema and never in the prose.

2. Check bad arguments for the same tool-and-client pair. If that's high, the model isn't failing to choose your tool — it's failing to fill your schema. Different fix.

3. Fix for the struggling client without breaking the good one. Watch both rows after the change. A rewrite that lifts one client from 21% to 60% and drops the other from 76% to 50% is not progress, and a server-wide average would have told you it was.

That last point is the whole argument in miniature. Without the split you can't even evaluate your own fix.

What it can't tell you

The client name is what the client says it is. It identifies software, not a model version — and a client can change its model without changing its name.

So when a client's numbers move overnight and your server didn't change, that's the first thing to suspect. It's also unfalsifiable from your side, which is worth knowing before you spend a day on it.

It also can't tell you why one model reads your description differently. Only that one does, on which tool, and by how much.

Which is still the part you had no way of seeing before.

I build MCPulse — two lines inside your server, never your arguments or your results. There's a live demo with the client filter on every panel, no signup.

Originally published at getmcpulse.com.

Counting MCP retries without counting pagination

MCPulse — Fri, 09 Oct 2026 06:01:30 +0000

A retry is the cheapest signal you'll ever get that a model didn't understand your tool.

It's also easy to count wrongly — and a wrong retry count is worse than no retry count, because it sends you off to rewrite descriptions that were fine.

Here's the definition that works, and why each part of it is there.

Same tool, within 30 seconds, in the same session, with different arguments.

Three conditions. Each one exists to throw out something that looks like a retry and isn't.

Same tool

A model that calls search_orders and then get_order hasn't retried anything. It drilled down.

That's a tool pair, and it's a design note about your result shape — maybe search_orders should have returned enough detail that the second call wasn't needed — rather than a sign of confusion.

Only the same tool twice counts. When the model reaches for a different tool after a bad result, that shows up elsewhere: in empty answers, or in the pairs.

Within thirty seconds

Long enough to cover a model reading a result, thinking, and calling again, with a slow tool on either side.

Short enough that the user's next, unrelated question doesn't land inside the window and get counted as a second attempt at the first one.

It's a judgement call. The important part is that it's stated rather than tuned per server, so a retry rate means the same thing everywhere.

With different arguments

This is the condition doing most of the work, and the one most likely to be got wrong.

Same arguments inside thirty seconds is a repeat, not a retry. Pagination. Polling. A client re-issuing a request it lost. A well-behaved paginating tool calls itself four times in ten seconds, and counting those as retries grades it as broken.

Different arguments means the model looked at what came back, decided it was wrong, and changed what it asked for. That's a rewording. A rewording is the model telling you it guessed.

Get this backwards and your best-behaved tools look like your worst.

Comparing arguments without keeping them

If you're building this yourself and you don't want to store argument values — and for an MCP server you probably don't, since tool arguments are whatever the conversation contained — you need a hash.

Two things matter:

Canonicalise first. {"a":1,"b":2} and {"b":2,"a":1} are the same arguments and different bytes. Without sorted keys, every repeat reads as a rewording and your retry rate is noise. RFC 8785 defines exactly one byte sequence per JSON value — sorted keys, specified number format, specified escaping.

Hash the wire bytes, not the parsed object. By the time arguments reach your handler the SDK may have applied schema defaults and dropped unknown fields. Hash after that and a client who omits limit and a client who explicitly sends limit: 50 produce the same hash — fine, arguably, until you change the default and every historical hash becomes incomparable with every new one. Silently.

What the client sent is what identifies the call.

Why it has to be computed after the fact

Whether a call was retried depends on what came after it. That can't be known when the call arrives.

So retries are a batch job, not a property you can stamp on a request as it lands. Walk each session's calls in order, look forward thirty seconds.

One detail that's easy to miss: the pass has to read thirty seconds past the end of the period it's processing. A call at 23:59:50 can be retried at 00:00:05. Stop at midnight and the last call of every single day gets judged with its follow-up missing.

The three shapes you'll actually see

A bad_args, then a successful call. The model couldn't fill in your schema, read the validation error, and corrected itself.

This is the most fixable kind, and slightly embarrassing: the error message told the model what your schema should have told it up front. Usually a parameter with no description, or one whose description just restates its name.

An empty answer, then a broader query. The model got [], assumed it had asked too narrowly, and loosened the filters.

Often the empty answer was correct — there genuinely were no matching records — and the description never said that nothing-matching was a possible outcome. One sentence fixes it.

Two or three successful calls in a row with different arguments. Every call returned 200. Every call returned data. The model still didn't get what it wanted.

This is the one no other metric catches. It looks perfectly green in every log you own. Usually it's a parameter whose meaning is ambiguous — a customer_ref that might be an ID, an email, or a name, and the model is working through the options.

Reading the number

Retries feed first-call success: a call followed by a retry isn't a first-call success, however it ended.

The useful view is per tool and per client. A server-wide retry rate tells you something is being guessed at somewhere, which is almost useless. The same rate isolated to one tool in one client tells you where to spend the afternoon.

Worth stating as a count rather than a percentage where you can. "Agents retried search_orders 2.4 times on average before getting a usable answer" lands harder than "62% first-call success", because a number of wasted attempts is easier to feel.

What it can't see

A model that gives up rather than retrying leaves no trace. It calls once, gets something unusable, and answers the user without your tool. That shows up as an empty answer at best and as nothing at worst.

A retry across sessions isn't visible. The user starts a new conversation and asks again — from inside the server, that's two unrelated first calls.

Both make the retry rate a floor rather than an estimate. Which is the right way round for a signal like this: when it's high, it's definitely telling you something.

I build MCPulse, which computes this from inside your server — two lines, never your arguments or your results.

Originally published at getmcpulse.com.

Your MCP tool schemas are billed on every session, used or not

MCPulse — Mon, 28 Sep 2026 10:09:57 +0000

When a client connects to your MCP server it calls tools/list. Every tool you registered comes back — name, description, and the full JSON Schema for its arguments — and all of it goes into the model's context window.

That happens before the user has typed anything. It happens on every session. It happens for tools nobody will call.

This is the cost of your server that nobody measures, and for most servers it's the largest one.

Doing the arithmetic

Take a plausible tool:

{
  "name": "search_orders",
  "description": "Search orders by customer, status, or date range. Returns up to 50 matching orders with their line items.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "customer_email": { "type": "string", "description": "Customer's email address" },
      "status": { "type": "string", "enum": ["pending", "shipped", "delivered", "cancelled"] },
      "from": { "type": "string", "format": "date" },
      "to": { "type": "string", "format": "date" }
    }
  }
}

Serialised, that's about 480 bytes — roughly 120 tokens.

Twelve tools of that size is around 1,400 tokens per session. Verbose descriptions and nested schemas push it much higher; servers in the 4,000–6,000 token range are common, and it isn't hard to find worse.

For context from a survey of 4,749 public MCP servers: the median server ships about 1,251 tokens of schema, the 90th percentile about 7,992, and the largest one measured is on the order of 216,000 — which doesn't fit a 200K context window at all, before anyone has asked a question.

At $3 per million input tokens the money is trivial. That isn't the cost that matters.

The cost that matters is attention

The real price is paid in the context window. Every token of schema is a token not available for the conversation, and — more importantly — one more thing the model has to read past to find the tool it needs.

This is the effect people underestimate. A model choosing between four sharply described tools picks well. The same model choosing between eighteen, several of which overlap, picks worse. You'll see it in your first-call success rate long before you see it on a bill.

So the question for every tool isn't "does this work" but "does this earn its place in every conversation".

Which is why dead tools are worse than they sound

Schema size comes from the startup payload — the serialised length of each tool's schema, sent once when the server boots. Put that next to call counts and you get a specific, slightly uncomfortable number:

export_report has never been called, but costs 480 tokens of schema every session.

A tool with no calls isn't neutral. It's a fixed tax on every conversation, paid in the scarcest resource the model has, in exchange for nothing at all.

Trimming without losing accuracy

Cut tools before cutting words. Removing one unused tool saves more than tightening the prose in five.

Descriptions are for disambiguation, not documentation. The model needs to know when to pick this tool over its neighbours. It doesn't need your changelog, your rate limits, or a worked example — those cost tokens on every session to serve a case that arises rarely.

Enums are cheap and worth it. Four values in an enum cost a handful of tokens and remove an entire class of argument-validation failures. This is one of the few places where more schema pays for itself.

Don't describe self-evident parameters. "customer_email": { "type": "string", "description": "Customer's email address" } — the name already said it. Delete the description and lose nothing.

Watch for duplication across tools. Five tools that each explain your pagination convention are paying for that explanation five times per session. One server in the corpus appends the same 51-word context block to all 275 of its tools.

The measurement that settles arguments

Sort your tools by schema bytes, then look at calls next to them. The tools at the top of the first list and the bottom of the second are your answer.

You're not looking for a small saving spread across everything. You're looking for the two tools costing 800 tokens a session between them and getting called once a week — because deleting those is free, and it makes every remaining tool easier for the model to choose correctly.

If you want your own number without installing anything, the schema checker reports your total tools/list size and per-tool breakdown from a paste of your JSON, scored against servers your own size. Runs in the browser, nothing uploaded.

Call counts need instrumentation — that's MCPulse, two lines inside your own process, never your arguments or your results.

Originally published at getmcpulse.com.

How to test whether a model can tell your MCP tools apart

MCPulse — Fri, 25 Sep 2026 10:00:53 +0000

You can read your own tool descriptions all day and not know whether a model can tell them apart. I spent a few weeks measuring 82,549 public MCP tool descriptions, and the most useful thing I learned is that the measurement has a ceiling.

Here are two tools from a real server:

score_resume             "Score a resume for ATS compatibility."
analyze_job_description  "Extract what a job posting actually screens on."

They share no content word. Not one, even after stemming. Every lexical metric I have scores them as perfectly distinct.

Their author says they're confusable, and he's right — because "check my resume for this job" names both objects at once. The collision doesn't live between the two descriptions. It lives between a sentence and a set of tools, and nothing computed from descriptions alone will ever see it.

So here's the test that does. It takes an afternoon, needs no traffic, and produces evidence you can put in a pull request.

The method came out of a comment thread on a previous post — credit where it's due, I'd been circling something much more expensive.

The shape of it

For each pair of tools you suspect might be confusable, write two sentences:

A control — a request that should route to exactly one of them, unambiguously.
A probe — a request that could fairly route to either.

Run each through a model with the full tool list, at fixed temperature, a handful of times. Record which tool gets picked.

That's the whole method. The interesting part is reading the results.

Reading the control

The control runs first, and it's a gate rather than a finding.

If the control doesn't route correctly, stop — you don't have an ambiguity problem between this pair. You have one description that's broken on its own terms, and that's a different fix. Rewriting it in relation to its sibling would be solving the wrong problem.

Most of the time the control passes and you move on. When it fails, it's usually worth more than anything the probe would have told you.

Reading the probe

This is where the instinct misleads.

If the probe routes to the same tool every time, that's a pass, even though the sentence was genuinely ambiguous to you. It means the model has found a distinction you can't see in the words. Maybe it's the parameter names. Maybe it's the tool name. Maybe it's something about the phrasing you didn't consciously encode. Doesn't matter — it works.

My instinct would have been to flag any non-uniform distribution as a collision. That's wrong, and acting on it would mean rewriting descriptions that are already doing their job.

If the probe splits, that's a real collision, and you now have the sentence that proves it. Not a suspicion. Not a metric above a threshold. An actual user request that lands in two different places depending on the roll.

That distinction — stable versus split — is the whole reason this is worth running. And a handful of runs at fixed temperature is enough to tell them apart, which is a far smaller budget than I'd assumed before someone spelled it out.

Picking the pairs

You can't do this pairwise on a large server. 191 tools is 18,145 pairs, and at even a handful of model calls each, that's not an afternoon.

This is where static analysis earns its place. It stops being the answer and becomes the filter that produces candidates.

Rank pairs by input overlap — do these two tools act on the same object? Compare the nouns in their descriptions plus their parameter names. Tools that share their input are the ones where a single sentence can route to either, which is exactly the failure the probe tests for.

Then gate that on output distinguishability. If two tools take the same input but their descriptions clearly state different returns — "flat details for many records" versus "a bounded evidence bundle" — that's good design, not a problem. Don't spend probes on it.

Rank, take the top fifty, probe those. The static metric was never going to find the collision; it's very good at telling you where to look.

What to do with a split

Three outcomes, and they're genuinely different kinds of work.

Rewrite for contrast. The cheapest fix and it changes no behaviour. The rule that works: every tool in a cluster shares its input by definition — that's what makes them a cluster — so describing the input describes what they have in common. Name the output instead. "Search reports" describes ten tools. "Search reports by client, including archived ones, returning IDs and titles only" describes one.

State the trigger condition. Call this one before acting, that one after a decision is made. No lexical measure sees this and it's frequently the real distinction. It's also the part a description template flattens, which is why collision rates climb sharply past about thirty tools — that's where people stop writing descriptions individually.

Merge into one tool with a mode parameter. Sometimes the split isn't telling you your wording is bad. It's telling you the sentence names an intent your tools have partitioned and the user hasn't. "Check my resume for this job" is not a failure of either description — it's a request that legitimately spans both. If two tools can ever be correct for the same sentence, you may not have two tools. You may have one tool with a parameter.

That last one is the conclusion I've now reached from two completely different directions: as a design rule about interfaces, and empirically from a probe that splits. The second version is the one you can put in a pull request without arguing, because it comes with the sentence attached.

Keep the failures

Every split is a regression case: the prompt, the expected tool, the tool actually picked.

Store them. Then a description rewrite stops being hopeful and becomes testable — you re-run the probe and see whether the split closed. And the next person who "improves" the wording six months from now finds out immediately if they've broken the distinction.

This is the thing missing from every schema-linting approach, mine included. A linter tells you a description looks risky. A regression suite tells you whether the fix worked.

The limits

Fixed temperature and a handful of runs gives you a signal, not a rate. If you want "this pair splits 60/40", that needs many more runs and it's probably not worth the budget — the useful information is binary.

Different models may split differently, which is itself worth knowing. If a pair is stable in one client and splits in another, the problem is more specific than your descriptions.

And a probe that doesn't split tells you about that sentence, not about every sentence. Someone will eventually phrase something you didn't think of. That's an argument for keeping the regression suite and adding to it, not for distrusting the method.

Why this is the interesting layer

I've published a fair amount of static analysis on MCP schemas: distinctive share, input overlap, name distinctiveness, parameter coverage. All of it measures properties of text.

None of it observes a model choosing a tool. Which means every number reports how often a pattern exists, not how often it costs you a call — and the gap between those two is where all the value is.

The probe closes that gap for a pair at a time, cheaply, before you have any production traffic at all. If you maintain an MCP server and you've been meaning to audit your descriptions, this is the thing I'd do first.

I build MCPulse, which reports what models actually do with your tools once real traffic arrives — retries, empty results, first-call success, schema cost. There's also a free schema checker that ranks your tool pairs by input overlap, which is the prefilter step above.

The control-and-probe design came from readers of a previous post. Best methodology suggestion I've had, and it came from people who run servers rather than measure them.

Readers took my MCP schema study apart. Here's what they found.

MCPulse — Mon, 21 Sep 2026 08:19:16 +0000

I published a survey of 4,749 public MCP server schemas a few weeks ago. The headline was that 17.7% of tools carry a description containing no word that distinguishes them from a sibling tool on the same server.

Then people who actually run MCP servers showed up in the comments, and over about ten days they took the measurement apart.

One found a bug. Several found a design flaw. Two of them found opposite design flaws that cancel each other out, which was the most useful thing that happened. And one found a ceiling the whole approach can't get past.

Here's all of it, because the corrections are worth more than the original number.

The bug: no stemming

The metric counts content words in a tool's description that appear in no other tool's description on the same server. It does exact string matching.

Someone pointed at two descriptions from his own server:

analyze_job_description   "Extract what a job posting actually screens on."
optimize_resume           "Rewrite a resume so it passes ATS screening."

screens and screening. Same word, two forms, and my counter treats them as unrelated — so these two tools score as distinctive on the one word that actually links them.

That's not a nuance. It's wrong, and it's wrong across all 82,549 tools. Stemming before counting is a small change with an unknown effect on the headline figure, which I'll report when I've re-run it.

The design flaw: lexical distinctness isn't ambiguity

The same author gave me his full ten-tool list and said three of them were confusable in practice. So I ran the metric on it.

 50.0%  score_resume              "Score a resume for ATS compatibility."
 60.0%  analyze_job_description   "Extract what a job posting actually screens on."
 60.0%  optimize_resume           "Rewrite a resume so it passes ATS screening."
 75.0%  search_jobs               "Return the user's job matches."
100.0%  update_job_preferences    "Set the roles and locations it hunts for."

His ambiguous cluster — score, analyze, optimize — comes out at 50–60%. My corpus calls that healthy.

My first fix was to compute distinctiveness on verbs alone, on the theory that verbs carry the action and nouns are shared boilerplate.

That fails harder. His ten verbs are score, extract, rewrite, produce, write, translate, return, set and build. All distinct. Verb-only distinctiveness scores every tool at 100%, including all three ambiguous ones.

The problem isn't granularity. Lexical distinctness and semantic distinctness are different properties, and no word-counting metric at any resolution closes the gap.

What does show the cluster is the shared input nouns — resume in four descriptions, job in three, ats in two. Those three tools are ambiguous because they act on the same object, not because they're worded alike.

His rule, better than my number: if two tools can ever be correct for the same sentence, you don't have two tools. You have one tool with a parameter.

So the fix isn't a better distinctive share. It's a second signal, measuring something else entirely. Distinctive share asks whether two descriptions look alike. Input overlap asks whether two tools could both be right.

The correction to the correction: overlap alone flags good design

Someone running a six-tool ticket server pushed back, with a pair he'd deliberately kept:

Batch retrieval and an analysis bundle. Identical input shape. Different intended output — flat details for more tickets, versus bounded evidence with optional comments and source anchors.

Input overlap would flag that pair. He'd be right to dismiss it, because the boundary is stated: a model reading those two descriptions has something to discriminate on even though the inputs are the same.

So overlap needs a partner. Overlapping inputs with distinguishable outputs is fine design. Overlapping inputs with indistinguishable outputs is the failure. Flagging the first is exactly the false positive that gets a signal switched off — and on a large server, a 10% false positive rate buries everything.

His test is better than any metric I can compute: can a model infer the intended output and boundary from the name, description and parameter contract alone, without knowing the repo or the author's intent?

The two corrections that fight each other

This is the pair I'd have got wrong silently.

One reader pointed out that the metric penalises focused servers. A memory server says "memory" in every description by design. A ticket server says "ticket". Those words count as shared, but they aren't a collision — they're the name of the server, repeated. Drop words appearing in more than 80% of a server's tools before counting, and small focused servers score higher.

Correct, and I was about to implement it globally.

Then the ten-tool author showed why that breaks the other signal. On a focused server, the collision is the domain noun. Strip resume from his descriptions and you delete the exact words that make score, optimize and analyze confusable. I'd have shipped a sharper distinctive-share number that quietly destroyed input overlap.

The resolution is two readings with different preprocessing, not one corrected score:

Domain-stripped distinctiveness — do these descriptions look alike?
Domain-intact input overlap — could two tools be right for the same sentence?

And the gap between them is the finding. A server that scores high on the first and badly on the second is focused, acts on one object throughout, and never says what its tools return. That's a common shape and it deserves its own line rather than being averaged away.

A third reader then improved the threshold out of existence. Frequency isn't a dial to tune — it's a distinction the same count already makes. A noun in nearly every tool names the server. A noun in a subset of tools names a cluster, and that subset is your candidate collision list.

Run it on the ten-tool server and the tightest group is ats, shared by score and optimize and nothing else. Which is the pair he flagged first. Same counter, no threshold, no tuning.

The ceiling

The same reader then found the thing none of this reaches.

analyze_job_description and score_resume share no word at all, stemmed or not. And they still collide, because "check my resume for this job" names both objects at once. The tools partition an intent that the sentence doesn't.

That collision exists between a request and a set of tools, not between two descriptions. Nothing computed on descriptions alone will ever see it.

That's a hard ceiling on static analysis, not a gap to close with a cleverer metric. The only artefact that catches it is a small labelled set of real requests — a handful of plausible sentences and which tool each should route to. Which is the one thing a corpus can't generate.

Two things I hadn't measured at all

Names. Two people independently said that when descriptions collide, models fall back to pattern-matching on tool names. One put it: "the model stops reading descriptions and pattern-matches on names, and then your error rate is really a naming problem."

That changes what a collision means. Gmail_DeleteDraftEmail and Gmail_SendDraftEmail have zero-distinctive descriptions — that pair was my headline example — but their names are clear. The names are carrying the load. A hypothetical pair with the same descriptions and names like delete_item and remove_item scores identically and is in far worse shape.

Annotations. MCP has readOnlyHint, destructiveHint, idempotentHint, openWorldHint. Two readers raised the same point: authors set them and write "Read-only" and "REPLACES" in the prose by hand, because the annotations are read by the client deciding whether to prompt for confirmation, while the description is read by the model deciding whether to call the thing. Two readers, two channels, and nothing requires them to agree.

Which makes a mechanical check available: descriptions mentioning constraints with no matching annotation set, and annotations set with nothing in the prose. The first means the client can't protect the user. The second means the model doesn't know a tool is destructive at the moment it's choosing.

What I'd take from this

Every correction came from someone who knew what the text was for.

I measured a property of the text and treated it as a proxy for ambiguity. Lexical distinctness looked like a proxy until someone showed me ten tools where it isn't. Populated descriptions looked like described parameters until someone pointed at one that says nothing. Descriptions looked like the whole interface until two people said names do the work when descriptions fail.

The original study was careful about one thing: it drew a hard line between what a schema shows and what models do with it, and refused to claim the second. That line held. What didn't hold was the assumption that measuring the text well is the same as measuring the thing that matters.

The published numbers stand as published — they measure what they say they measure, and they're all floors. The more useful metrics are the ones nobody had asked for yet.

Original study, data and analysis scripts: https://github.com/getmcpulse/mcp-schema-study

The schema checker runs these measurements on your own tools/list in the browser. The corrections above are being added to it.

I build MCPulse, an SDK that reports what models actually do with your tools under real traffic. If something here matches what you've seen on your own server, I'd like to hear it — every improvement above came from exactly that.

How deep do MCP input schemas nest?

MCPulse — Fri, 18 Sep 2026 07:18:09 +0000

A model filling in a tool call has to construct whatever shape your inputSchema describes. A flat object of strings is one thing; an array of objects each containing an object is another.

So how nested are real MCP schemas? We measured depth across 74,666 tools that take parameters, following properties, items, and anyOf/oneOf/allOf branches.

Then we split the undescribed-parameter rate by depth, expecting deep schemas to be worse. They're better, and the reason is more interesting than the number.

Almost everything is flat

Depth	Tools	Share
1	62,054	83.1%
2	9,279	12.4%
3	2,333	3.1%
4	599	0.8%
5	216	0.3%
6	77	0.1%
7–13	~108	<0.1%

Five tools in six are a flat object of scalars. No nesting at all — {"query": string, "limit": integer} and nothing more.

Add depth 2 and you have 95.5% of the corpus. Depth 2 is the array-of-strings and the single options object: {"tags": string[]}, {"filter": {"from": string}}.

The deepest schema in the corpus is thirteen levels. There are five of those.

The finding that reverses the expectation

We expected deep schemas to be worse documented — more structure, more fields, more places to skip a description.

The opposite:

Tools at depth 3 or more: 18.2% of top-level parameters undescribed
Tools below depth 3: 21.8%

Deep schemas are better documented than flat ones, by 3.6 points.

Our reading is that it's about who writes them. A depth-5 schema isn't something you arrive at casually. It's either generated from a typed source — an OpenAPI spec, a Zod schema, a protobuf definition — where descriptions come along for the ride, or hand-built by someone modelling a genuinely structured input and paying attention.

Flat schemas are where the two-line afterthought lives. The single-parameter tool is the worst-described shape in the whole corpus, at 27.0%.

So depth isn't a warning sign. It correlates with care.

The caveat that matters more than the finding

That comparison counts top-level inputSchema.properties only. A field three levels down isn't in the denominator.

That was the right call for the survey — per-tool parameter counts had to mean one consistent thing — but it means the corpus number understates the real gap. A tool whose top-level parameters are all described and whose nested object is completely bare scores clean.

And an undescribed field at depth 4 is exactly as invisible to a model as one at depth 1. It has a name, a type, and nothing else.

What this means practically

Don't flatten a schema to look simpler. Depth isn't the problem the data points at. If your input genuinely has structure — a filter object, a list of line items — modelling it honestly beats a dozen underscore-joined top-level fields. The corpus suggests people who do it describe their fields better anyway.

Do describe the nested fields. They're the ones most likely to be missed, because most tooling shows you the top level. This is the one place where our own methodology would let you off and a model wouldn't.

Past about four levels, ask what the model is meant to construct. 0.4% of the corpus goes deeper than four. A model generating a five-level nested object has many more ways to get the shape wrong than right, and there's usually a flatter representation of the same request. Not always — but at that depth it's worth checking.

Watch anyOf and oneOf specifically. They're depth without looking like depth. A parameter that is "either a string or an object with three fields" is two shapes the model has to choose between, and the choice is rarely documented. A description on the branch point is worth more than one on either branch.

What this can't tell you

We measured depth, not difficulty. A depth-3 schema of well-named described fields is easier for a model than a flat one with two bare id parameters, and nothing here captures that.

The correlation between depth and better documentation is a correlation. The generated-from-typed-source explanation is our reading of it, not something the data shows — we can't see how a schema was produced.

And as always: no model was run against any of these servers. Whether nesting depth actually causes malformed arguments is unmeasured, and it's one of the more testable things on our list.

Measuring your own

The schema checker walks your schema to every depth — through properties, items and union branches — and names undescribed fields by their full path: filter.from, tags[].label. It reports the top-level count separately, so the number scored against the corpus stays comparable while the list of things to fix doesn't leave anything out.

Runs in your browser, nothing uploaded. Data in getmcpulse/mcp-schema-study.

Originally published at getmcpulse.com.

Why the model won't call your tool

MCPulse — Wed, 16 Sep 2026 08:26:29 +0000

Your tool is registered. It appears in tools/list. The model never calls it, or calls a different one instead.

There are six reasons this happens. They are not equally likely, and the instinct — assume the description needs to be better — is right about a third of the time.

Here they are in order of how often they turn up across 4,749 public MCP servers.

1. Another tool is winning a competition you cannot see

Most likely. 17.7% of public tools have a description containing no word that distinguishes them from a sibling. On servers with more than sixty tools it is 32.4%.

The tell is that a neighbouring tool gets called instead of yours, consistently. Not randomly — the same substitution every time.

How to check. Print your whole tool list and read your tool's description next to the one that keeps getting called. If you can swap the two descriptions and both still read correctly, the model has nothing to choose on.

The fix. Lead with the difference, not the category. "Search orders" describes four of your tools. "Search orders by customer, including cancelled ones" describes one. Naming the return shape is the single most distinguishing sentence available and almost nobody writes it.

2. The parameters are unfillable

Second most likely, and the one people check last. 21.5% of parameters across the corpus have no description at all.

A model that cannot work out what to put in a required field will avoid the tool rather than guess. This looks identical to a description problem from outside, and it is not.

The clearest version:

fetch    id, document_id      (neither described)

A model has to pick one and nothing tells it which, or whether they differ. So it picks a different tool.

How to check. Cover the parameter names and look only at types and descriptions. If you could not supply a value, neither can the model.

Watch for the internal identifier specifically. A tool requiring a warehouse_id is unreachable from a conversation where nobody has ever seen a warehouse ID. That is not a description problem — it is a tool that needs a lookup step before it, or a name-based alternative.

3. The name is doing all the work and losing

When descriptions collide, the name is what decides — in 89.6% of collision cases. So a colliding description plus a generic name is the case where nothing is choosing at all.

get leads 15.6% of all public tool names. If your tool is get_thing and a sibling is fetch_thing, those are synonyms in English and the model is guessing.

The fix. One verb per operation, across the whole server. Never ship two words from the same row:

get / fetch / read / retrieve
list / getAll
search / query / find
delete / remove

4. Your tool list is too long to hold

Schema goes into the context window on every connection. The median server spends about 1,251 tokens; the 90th percentile spends 7,992. The largest in the corpus spends around 216,000 — which does not fit a 200K window at all.

The failure here is not that your tool is badly described. It is that it is the fortieth candidate in a list the model is skimming, and attention is finite.

How to check. Count your tools. Under fifteen, this is not your problem. Past thirty, it is probably your main one.

5. A closed set was written in prose

The model sends "Direct" where you accept "direct", your server rejects it, and from the model's side that reads as a broken tool rather than a fixable mistake — so it stops trying.

857 parameters in the corpus name their valid values in the description and leave the schema as an open string. Moving them into an enum is one line.

The tell for this one is distinctive: the tool does get called, once, and then never again in that session.

6. It is actually unreachable

Two rarer causes worth ruling out.

A duplicate tool name. Only 0.19% of servers — nine in the whole corpus — but when it happens, one of the two tools cannot be called at all, and the symptom points somewhere else entirely. Almost always a generated tool list. One assertion in CI catches it forever.

The tool is not in the response you think it is. Servers that build their list conditionally can emit something different from what the code appears to register. Read the real tools/list output rather than the registration code.

The order to actually work through

Read your whole tool list as one block. Which tool would you pick? — catches causes 1 and 3.
Cover the parameter names, read only types and descriptions — catches cause 2.
Count your tools — catches cause 4.
Grep descriptions for valid values / options: — catches cause 5.
Assert unique names in CI — catches cause 6.

That is fifteen minutes and it covers all six.

What none of this can tell you

Every cause above is a property of your schema, and this is the honest limit: we have never observed a model choosing a tool on any of these servers. The rates say how often each pattern exists, not how often it costs you a call.

It is entirely possible your tool is skipped for a reason no schema shows — the client truncates long lists, the model was mid-task, the user phrased something unusually. Distinguishing those from the six above needs data from inside your server: which tools get called, which get retried, which return empty. A schema check is what you can do before you have any of that.

The schema checker tests causes 1, 2, 3, 5 and 6 on a paste of your tools/list, scored against servers your own size. It names the sibling each colliding tool is hardest to tell apart from — which for cause 1 is usually the whole answer. Runs in your browser, nothing uploaded.

Originally published at getmcpulse.com.

How many tools should an MCP server have?

MCPulse — Mon, 14 Sep 2026 06:52:52 +0000

If you maintain an MCP server, at some point you ask this. You've got twenty tools, you're about to add five more, and something feels wrong about it — but you can't say what, and there's no guidance anywhere.

I read the tool schemas of 4,951 public MCP servers to answer it. 87,146 tools, 270,487 parameters.

The short answer: around thirty. Past that, the thing that breaks isn't your server. It's whether a model can tell your tools apart.

The measurement

For every tool, I took the content words in its description and counted how many appear in no other tool's description on the same server. Call it the tool's distinctive share.

If that number is zero, every word in the description is a word its siblings also use. A model choosing between your tools has nothing in the descriptions to choose on — the names are doing all the work.

Then I split the corpus by how many tools each server publishes.

Tools on the server	Servers	Zero-distinctive tools	Parameters with no description
1–3	1,256	0.5%	14.8%
4–7	1,292	1.6%	22.4%
8–15	1,044	4.3%	21.9%
16–30	767	7.7%	24.8%
31–60	377	16.3%	22.2%
61+	215	31.3%	20.5%

A factor of sixty, rising monotonically. On servers with more than sixty tools, nearly one tool in three has no distinguishing word at all.

Why it gets worse

Part of it is arithmetic. More tools means more chances that two of them collide, and that would happen even if every author wrote carefully.

But the curve steepens around thirty, and arithmetic alone doesn't explain that. What happens around thirty is that authors stop writing descriptions one at a time and start generating them from a pattern. A template is a machine for producing tools that read alike.

The most extreme case in the corpus: one server appends the same 51-word context block to all 275 of its tools.

3land_createCollection   "Create a new NFT collection on 3.Land marketplace.
                          SAP MCP context: Protocol 3land; operation class
                          write. Use for 3.Land NFT collection, minting,
                          listing, cancellation, and purchase flows…"

3land_buyNFT             "Purchase an NFT from a 3.Land listing.
                          SAP MCP context: Protocol 3land; operation class
                          write. Use for 3.Land NFT collection, minting,
                          listing, cancellation, and purchase flows…"

Creating a collection and buying one are different operations, and the opening sentence says so — in 8 words out of 59. The other 51 are identical across both, and across all 275.

That block was added deliberately, to help.

Zero doesn't mean badly written

This is the part worth internalising, because it's counterintuitive.

Four tools from a widely-installed Gmail server:

Gmail_DeleteDraftEmail   "Delete a draft email using the Gmail API."
Gmail_SendDraftEmail     "Send a draft email using the Gmail API."
Gmail_ListLabels         "List all the labels in the user's mailbox."
Gmail_SearchThreads      "Search for threads in the user's mailbox."

Every one of those is clear, correct English. No reviewer would flag them. Every one is also built entirely out of words the other tools use — delete, draft, email, gmail, api, list, search, threads, mailbox all recur across the set.

The description tells you what the tool does. It doesn't tell you what this tool does and the others don't. That second thing is what a model needs at the moment it's choosing, and it's a different question from "is this description good."

The column that argues against splitting

Look at the right-hand column again. Parameters with no description at all sit between 20% and 25% at every size above the smallest bucket. It doesn't improve as servers get smaller.

A two-tool server has the habit about as much as a two-hundred-tool one.

So the two failures are independent. Splitting a large server reduces your description collisions and does nothing whatsoever for your undescribed parameters. They need separate fixes, and the split only buys you one of them.

So should you split?

Split when the collisions are real, not because you crossed a number.

The threshold in the data is around thirty, but that's a population average and your server isn't the population. A server with forty tools that all do genuinely different things to genuinely different objects may be fine. A server with twelve tools where four of them are variations on "search" is not.

The test that actually tells you: read your tool list as one block, the way a model receives it. Nothing else. No README, no repo, no memory of what you meant. Then ask which tool you'd pick for a request that could plausibly go to two of them.

Three options when the answer is "I can't tell":

Rewrite for contrast rather than clarity. Not "is this description clear" but "is it clear which of my tools this is." If you have a shared preamble on every tool, it's costing more than it's buying — the distinguishing sentence shouldn't be a seventh of the text.

Collapse near-identical tools into one with a mode parameter. Four search variants become one search with a scope enum. Fewer things to choose between, and the choice the model has to make moves from "which tool" to "which value," which an enum can constrain and a description can't.

Split the server. Fewer tools per connection, more servers to maintain. Worth it when the tools genuinely belong to different domains, less so when you're just cutting an arbitrary list in half.

The limits of this

This is static analysis. I never ran a model against any of these servers, so I can't tell you how often collisions actually cost anything. A zero-distinctive description might be harmless when the tool name is unambiguous, and expensive when it isn't. Ranking these signals by how well they predict a real mistake needs a model in the loop, which is the next study.

One more hole, found by a server author after I published: the undescribed-parameter figure measures absence only. A parameter described as "query: The query" counts as described and passes. Restating the parameter name is arguably the more common failure and it passes every linter, so 21.8% is a floor.

Data and analysis scripts: https://github.com/getmcpulse/mcp-schema-study

If you want your own numbers rather than the corpus averages, there's a free checker at https://getmcpulse.com/check — paste your tools/list JSON and it scores your distinctive share, undescribed parameters, and token cost against all 4,951 servers. Browser only, nothing uploaded.

I'm building MCPulse, an SDK that reports what models actually do with your tools once real traffic arrives. Everything above came from outside the server, which is exactly its limit: a schema can tell you a model has nothing to choose on, but only traffic tells you whether it chose wrong.

Originally published at getmcpulse.com.

We read the schemas of 4,951 public MCP servers

MCPulse — Wed, 09 Sep 2026 09:14:53 +0000

Your MCP server's tool schema is the entire interface a model has. No README, no repo, no idea what you meant — just the JSON from tools/list.

We read that JSON for 4,951 public servers: 87,146 tools and 270,487 parameters. Here's what's in it.

The short version: one tool in six carries a description containing no word that distinguishes it from a sibling tool on the same server. On servers with more than sixty tools, it's nearly one in three.

What this is, and what it isn't

This post is about what models are given. It is not about what they do with it.

We did not run a model against these servers. We never observed a tool selection, an argument, or a retry, so nothing here can tell you how often models actually get it wrong. Every number below is a property of a schema sitting still.

We're drawing that line hard because it's the line the interesting claim sits on. "A model has nothing to discriminate on" is a fact about a schema. "A model therefore picks wrong 30% of the time" is a fact about traffic, and we don't have it.

How we did it

We took the tool schemas from the Smithery registry, which stores the tools/list response for every server it has scanned — inputSchema included. That's byte-for-byte the JSON a model receives.

The obvious alternative was to boot each server in a container and call tools/list ourselves. We didn't, and the reason is the finding underneath the method: a large share of public MCP servers won't start without real credentials. The set that boots cleanly on a machine with no API keys isn't a random subset of anything.

Of 5,123 servers in the frame, 154 detail requests failed, 5 had never been scanned, and 13 published no tools at all. That leaves 4,951 servers with schemas.

The check that nearly ended the study

Before any of this counts for anything, one question has to be answered: does the registry hand back the schema the server actually serves?

We had reason to think it might not. Across the first 1,377 tools we collected, not one carried a top-level required array. Real MCP servers mark parameters required constantly.

So we booted the four official reference servers locally — they need no credentials and no network — took their real tools/list over stdio, and diffed field by field.

Everything survives except required, which is stripped. On one server that's 27 of 37 tools with a required array on the real server, and 0 via the API.

So this study measures nothing about required-versus-optional parameters. If you're doing your own analysis on registry data, that field is not there, and it does not announce itself.

1. One parameter in five has no description at all

59,038 of 270,487 parameters — 21.8% — ship with no description. They appear on 33.7% of servers.

A parameter with no description is a parameter the model guesses at. It has the name, it has the type, and that's the entire brief. Sometimes the name carries it: query on a search tool isn't mysterious. Often it doesn't — we found plenty of bare id, type, mode and filter parameters with nothing to say which of several plausible things they meant.

The striking part is the contrast with tool descriptions. Only 0.4% of tools have no description. Authors describe the tool and forget the arguments — which is understandable, because the tool is the thing you're thinking about when you write it, and the arguments are the thing the model has to fill in.

2. One tool in six has nothing to tell it apart from its neighbour

For every tool, we took the content words in its description and asked how many appear in no other tool's description on the same server. Call it the tool's distinctive share.

The median tool's description is 26% distinctive. Three-quarters of the words it spends are words its siblings also use.
26.7% of tools are under 10% distinctive.
17.4% are exactly zero. 23.1% of servers have at least one.

Zero doesn't mean the description is bad. Here are four from one widely-installed Gmail server:

Gmail_DeleteDraftEmail   "Delete a draft email using the Gmail API."
Gmail_SendDraftEmail     "Send a draft email using the Gmail API."
Gmail_ListLabels         "List all the labels in the user's mailbox."
Gmail_SearchThreads      "Search for threads in the user's mailbox."

Every one of those is clear, correct English. Every one is also built entirely from words the other tools use — delete, draft, email, gmail, api, list, search, threads, mailbox all recur across the set. The description tells you what the tool does. It doesn't tell you what this tool does and the others don't, and that second thing is the one a model needs when it's choosing.

The pattern that produces the most extreme cases is shared boilerplate. One server appends the same 51-word context block to all 275 of its tools. "Create a new NFT collection" and "Purchase an NFT from a listing" differ by 8 words out of 59. That block was added deliberately, to help.

3. It gets worse the more tools you ship

Tools on the server	Servers	No description	Zero-distinctive tools
1–3	1,256	14.8%	0.5%
4–7	1,292	22.4%	1.6%
8–15	1,044	21.9%	4.3%
16–30	767	24.8%	7.7%
31–60	377	22.2%	16.3%
61+	215	20.5%	31.3%

Description collision rises monotonically and by a factor of sixty. Some of that is arithmetic — more tools means more chances for two to collide — but it's also the point at which authors start generating descriptions from a template, and a template is a machine for producing tools that read alike.

And notice the column that doesn't move. Missing parameter descriptions sit between 20% and 25% at every size above the smallest bucket. It's not a scale problem; it's a habit. The two failures are independent, which means shipping fewer tools won't fix your undescribed parameters and writing better descriptions won't fix your collisions.

4. The median tool list costs about 1,250 tokens before anyone asks a question

Tool schemas are sent on every connection, whether or not a single tool gets called.

Median server: 4,991 bytes, roughly 1,250 tokens
90th percentile: 32,636 bytes, roughly 8,200 tokens
Largest in the corpus: 1,145,575 bytes — on the order of 280,000 tokens of schema, from one server, before the conversation starts

Median tools per server is 7 and the mean is 17.6. One server publishes 2,530 tools.

5. Some parameters name their valid values and then don't enforce them

8.0% of all parameters carry an enum. What you can find from outside is the case where the author wrote the values down in prose and left the schema as an open string:

outcome_attribution   "Attribution type for the outcomes.
                       Valid values: "direct", "influenced",
                       "unattributed", "total"."

commitment            "Optional processed|confirmed|finalized commitment"

895 parameters, on 4.6% of servers. That's a floor rather than an estimate: it only catches authors who documented the set.

Worth reporting how we got there. Our first version of this measurement said 108 hits in a 40-server sample. It was matching text like Filter by line (e.g. "1", "A", "F"), which is an illustration, not a closed set. Requiring explicit closed-set language took that sample from 108 to 9, and all nine were real.

One signal we expected to find and didn't: duplicate tool names within a server, on 0.2% of servers. Effectively nobody does this. If it's on your review checklist, take it off.

What we couldn't see

This is static analysis, and we never observed a single real request to any of these servers.

That means we missed everything that only shows up under traffic. The tool that works in isolation but gets called in the wrong order. The parameter that's fine until someone phrases a request unusually. The retry loop that only triggers on a specific error path.

We also can't tell you the thing you most want to know, which is how much of this matters. A zero-distinctive description might cost nothing when the tool name is unambiguous, and a great deal when it isn't.

One more hole, found by a server author after publication: the 21.8% measures absence only. A parameter described as "query: The query" counts as described and passes. Restating the parameter name is the more common failure in his experience, and it passes every linter. So that number is a floor too.

If you maintain a server

Three things, ordered by how common the problem is in the data. None of them changes your server's behaviour — they're all changes to the text a model reads.

Describe every parameter. The most common gap by a distance, and it doesn't get better at any size. If you do one thing, do this one.

Make each description say what the others don't. Not "is it clear" — is it clear which of my tools this is. Read your tool list as one block and ask what a reader with only that block would use to choose.

If you have more than about thirty tools, audit for collisions specifically. Above that size, roughly one tool in six has no distinguishing word, rising to one in three past sixty.

The data

Analysis scripts and raw data: https://github.com/getmcpulse/mcp-schema-study

Every figure is in analysis.json, with the servers and tools behind each one in examples.json. The collector rebuilds the whole corpus from a public API in about twenty minutes.

There's also a browser-based checker that runs these measurements on your own tools/list — paste the JSON, get your numbers scored against the corpus: https://getmcpulse.com/check

We ran this because we're building MCPulse, an SDK that reports what actually happens under real traffic. Everything in this post came from outside the server, which is exactly its limit. A schema can tell you a model has nothing to choose on. Only traffic can tell you whether it chose wrong.

Originally published at getmcpulse.com.