Say a tool's first-call success is 62%. What do you change?
You can't tell. 62% might be a tool every model finds mildly confusing, in which case the description needs work. Or it might be a tool one client reads perfectly and another reads badly, which is a completely different problem with a completely different fix.
The number is identical either way.
An MCP server has several readers, not one
Your server isn't called by "the model". It's called by whatever clients your users connect from — Claude Desktop, Cursor, Codex, and frequently something you didn't expect — each putting a different model, a different system prompt and a different tool-selection strategy in front of the same descriptions.
Those readers disagree with each other. A description that's unambiguous to one can be read two ways by another. A required parameter one client fills from context, another has to guess at.
And all of it lands in the same column of your metrics, averaged together.
What the average costs you
Here's the shape, from a deliberately-constructed sample server — treat the numbers as an illustration rather than a measurement:
| Client | First-call success on one tool |
|---|---|
claude-desktop |
76% |
cursor |
21% |
| All clients | 62% |
Same tool. Same description. Same day.
Three readings of that:
From the server-wide figure, the tool is a bit weak. You'd reword the description in general terms and hope.
From the split, the tool is fine in one client and badly broken in another. The handler isn't the problem. The description isn't generally the problem either — something specific in it is being read differently by one reader.
And the larger client hides the smaller. The good client's traffic pays for the bad one's, so the average looks survivable while one group of your users is having a terrible time.
The gap between those two rows is the finding. A single number cannot contain it.
62% is also the worst place for a metric to sit: too high to look urgent, too low to look healthy. You glance at it, decide it's not today's problem, and move on. Which it isn't — unless you're one of the Cursor users.
It's not just first-call success
That's the sharpest case, but nothing is immune:
- Bad arguments concentrated in one client point at a parameter that client's model fills differently.
- Empty answers in one client usually mean it's passing narrower filters.
- Latency can differ by client, when one sends larger requests or calls tools in a different order.
- Dead tools are sometimes dead in one client only, because its model always reaches for a sibling instead.
Where the client name comes from
You don't have to guess it. Every MCP connection opens with an initialize request, and the client names itself in clientInfo.name. Read it from there, attach it to every call in the session.
The cost of keeping it is a column — but which column matters, and this is worth getting right the first time.
Counters stored one row per hour, per tool, per client let you ask "how is search_orders doing in Cursor only". Tool counts and client counts in two separate tables cannot answer that question at all — you can filter by tool, or filter by client, and never both. That was a real fix on our side rather than a hypothetical, and retrofitting it meant a migration.
If you're building this yourself, key by both from the start.
What to do when you find a gap
When one client is far below the others on one tool:
1. Read the description as that client would. Look for the word with two meanings, the parameter whose format is implied rather than stated, the constraint that lives only in the schema and never in the prose.
2. Check bad arguments for the same tool-and-client pair. If that's high, the model isn't failing to choose your tool — it's failing to fill your schema. Different fix.
3. Fix for the struggling client without breaking the good one. Watch both rows after the change. A rewrite that lifts one client from 21% to 60% and drops the other from 76% to 50% is not progress, and a server-wide average would have told you it was.
That last point is the whole argument in miniature. Without the split you can't even evaluate your own fix.
What it can't tell you
The client name is what the client says it is. It identifies software, not a model version — and a client can change its model without changing its name.
So when a client's numbers move overnight and your server didn't change, that's the first thing to suspect. It's also unfalsifiable from your side, which is worth knowing before you spend a day on it.
It also can't tell you why one model reads your description differently. Only that one does, on which tool, and by how much.
Which is still the part you had no way of seeing before.
I build MCPulse — two lines inside your server, never your arguments or your results. There's a live demo with the client filter on every panel, no signup.
Originally published at getmcpulse.com.
Top comments (4)
The rewrite in step 3 is a case the server-wide figure catches rather than hides. If the two rows are the whole of that tool's first calls, 62 percent puts about three quarters of them in claude-desktop, and after the rewrite the average falls from 62 to about 52.5 percent. Under any mix of the two clients it lands somewhere between 50 and 60, so the average reports that change as a regression, not as progress.
What it hides is the opposite move. At that mix one point gained in claude-desktop covers almost three points lost in Cursor. A rewrite that takes claude-desktop from 76 to 78 percent and Cursor from 21 to 16 moves the average from 62.0 to 62.2, a small improvement in the server-wide figure while Cursor loses close to a quarter of its successful first calls. That is the larger client hiding the smaller one, and it is the release where the per-client rows have to decide.
The mix also moves the average on its own. If Cursor's share of first calls on that tool goes from about a quarter to half while neither client's rate changes, the average drops from 62 to 48.5 percent. A before and after comparison across a week in which a client's share moved reports a change that no description caused, which is one more reason to compare the rows rather than the total.
The example where a rewrite lifts one client from 21% to 60% while dropping another from 76% to 50% is the strongest argument here, since a single average would have called that change progress. It may also help to keep a small fixed set of test prompts for each client, so a description change can be checked against every client before release, especially given that the client name identifies software and not a model version, as you note.
if you break the avg down per client, what strategy do you use to keep the extra telemetry cheap?
Client is a cheap dimension to add, for one reason: it's low cardinality. A handful of names, not one per user. That's what makes it different from splitting by session or by caller identity, which genuinely would be expensive.
Concretely, counters rather than events. One row per hour, per tool, per client, and increment it. Worst case per day is 24 x tools x clients, and it's sparse in practice because most tools aren't called by most clients every hour. Twenty tools and four clients is under 2,000 rows a day. The same traffic as raw events is 24,000 rows for 24,000 calls, and you'd be aggregating at read time forever.
Two things that will blow it up if you're not careful, both learned the hard way:
Normalise the name. clientInfo.name sometimes arrives with a version attached, and "claude-desktop/0.7.1" versus "claude-desktop/0.7.2" turns a dimension with four values into one with hundreds. Strip the version into its own column if you want it, or drop it.
Cap the distinct set. The client name is a string the client chooses — you don't control it, and nothing stops something sending a unique value per connection. Keep the top N by volume and bucket everything else as "other". Without that you have an unbounded dimension fed by an untrusted field, which is a cardinality explosion waiting for someone to trigger it, deliberately or not.
The expensive version is keeping raw per-call rows so you can slice by anything later. Worth deciding up front that you won't, because the dimensions that matter here are known in advance.