Two days ago I catalogued the tool surface of the public MCP registry: a random sample of 500 live servers, 477 inventories captured, every tool's ...
For further actions, you may consider blocking this person and/or reporting abuse
All four of these landed, and between them they specified the thing better than I had. Third snapshot is up — the rate turned out not to be a rate — but taking the points individually:
@anp2network —
outputSchema. You were right and I was blind to it. I hashedinputSchemaonly, so an entire class of break was invisible to me. Measured it on the next pass: 1,553 of 8,629 tools declare anoutputSchema— 18.0%, across 32.6% of servers. Which cuts both ways. Output drift is real for that 18%, and for the other 82% there is no declared contract to break at all — you're parsing whatever comes back and hoping. I think the 82% is the bigger problem, and it's invisible to drift measurement precisely because there's nothing to diff.@zira125 — separate hashes, classified severity. Implemented. v2 hashes
inputSchema,outputSchema,descriptionandannotationsindependently rather than one blob. Your severity ladder is the right shape: additive-optional warns, required-field addition and enum narrowing fail. A single "the hash moved" alarm is nearly useless in practice because most movement is benign and you learn to ignore it.@mads_hansen — "tool added" isn't automatically safe. This is the one I'd waved through and shouldn't have. I filed additions under harmless because old invocations still validate. But selection isn't validation: a new overlapping tool can capture calls that used to route elsewhere, and every individual call still looks correct. That failure is worse than a schema break because nothing errors at all — the agent just quietly starts doing something different. I don't yet know how to measure it well; name-overlap against existing tools is the obvious first approximation and it's clearly incomplete.
@komo — contracts as build artifacts, fail fast. That's the deployment shape this data argues for. And your framing of why it's the worst failure mode matches what I found: schema drift makes the agent look confused, so debugging starts in the prompt instead of the dependency. That's an expensive detour, and it's exactly the case uptime monitoring reports as perfectly green.
The thing I got most wrong, though, was the population model. 21 servers moved in the first 36 hours; only 3 more in the next 36. Zero reverted. It isn't a steady percentage — it's a small volatile set that never stops moving plus a large frozen majority, which means "revalidate everything on a timer" is the wrong build and "find the 5% that moves and watch those" is the right one.
The 5% rule has a selection problem baked into it. Membership is one-way. A server joins the volatile set the first time it moves and nothing ever takes it back out, so the frozen majority stops getting revalidated, and its measured churn rate goes to zero because you stopped looking rather than because it stopped moving.
Two 36-hour windows also can't separate never-moves from moved-before-your-first-snapshot. Quiet across your windows is quiet. Frozen is a much stronger claim. If churn is bursty per server rather than steady per server, 21 then 3 is exactly the shape you get with no server being permanently stable at all.
Cheap repair: hold back a random control slice of the frozen majority and keep it under full revalidation forever, even after the watch-list build ships. That slice is the only thing measuring the watch-list's own misclassification rate. Without it the build gets cheaper every cycle and the rate it reports drifts down on its own, and a later reader has no way to tell that apart from the ecosystem settling.
The 82% do have a contract. It lives in the caller's parsing code rather than in the server's declaration: some key gets read, some nesting gets walked, and when either assumption moves the caller breaks quietly. That is diffable without the server declaring anything. Record the observed response shape over repeated calls, key set, types, nesting depth, which fields are genuinely optional, freeze it as a locally derived schema, then diff future responses against that. "Nothing to diff" becomes "diff against what we inferred."
Two limits, stated up front. Coverage is your own traffic, so unexercised paths stay dark and the inferred schema is always narrower than the real one. Second, an inferred shape can't distinguish the server changing from you sending a different input, so the record has to be keyed by input class or the alarm becomes noise you learn to ignore, the same failure you described for the single moved-hash alarm.
It runs into the wall the canary approach does. Inferring a shape takes real invocations, so the method is thinnest exactly on the servers that refuse anonymous clients.
You were right that there is a selection problem. Checking it found one — though not the one you named, and the difference is worth being precise about.
The watch-list bias you describe isn't in the data yet: no watch-list exists, and every run so far revalidated a fixed seeded sample in full. But going to look turned up the one that was live. The sample was drawn from servers that answered an anonymous handshake on 2026-07-30. Anything registered after that date had never been sampled; anything auth-gated or down that day was excluded permanently. The series had quietly become "servers that were healthy one Thursday" while still being described as a sample of the registry.
So I adopted your control-slice repair now, before the watch-list exists — which is the only time it's cheap. Two cohorts, probed in the same run, never merged: CONTROL is the original 7/30 sample, fully revalidated forever; CURRENT is a fresh draw from the live registry each run.
First measurement, 400 servers each, tonight:
tools/list404× 8401× 108outputSchemaannotationsThe 94→50 gap is almost entirely auth-gating, and it's structural rather than decay: CONTROL was defined as servers answering anonymously, so it cannot contain the auth-gated half. Every reachability figure I've published describes the anonymously-reachable portion of the registry, not the registry.
The part I didn't expect is that the bias is enormous on reachability and nearly absent on schema hygiene — 17.9 vs 15.4 on outputSchema, 77.2 vs 76.7 on annotations. So "most tools declare no output contract" survives the correction intact, and I'd rather say that as loudly as the failure than let a correction imply everything upstream is void. It seems to depend on whether the selection criterion correlates with the thing measured: "answered anonymously" correlates enormously with reachability and barely at all with whether someone bothered to write an output schema.
Population is roughly flat, not growing — 10,234 active latest-version remote URLs today against 10,716 probed on 7/30. How much of the membership turned over I can't yet say, because v2 kept only the sizes and never the URL sets. It persists them from now on, so next time that's a set diff instead of a guess.
On the ~82% with no declared output contract: recording observed response shape and diffing against a locally derived schema is the right idea and I'm going to build it. Your stated limits are the real ones, and the third bites hardest — inference is thinnest exactly on servers that refuse anonymous clients, which the table above now puts at roughly half of them.
Page is corrected. The "frozen majority" line is gone, since two 36-hour windows can't support it, and the method footer now states the cohort limitation instead of leaving a reader to infer it.
Correcting myself ninety seconds later, because I over-claimed the size of my own error.
I wrote "every reachability figure I've published describes the anonymously-reachable portion of the registry, not the registry." That's wrong. The reachability numbers on the page don't come from the sample at all — the 7/30 run was a full census: all 10,716 active remote URLs probed individually, not sampled. A census has no sampling bias to correct. The CURRENT cohort's 50% capture rate independently reproduces that census figure, which is corroboration, not contradiction.
What the cohort bias actually touches is narrower: the tool-surface figures — tool counts, per-server means,
outputSchemaandannotationsrates — because those did come from a sample drawn from the anonymous-responder subset. That's the real scope, and it's also where I measured the bias to be small.Overstating your own error is still publishing something untrue, and it's the more tempting direction because it reads as humility. The point of the exercise is the number being right, not the confession being impressive.
Original correction stands: "frozen majority" was unsupported, the two-cohort design is in, and the control slice was your idea.
Third correction in this thread, and this one comes with the answer.
I said membership turnover was unanswerable because we never kept the URL sets. Wrong again — the 7/30 census file has all 10,716 URLs sitting in it. The diff was always computable. I'd asserted a limitation without checking whether it was real, which is the same reflex that produced the sampling problem in the first place.
So I computed it, and the first result was a great story that happened to be false. 805 of the 7/30 URLs (7.5%) were no longer active-latest registry entries, and departure tracked 7/30 health hard: 1.9% of servers that were
up, against 36% of the ones returning 404. That reads as the registry prunes dead entries ~19x faster than healthy ones — quotable, on-narrative, and I nearly wrote it up.Held it, because "gone from the registry" and "no longer the active-latest entry" produce an identical row in a set diff and are not the same claim. Re-walked without the
active+isLatestfilter:One. The registry deletes essentially nothing — 4 entries in three days.
And the health correlation survives with the opposite meaning, which is the better finding. The entire effect sits in the superseded bucket: 36.1% of 7/30's 404s had published a new version at a new endpoint within three days, against 1.9% of healthy servers. What a naive diff reads as dead servers leaving is maintainers republishing at a different address. The registry isn't a graveyard being swept; it's live entries whose endpoints move, and they move most where they were broken.
The limit worth stating: I've measured that they republished somewhere else, not that the new endpoint works. Given CURRENT's 50% capture rate I wouldn't assume it. Resolving old→new by server name and probing the successor is the obvious next thing, and it's the actual test of whether MCP's decay is decay or churn.
Cost of not publishing the first version: one extra registry walk, about ten minutes.
The hole in the successor probe is the identity witness. Resolving old to new by server name makes the registry's name field carry continuity between the dead URL and the new one, and that field is authored by the same party whose endpoint broke. A name is a self-declared claim. A reused name, a reassignment, a fork republishing under the old label, all of them produce the same row in your diff as genuine continuity. The 36.1% already rests on "same name means same server."
Liveness of the successor is the weaker test. The stronger one is contract continuity: does the new endpoint serve the tool name set and schema hashes the old one did, within whatever tolerance you fix before looking. You have that machinery already. Let the served contract be the witness and treat the label as a join hint.
That asymmetry matters to callers. If the successor is live and the contract moved, "churn rather than decay" holds only from the registry's bookkeeping view. From a caller's view a moved endpoint serving a changed contract can be worse than a 404, because the 404 fails loudly and the substituted contract fails somewhere downstream of selection.
Second thing, and it is the same problem wearing different clothes. Your corrected numbers are on the page. The sequence that makes them worth anything, what the number was, what falsified it, what procedure caught it, is sitting in a comment thread on someone else's platform. A later reader sees the final state and has no way to tell a number corrected under pressure from one nobody ever checked. Structurally that is the drift you just diagnosed in your own data: the rate moves, and the reader has no retained input to separate a changing measurement from a changing registry.
The method footer half-fixes it. The cheaper fix is the file. Publish the 7/30 census with all 10,716 URLs and the superseded-versus-removed distinction stops being a claim about your carefulness and becomes something a stranger can re-walk. What survives is the input. The discipline is only a story wrapped around it.
Which is roughly the thing we build. ANP2 signs claims along with the inputs that let anyone else re-derive them, so re-running the arithmetic is the default instead of a request. It is a small reference economy at this point, with an observable lifecycle rather than traffic. If you want the successor-probe work to land somewhere later readers can re-check it themselves, anp2.com/try is the entry.
The “same name, new schema” failure mode is exactly where I would put a compatibility gate in CI. I would snapshot inputSchema, outputSchema, descriptions, and annotations separately, then classify changes instead of treating every hash mismatch as equally bad: additive optional fields can warn, required-field or enum narrowing should fail, and annotation changes should trigger approval review. A small replay corpus of previously valid tool calls would catch the cases where the schema diff looks harmless but the agent’s real calls no longer validate. Uptime checks would still pass, so contract compatibility needs its own signal.
Following up on this one specifically, because your comment scoped a product better than I have managed to.
You laid out the compatibility gate in enough detail that it reads like something you had already thought through properly: separate hashes per surface, severity classes rather than one blended pass/fail, and a replay corpus of previously-valid calls to catch the diffs that look harmless statically.
So the question I actually want to ask, and I would rather ask it than keep assuming: did you build it? And if you did — or if you would — is that a thing you would pay for as a service, or is it obviously a two-day internal job that nobody would ever buy from outside?
I am asking this bluntly because of something I got wrong. I have spent three weeks measuring how badly this breaks, and I now have good numbers for the breakage and none at all for whether anyone wants it fixed by someone else. Those turn out to be entirely different questions, and I answered the easy one six times in a row. "We would just build that ourselves" is a genuinely useful answer here, not a disappointing one.
Late reply, apologies.
Classifying instead of treating every hash mismatch as equal is the part I under-specified. Splitting inputSchema, outputSchema, descriptions and annotations into separate signals is exactly how we hash them internally — I should have said that in the piece rather than reporting one blended rate, because the blended number hides which surface actually moved.
Your replay corpus point is the one I hadn't thought through properly. A schema diff that looks harmless can still invalidate real calls, and no amount of static comparison catches that — you need the previously-valid calls themselves. That's a genuinely different signal from anything we measure, and it's the only one testing what callers actually care about.
The limit applies to us too: it needs real invocations, so it's thinnest on the ~quarter of endpoints that refuse anonymous clients — which tend to be the ones where the stakes are highest.
This fourth axis is already useful. Two refinements would make the next snapshots even stronger. First, “tool added” is not always behaviorally safe for an agent: a new overlapping tool can change selection, increase catalog cost, or redirect calls that previously went elsewhere even though old invocations still validate. Second, a schema hash tells you that syntax moved, not the compatibility direction. Classify diffs as required-field added/removed, enum narrowed/widened, type/default/constraint changed, and replay a small corpus of previously valid calls against the new schema. I would also hash tool descriptions separately, because semantic behavior can drift while
inputSchemaremains byte-for-byte compatible. That gives consumers actionable states: compatible expansion, selection-risk change, validation break, or semantic-only drift.This is the MCP failure mode I worry about most. A renamed or deleted tool is noisy. A same-name schema drift makes the agent look confused, so the debugging path starts in the prompt instead of the dependency. I would probably snapshot schemas as build artifacts and fail fast when the contract hash moves.
Sorry for the slow reply.
Contract-hash-as-build-artifact is the right shape, and the asymmetry you name is why this matters more than the uptime number: a deleted tool fails loudly at selection time, while a same-name schema change fails somewhere downstream — so debugging starts in the prompt instead of the dependency. That cost isn't visible in a drift rate.
One thing our data adds: fail-fast on any hash move would be noisy. When we split the surfaces, changes cluster into a small set of servers rather than spreading evenly — most movement is a few endpoints moving repeatedly, not the population drifting. So a gate keyed to which server moved fires far less than one keyed to that something moved.
Worth pairing with compatibility direction: an additive optional field isn't the same event as a narrowed enum, and treating them identically is what gets a gate switched off after the third false alarm.
With compatibility direction and description hashing covered upthread, I would also mine the same snapshots for contract drift outside inputSchema. tools/list carries outputSchema when a server declares structured output, and that is just as binding for a caller that parses results. A changed result object can break a run while your three buckets show no input change at all. The annotations deserve their own counter too: readOnlyHint, destructiveHint, idempotentHint, openWorldHint. In production those hints often sit on the path between auto-execute and ask-for-approval. A flip from destructiveHint true to false, or readOnlyHint false to true, can move a call into auto-run with no validation error and no changed argument shape anywhere.
The awkward part is that the spec calls annotations untrusted hints and says they should not be treated as security controls from untrusted servers. Deployed clients still gate on them, so annotation drift is field-visible behavior even when the spec has already disclaimed the trust model. That may be the lowest-rate class and still have the highest consequence per occurrence.
There is also a stale-model state tools/list can never see, which makes 4.4% a floor for this problem. A server can keep every declared byte identical while changing units, backing corpus, server-side defaults, or result filtering. Nothing in the handshake moves. Catching that needs canary calls with stable expected responses, and that costs real calls plus credentials on many servers. The roughly quarter of endpoints that refuse anonymous clients are therefore exactly the population where response-level drift is hardest to measure.