Thank you, this is a generous answer. After reading it I checked my own listing: the registry copy and the live worklore server both say 0.1.0, so a version diff passes cleanly. But the server has changed quite a bit since it was listed: tools/list now answers without a token, and the auth rules moved into its instructions. I never bumped the version. So the drift can hide one level deeper: the version matches, while what the server actually does has moved on. Your scheduled diff is going on my list; I'd just compare more than the version.
Data scientist building open-source tools (MCP servers, R packages) for public data — Brazilian government APIs (DataSUS, IBGE, BCB, INEP) and global medical terminologies (WHO, NLM).
That's a sharper version of the drift, and I think you're right that the version is the weakest thing to diff on.
I checked what each copy can actually be compared on. The LobeHub manifest carries the whole surface, so there my test compares tools/list, resources/list and prompts/list field by field, descriptions and schemas included, not the version. The official MCP Registry's server.json carries no surface at all: name, version, packages, remotes. Against that copy the version is the only thing there is to diff, which turns your case around: the check is only as good as the habit of bumping the version whenever the surface moves. Nothing in my pipeline enforces that either; the version goes up because I run the release step, not because a test noticed the surface changed.
Your example also finds a gap in mine: both of your changes live outside tools/list. Instructions come back in initialize, which my manifest test doesn't read, and "answers without a token" is behaviour rather than declaration, so no listing would show it. The cheap fix I can see is to hash the initialize result together with the three lists, commit the hash, and fail CI when it changes without a version bump. That would turn "I never bumped the version" into a red build instead of a silent drift.
I built it. The test hashes the initialize result together with tools/list, resources/list and prompts/list, plus which methods answer without a token. It commits the hash next to the version, and the deploy now refuses to run while it's red.
Then I recorded the surface exactly as the server was when I listed it on 09-15 and ran it against today's code. It failed, and on something worse than I'd described: the registry said 0.1.0 for a server with four read-only tools, and the server behind that number now has six, including two that write on the user's behalf. Bumped to 0.2.0, deployed, and the registry shows it now.
Thank you, that's one comment thread that turned into a real fix.
Data scientist building open-source tools (MCP servers, R packages) for public data — Brazilian government APIs (DataSUS, IBGE, BCB, INEP) and global medical terminologies (WHO, NLM).
That replay is the part I'd have skipped, and it's the part that found the real problem. A version diff would have passed forever: same number, but four read-only tools had quietly become six, two of them writing on the user's behalf. That's the case a registry listing exists to rule out.
Both of your additions are going into my own version, which isn't built yet. Hashing which methods answer without a token covers the gap I said no listing could reach. And making the deploy refuse to run, instead of only failing CI, is the right place for the gate. I'll also borrow the replay: record the surface each published version had and run it against today's server, so the first run checks the whole history rather than just the next release.
Data scientist building open-source tools (MCP servers, R packages) for public data — Brazilian government APIs (DataSUS, IBGE, BCB, INEP) and global medical terminologies (WHO, NLM).
Follow-up, since I said I'd build it: it's live now on all seven of my servers.
Same shape as yours. A lock file next to the version holds a hash of the initialize result (instructions and capabilities included) plus tools/list, resources/list and prompts/list. It also records which methods answer without a token, on every MCP route, with the API key unset and set. If the surface changes and the version doesn't, the test goes red. The script that rewrites the lock refuses to do it under the old version, and the deploy won't run. I added one step after the deploy: it queries the live endpoint and checks it against the lock. Green tests prove the code; that step proves what's actually being served.
The replay covered 142 published versions. In all seven, the live server matches the version it reports. It also found two things I didn't know: two old releases of one server that don't even start (a broken bundle nobody had noticed), and a minor release that removed six tools when they moved behind a licensing flag.
One thing your version may not have hit yet: the probe has to be checked before it's written, not after. In one server the backend sits behind an edge proxy, and my first test double answered result to every method, so it locked "prompts/list answers" for a server that has no prompts. The live check caught it. The sanity checks now run before anything is written.
It's packaged as @sbissoli/mcp-surface if it's useful to anyone. Thanks again, this thread changed how all seven ship.
Thank you for coming back with this, and for the package. Seven servers and 142 versions is a far better test of the idea than my one server was.
Your point about checking the probe before writing the lock landed. My lock is written from a test double too, so I checked it against the live server just now: the methods that answer without a token match what's locked (initialize, tools/list and ping answer; resources/list and prompts/list return 401). It matches today, but only because nothing has drifted yet — I have no step that would notice if it did. Your post-deploy check against the live endpoint is the piece I'm missing, and I'm adding it.
The two releases that don't start are the finding I keep thinking about. A version diff would have called them fine forever.
I also wrote up the whole MCP-server experience on Habr (in Russian), and this idea is in it with your name on it: habr.com/ru/articles/1089802/
Thanks again. This thread changed how mine ships too.
Data scientist building open-source tools (MCP servers, R packages) for public data — Brazilian government APIs (DataSUS, IBGE, BCB, INEP) and global medical terminologies (WHO, NLM).
Thank you for the Habr write-up, and for putting my name next to the idea. That's very kind, and I'm glad it travelled further than this thread.
Your check against the live server is the same situation I was in before the post-deploy step: everything matched, but only because nothing had drifted yet. Nothing was standing guard. Once that step is in, "it matches today" becomes something the pipeline re-proves on every deploy, instead of something you happened to check once.
The two releases that don't start still bother me too. Nobody would have installed them on purpose, but a registry would have offered them as valid versions forever.
Thanks for this exchange. Both our servers ship better because of it.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Thank you, this is a generous answer. After reading it I checked my own listing: the registry copy and the live worklore server both say 0.1.0, so a version diff passes cleanly. But the server has changed quite a bit since it was listed: tools/list now answers without a token, and the auth rules moved into its instructions. I never bumped the version. So the drift can hide one level deeper: the version matches, while what the server actually does has moved on. Your scheduled diff is going on my list; I'd just compare more than the version.
That's a sharper version of the drift, and I think you're right that the version is the weakest thing to diff on.
I checked what each copy can actually be compared on. The LobeHub manifest carries the whole surface, so there my test compares tools/list, resources/list and prompts/list field by field, descriptions and schemas included, not the version. The official MCP Registry's server.json carries no surface at all: name, version, packages, remotes. Against that copy the version is the only thing there is to diff, which turns your case around: the check is only as good as the habit of bumping the version whenever the surface moves. Nothing in my pipeline enforces that either; the version goes up because I run the release step, not because a test noticed the surface changed.
Your example also finds a gap in mine: both of your changes live outside tools/list. Instructions come back in initialize, which my manifest test doesn't read, and "answers without a token" is behaviour rather than declaration, so no listing would show it. The cheap fix I can see is to hash the initialize result together with the three lists, commit the hash, and fail CI when it changes without a version bump. That would turn "I never bumped the version" into a red build instead of a silent drift.
I built it. The test hashes the initialize result together with tools/list, resources/list and prompts/list, plus which methods answer without a token. It commits the hash next to the version, and the deploy now refuses to run while it's red.
Then I recorded the surface exactly as the server was when I listed it on 09-15 and ran it against today's code. It failed, and on something worse than I'd described: the registry said 0.1.0 for a server with four read-only tools, and the server behind that number now has six, including two that write on the user's behalf. Bumped to 0.2.0, deployed, and the registry shows it now.
Thank you, that's one comment thread that turned into a real fix.
That replay is the part I'd have skipped, and it's the part that found the real problem. A version diff would have passed forever: same number, but four read-only tools had quietly become six, two of them writing on the user's behalf. That's the case a registry listing exists to rule out.
Both of your additions are going into my own version, which isn't built yet. Hashing which methods answer without a token covers the gap I said no listing could reach. And making the deploy refuse to run, instead of only failing CI, is the right place for the gate. I'll also borrow the replay: record the surface each published version had and run it against today's server, so the first run checks the whole history rather than just the next release.
Follow-up, since I said I'd build it: it's live now on all seven of my servers.
Same shape as yours. A lock file next to the version holds a hash of the initialize result (instructions and capabilities included) plus tools/list, resources/list and prompts/list. It also records which methods answer without a token, on every MCP route, with the API key unset and set. If the surface changes and the version doesn't, the test goes red. The script that rewrites the lock refuses to do it under the old version, and the deploy won't run. I added one step after the deploy: it queries the live endpoint and checks it against the lock. Green tests prove the code; that step proves what's actually being served.
The replay covered 142 published versions. In all seven, the live server matches the version it reports. It also found two things I didn't know: two old releases of one server that don't even start (a broken bundle nobody had noticed), and a minor release that removed six tools when they moved behind a licensing flag.
One thing your version may not have hit yet: the probe has to be checked before it's written, not after. In one server the backend sits behind an edge proxy, and my first test double answered result to every method, so it locked "prompts/list answers" for a server that has no prompts. The live check caught it. The sanity checks now run before anything is written.
It's packaged as @sbissoli/mcp-surface if it's useful to anyone. Thanks again, this thread changed how all seven ship.
Thank you for coming back with this, and for the package. Seven servers and 142 versions is a far better test of the idea than my one server was.
Your point about checking the probe before writing the lock landed. My lock is written from a test double too, so I checked it against the live server just now: the methods that answer without a token match what's locked (initialize, tools/list and ping answer; resources/list and prompts/list return 401). It matches today, but only because nothing has drifted yet — I have no step that would notice if it did. Your post-deploy check against the live endpoint is the piece I'm missing, and I'm adding it.
The two releases that don't start are the finding I keep thinking about. A version diff would have called them fine forever.
I also wrote up the whole MCP-server experience on Habr (in Russian), and this idea is in it with your name on it: habr.com/ru/articles/1089802/
Thanks again. This thread changed how mine ships too.
Thank you for the Habr write-up, and for putting my name next to the idea. That's very kind, and I'm glad it travelled further than this thread.
Your check against the live server is the same situation I was in before the post-deploy step: everything matched, but only because nothing had drifted yet. Nothing was standing guard. Once that step is in, "it matches today" becomes something the pipeline re-proves on every deploy, instead of something you happened to check once.
The two releases that don't start still bother me too. Nobody would have installed them on purpose, but a registry would have offered them as valid versions forever.
Thanks for this exchange. Both our servers ship better because of it.