Between August 10 and September 3, the number of tools my MCP server answers without an API key went from two to three to four to five. Three public documents stated that number, tests guarded it, and those tests passed the whole time - including every day the documents were wrong.
This is about why that happens, why the obvious tests cannot catch it, and the small script that finally does.
The setup
I run a marketplace API for software agents, and it ships an MCP server so an agent inside Claude, Cursor or any other MCP client can use it. The server lives in its own repository, is published to npm, and releases on its own cadence.
The API repository describes that server in three places a machine reads: the A2A Agent Card, llms.txt, and llms-full.txt. The most load-bearing sentence in all three is how many of its tools work with no key and no account, because that is the one fact that lets a reader try it before deciding anything.
The obvious engineering move is to not write the number three times. So the count lives in one Go constant:
// MCPKeylessToolCount is how many tools the server answers with no API key
// and no account. Documents spell it as a word, and the Go test asserts the
// word matches this value.
MCPKeylessToolCount = 2
The documents built from code read it directly, and tests assert the prose in every one of them agrees - down to spelling the number as a word, so "Two of its tools" and the constant cannot disagree.
That test is correct. It is also nearly useless for the problem that actually happened.
A test that passes the moment you edit the constant
Look at what the guard compares: documents against a constant, both in the same repository. On August 12 the MCP server shipped a release with a third keyless tool, and nothing in this repository changed. The constant still said two, the documents still said two, they agreed perfectly, and the test passed. They stayed wrong together for a week.
And when someone does notice and edits the constant, the test passes again immediately, whether or not the new number is true. A check that compares your claims to your own record of the claims can only ever tell you that you are consistent. It cannot tell you that you are right, because the thing that defines "right" lives somewhere else - in this case in a package on npm, published from another repository, which nothing in the test run can see.
That was the real lesson for me: agreement is not accuracy, and a guard whose reference value is under your own control will happily guard a lie.
Ask the thing itself
The fix is to do exactly what a reader of those documents would do. Install the package by the name the documents give, start it the way they tell people to start it, and ask it what it can do.
MCP makes that cheap, because the server will describe itself over stdio:
const child = spawn('npx', ['-y', `${pkg}@latest`], { env, stdio: ['pipe', 'pipe', 'pipe'] })
send({ jsonrpc: '2.0', id: 1, method: 'initialize', params: { /* ... */ } })
// on the initialize result:
send({ jsonrpc: '2.0', method: 'notifications/initialized' })
send({ jsonrpc: '2.0', id: 2, method: 'tools/list' })
send({ jsonrpc: '2.0', id: 3, method: 'prompts/list' })
send({ jsonrpc: '2.0', id: 4, method: 'resources/list' })
The script does that twice - once with no key in the environment and once with an obviously fake one. What the server lists with no key is the keyless set; what it lists with the fake key is everything. Then it reads the claimed count straight out of the Go constant, rather than restating it, so the checker cannot drift from the documents it is checking.
A few details turned out to matter more than the happy path:
- Strip the environment first. Every project-prefixed variable is removed before the child starts. A developer with a staging URL exported in their shell would otherwise be checking a different deployment than the documents describe, and the check would quietly mean something else.
-
An older server's "method not found" is an answer. Prompts and resources are optional in MCP, so a version that predates them returns
-32601. Treating that as an empty list keeps the script honest against old releases instead of crashing. - Any non-JSON byte on stdout is a failure. MCP over stdio is a protocol stream, so a published build that logs a stray line to stdout breaks every client. The server has a unit test forbidding it; the drift check catches the regression in the build people actually install, which is where it matters.
- Classify every tool, and fail on strangers. The script keeps an allowlist marking each tool as keyless, keyed, or one that spends money. A new tool that ships unclassified fails the check. That turned a count check into a "do the documents still describe what exists" check, which is the question I actually cared about.
Three exit codes, not two
The script exits 0 when every claim holds, 1 when a claim is false, and 2 when it could not finish - npm unreachable, the package failed to start.
Collapsing 1 and 2 into "failed" is the easy mistake, and it kills checks like this. A check that downloads a package and talks to a subprocess will sometimes fail on a flaky network. If that failure looks identical to "your documents are lying", people learn to rerun it until it goes green, and then they learn to ignore it.
So the release pipeline treats them differently. Exit 1 stops a production deploy, because a false public claim is worth stopping a release over. Exit 2 prints a warning and proceeds, because nothing was proven either way.
The part that stung
The script was written on August 10, in the same commit as the constant. It was correct from the first day.
It ran nowhere. It was not in the test suite, not in CI, not in any release step. For nine days a correct check sat in the repository wired to nothing - and for the last seven of them, after a release on August 12 added a third keyless tool, the documents said "two" while the published server had three. On August 19 it went into the pull-request checks as a non-blocking signal; on August 23 it became a gate on production deploys.
A correct check that nothing runs is exactly as useful as no check. The work that mattered was not writing it - it was wiring it into the one place a failure could not be scrolled past.
The gate earned its place on September 3. A release shipped a fifth keyless tool, the documents still said four, and production deploys stopped until they were corrected. That is annoying, and it is also the entire point.
There is an ordering trap in there
Once the gate exists, the order of operations matters. Update the constant when the package publishes, not when the API deploys.
Bump it early and the check correctly reports that the documents claim a tool npm does not serve yet. Bump it late and every production deploy is blocked until someone catches up. Neither is a bug in the check; both are the check doing its job on a process that has two repositories releasing independently.
The copy you cannot edit
The same number did damage somewhere the gate cannot reach. I had drafted a guest article that described the server, and it said "two of its tools need no key." When I wrote it, the server already had three.
I corrected it to three. Two days later there were four. I corrected it to four, and on the day I checked, a release made it five.
That is survivable in a document you own and can redeploy. It is not survivable in copy printed on someone else's site, which you will never be able to edit again. So the final version names no number at all: it says several of the tools, including the discovery and reputation lookups, need no key. That sentence stays true at any count, as long as those two stay keyless - and that is a much narrower thing to check before submitting.
What I would keep
- Never let a guard compare your claims to your own record of them. Point it at the artifact that defines truth - the published package, the live endpoint - or it will guard a lie.
- Describe the thing by asking it. If a component can describe itself, the check should do what a user would do: install it by the documented name and ask.
- Separate "false" from "could not check". A check that cannot tell a flaky network from a wrong claim gets muted.
- Wire it into the step that cannot be scrolled past. A correct check nothing runs is no check.
- Do not bake a moving number into copy you cannot edit. Write the sentence so it survives the next release.
Drafted with AI assistance; every date, count and commit above was checked against the repository history before publishing.
Top comments (3)
@akashy, asking the published package is the right move, but
@latestalso makes the check’s evidence time-dependent. I’d resolve the npm version once, record the tarball integrity digest, then run both no-key and fake-key probes against that immutable artifact; the documentation update can be reviewed against the exact package it claims to describe. Does the deployment gate persist that resolved version and tool-list output, or can a retry silently test a newer release?The three-exit-codes point is the part most people skip and the part that decides whether a check survives contact with reality. A script that exits 1 for both "the docs lie" and "the network was down" gets muted inside a week, and then it guards nothing. Your rollout order matches what I've learned the hard way too: non-blocking signal for two weeks before it became a deploy gate — that window is what catches "the check itself is broken" before it can block a real release. "A correct check that nothing runs is exactly as useful as no check" should be on a poster. The allowlist that fails on unclassified new tools is the clever bit: it turns a count assertion into a "do the docs still describe what exists" assertion, which was the question all along. When the gate caught the fifth keyless tool on Sept 3 and stopped deploys — did correcting the constant ride the same PR pipeline, or was it a manual step someone had to remember to do before publishing?
This is a failure I keep meeting: the constant was true when written, the test tested the constant, and the docs inherited a lie with CI's blessing. Checking claims against the published package - not the source repo - is the step people skip, and it is the only version a reader ever meets. The agent-card / llms.txt drift problem will only grow as more agents read those files before a human ever does.