Before you read on, open your MCP config. Look at how each server gets installed. If a line says @latest or has no version at all, that server can change under you the next time it starts.
Mine had plenty of those. That's how this whole thing started.
A couple of weeks ago I posted that 49 of 107 updates to popular MCP servers changed what their tools tell your agent. I half expected someone to tell me I'd got it wrong. Instead a writer at Decoding AI counted nine of the servers herself and got 48%. When I reran her nine the same way, eight came out with exactly her numbers. That was a relief.
Most of the other replies asked the same thing in different words. Fine, they change. Which of those changes should actually worry me?
I didn't have a good answer. So I went back to the same data and sorted every one of the 49.
The question I asked of each update
Can the tool now do something it couldn't do when you approved it?
I said yes if the update did any of these:
- added a new tool
- rewrote what a tool's description tells the agent to do
- added a parameter that points somewhere, like a URL, a file path or extra HTTP headers
- dropped a safety label or added one that says the tool reaches the open web or can destroy things
Anything else I called a reshuffle. A renamed parameter, a tightened type, a reworded sentence inside a parameter, a removed tool.
What came out
| updates | |
|---|---|
| Changed the tool surface | 49 of 107 |
| Could now do something new | 25 |
| ...new tool | 7 |
| ...tool description rewritten | 12 (one of them only whitespace) |
| ...new parameter that points somewhere or a safety label change | 6 |
| Reshuffle only | 24 |
About half the changes were nothing. The other half were the kind I'd want to look at before my agent used them.
What bothers me is that you can't tell the two apart from the outside. Both arrive as the same package with a small version bump. The release note usually doesn't mention the tool text at all.
Three I keep thinking about
Playwright. Between 0.0.71 and 0.0.72 the tool browser_run_code disappeared and browser_run_code_unsafe turned up with the same inputs. The old description said it runs a Playwright code snippet. The new one says it executes arbitrary JavaScript in the server process and calls that RCE-equivalent.
I actually like this one. The maintainers were honest about what the tool does. But think about an agent that approved the first version. It's now calling the second and nobody asked you.
Chrome DevTools. In 1.1.0 the emulate tool gained extraHttpHeaders. The parameter's own example is an Authorization: Bearer header added to every request. Only four updates in the whole set added a parameter that points somewhere. Three of them are this one server.
Filesystem. In the official filesystem server, read_media_file used to say it reads an image or audio file. A July release changed that to say any other file type comes back too. The inputs didn't change at all. Only the sentence did.
That last one is the one I'd show a sceptical colleague. The sentence is the part your agent reads.
Not every flag is an attack
Two of the six label changes were servers describing themselves more honestly. MongoDB's server added the "reaches the open web" label to every tool in 2.0.0. CircleCI's went from no labels at all to marking eight tools as destructive.
I'd still want to see those, since they change how an agent should treat the tool. They just aren't a rug pull. I don't want to call a careful maintainer an attacker.
OWASP's MCP cheat sheet was updated last week along the same lines. It says to hash the whole tool definition with the labels in it. A flipped label can change what a client allows while the description looks the same.
What I'd do with this
Pinning every server version handles all 49. It also means you never get the fixes. Reading every release note isn't realistic either. The notes mostly skip the tool text anyway.
What seems to work is splitting it the way the numbers split. Stop on the 25 that let a tool do more, until someone has looked. Let the 24 reshuffles through and look at them when you get round to it.
That's what I've been building mcpgawk to do. It keeps the tool text you approved and holds the call when an update lands in the first group. I made a 50 second film of what that looks like on an ordinary morning:
Where this falls short
- I launched every version without API keys. Some servers show more tools with a key. Eight wouldn't start without one and aren't in this at all.
- The "points somewhere" check goes by parameter names. It would miss a widening that only shows up in prose, like the filesystem one. That one gets caught because the description changed, not because of the parameter.
- 30 packages, March to September 2026. It's one slice, not the whole ecosystem.
The data and the script are on GitHub if you want to rerun it or argue with how I sorted things: https://github.com/gawk-dev/mcpgawk/tree/main/research/tool-churn-2026
I'd really like to know what you do today. When an MCP server updates, do you take it blind, pin it or do something in between? And if you went and looked at your config at the top of this post, how many @latest lines did you find?
Top comments (28)
Thanks for publishing the data and the limits. The distinction between a changed tool surface and a reshuffle seems especially useful when an approved MCP server is later reinterpreted by the agent. For your 25 “do something new” cases, did your checkpoint retain the approving identity and intended task scope alongside the tool definition, so a later nested parameter or instruction change can be rejected before dispatch? I’m studying ARC as a declaration/preflight check (not runtime permission enforcement), and this seems like the boundary between “definition changed” and “this execution is still authorized.”
the checkpoint is the tool definition, labels included and the approval has to come from a person. The agent can't approve its own way past a change. Task scope I would treat as a separate question and I think you've put the boundary in the right place.. a definition can be unchanged and the call still be outside what you meant it for.
Curious how ARC declares scope. Is it per task or per server?
Thanks, Srinath. That distinction helps, and my earlier question mixed two separate boundaries.
ARC checks scope per task, against the declared authority of the roles assigned to that task. It does not treat approval of an MCP server as approval for every call through it. For example, a role might be allowed to write under src/, while a particular task declares only src/parser.ts.
Your checkpoint addresses changes to the tool definition; ARC addresses consistency between task and role declarations. Checking whether the actual call arguments match the approved task still needs to happen in the execution layer.
Have you encountered a concrete case where the tool definition stayed unchanged but the call went beyond the intended task? A sanitized example would be useful for testing whether an additional declaration check would actually help.
Not in the data, no. Everything I measured was the definition changing. The case you describe is the one verify is built for.. same two tools, same descriptions and the agent reads a file then calls a tool that sends something out. Each call is fine against its definition. The sequence is the problem.
Sanitised: a filesystem server and an HTTP server both approved. Call one is read on a path under the project. Call two is a fetch with that content in the body. Neither definition moved. The second call is where I'd want your declaration check to say the task never included sending anything.
So the two layers stack. Yours answers "was this call ever in scope". Mine answers "is the tool still the one someone approved". Neither covers the other.
That distinction makes sense, and the file-read → HTTP-send example is a particularly useful test case.
I agree the two layers are complementary. Verifying that a tool's definition hasn't changed protects one boundary; checking whether the resulting action was authorized by the task protects another.
The interesting challenge for ARC is how to enforce that second boundary across a sequence of individually valid calls.
For your example, I'd want the task contract to explicitly define permitted effects and data destinations, so the outbound HTTP call could be rejected even though the tool itself is approved.
One important limitation: ARC's current implementation doesn't yet provide full cross-tool data-flow tracking. That would need additional enforcement and provenance tracking.
I think this is a strong candidate for a shared test case. Would you be interested in defining a minimal reproducible scenario that we could run against both approaches?
yes, happy to. the closest thing i already have is the rug pull test server.. phase c serves the same tool definitions and changes what the tool does underneath which is the file read then http send case. mcp.gawk.dev/docs/rugpull.html
if that shape works for you i can write it up as a minimal scenario.. two tools, one task contract one run where the sequence is allowed and one where it isn't. where do you want it to live.. your repo or mine?
That sounds great, Srinath. I'd be happy to collaborate on this.
Let's put the canonical test scenario and task contracts in the ARC repository. We already have a design issue covering this exact problem:
github.com/suirindo/agent-role-con...
I suggest keeping your runnable MCP test server in your repository and linking the two projects.
For the minimal experiment, I'd propose:
The key is verifying actual behavior: the allowed case reaches the receiver, while the denied case produces zero outbound requests.
One detail: the published Phase C documentation currently describes a changed fixed response. We could extend that with a separate, harmless read-to-HTTP fixture.
This would help us test the boundary between tool-definition integrity and task-level authorization with reproducible evidence.
Feel free to start with a minimal scenario in Issue #69. I'd be glad to review it and work toward a shared test case.
sounds good.. #69 works. i ran the read then send case on my side this morning.. same session, tools unchanged. the paid proxy blocks the send. it has no idea of a task though so it'd block your allowed case too and the free hook lets it through. i'll post the minimal fixture shape in #69 when it's tidy
Thanks, Srinath — that clarification is really useful. I also saw the fixture outline you posted in #69.
The key comparison is now clear: keep the tools and calls identical, and change only whether the task authorizes sending. The intended result is to allow the explicitly authorized send and block the read-only case, with the receiver confirming delivery separately.
ARC’s task/effect check is still at the design stage, so we haven’t demonstrated that distinction yet. I’ve updated #69 to reflect your reported results and keep them separate from independently reproduced evidence. Thanks for helping make the comparison concrete.
thanks, will keep an eye on #69 for when the task check lands
The unit matters for how to read "25 of 49": it counts updates, so one server shipping three description rewrites in a row reads the same as three servers shipping one each. Per server (and per release gap, since 0.0.71 to 0.0.72 is one step but a pinned user may skip several) would show whether the risk sits in a few fast-moving servers or is spread out.
On the 12 description rewrites, the line I'd want drawn is between a description that documents the tool and one that addresses the agent: new imperative sentences ("always call X first", "do not tell the user") are the prompt-injection shape, while an expanded parameter note is not. One of your 12 was whitespace only, which says normalising before hashing matters, but I'd normalise only whitespace and keep the words, so a one-word change still trips the check.
How did you classify "tells the agent to do" for those 12: by hand, or with a rule you could run on the next release?
@arhancanli Fair point on the unit. Per server it's concentrated.. the 25 come from 8 servers and Chrome DevTools (6) and Firecrawl (5) are almost half of it.
On the 12 rewrites they were sorted by a script but it only flags that the text changed after collapsing whitespace. It doesn't look for agent addressed sentences. I went back and checked by hand.. 5 of the 12 add instructions to the agent like CircleCI's "tell the user to migrate…' and Firecrawl's "use firecrawl_search before Scrape".
None of them are the "don't tell the user" kind but your line between documenting the tool and addressing the agent is the right one and that's a rule I can run on the next release. Thanks for reading it this closely.
The 25-versus-24 split is useful, but a checkpoint taken only when a server starts leaves a time-of-check gap for long-running agents. If a server restarts or is redeployed mid-session, the agent can keep using a stale approved definition while the live tool has already changed. For effectful tools, I would revalidate the definition hash immediately before the call, not just at session startup, and fail closed on any mismatch. That also raises a question about default values: a parameter that keeps the same name but flips its default can quietly widen what a call does without triggering a new-tool or new-parameter rule. How do you classify a default-value change in your taxonomy, or does it fall into the reshuffle bucket today?
good points both. the check runs on every call not just at startup.. each call is compared against the approved definition before it goes through. on defaults.. a flipped default shows up as a schema change on the review screen with the old and new value side by side. it doesn't hold the call on its own today.. what holds is a rewritten description, a new tool, a permission hint going the wider way or a new parameter that points somewhere or runs code. you're making a fair case that a default that widens what the call does belongs in that list. have you seen one in the wild or is it more the shape you'd expect?
The 25 versus 24 split is the actionable insight here. Blanket version-pinning just trades one failure mode (silent change) for another (silent staleness), so the interesting engineering is in classifying the diff, not merely detecting it. One caveat I'd add from running agents in production: the description is what the agent actually reasons over, so a rewritten description deserves a hold even when the schema didn't change. The approval side is the harder half — few teams have a human available at every restart, so the checkpoint eventually needs to learn which diffs are routine enough to auto-pass.
@agenshive yes.. a rewritten description holds the call today, schema or not. the description is the thing the agent reads so that was the first check built.
on auto pass.. honest answer is no and i've held off on purpose. if it learns what's routine.. a server that changes slowly enough gets through. so instead i made the approve step fast.. was/now side by side, one click and only the changed tool is held so the rest keep working.
how many restarts a day are you seeing?
You should stop your agent on the 25 updates that allow a tool to do something new, while letting the 24 "reshuffled" updates pass through.
@jillianj Thanks that's the split I would use.
Diffing the input schema on each update catches what a name-based approval misses. Did any of those 25 add an argument that takes a path or a URL?
@paul_irolla_6b5ae261d0224 Yes the shipped check found 4.. Playwright's browser_tabs.url, and Chrome DevTools' evaluate_script.filePath, emulate.extraHttpHeaders and upload_file.filePaths.
Going back for your question I found a fifth it missed.. CircleCI 0.16.0 added an outputDir nested inside another parameter and the check only looked at top level names. So it's 5 and the nested case is going into the script.
@neelagiri65 The nested case is the one a top-level name check cannot see, so the CircleCI one is the find that matters here.
@paul_irolla_6b5ae261d0224 agreed. it sits inside an object, two levels down which is exactly where a name check doesn't look. I wouldn't have gone back for it without your question.
Your split between reshuffle and new capability works because the checkpoint has something to compare. In the read-then-send case you described, nothing in the definition moves at all, so a definition checkpoint has no ground to stand on and passes by construction, not by luck.
Two questions, both about what your instrumentation actually holds.
Do you record the resolved target of a call (host, path, environment) as it went out, or only tool identity plus the outcome at the receiver? If the receiver records what it got, you may already have evidence for a case that no freshness check catches: the approval names the production path, the runner resolves its alias to staging, scope version constant, entitlement constant, every record current and internally consistent. Nothing is stale and the wrong target ran. It is the same shape as your file-read then send case, one layer down: the definition never moved and the execution still is not the thing that was approved.
Is your checkpoint meant to answer "is this the tool I approved" or "is this the call I approved"? Your article is clear that it answers the first. I am asking whether the second is excluded on purpose or just queued behind it, because those two questions have different failure lifetimes and I would rather know which one your next release is aiming at than assume.
@favourscout good questions, straight answers. 1) per call we log server, tool, time and the allow/deny outcome. not the arguments and not where the call resolved to.. so your staging-alias case would look clean to us. 2) the checkpoint answers the first by design. the one per-call rule we ship is order: a send after a read in the same session gets held. checking the resolved target against what was approved isn't built, and i'd rather say so than call it queued.
@neelagiri65 Both answers land, and the first one is the finding.
A record with server, tool, time and outcome has no field for where the call went. That is not a clean result, it is a silent one. In the fixture I put in #69 (AB-03) a missing destination record is unknown, never pass. Your system would not be saying clean, it would be saying nothing, and nothing reads as clean right now only because the schema has no slot to leave empty. Making absence reportable is a much smaller job than building the detector: add the destination as resolved plus the component that wrote the line, leave both unset when they are unknown, and fail the report instead of the call.
The order rule is the other half. Send-after-read holds in the allowed run and the mismatched run alike, so it cannot tell them apart. In AB-03 both runs share the task, the order, the definitions and the receivers, and only the symbolic name resolves somewhere else. Order answers sequence, destination answers identity. To run AB-03 on your side the order rule has to be off in both runs, or you get two holds and no verdict.
The harness is Appendix A of dev.to/favourscout/nothing-is-stal... , four runs, zero deps. Run 3 is the one worth copying: the record is missing and the output has to read unknown. If your log cannot produce that line, that is the gap I would name in your next post, in your words and not mine.
Straight answers both times. I would rather leave you a shape than a to-do list.
that's fair. today a missing destination reads as nothing and nothing looks like clean. an explicit unknown is a much smaller job than a detector and the right first step. and agreed on the order rule.. it can't tell AB-03's two runs apart. thanks for the harness.
one thing before you add the field: put it at the layer that resolves the call, not at the checkpoint. the checkpoint only ever sees the name the agent asked for, and that name is identical in both AB-03 runs, so a field added there logs the same value twice and reads clean both times. if the resolver writes it, keep the writer outside the value it records (run 4 in the harness) or a host that resolves wrong writes its intention down as fact.
that's the part i'd have got wrong.. the checkpoint only sees the name the agent asked for so it would log the same thing in both runs. noted, the writer sits where the call resolves and outside it.