The failure mode that worries me most in agent tooling is not a crash. It looks like success.
An operator, somewhere, asks their agent to put the summer collection on sale. The agent discovers the products, loops over the variants, fires four hundred write calls, collects four hundred 200s, and reports back, confident and polite: "Done! All prices are updated." And they are. On the wrong collection.
Nothing in that transcript failed. The transport worked. The writes persisted. The agent did not hallucinate a single call. Yet the one thing the operator actually wanted, "the summer collection, on sale, nothing else", is not what happened. If you have shipped distributed systems you know this smell: every component is green and the system is wrong.
I run engineering at GoodBarber, an app platform, and our MCP server exposes a live production surface: content, commerce, push notifications, real apps with real users. So this is not a thought experiment for me. And judging by the comments on my last piece and the threads landing on r/mcp lately, it is not a thought experiment for many of us anymore.
Three different things we call "success"
When an agent reports "done", that word is doing a lot of undeclared work. Unpack it and there are three separate claims stacked on top of each other:
- Transport success. The request arrived, the server answered, nothing threw. HTTP 200, no protocol error. This is the only one of the three the agent gets for free.
- State success. The write persisted: the article exists, the price changed, the push is scheduled. This is what a verified read-back proves.
- Intent success. What persisted is what the operator meant, within the policies of the account. The right collection. The right segment. The right amount.
Language models are spectacular at conflating the three. A 200 becomes "done"; "done" becomes "what you wanted". Not because the model lies, but because nothing in the loop ever represented intent explicitly, so the model reports the only success it can observe.
After my last piece, an engineer I traded notes with put the gap in one line I have not managed to improve on: given only transport success, the agent will "faithfully confirm the wrong thing." Confirmation is not correctness. A model that is perfectly honest about what it saw can still be wrong about what it did.
Where verified read-back stops
In my last piece I argued that an application MCP server should verify writes as a matter of server contract. On our server, every write result carries _mcp_policy.verification_required: true: the server itself instructs the agent to read the object back through the public read path and confirm the result before declaring success. I still believe every word of that. Read-back closes the gap between transport and state, and it catches the most common agent failure in the wild, which is declaring success on a write that never landed.
But I want to be honest about where it stops. Read-back proves the object changed the way the call said it would. It cannot prove the call was the right call. In the repricing story above, read-back passes four hundred times in a row. Wrong collection, verified thoroughly.
Necessary, not sufficient. That exchange, and those threads, shaped the list below: the layers that close the rest of the gap.
The layers that close the gap
None of this is exotic. Most of it is decades of transactional common sense applied at a new boundary: the line between a language model and your production system. Each layer kills one specific class of failure.
1. Scope the credential, not the prompt
"Please only touch the blog" is not a permission model. The prompt is reachable territory for anyone who can get text in front of your agent: a product review, a support ticket, a pasted document. If the prompt is the only thing standing between the agent and a tool family it should never use, you do not have scoping, you have a suggestion.
Scoping has to live where the prompt cannot reach it: in the credential. On our server that takes two forms. Sessions are bound by OAuth to a single app, so an agent connected to app A cannot discover that app B exists. And the tool list is generated from what the app has enabled: no shop, no shop_ tools; push not configured, no push tools. The families the agent must not touch are not denied, they are absent from the inventory. Billing, design, and the build pipeline are not on the surface for anyone. The server card is public if you want to see what a feature-gated inventory looks like.
What this buys you: a compromised or confused session cannot escalate. There is nothing to escalate into.
2. Idempotency, because agents retry
Agents retry. Clients time out and replay. Runtimes resume interrupted loops. In human API usage a duplicate submit is rare; in agent usage, retry-on-ambiguity is standard behavior, which means "the network blinked between the request and the response" must never be able to mean two pushes or two charges.
The first half of the answer is declarative, and MCP already has the vocabulary for it: tool annotations. Every tool on our card declares idempotentHint (along with readOnlyHint and destructiveHint): updates are marked safe to replay, creates and push are not, and the server-side failure policy the meta tool returns caps retries at exactly one. A client that respects the annotations will not loop a non-idempotent write on its own initiative.
The second half is where most surfaces, ours included, still have work to do: true idempotency keys on the non-idempotent writes. The mechanism transfers unchanged from classic API design: the client sends a key with the write, the server stores the outcome under that key, and a replay returns the stored outcome instead of running the write again. Stripe drilled this into a generation of engineers. MCP write tools need it for a bigger reason: the thing doing the retrying now improvises.
3. Bind the write to the version you read
Between the agent's read and the agent's write, the ground can move: another agent, a colleague in the back office, a scheduled job. A write built on a stale read should be rejected, not applied. The mechanism is old and boring: the write carries the version (or etag, or updated-at) of the object as the agent read it, and the server refuses with a conflict if the object has moved since. The agent re-reads and re-decides, with current facts.
There is a coarser flavor of precondition you can get for free at schema-design time: make illegal transitions unrepresentable. On our order tooling, the shipping-status enum only contains FULFILLED and DELIVERED, and the state machine runs one way. An agent cannot un-deliver an order, however confused it gets. If you do nothing else from this section, enum your state machines.
4. High-impact writes get a plan, then a commit
A draft article is a non-event; delete it and nothing happened. A broadcast push has no undo: our card marks it destructiveHint: true, idempotentHint: false, which is the formal spelling of "no undo, no retry". Bulk repricing touches four hundred objects in one intent. Above a certain blast radius, one-shot writes are the wrong shape entirely.
The shape that works is plan-before-commit. The first call returns a plan: what will change, how many objects, a sample of them, bound to the exact context it was computed in, with an expiry:
{
"plan_id": "pl_9f27",
"action": "reprice",
"objects_matched": 412,
"sample": ["Home Kit Hoodie 2026", "Away Kit Hoodie 2026", "..."],
"bound_to": {
"app": "my-shop-app",
"object_set_hash": "sha256:c41d...",
"tool_schema_version": "2026-07-01",
"policy_version": 14
},
"expires_at": "2026-08-05T18:00:00Z"
}
Commit is a second call that references plan_id and fails if any binding moved: the object set changed, the schema changed, the policy changed, the plan expired. Between the two calls sits a human or an explicit policy, looking at "412 objects" and deciding whether that number matches the intent.
We are partway there in production and heading the rest of the way. Every tool on our server publishes its recommended discover, call, verify sequence through a meta tool (meta_get_tool_plan), and my own standing rule for push is send: "at", never now: the agent schedules, which leaves a review window between the ask and the broadcast. The plan object is where this converges, because "a number a human saw before commit" is the single cheapest defense against the wrong-collection story.
5. Verified read-back stays
Everything above is why read-back is not enough. None of it is a reason to drop read-back: it remains the floor. After every write, read the object back through the same read path anyone else would use, and compare against what was sent. It is the layer that catches the write that silently did not land, the truncation, the default that filled a field you did not send. Keep it, make it the server's contract rather than the client's memory, and stack the rest on top.
One requirement I did not fully appreciate until I ran this article's checklist against our own server: the read path you verify through must be read-your-writes consistent. While testing, I hit the mirror image of the 200-that-failed: a delete succeeded, and the immediate read-back served a seconds-stale cached copy of the object, so the verification step swore the delete had not happened. An agent following the discipline to the letter would have retried a write that had already landed, which is exactly the failure class layer 2 exists for. Read-back through a cache lies in both directions. Verify against the source of truth, or invalidate on mutation for the writing session.
6. Make the 200-that-failed observable
The nastiest property of an intent failure is that your telemetry approves of it. A standard span records transport: status code, latency, maybe payload size. It has no opinion about whether the business outcome matched the ask. On a default dashboard, the span for the wrong-collection repricing is indistinguishable from the span for the right one. All green.
The signal has to be emitted on purpose, in two places. In the tool result, where the agent can see it: a server-side outcome flag, plus the read-back requirement, so "it worked" is a claim the server participates in. And on the span, where your on-call can see it:
http.status_code = 200 # transport: the easy part
mcp.write.readback = "match" # state: persisted as sent
mcp.write.outcome = "ok" # intent: preconditions held, plan honored
Then alert on the combination that matters: status 200 with an outcome that is not ok. That alert is the entire point of this post compressed into one rule: the failures worth paging on are the ones that returned success.
You do not remove the human. You move them.
It would be easy to read all this as "add friction everywhere", and that would be the wrong lesson. Draft states, scheduled sends, version conflicts and plan gates are all ways of sorting writes into two lanes: low-impact writes flow without ceremony, high-impact writes pause at exactly one place, the commit line.
That is not less automation. It is oversight spent where it changes the outcome. The content agents I run create drafts all day without me. Nothing reaches a user's lock screen without a human having seen the plan. Both of those sentences are policy, and I can defend each one, which is more than I could say for "the agent seems careful."
The checklist
Before an agent gets write access to a production system, you should be able to answer yes six times:
- ☐ Scoped: can this credential reach only the tool families this agent needs, with everything else absent from the inventory, not just denied?
- ☐ Idempotent: does the same write, retried, produce one change instead of two?
- ☐ Version-bound: does a write built on a stale read get rejected instead of applied?
- ☐ Previewable: do high-blast-radius writes produce a plan that a human or a policy approves before commit?
- ☐ Read-back verified: is post-write verification part of the server contract, not client etiquette?
- ☐ Semantically observable: would a 200 that did the wrong thing show up on a dashboard, or only in a customer email?
Six noes is a demo. Six yeses is something you can hand to a customer.
Takeaway
Getting an agent to call your write tools and collect 200s is the easy 80 percent, and it is the part every MCP server demo shows. The remaining 20 percent, making sure the right thing happened and making it survivable when it did not, is unglamorous server work: scopes, keys, versions, plans, read-backs, span attributes. It is also, I think, where production MCP surfaces will actually differentiate from here on.
The surface these notes come from is documented at goodbarber.com/mcp if you want to see the choices in context.
What is in your write-safety stack that I missed? Genuinely curious. The comments on the last piece made this one better.
Top comments (1)
This separation between transport, state, and intent success is exactly the right model. One useful addition is a set-level postcondition for bulk writes. The plan can carry the target predicate, object-set hash, and invariants such as
changed_count = 412andoutside_target_changed_count = 0. After commit, verify those against a server-side change log or audit snapshot—not only by reading each object through the same path that produced the plan.That catches both partial execution and the “wrong collection, verified 400 times” failure. An idempotency key can then be bound to the same intent manifest, so a retry cannot silently recompute a different target set. In other words, intent success becomes an explicit precondition/postcondition contract rather than a judgment the model has to infer.