A production bug I shipped, why the obvious fix was incomplete, and what actually closes it.
The incident
I build an AI assistant for a school-management platform. Teachers and school admins ask it questions in natural language, and it can also perform write operations: create an exam room, correct attendance, and so on. The stack is a LangGraph agent in Python calling an MCP server written in Go. The MCP server exposes 26 tools. Twelve of them write.
A few months after launch we found duplicate rows. Two exam rooms where the user had created one. Two attendance corrections where the user had made one.
Nothing in the logs looked wrong. Both writes were authenticated. Both were authorized. Both contained exactly what the user had asked for. There were just two of them.
The cause was mundane. Sometimes the user clicked again because the response felt slow. Sometimes the model called the same tool twice inside one turn. The user approved one action. The system performed two.
The first fix
I split every write tool into two phases.
The first call does not write anything. It returns a preview of what will happen, plus a confirmation token. The second call replays that token, and only then does the write happen.
The token is an HMAC. It is bound to the user ID, the tool name, and a hash of the parameters. It expires after five minutes. The nonce inside it is burned the moment it is used.
# phase 1 - preview only
token = hmac(secret, f"{user_id}|{tool}|{sha256(params)}|{nonce}|{exp}")
return {"preview": render(params), "confirm_token": token}
# phase 2 - execute
assert verify_hmac(token)
assert burn_nonce(token.nonce) # fails if already used
write(params)
The parameter hash matters. Without it, the model could show the user one thing and execute another. With it, what the user approved is exactly what runs.
I also strip the token out of the SSE stream before it reaches the browser. The model never sees it. The user never sees it.
This worked. The duplicate writes stopped. I considered the problem solved for a long time.
What I missed
The nonce protects the token. It does not protect the action.
Burning a nonce means one specific token cannot be used twice. But the agent does not need the old token. The agent re-plans.
It calls phase one again, for the same action. It receives a new token. A completely valid one. Then it replays that token, and the write happens a second time.
Walk through the checks with me. HMAC signature: valid, it is a fresh token. Nonce: unused, it is a fresh nonce. Expiry: fine, it was issued a second ago. Parameter hash: matches, because it is the same action with the same parameters.
Every check passes. That is the uncomfortable part. The parameter hash matching is the strongest evidence available that this is a duplicate, and my design treated it as proof of correctness instead.
The root cause is that my state was keyed by the token, not by the action. The issuer remembered which tokens had been spent. It remembered nothing about which actions had been performed.
Recent work has a name for this: semantic replay. Xu et al. define it as "exceeding the execution budget of a token-independent authorization instance rather than merely reusing an old token identifier" (Beyond Single-Use Tokens, arXiv:2608.01710). Their phrase for the failure mode is precise: identifier-local tokens permit fresh semantic reissuance.
That is exactly what my design did. It made reissuance cheap and invisible.
Why more token hardening does not help
The instinct is to tighten the token. It does not work.
A shorter expiry does not help, because the second token is brand new. A stronger nonce does not help, because no nonce is reused. Signing more fields does not help, because every field is legitimately identical.
Asking the user to confirm again feels like a fix, and it is the worst one. In a chat interface, the second confirmation prompt looks exactly like the first. The user clicks yes. That is not oversight. That is a rubber stamp with an audit log attached.
Every fix at the token layer protects the token. The problem is not at the token layer.
What actually closes it
Key the state to the action.
action_key = sha256(user_id | tool_name | canonical(params) | turn_id)
canonical(params) is not optional. Sort the keys, normalise the types. Otherwise {"a":1,"b":2} and {"b":2,"a":1} produce different keys and the mechanism does nothing.
Then keep a durable record per action key, and let the state move in one direction only.
- No record: create one in
ISSUEDstate, return a new token. - Already
ISSUED, not committed: return the same token again. Issuing is idempotent. You do not hand out a second authorization for an action that already has one. - Already
COMMITTED: refuse, and return the result of the first execution.
That third case is the one people forget. A duplicate request should not fail loudly. It should return what happened the first time.
Commit is a transaction. Flipping ISSUED to COMMITTED and performing the write happen together, or neither happens.
In my case the write goes out over HTTP, to the same REST endpoints the browser uses, so a shared database transaction is not available. The answer there is to derive an idempotency key from the action key and pass it to that endpoint.
Two layers, doing two different jobs. The ledger prevents duplicate admission. The idempotency key prevents duplicate effects. You need both, because the ledger can commit and the HTTP call underneath it can still be retried.
If this shape feels familiar, it should. It is at-least-once delivery with idempotent consumers. Agent frameworks are rediscovering a problem that message queues solved twenty years ago.
The cost
There is a trade-off here, and I would rather state it than hide it.
Including turn_id in the action key means a user can genuinely create two identical exam rooms in two different turns. That is correct behaviour. It is also a gap.
Removing turn_id closes the gap and breaks legitimate repetition.
There is no universally correct answer. There is only an explicit one. The choice I favour: keep turn_id, and when two action keys differ only by the turn while the parameters are identical inside a short window, surface a warning to the user instead of silently proceeding.
The takeaway
The risk in agent systems sits in the authorization layer and in the side effects, not in the model.
Human-in-the-loop tells you that an action was approved. It does not tell you that the action happened once. Those are different guarantees, and they are constantly confused.
So ask your stack one question. If the model re-plans, can it obtain a second valid approval for the same action? If you do not know the answer, it can.
Top comments (3)
The key insight — "the nonce protects the token, not the action" — is the part almost everyone misses, because the confirm-token pattern feels airtight until you remember the agent can just re-plan and mint a fresh, fully-valid token for the same action. You closed the replay hole and left the re-execution hole wide open.
The fix that generalizes is to move idempotency onto the action, not the confirmation. Derive an idempotency key from (user, tool, param-hash, and a stable "intent" id for that turn) and enforce uniqueness at the write boundary — ideally a unique constraint in the database, so even a concurrent double-fire collapses to one row. The token authorizes; the idempotency key deduplicates. They're two different jobs and conflating them is what bit you.
One subtlety worth flagging: the param-hash has to be canonicalized, or a reordered-but-equivalent payload from a re-planning model slips through as a "different" action. Did you end up keying on a semantic intent id, or purely on the serialized params?
The distinction between admission control (the agent runtime ledger) and effect deduplication (downstream idempotency key) is the crux here that most framework architectures skip.
We ran into an almost identical edge case with browser-driven agent mutations where a network call timed out at the transport layer after the remote service had already processed the payload. If the planner re-evaluates and issues a new request, you get duplicate state unless the downstream handler recognizes the semantic identity of the action.
Deriving the downstream idempotency key directly from the canonical action hash—rather than an ephemeral turn nonce—ensures that even if the planner loops or retries mid-flight, the target API treats the retry as an idempotent replay rather than a second distinct write. Great breakdown of semantic replay.
The action_key is still identifier-local, one level up. It binds canonical(params), the spelling of the arguments, rather than what the write does to the row. Sorting keys and normalising types gets you past {"a":1,"b":2} versus {"b":2,"a":1}. It does not get you past "Room 5A" against "Room 5a", or a start time sent as an offset on one pass and as Z on the next, or an optional field the planner fills in once and omits the second time. With twelve write tools, more than one tool_name can also reach the same table.
That matters because your reproduction depends on the re-plan emitting byte-identical parameters. A re-plan regenerates the arguments from the model, so identical bytes across two plans is the lucky case. The paraphrased duplicate is the ordinary one, and it passes the ledger with a fresh action_key and a fresh token, the same way a fresh nonce passed the old design. Xu et al.'s sentence survives your fix with one word changed: params-local keys permit fresh semantic reissuance by paraphrase. The identity that holds still has to come from the effect, a natural key on the target row with a uniqueness constraint behind it.
Then the atomicity. You write that flipping ISSUED to COMMITTED and performing the write happen together or neither happens, and a paragraph later that the write goes over HTTP so a shared transaction is not available. Both orderings leave a window. Write then flip: a crash in between strands the record in ISSUED, your ISSUED rule hands back the same token, and the next attempt replays. Downstream idempotency may swallow the effect, but the ledger learns nothing, and its "return the result of the first execution" branch is unreachable for that key forever. Flip then write: a crash in between leaves COMMITTED with nothing behind it, and every later attempt is refused and handed a stored result for a write that never happened. That one is silent.
An idempotency key deduplicates effects. It cannot repair the ledger's belief about them, which is why ISSUED has to be read as unknown and reconciled against the downstream rather than as not yet done.
When the ledger says ISSUED and the write already landed, which of the two decides what your user's next request sees?