The tool times out, the agent retries. Every framework does this and it is the right default for a read.
Run it: https://dev48.infy.uk/ai/days/day78-tool-call-retries.html
Nothing here simulates language. The engine simulates the call — a request that may not arrive, a server that may have applied it anyway, a reply that may not come back — over 4,000 enumerated timelines per policy. The agent sees only the reply. The world knows what happened.
Three failures, and only one makes a retry dangerous
never arrived -> nothing applied, agent sees nothing
applied, reply lost -> APPLIED, agent sees nothing <- this one
refused -> nothing applied, agent sees a refusal
From the agent's side the first two are indistinguishable.
The metric and the world move in opposite directions
| reported success | applied exactly once | |
|---|---|---|
| no retry | 81.92% | 89.95% |
| retry up to 3 | 96.95% | 89.42% |
The agent's own success rate improves by fifteen points and the share of calls that applied exactly once goes down.
What paid for it: 334 calls of 4,000 applied more than once — and the success metric cannot see a single one. From inside the agent, a duplicated write and a clean write are the same event. The success rate is a measurement of the replies, not of the world.
The honest half
It would be wrong to stop at "retries are bad". Without them the agent reports a failure for a call that actually succeeded 321 times in 4,000 — the write landed, the reply was lost, and whatever the agent does next is built on a false picture.
Retrying cuts that to 33. Both problems are real; retries trade one for the other, and the trade is good for reads and bad for un-keyed writes.
The fix changes nothing the agent can see
| retry 3 | retry 3 with an idempotency key | |
|---|---|---|
| reported success rate | 96.95% | 96.95% |
| tries per call | 1.18 | 1.18 |
| applied exactly once | 89.42% | 97.78% |
| applied twice or more | 334 | 0 |
An idempotency key does not prevent the retry and does not change the reply. It stops the second application. So the reported rate is identical — the same count, from the same 4,000 timelines — while duplicates go to zero.
Any evaluation that scores an agent on its own success rate rates those two configurations exactly equally. No metric computed from replies can distinguish them, which is worth knowing before you trust one.
159 verifier asserts, 15 in-page checks, 0 failures.
Top comments (0)