Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Search for "what does idempotent mean" and you'll find a definition, but not always one you can use. Here's the version that matters: idempotent means doing the same thing twice gives you the same result as doing it once.
This is essential infrastructure vocabulary. It's the difference between "safe to retry" and "dangerous to retry," and that difference determines whether your system charges customers twice, sends duplicate messages, corrupts data, or works reliably when the network fails.
Part 1: The Core Idea
Imagine you're playing chess. You tell someone: "Move my pawn to e4." They do it. If you say it again, does the pawn move twice? No — it stays on e4. That's idempotent. The board state is the same whether you made that request once, twice, or a hundred times.
Now imagine a different instruction: "Give me $10." If someone follows that twice, you get $20, not $10. That's not idempotent.
The key insight: idempotency is about state, not about how many times you ask. If the end state is the same, the operation is idempotent.
Why does this matter? Because computers fail. Networks drop packets mid-transmission. Servers timeout. When something goes wrong mid-request, the safe thing to do is retry. But if your operation isn't idempotent, retrying it can corrupt data, double-charge customers, send duplicate notifications, or delete things twice. In production, at scale, with real money and real data.
The term comes from Latin — idem (same) + potent (power). It was coined in mathematics and abstract algebra, but in the last 15 years it's become essential vocabulary for anyone building distributed systems, APIs, or agents that need to survive real networks.
Part 2: REST APIs and the Exactly-Once Problem
In HTTP, different methods have different idempotency guarantees:
GET — Reading data is idempotent. When you fetch /api/users/123 twice, you get the same user object back. The server doesn't change anything; you're just reading. Safe to retry forever.
POST — Creating new resources is not idempotent. When you POST /api/orders with an order object, the server creates a new order and returns it. If you POST the same data again, you get a new order. Same request, different outcome. POST twice, you've charged the customer twice.
PUT — Replacing a resource is idempotent. When you PUT /api/users/123 {name: "Alice"}, you're saying "set this user's name to Alice." Do it again? It's still Alice. The state is the same. PUT is safe to retry.
DELETE — Removing a resource is idempotent. DELETE /api/users/123 removes the user. Call it again? The user is already gone, so the end result is the same: the user doesn't exist. DELETE is safe to retry.
PATCH — Partial updates are usually not idempotent. If your PATCH says "increment balance by $10," and you retry, the balance goes up by $20. But PATCH can be idempotent if it's replacing a field instead of modifying it (e.g., "set balance to $100" vs. "add $10 to balance"). When in doubt, don't retry PATCH without explicit safeguards.
The Real-World Problem
Here's a concrete scenario: A customer clicks "buy," their client sends a POST request to create an order, and the network connection drops mid-transmission. The server might have received the request, or it might not. The client doesn't know, so it retries.
Without idempotency protection, the result is a 50/50 coin flip: either the order succeeded and the retry creates a duplicate, or the order failed and the retry succeeds. You can't reliably know which happened. At scale, with thousands of transactions, "coin flip" becomes "reliably lose money."
Let's make this concrete. An e-commerce site processing 100 orders per minute during peak hours sees roughly 1–3% of requests fail mid-transmission. That's 1–3 orders per minute that might be retried without idempotency protection. Over a single busy day, that's 1,440 to 4,320 duplicate charges. Your refund queue explodes, your support team drowns in angry emails, and your payment processor flags your account for high dispute rates. Some payment systems will actually block your account entirely if your chargeback rate climbs above a threshold — losing your ability to process any orders because you didn't design for idempotency.
The solution is idempotency keys. The client generates a unique ID (a UUID, for example) and includes it in the request header:
POST /api/orders
Idempotency-Key: "a1b2c3d4-e5f6-47a8-b9c0-d1e2f3a4b5c6"
Content-Type: application/json
{
"user_id": 42,
"items": [{"sku": "WIDGET-1", "qty": 2}],
"total": 199.99
}
The server checks: "Have I seen this key before?" If yes, it returns the stored response without executing anything. If no, it executes the operation (creates the order, charges the card, sends a confirmation email), stores the result, and returns it. Same key arriving twice = identical response both times, guaranteed.
Stripe's API requires Idempotency-Key on all POST operations. AWS Lambda has idempotency decorators baked into their SDK. Google Cloud Tasks offers exactly-once delivery semantics backed by idempotency. These aren't optional niceties — they're table stakes for production APIs that handle money or data that matters. If you're building an API and not supporting idempotency keys, you're shipping a ticking time bomb.
The Common Mistake
Developers sometimes confuse "safe to retry" with "has no side effects." That's backwards.
A PUT request that updates a user's name in the database does have a side effect — it changes the database. But it's still idempotent because doing it twice leaves the same state: the user's name is set to the new value. Side effects and idempotency are not opposites. What matters is whether repeating the action gives the same final state.
Part 3: Agents, Tool Calls, and Retry Logic
When you use an agent — whether it's Claude in an agentic loop, an autonomous workflow, or a custom agent framework — the agent calls tools in a loop. Network failures happen. Timeouts happen. The agent retries automatically.
This is where idempotency becomes critical. If the tool being called is not idempotent, retries don't just waste tokens — they break data. Here are three common cases and how to fix them:
Sending a notification: A tool that sends an email to a customer is not idempotent by default. Scenario: The agent calls send_email(user_id=123, template='welcome'). The email sends successfully, but the network drops before the tool returns. The agent retries the call. Now the customer receives two identical welcome emails. If this tool is called by an automated onboarding agent across thousands of new users, you're sending duplicate welcome emails to everyone.
Fix: The tool accepts an idempotency_key parameter. It stores a record of what it did for that key (e.g., "For key X, we sent email Y at time Z"). On the second call with the same key, it returns the previous result without re-sending. One email sent, two calls made, same outcome.
Updating a value: A tool that increments a database field is not idempotent by default. Scenario: The agent needs to credit 10 points to a user's account, so it calls add_points(user_id=456, points=10). The operation executes, but the response times out. The agent retries. The user now has 20 extra points instead of 10. If this happens across hundreds of daily operations in a rewards program, you've given away thousands of dollars in unearned points.
Fix: Wrap the operation in a database transaction. Before executing the increment, check if that specific operation already ran (usually stored in an idempotency_key field in the transaction log). If it did, return the stored result. If not, execute and record it.
Deleting a record: A tool that deletes a resource is idempotent by default. Scenario: The agent calls delete_file(file_id=789). The file is deleted, but the response is lost. The agent retries. The file is already gone, so the second call finds nothing to delete, but the end state is the same: the file doesn't exist. Safe to retry without additional logic.
Agent Frameworks and Idempotency
Most modern agent frameworks assume tools are not idempotent by default, because most tools aren't. Claude's official tool-use examples show wrapping tool calls in undo/redo layers or checkpoint systems to handle retries safely. Google's Sheets API, when called through agentic interfaces, wraps mutations in transaction-aware layers that track what's already executed. The pattern is universal: the agent framework or the tool itself must provide idempotency guarantees.
When you design a tool for an agent to call, the question isn't "is this tool idempotent?" It's "who ensures it's idempotent — the tool or the framework?" If the tool doesn't handle it, the agent framework has to. If neither does, you have a bug.
For the specific implementation patterns when building agent-facing tools — code examples, database transaction patterns, and when to use idempotency keys vs. other mechanisms — see the idempotency keys for agents post. This post covers the concept; that one covers the implementation.
Part 4: When and Why You Should Care
You should care about idempotency in three situations:
1. You're building an API. Any endpoint that mutates data (POST, PUT, DELETE, PATCH) needs an idempotency story. If you're not thinking about it, you're relying on the idea that your network never fails and your clients never retry. Neither assumption holds at scale. A good starting point: support idempotency keys on all mutation endpoints. Document which endpoints are idempotent by design (GET, PUT, DELETE) and which require explicit key support (POST, PATCH). Test your idempotency logic: write a test that calls the same endpoint twice with the same request body and verifies the result is identical.
2. You're deploying agents or automation. If a system can retry, every tool it calls must be idempotent. This is non-negotiable. Test it explicitly: call the same tool twice with the same arguments and verify the outcome is identical. Better yet, make it part of your tool's test suite. If a tool's side effects aren't idempotent, it shouldn't be called by an agent without explicit idempotency wrapping.
3. You're integrating with another system. Read the API docs. Which endpoints are safe to retry? Stripe says POST charges are idempotent if you include an idempotency key. PayPal says some endpoints are, others aren't. Some APIs are idempotent by default; others require you to pass an idempotency key. Some don't support idempotency at all, which means you can't safely retry them — you have to live with the risk. Know before you retry, and document it.
You can de-prioritize idempotency if:
- Your system genuinely never retries and your network never fails. (Unrealistic, but if true: you can relax this.)
- You only read data. GET is idempotent; no special handling needed.
- Your failure rate is so low and your consequence so mild that duplicate operations are acceptable. (Rare.)
How to Test Idempotency
If you're unsure whether a system is actually idempotent, test it:
- Execute an operation and record the result (e.g., order ID, new balance, confirmation number).
- Execute the identical operation again.
- Verify the result is identical. Same order ID, same balance, same confirmation number.
If step 3 fails — if the two calls produce different results — your system is not idempotent, and retrying it is dangerous.
The Rule
If something might be retried — whether by the client, the server, an agent, or a proxy — it must be idempotent. If it's not idempotent, your retry logic is a bug waiting to happen. You're betting on the idea that failures never occur, and in production, that's a bet you'll lose.
Recap
Idempotency is a property of an operation: doing it twice has the same effect as doing it once.
By HTTP method:
- GET, PUT, DELETE: idempotent by design
- POST, PATCH: not idempotent without explicit safeguards (idempotency keys, transactions, etc.)
By context:
- REST APIs: support idempotency keys on mutation endpoints
- Agents and tools: mutation operations need explicit idempotency via keys or transaction tracking
- Integrations: always check the other system's documentation before retrying
The business impact:
- Duplicate charges, duplicate messages, data corruption
- Customer trust erosion, refund queues, chargeback disputes
- Payment processor flags and account restrictions
The fix:
- Design for idempotency from the start, don't retrofit it later
- Test idempotency explicitly: call operations twice and verify identical results
- Document which operations are safe to retry and which aren't
- If something can be retried, it must be idempotent
The one thing worse than a system that crashes is a system that silently succeeds twice when it should succeed once. Don't ship that system.
Top comments (1)
The distinction between “safe to retry” and “no side effects” is probably the most important point here.
I’ve seen this become especially painful with agent/tool workflows. A timeout doesn’t tell you whether the operation failed it only tells you that you didn’t receive the result. Retrying a mutation at that point without an idempotency strategy is where the real problems start.
I also like the framing of asking “who guarantees idempotency?” rather than assuming the framework handles it. For payments, emails, credits, or other external side effects, that guarantee needs to be explicit and tested.
One thing I’d add is testing the failure window itself: execute the mutation, deliberately lose the response, then retry with the same key. That’s much closer to the failure mode you actually need to survive in production.
Great write-up. The agent/tooling angle makes this especially relevant now.