AI agents can retry a failed request without knowing whether the first attempt already succeeded. That uncertainty becomes dangerous when APIs create records, send messages, or move money.
The Timeout That Creates Two Support Tickets
Imagine an AI agent responsible for handling customer complaints. It reads an incoming email, identifies the issue, and calls an API to create a support ticket. The API creates the ticket successfully, but the network connection drops before the agent receives the response.
From the agent's perspective, the operation failed. It retries the request. The API creates another ticket.
Now two employees may investigate the same complaint. If the operation involved issuing a refund, the consequences could be more serious.
This is not primarily a problem with the language model. It is a problem with how the application handles uncertain outcomes. Distributed systems have faced this issue for years, but AI agents make it easier to encounter because they can initiate sequences of API calls without a human confirming each action.
Before an agent can safely modify business data, the application needs a reliable way to recognize repeated attempts at the same logical operation.
That is where idempotency keys come in.
What an Idempotency Key Actually Does
An idempotency key is a unique identifier attached to a request that performs a state-changing operation. The server records the key and associates it with the operation's result.
If the same request arrives again with the same key, the server recognizes that the operation has already been handled. Instead of performing the action again, it can return the stored result.
For example, an agent creating a support ticket might send:
POST /tickets HTTP/1.1
Content-Type: application/json
Idempotency-Key: run-482-step-create-ticket
{
"customer_id": "cust_104",
"subject": "Duplicate invoice charge"
}The first request creates the ticket. A later retry using the same key should return the original ticket identifier rather than create a second record.
The key identifies the logical action, not merely one network request. A retry and the original request belong to the same action, even when they arrive through different connections or worker processes.
Why AI Agents Make Retries More Complicated
Traditional applications usually have predictable control flow. A developer writes code that calls an endpoint and defines what happens after a timeout.
AI agents introduce another layer of decision-making. An agent may choose tools based on model output, repeat a step after an apparent failure, or resume a partially completed workflow after a worker restart.
Consider an agent performing three actions: create a customer record, issue an invoice, and send a confirmation email. If the second action times out, the agent needs to know whether the invoice was created before deciding what to do next.
Retrying every action is unsafe. Restarting the entire workflow is also unsafe because earlier steps may have succeeded.
The solution is to give each state-changing step a stable identity that survives retries and restarts.
Generate Keys From Stable Intent, Not Each Attempt
A common mistake is generating a fresh UUID immediately before every API request. That gives every retry a different key, preventing the server from recognizing duplicates.
Instead, create the operation identifier when the workflow step is scheduled, store it durably, and reuse it for every attempt.
import uuid
def create_operation(workflow_id, step_id):
return {
"workflow_id": workflow_id,
"step_id": step_id,
"idempotency_key": str(uuid.uuid4())
}
# Persist this record before executing the tool.
operation = create_operation(
workflow_id="run-482",
step_id="create-ticket"
)
# On retry, load the existing operation.
# Do not call create_operation() again.A deterministic identifier such as run-482:create-ticket can also work when workflow and step identifiers are guaranteed to be unique within the appropriate scope.
Avoid deriving the key solely from the model-generated request body. An agent may rewrite a subject line or change its wording during a retry, even though it is trying to complete the same business action.
The key should answer: "Is this another attempt to perform the same intended operation?"
Build a Small Idempotent API With FastAPI
Let's build a minimal ticket-creation endpoint using FastAPI and SQLite. This example stores the idempotency record and ticket in the same database transaction.
That matters because the ticket creation and its deduplication record must succeed or fail together. Otherwise, a crash between those writes could leave the server unable to recognize a completed operation.
Install the dependency:
pip install fastapi uvicornCreate main.py:
import hashlib
import json
import sqlite3
import uuid
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field
app = FastAPI()
DB_PATH = "tickets.db"
class TicketRequest(BaseModel):
customer_id: str = Field(min_length=1)
subject: str = Field(min_length=1)
def connect():
db = sqlite3.connect(
DB_PATH,
timeout=10,
isolation_level=None
)
db.execute("PRAGMA busy_timeout = 10000")
return db
def init_db():
with connect() as db:
db.execute("""
CREATE TABLE IF NOT EXISTS tickets (
id TEXT PRIMARY KEY,
customer_id TEXT NOT NULL,
subject TEXT NOT NULL
)
""")
db.execute("""
CREATE TABLE IF NOT EXISTS operations (
tenant_id TEXT NOT NULL,
operation_key TEXT NOT NULL,
payload_hash TEXT NOT NULL,
response_json TEXT NOT NULL,
PRIMARY KEY (tenant_id, operation_key)
)
""")
init_db()
@app.post("/tickets", status_code=201)
def create_ticket(
payload: TicketRequest,
idempotency_key: str = Header(
min_length=8,
max_length=200
)
):
# Demo only. In production, derive tenant
# identity from authenticated credentials.
tenant_id = "demo-tenant"
body = payload.model_dump_json()
payload_hash = hashlib.sha256(
body.encode("utf-8")
).hexdigest()
db = connect()
try:
# Serialize competing writers in SQLite.
db.execute("BEGIN IMMEDIATE")
existing = db.execute(
"""
SELECT payload_hash, response_json
FROM operations
WHERE tenant_id = ?
AND operation_key = ?
""",
(tenant_id, idempotency_key)
).fetchone()
if existing:
old_hash, response_json = existing
if old_hash != payload_hash:
db.execute("ROLLBACK")
raise HTTPException(
status_code=422,
detail="Key reused with different payload"
)
db.execute("COMMIT")
return json.loads(response_json)
ticket_id = str(uuid.uuid4())
db.execute(
"""
INSERT INTO tickets
(id, customer_id, subject)
VALUES (?, ?, ?)
""",
(
ticket_id,
payload.customer_id,
payload.subject
)
)
result = {
"ticket_id": ticket_id,
"status": "created"
}
db.execute(
"""
INSERT INTO operations
(tenant_id, operation_key,
payload_hash, response_json)
VALUES (?, ?, ?, ?)
""",
(
tenant_id,
idempotency_key,
payload_hash,
json.dumps(result)
)
)
db.execute("COMMIT")
return result
except HTTPException:
raise
except Exception:
if db.in_transaction:
db.execute("ROLLBACK")
raise
finally:
db.close()Start the API with uvicorn main:app --reload. The first request inserts a ticket and its operation record within one transaction. A repeated request using the same key returns the saved ticket identifier.
The example deliberately keeps the side effect inside SQLite. It does not solve the harder problem of coordinating a database transaction with an external payment provider or email service.
Test the Retry Behavior
Send the same request twice with the same key:
curl -X POST http://localhost:8000/tickets \
-H "Content-Type: application/json" \
-H "Idempotency-Key: demo-ticket-001" \
-d '{"customer_id":"cust_104",
"subject":"Duplicate invoice charge"}'Repeat the command without changing the key or body. Both responses should contain the same ticket_id.
Now change the subject while keeping the original key. The API returns HTTP 422 because the key has already been associated with a different payload.
This prevents an agent from accidentally using one operation identifier for two different actions.
Concurrency Is Where Simple Deduplication Breaks
Checking whether a key exists is not enough if two requests can execute simultaneously. Both requests might read an empty record and proceed to perform the action.
The example uses BEGIN IMMEDIATE to acquire SQLite's write reservation before checking the key. Competing writers are serialized, and the primary key prevents duplicate operation records.
For a production database such as PostgreSQL, use a transaction with a unique constraint and an appropriate insert-or-conflict strategy. Do not rely on an in-memory dictionary or an application-level lock when multiple workers or servers can receive requests.
Idempotency is a persistence and concurrency problem, not just an HTTP-header convention.
External APIs Need an Operation State Machine
Now consider an agent that calls a payment provider. Your application cannot atomically commit its own database transaction and the provider's payment operation using the simple pattern above.
If the provider charges a customer but your worker crashes before saving the response, your database may still show an unfinished operation.
A safer design uses explicit states such as pending, in_progress, completed, and needs_reconciliation.
Before making the external call, persist the operation and its stable key. Pass that key to the provider if the provider supports idempotent requests. After a timeout, query the provider's operation status or reconcile against a durable transaction identifier before attempting another side effect.
If the provider offers neither idempotency nor reliable status lookup, automatic retries may not be safe. The application may need to pause the operation for human review.
Exactly-once execution across independent services is not something an HTTP header can guarantee by itself.
Keep the Safety Rules Outside the Model
An AI agent should not decide whether a repeated payment or deletion is safe based on a conversational explanation alone.
The tool gateway should enforce permissions, operation identities, request validation, and retry behavior. The model can request an action, but deterministic application code should decide whether that action is permitted and whether it has already happened.
This separation matters in AI application development, where model-generated decisions may eventually trigger changes in existing databases and business systems. The surrounding software must enforce the rules that keep those changes predictable.
It also affects team responsibilities. Organizations using dedicated development teams for agent-enabled systems need clear ownership of API contracts, operation records, incident recovery, and production testing. Those responsibilities cannot be delegated to the model.
What to Check Before an Agent Goes Live
Before allowing an agent to call a state-changing endpoint, verify that repeated requests use a stable key, that the key is scoped to the authenticated tenant and operation, and that the server rejects conflicting payloads.
Test simultaneous requests, worker restarts, network timeouts, and failures after an external service accepts an action. Define how long operation records are retained and what happens when a retry arrives after that period.
Record operation identifiers in logs and traces so engineers can reconstruct what happened without exposing sensitive request data.
Most importantly, decide which actions can be retried automatically and which require reconciliation or human approval.
Make Repeated Requests Boring
AI agents are useful because they can carry out multi-step work with less direct supervision. That also means their mistakes can reach real systems quickly.
A timeout should not create another invoice. A restarted workflow should not send the same email twice. Two workers should not be able to perform one logical action independently simply because they received the same task.
Idempotency keys give APIs a practical way to distinguish a retry from a new request. They do not replace authorization, validation, or recovery planning, but they provide a critical boundary between uncertain agent behavior and durable business actions.
Before an AI agent gets permission to write to your systems, make sure your API knows how to say: "I've already done that."
Top comments (0)