AI Agent Idempotency: Prevent Duplicate Charges, Emails, and Records

A timeout is not a failed action. It is an unknown action.

That distinction matters the first time an agent calls a payment API, sends a customer email, or creates a CRM record—and loses the response. If the agent retries blindly, it may double-charge a card or send the same message twice. If it does nothing, the user’s request may never finish. A prompt telling the model to “avoid duplicates” cannot settle that ambiguity.

The fix is idempotency: make each logical write happen once, even when workers crash, queues redeliver, SDKs retry, or the model asks to try again. This guide shows a practical design for putting that guarantee in the tool layer, where it can be tested and enforced.

Why agents make ordinary retry bugs worse

Every distributed system has ambiguous failures. A request may reach the target, commit its side effect, and then lose its response. Traditional backends solve this with idempotency keys, unique constraints, durable jobs, and reconciliation.

Agents amplify the problem because they are built to keep trying. A model sees timeout, changes its plan, and asks the same tool to run again. Meanwhile, a queue may redeliver the job after a lease expires, and an HTTP client may retry underneath both of them. One user intent can become four writes.

Treat these as distinct outcomes:

Outcome	What the caller knows	Safe next step
Rejected before send	The target did not receive it	Correct input, then retry if appropriate
Explicit failure	The target responded with a permanent error	Stop or request correction
Explicit success	The target returned a durable receipt	Store and return the receipt
Unknown commit state	The request may have succeeded	Reconcile before retrying

The last row is the important one. A timeout after send must never be silently mapped to “failed.”

The design rule: one key per logical action

An idempotency key identifies the business action, not a network attempt.

For example, “send the approved invoice email for invoice inv_123 revision 4” is one action. It should keep the same key if a worker restarts five times. A later resend after a user edits the invoice is a new action and needs a new key.

Good keys are stable, scoped, and inspectable:

tenant:acme | run:run_8f2 | step:email_invoice | invoice:inv_123 | revision:4

Hash that canonical representation if it contains sensitive identifiers. Do not generate a new UUID on every retry; that turns a dedupe mechanism into a duplicate generator. Also do not use only the user prompt. “Email this invoice” can be a valid request more than once.

An action should usually include:

• tenant and actor identity
• workflow run and step name
• target resource identity and version
• normalized arguments or an argument hash
• expiry policy and final result

Put a write firewall in front of every agent tool

The agent should request a business action, not manipulate retry behavior directly. Place a deterministic action runner between the model and every state-changing integration.

agent plan
   -> tool request (business intent)
   -> action runner (policy + operation ledger)
   -> external API / database
   -> receipt + reconciliation result
   -> agent

This boundary is useful even if your tools are ordinary functions. It gives the system one place to validate tenant access, lock the operation, pass provider keys, classify failures, and hide unsafe retry choices from the model.

Reads can often be retried. Writes need an operation record first.

Start with a small operation ledger

Postgres is enough for many teams. Create the operation before calling the external service; keep it until the action’s replay window is over.

create table agent_operations (
  id uuid primary key,
  tenant_id uuid not null,
  idempotency_key text not null,
  action_type text not null,
  args_hash text not null,
  status text not null check (status in (
    'pending', 'running', 'succeeded', 'unknown', 'failed', 'needs_review'
  )),
  provider_reference text,
  result_json jsonb,
  error_json jsonb,
  lease_expires_at timestamptz,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (tenant_id, idempotency_key)
);

The unique constraint is a hard concurrency boundary. Two workers can receive the same job, but only one can own the logical action. Store an args_hash too: if the same key arrives with different arguments, fail closed. Reusing a key with changed data is almost always a caller bug.

Use status as evidence, not a guess

pending means no worker has started the side effect. running means a worker holds a short lease. succeeded stores the exact receipt to return on a duplicate call. unknown means the downstream result is ambiguous, so a reconciliation job—not an agent—must decide what happened.

Avoid marking an operation failed simply because the HTTP request timed out. That destroys the evidence needed to prevent a duplicate.

A TypeScript action wrapper

Here is a simplified pattern. In production, the claim query should use a transaction and a lease so a crashed worker can be recovered safely.

type ActionResult = { receiptId: string; status: "sent" | "already_sent" };

async function sendInvoiceEmail(input: {
  tenantId: string;
  runId: string;
  invoiceId: string;
  revision: number;
  to: string;
}): Promise<ActionResult> {
  const key = `invoice-email:${input.tenantId}:${input.invoiceId}:${input.revision}`;
  const argsHash = sha256(JSON.stringify({ to: input.to, revision: input.revision }));

  const operation = await operations.claim({
    tenantId: input.tenantId,
    key,
    actionType: "invoice_email",
    argsHash,
    leaseSeconds: 60,
  });

  if (operation.status === "succeeded") return operation.resultJson;
  if (operation.status === "unknown") return reconcileInvoiceEmail(operation, input);
  if (operation.status !== "running") throw new Error("Action is not safe to execute");

  try {
    const response = await emailProvider.send({
      to: input.to,
      template: "invoice",
      metadata: { operationId: operation.id },
      idempotencyKey: key,
    });

    return await operations.succeed(operation.id, {
      receiptId: response.messageId,
      status: "sent",
    });
  } catch (error) {
    if (isAmbiguousTransportError(error)) {
      await operations.markUnknown(operation.id, serialize(error));
      return reconcileInvoiceEmail(operation, input);
    }
    await operations.fail(operation.id, serialize(error));
    throw error;
  }
}

Notice what is missing: no model instruction decides whether an unknown write gets replayed. The wrapper returns a receipt, a verified “already completed” result, or an explicit escalation.

Reconcile when the provider lacks idempotency keys

Many third-party APIs do not support idempotency keys—or claim to accept them without making them searchable. Your internal ledger still helps, but it cannot prove the external effect occurred.

Make the action observable at the target. Depending on the system, reconciliation can query:

• a provider object by a stored request or message ID
• a database row with a unique operation ID
• a webhook event carrying correlation metadata
• a target record by a natural business key and version
• an outbox entry that is atomically written with your local state

For an email, attach operationId as provider metadata and persist the provider message ID. For a CRM create, send an external ID derived from the operation key. For an internal database mutation, use a unique operation_id column or an INSERT ... ON CONFLICT pattern.

If reconciliation cannot prove success or failure, keep the operation in needs_review. This is safer than inventing an answer. The agent can tell the user that the action is pending verification rather than claiming it completed.

Separate retries by failure class

“Retry three times” is too blunt. A safe policy distinguishes failure modes.

Failure class	Example	Retry policy
Safe read	Search endpoint returns 503	Exponential backoff with a limit
Validation error	Missing required field	Do not retry; return correction needed
Rate limit	HTTP 429	Wait for the provider signal, then retry same operation
Ambiguous write	Socket closes after request body	Mark unknown and reconcile
Known transient write failure	Provider confirms no commit	Retry same operation key
Auth or policy denial	Scope removed	Stop and escalate

Retry budgets should also be outside the model. Put limits on attempts, elapsed time, spend, and allowed action types. This makes a bad downstream day finite instead of turning it into an overnight retry loop.

Test the ugly paths before customers find them

Unit tests that expect a 200 response are not enough. Build a small fault-injection suite around every write tool.

Test at least these cases:

1. Two workers claim the same operation at once.
2. The provider succeeds but the response is dropped.
3. The process dies after the provider call and before succeeded is stored.
4. A queue redelivers a completed job.
5. The same key arrives with different arguments.
6. Reconciliation finds a target effect after a timeout.
7. Reconciliation finds no effect and allows a same-key retry.
8. The provider returns a permanent validation or authorization error.

An effective assertion is simple: after any number of retries, the target contains exactly one effect for the operation key. Also test the user-visible result: a duplicate request should return the original receipt, not an opaque error.

Make idempotency visible in operations

Track these counters by tenant, tool, and provider:

• duplicate calls served from the ledger
• operations entering unknown
• reconciliation success and failure rate
• lease expirations and takeovers
• key/argument mismatches
• age of operations waiting for review

These are more useful than raw timeout counts. A rise in unknown commits may reveal a provider regression, an overly short client timeout, or a worker shutdown problem. A rise in duplicate calls can mean your queue is redelivering as designed—or that an upstream client is misbehaving.

Add an audit event each time an operation moves state. Include the actor, agent run, tool version, key hash, arguments hash, provider reference, and decision made by reconciliation. Do not store sensitive prompt or customer data just for convenience.

Roll this out without freezing your product

Start with the highest-impact tools: payments, email sends, record creation, access changes, and anything that triggers work outside your system. Inventory every write-side tool and give it an action type, stable key recipe, reconciliation strategy, and owner.

Then ship in stages:

1. Observe: log proposed keys and duplicate attempts without changing behavior.
2. Protect internal writes: add unique operation IDs and return stored receipts.
3. Protect provider writes: pass stable provider idempotency keys and record references.
4. Reconcile ambiguity: route timeout-after-send cases through target checks.
5. Enforce: reject direct write tools that bypass the action runner.

This is a better investment than ever more prompt rules. Prompts can help an agent choose a valid action; they cannot offer exactly-once delivery across a network.

FAQ

What is AI agent idempotency?

AI agent idempotency means repeating the same logical agent action produces one durable effect, not multiple ones. It protects state-changing tools from retries, worker crashes, duplicate queue messages, and repeated model tool calls.

Should every agent tool use an idempotency key?

Every state-changing tool should have an idempotency strategy. Read-only tools can normally use conventional retry policies. For writes, use provider keys, internal unique constraints, or an operation ledger plus reconciliation.

Can a system prompt prevent duplicate agent actions?

No. A prompt influences model behavior but cannot coordinate concurrent workers, recover a lost response, or prove whether a remote API committed an action. Enforce deduplication in the action runner and target system.

What should happen after a timeout on a write tool?

Treat it as an unknown commit state. Record the ambiguity, query the system of record using a correlation ID or business key, and retry only if you can establish that no effect occurred.

How long should idempotency records be retained?

Keep them at least as long as every possible replay window: queue retention, client retries, scheduled job retries, and provider webhook delays. High-risk actions such as payments often need longer retention and durable audit evidence.

Is idempotency the same as exactly-once delivery?

No. Most infrastructure offers at-least-once delivery. Idempotency makes repeated delivery safe by ensuring the receiver commits a logical action once and returns the original result to later attempts.

Make unknown explicit

Reliable agent systems do not pretend a timeout means failure. They preserve the operation, reconcile the target, and only then decide whether a retry is safe. Give every write tool a stable identity and a durable receipt, and an agent can be persistent without becoming destructive.

Comments

[Ali Khater]

Strong framing: a timeout is an unknown outcome, not a failed action. One wrinkle worth making explicit is that the internal operation ledger and an external provider cannot share an atomic transaction, so the state machine must survive a crash after external success but before the local receipt is stored. Reconciliation needs its own idempotency and lease-takeover tests too. Treating unknown as a durable state instead of an error class is the key idea here.

[Jo Do]

The distinction between a logical action and an attempt is where most implementations break. One extension I have found useful is to return the operation identity even for an unknown outcome, then prevent the planner from inventing a replacement action while reconciliation is open. Otherwise the infrastructure deduplicates one key perfectly while the model paraphrases the goal into a second valid key. The ledger needs a user-intent boundary above provider idempotency, especially for messages and payments.