An AI agent hit a timeout. It retried the payment API call. The first request had already succeeded.
Every monitoring dashboard stayed green.
This is a silent failure class that traditional SRE metrics miss completely. We ask "did the request return HTTP 200?" The question that matters for agents with write access is:
Did the agent execute the same irreversible action twice?
When an LLM agent operates under non-deterministic retries, status codes stop telling you the truth about what happened.
Why agents make this worse
Normal retry logic is boring and predictable. Agents are not:
Retries stack. The HTTP client retries, the orchestration framework retries, and the model decides to "try again". Each layer assumes it is the only one.
Retries get rephrased. After a timeout the agent may re-plan and re-issue the action with reordered or slightly different parameters, so no lower layer recognizes it as a duplicate.
A timeout means "unknown", not "failed". The agent can't tell if the write landed, and it tends to guess "do it again."
Three rules for critical write paths
- Deterministic action keys
Fingerprint the action before dispatching it:
key = hash(action_name + sorted_parameters + caller_scope)
If the payload matches, the agent cannot emit a duplicate mutation. Sort keys and normalize types so logically identical payloads hash the same. Include the caller scope (tenant, session, workflow run) so unrelated users don't collide, and an intent ID if the same action can legitimately repeat.
- Sliding TTL deduplication
Block repeated signatures inside a bounded window (say, 300 seconds).
Don't swallow duplicates silently. That hides the bug behind them. Intercept the call, reject it with an explicit error, and increment a counter. "Sliding" means each blocked attempt refreshes the window, so an agent stuck in a loop stays blocked.
- Duplicate Action Rate (DAR) as a core SLI DAR = blocked_duplicate_actions / total_action_attempts
If DAR spikes, don't blame the agent. It is almost always compensating for unhandled timeouts, slow database locks or upstream lag. DAR tends to show you those problems before your customers do.
A minimal gate in Python
python
import hashlib
import json
import threading
import time
class DuplicateActionError(Exception):
"""Raised when an identical action was already submitted inside the window."""
class DedupGate:
def init(self, ttl_seconds: float = 300, clock=time.monotonic):
self.ttl = ttl_seconds
self._clock = clock
self._seen: dict[str, float] = {}
self._lock = threading.Lock()
self.allowed = 0
self.blocked = 0
@staticmethod
def action_key(action_name: str, params: dict, caller_scope: str) -> str:
canonical = json.dumps(
{"a": action_name, "p": params, "s": caller_scope},
sort_keys=True, separators=(",", ":"), default=str,
)
return hashlib.sha256(canonical.encode()).hexdigest()
def acquire(self, action_name: str, params: dict, caller_scope: str) -> str:
key = self.action_key(action_name, params, caller_scope)
now = self._clock()
with self._lock:
# evict expired entries
self._seen = {k: t for k, t in self._seen.items() if now - t < self.ttl}
if key in self._seen:
self._seen[key] = now # sliding: refresh the window
self.blocked += 1
raise DuplicateActionError(
f"'{action_name}' already submitted within {self.ttl:.0f}s"
)
self._seen[key] = now
self.allowed += 1
return key
def release(self, key: str) -> None:
"""Call only on a *confirmed* failure, so a legitimate retry can proceed."""
with self._lock:
self._seen.pop(key, None)
@property
def dar(self) -> float:
total = self.allowed + self.blocked
return self.blocked / total if total else 0.0
Using it around a payment call:
python
gate = DedupGate(ttl_seconds=300)
def charge(order_id: str, amount_cents: int, tenant: str):
params = {"order_id": order_id, "amount_cents": amount_cents}
key = gate.acquire("create_charge", params, caller_scope=tenant)
try:
# also pass key as the provider's idempotency key if it supports one
return payments.create_charge(**params, idempotency_key=key)
except PaymentDeclined:
gate.release(key) # definitive failure: safe to retry
raise
# on a timeout: do NOT release. Reconcile by looking up the charge instead.
The agent gets a typed error it can reason about ("already submitted, check status") instead of looping.
Gotchas
Ambiguous timeouts. Don't release the key on a timeout. Reconcile against the downstream system first. Release only on definitive failure.
In-memory state doesn't scale out. This gate lives in one process. With multiple workers, back it with a shared atomic store, for example Redis SET key value NX EX 300, or a DynamoDB conditional write.
Defense in depth. Forward the action key as the provider's idempotency key where available. The gate protects you from your agent, and the provider key protects you from everything else.
Legitimate repeats. Two deliberate $10 transfers are not duplicates. Put an intent ID into the key.
Where this lives
I integrated this deduplication gate into the open-source agentsre telemetry framework so DAR sits next to the other signals an agent emits: https://lnkd.in/e7Qpux2j
Discussion
How is your team handling idempotency and duplicate execution as agents move from read-only assistants to production write systems? Enforced at the agent layer, the API layer, or both? I'd like to hear the war stories.
Top comments (2)
The release-on-failure branch is where this gets tricky in practice.
If the downstream service rejects the payload with a hard schema validation error or an explicit card decline, releasing the action key lets an agent fix a parameter and retry. But if the failure comes back as a connection reset or a 502 from an edge proxy, you hit the same ambiguity as a raw timeout: the gateway might have dropped the socket after passing the mutation to the database. Releasing the gate on every caught exception turns proxy disconnects right back into double charges.
Passing the hash straight into the upstream provider as the idempotency header is what actually saves you there. When the gateway and the external API share that same token, a retry after a dropped socket collapses into the original record rather than creating a second one.
tr.ee/dev-to