Here's a scenario that becomes interesting as soon as an AI agent starts doing more than just answering questions.
Imagine an agent connected to an order system.
A user says:
"Create a refund for order #48291."
The agent decides to call:
create_refund(
order_id=48291,
amount=249.00
)
The API receives the request and processes it.
But the response times out.
From the agent's perspective, all it knows is:
Agent → API → timeout
So what happens next?
If the agent retries the request, we could end up with:
create_refund()
↓
timeout
↓
retry
↓
create_refund()
Did we create one refund or two?
That's not really an AI hallucination problem. It's a distributed systems problem.
A timeout doesn't necessarily mean the operation failed. The server may have processed the request successfully even though the response never made it back to the client.
The same issue can show up with invoices, payments, orders, emails, support tickets, and other operations that create side effects.
Make the operation idempotent
One way to handle this is with an idempotency key:
create_refund(
order_id=48291,
amount=249.00,
idempotency_key="refund-48291"
)
The server records that key with the result of the operation.
If the same request arrives again, it can return the existing result instead of performing the refund again.
So the flow becomes:
Agent
↓
create_refund()
↓
API
↓
Timeout
↓
Retry with same idempotency key
↓
Existing result
The agent doesn't need to understand the underlying failure mode. The execution layer can make the operation safe to retry.
But not every operation is that simple
Consider:
send_email()
The first request succeeds, but the response times out.
The agent retries.
Now the customer might receive the same email twice.
Or:
charge_card()
A duplicate request could have a much bigger consequence.
This is where I'd start separating agent reasoning from action execution.
Something like:
AI Agent
│
│ Intent
▼
Execution Layer
│
┌───────────┼───────────┐
▼ ▼ ▼
Authorization Validation Idempotency
│ │ │
└───────────┼───────────┘
▼
Tool/API
│
▼
Result
The agent decides what it wants to accomplish.
The execution layer should control how that action is safely performed.
That layer can handle authentication, authorization, validation, retries, idempotency, rate limits, transaction handling, and audit logging.
For high-impact operations, you might also add an approval step before execution.
This changes how I think about agent permissions
Instead of treating tools as simply:
allowed / not allowed
there could be different levels of execution:
READ
↓
SUGGEST
↓
DRAFT
↓
EXECUTE
↓
REQUIRE APPROVAL
Reading an order is one thing.
Creating a refund is another.
Deleting production data or moving money is something else entirely.
The interesting part is that the risk isn't always obvious from the tool name.
A tool like:
update_customer()
might look harmless.
But that update could trigger billing changes, notifications, workflows, or updates in other systems.
So perhaps the real question isn't:
"Can the agent call this tool?"
It's:
"What happens if the agent is wrong, the request times out, or the operation gets executed twice?"
We're getting very good at giving agents more tools and more autonomy.
But once those tools can change things in production, we also need to think about the engineering underneath those tool calls.
The model can be probabilistic.
The execution layer probably shouldn't be.
I'm curious how others are handling this.
If your AI agent can perform write operations against production systems, where do you handle retries and idempotency: inside the tool, the agent framework, or the application layer?
Top comments (2)
Putting idempotency in the execution layer is the right boundary because the model cannot reliably distinguish “failed” from “completed but response lost.” The subtle part is making the key represent semantic intent and preserving it across retries, resumptions, and even a different worker process. I’d also give tools a status lookup so an unknown outcome can be reconciled before another write is attempted. Do you mint the operation ID when the user intent is accepted, before the model plans, or when the executor receives a specific tool call?
At the specific tool call, hashed from tool name plus arguments, not at intent or planning time. Intent is fuzzy, the model can restate the same goal differently across a retry, but the actual call it emits is concrete and hashable. This is roughly how Cadence works in DataGrout.
It also tiers by consequence rather than one blanket policy. Reads tolerate a few identical calls. Writes get one execution, the second identical call returns a confirmation gate instead of re-running. Destructive ops hard-block after the first. So create_refund doesn't get the same tolerance as reading an order.
Your status-lookup point is the harder gap though: the key is scoped to the session. If a worker dies and a new session resumes the same logical operation, nothing ties the new key back to the old one automatically. Real gap, not solved by the tiering.