Your AI agent timed out. Did the action fail?

Here's a scenario that becomes interesting as soon as an AI agent starts doing more than just answering questions.

Imagine an agent connected to an order system.

A user says:

"Create a refund for order #48291."

The agent decides to call:

create_refund(
    order_id=48291,
    amount=249.00
)

The API receives the request and processes it.

But the response times out.

From the agent's perspective, all it knows is:

Agent → API → timeout

So what happens next?

If the agent retries the request, we could end up with:

create_refund()
     ↓
   timeout
     ↓
   retry
     ↓
create_refund()

Did we create one refund or two?

That's not really an AI hallucination problem. It's a distributed systems problem.

A timeout doesn't necessarily mean the operation failed. The server may have processed the request successfully even though the response never made it back to the client.

The same issue can show up with invoices, payments, orders, emails, support tickets, and other operations that create side effects.

Make the operation idempotent

One way to handle this is with an idempotency key:

create_refund(
    order_id=48291,
    amount=249.00,
    idempotency_key="refund-48291"
)

The server records that key with the result of the operation.

If the same request arrives again, it can return the existing result instead of performing the refund again.

So the flow becomes:

Agent
  ↓
create_refund()
  ↓
API
  ↓
Timeout
  ↓
Retry with same idempotency key
  ↓
Existing result

The agent doesn't need to understand the underlying failure mode. The execution layer can make the operation safe to retry.

But not every operation is that simple

Consider:

send_email()

The first request succeeds, but the response times out.

The agent retries.

Now the customer might receive the same email twice.

Or:

charge_card()

A duplicate request could have a much bigger consequence.

This is where I'd start separating agent reasoning from action execution.

Something like:

                AI Agent
                   │
                   │ Intent
                   ▼
            Execution Layer
                   │
       ┌───────────┼───────────┐
       ▼           ▼           ▼
 Authorization  Validation  Idempotency
       │           │           │
       └───────────┼───────────┘
                   ▼
               Tool/API
                   │
                   ▼
                Result

The agent decides what it wants to accomplish.

The execution layer should control how that action is safely performed.

That layer can handle authentication, authorization, validation, retries, idempotency, rate limits, transaction handling, and audit logging.

For high-impact operations, you might also add an approval step before execution.

This changes how I think about agent permissions

Instead of treating tools as simply:

allowed / not allowed

there could be different levels of execution:

READ
  ↓
SUGGEST
  ↓
DRAFT
  ↓
EXECUTE
  ↓
REQUIRE APPROVAL

Reading an order is one thing.

Creating a refund is another.

Deleting production data or moving money is something else entirely.

The interesting part is that the risk isn't always obvious from the tool name.

A tool like:

update_customer()

might look harmless.

But that update could trigger billing changes, notifications, workflows, or updates in other systems.

So perhaps the real question isn't:

"Can the agent call this tool?"

It's:

"What happens if the agent is wrong, the request times out, or the operation gets executed twice?"

We're getting very good at giving agents more tools and more autonomy.

But once those tools can change things in production, we also need to think about the engineering underneath those tool calls.

The model can be probabilistic.

The execution layer probably shouldn't be.

I'm curious how others are handling this.

If your AI agent can perform write operations against production systems, where do you handle retries and idempotency: inside the tool, the agent framework, or the application layer?