DEV Community

satya-anguluri
satya-anguluri

Posted on

The Agent Charged the Customer Twice. The Logs Said Everything Was Fine.

#architecture #backend #java #softwareengineering

The Agent Charged the Customer Twice. Then Reported Success.

I run EngineerPrep, where I build executable labs around production AI-agent failure patterns. This lab reproduces a dangerous distributed-systems failure: an agent mistakes a timeout for a failed payment and charges the customer again.

The setup

An AI agent handles an order workflow.

To complete an order, it calls a payment tool:

capture_payment(
    order_id,
    amount,
    idempotency_key
)
Enter fullscreen mode Exit fullscreen mode

The tool calls a payment processor through a Spring Boot service.

The agent is allowed to retry when a tool call fails. That sounds reasonable—until the tool performs an irreversible side effect.

What the agent saw

For order ORD-1001, the execution trace showed:

AGENT  action=capture_payment
       order=ORD-1001
       amount=4999
       idempotency_key=pay-0001

TOOL   result=TIMEOUT
       message="No response received within 2000ms"

AGENT  decision="Payment attempt failed. Retrying capture."

AGENT  action=capture_payment
       order=ORD-1001
       amount=4999
       idempotency_key=pay-0002

TOOL   result=SUCCESS
       receipt=rcpt-pay-0002

AGENT  final="Payment completed successfully."
Enter fullscreen mode Exit fullscreen mode

From the agent’s perspective, the first attempt failed and the second succeeded.

The workflow ended successfully. The order was confirmed. No application error remained open.

But the customer was charged twice.

What actually happened

The payment processor’s ledger showed:

CAPTURED  ORD-1001  4999  key=pay-0001
CAPTURED  ORD-1001  4999  key=pay-0002
Enter fullscreen mode Exit fullscreen mode

The first payment did not fail.

The processor captured $49.99, but the network connection reset while returning the response. The agent received a timeout even though the financial side effect had already happened.

The agent interpreted:

No response received = payment failed
Enter fullscreen mode Exit fullscreen mode

But the correct interpretation was:

No response received = payment outcome unknown
Enter fullscreen mode Exit fullscreen mode

That difference cost the customer another $49.99.

The agent’s mistake

The agent made two unsafe decisions.

1. It classified TIMEOUT as FAILED

A timeout describes what the caller observed. It does not establish what happened inside the payment processor.

The payment could have:

  • Failed before reaching the processor
  • Reached the processor but failed
  • Completed successfully while its response was lost
  • Still been processing

The agent had insufficient evidence to call the operation a failure.

2. It created a new identity for the retry

The first call used:

idempotency_key=pay-0001
Enter fullscreen mode Exit fullscreen mode

The retry used:

idempotency_key=pay-0002
Enter fullscreen mode Exit fullscreen mode

To the processor, those were two different payment operations. It therefore captured both.

The agent believed it was retrying the same action. The processor saw a new purchase.

Why this is an agent failure

The payment service did not independently initiate the second charge.

The agent did.

It examined an ambiguous tool result, concluded that the payment had failed, constructed another tool call with a new operation identity, and executed it.

Each local decision looked plausible:

  • The tool timed out
  • The workflow still needed payment
  • Retrying transient failures was allowed
  • A new request object received a new identifier

The failure appeared only when those decisions were composed around a side-effecting tool.

That is exactly what makes agent failures dangerous.

What the agent should have done

A timeout on a side-effecting operation should move the workflow into an UNKNOWN state:

TIMEOUT
   ↓
Mark payment outcome UNKNOWN
   ↓
Query payment status using pay-0001
   ↓
CAPTURED → continue without another charge
NOT_FOUND → retry using pay-0001
UNKNOWN → wait or request human review
Enter fullscreen mode Exit fullscreen mode

If a retry is appropriate, it must reuse the original idempotency key:

AGENT  action=capture_payment
       order=ORD-1001
       amount=4999
       idempotency_key=pay-0001
Enter fullscreen mode Exit fullscreen mode

The processor can then recognize the duplicate request and return the original result instead of charging the customer again.

But prompting the agent correctly is not enough

We should not rely on an LLM to remember this rule every time.

The payment tool must enforce it deterministically.

String idempotencyKey =
    paymentOperationRepository.getOrCreateKey(order.id());

processor.capture(order, amount, idempotencyKey);
Enter fullscreen mode Exit fullscreen mode

The key must be persisted for the logical payment operation and reused across:

  • Agent retries
  • Service retries
  • Process crashes
  • Queue redelivery
  • Workflow restarts
  • A second application instance
  • Replanned tool calls

The agent may request another capture, but the application must prevent that request from becoming another charge.

The production lesson

The agent caused the unsafe retry.

The system allowed the agent’s incorrect inference to become a duplicated financial side effect.

Both matter.

“Improve the prompt” is not an adequate safeguard for payments, refunds, emails, infrastructure changes, database writes, or any other action that cannot be safely repeated.

Agentic systems need deterministic boundaries around nondeterministic reasoning:

  • Stable operation identities
  • Provider-side idempotency
  • Explicit UNKNOWN states
  • Outcome reconciliation
  • Restricted retry policies
  • Side-effect ledgers
  • Human escalation when the outcome cannot be established

The rule I took from this

A timeout is not a failure. It is an unknown outcome.

Before an agent repeats a side-effecting tool call, the system must answer:

If the first operation succeeded but its response was lost, what prevents this next call from performing the side effect again?

If the answer is “the agent should realize that,” the system is not safe.

You can reproduce the incident, inspect the agent trace and payment ledger, and test the remediation in the free interactive lab:

engineerprep.io/failure-labs/agent-charged-twice

It’s one of three production incident labs currently available on EngineerPrep.

aiagents #java #springboot #distributedsystems

Top comments (2)

Collapse
 
hannune profile image
Tae Kim •

We actually ran into this before we'd thought through the idempotency scope properly. The retry was exactly the kind of "retry on transient failure" pattern every guide recommends, but it'd generate a new key each time, and the one time a payment processor response got dropped mid-flight we ended up with two charges and three hours of log review. Persisting the key at the order level before the first call sounds obvious in hindsight but it wasn't obvious until we were already explaining it to a confused customer. A timeout doesn't tell you the payment failed, only that the response didn't arrive, and that difference is expensive to figure out while the customer's waiting on a refund.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.