DEV Community: satya-anguluri

The Agent Charged the Customer Twice. The Logs Said Everything Was Fine.

satya-anguluri — Sun, 13 Sep 2026 21:14:26 +0000

The Agent Charged the Customer Twice. Then Reported Success.

I run EngineerPrep, where I build executable labs around production AI-agent failure patterns. This lab reproduces a dangerous distributed-systems failure: an agent mistakes a timeout for a failed payment and charges the customer again.

The setup

An AI agent handles an order workflow.

To complete an order, it calls a payment tool:

capture_payment(
    order_id,
    amount,
    idempotency_key
)

The tool calls a payment processor through a Spring Boot service.

The agent is allowed to retry when a tool call fails. That sounds reasonable—until the tool performs an irreversible side effect.

What the agent saw

For order ORD-1001, the execution trace showed:

AGENT  action=capture_payment
       order=ORD-1001
       amount=4999
       idempotency_key=pay-0001

TOOL   result=TIMEOUT
       message="No response received within 2000ms"

AGENT  decision="Payment attempt failed. Retrying capture."

AGENT  action=capture_payment
       order=ORD-1001
       amount=4999
       idempotency_key=pay-0002

TOOL   result=SUCCESS
       receipt=rcpt-pay-0002

AGENT  final="Payment completed successfully."

From the agent’s perspective, the first attempt failed and the second succeeded.

The workflow ended successfully. The order was confirmed. No application error remained open.

But the customer was charged twice.

What actually happened

The payment processor’s ledger showed:

CAPTURED  ORD-1001  4999  key=pay-0001
CAPTURED  ORD-1001  4999  key=pay-0002

The first payment did not fail.

The processor captured $49.99, but the network connection reset while returning the response. The agent received a timeout even though the financial side effect had already happened.

The agent interpreted:

No response received = payment failed

But the correct interpretation was:

No response received = payment outcome unknown

That difference cost the customer another $49.99.

The agent’s mistake

The agent made two unsafe decisions.

1. It classified `TIMEOUT` as `FAILED`

A timeout describes what the caller observed. It does not establish what happened inside the payment processor.

The payment could have:

Failed before reaching the processor
Reached the processor but failed
Completed successfully while its response was lost
Still been processing

The agent had insufficient evidence to call the operation a failure.

2. It created a new identity for the retry

The first call used:

idempotency_key=pay-0001

The retry used:

idempotency_key=pay-0002

To the processor, those were two different payment operations. It therefore captured both.

The agent believed it was retrying the same action. The processor saw a new purchase.

Why this is an agent failure

The payment service did not independently initiate the second charge.

The agent did.

It examined an ambiguous tool result, concluded that the payment had failed, constructed another tool call with a new operation identity, and executed it.

Each local decision looked plausible:

The tool timed out
The workflow still needed payment
Retrying transient failures was allowed
A new request object received a new identifier

The failure appeared only when those decisions were composed around a side-effecting tool.

That is exactly what makes agent failures dangerous.

What the agent should have done

A timeout on a side-effecting operation should move the workflow into an UNKNOWN state:

TIMEOUT
   ↓
Mark payment outcome UNKNOWN
   ↓
Query payment status using pay-0001
   ↓
CAPTURED → continue without another charge
NOT_FOUND → retry using pay-0001
UNKNOWN → wait or request human review

If a retry is appropriate, it must reuse the original idempotency key:

AGENT  action=capture_payment
       order=ORD-1001
       amount=4999
       idempotency_key=pay-0001

The processor can then recognize the duplicate request and return the original result instead of charging the customer again.

But prompting the agent correctly is not enough

We should not rely on an LLM to remember this rule every time.

The payment tool must enforce it deterministically.

String idempotencyKey =
    paymentOperationRepository.getOrCreateKey(order.id());

processor.capture(order, amount, idempotencyKey);

The key must be persisted for the logical payment operation and reused across:

Agent retries
Service retries
Process crashes
Queue redelivery
Workflow restarts
A second application instance
Replanned tool calls

The agent may request another capture, but the application must prevent that request from becoming another charge.

The production lesson

The agent caused the unsafe retry.

The system allowed the agent’s incorrect inference to become a duplicated financial side effect.

Both matter.

“Improve the prompt” is not an adequate safeguard for payments, refunds, emails, infrastructure changes, database writes, or any other action that cannot be safely repeated.

Agentic systems need deterministic boundaries around nondeterministic reasoning:

Stable operation identities
Provider-side idempotency
Explicit UNKNOWN states
Outcome reconciliation
Restricted retry policies
Side-effect ledgers
Human escalation when the outcome cannot be established

The rule I took from this

A timeout is not a failure. It is an unknown outcome.

Before an agent repeats a side-effecting tool call, the system must answer:

If the first operation succeeded but its response was lost, what prevents this next call from performing the side effect again?

If the answer is “the agent should realize that,” the system is not safe.

You can reproduce the incident, inspect the agent trace and payment ledger, and test the remediation in the free interactive lab:

engineerprep.io/failure-labs/agent-charged-twice

It’s one of three production incident labs currently available on EngineerPrep.

aiagents #java #springboot #distributedsystems

I built an AI engineering course where every lesson is a runnable Spring Boot project (Ollama, no API key)

satya-anguluri — Thu, 27 Aug 2026 19:19:03 +0000

Hi everyone — I’m a Java/Spring Boot engineer with 17+ years of experience, and I’ve spent the past several months building EngineerPrep.

I kept running into the same problem while learning AI engineering: many courses either stay at the conceptual level or demonstrate everything inside notebooks. They explain what tokens, embeddings, RAG and agents are, but not how these pieces behave inside a production application.

So I built the kind of learning path I wanted:

73 focused AI engineering lessons

Short chapters instead of long video lectures

Visual walkthroughs of what happens inside the system

Production incidents and failure scenarios

Hands-on implementation labs

Runnable Maven projects with local Ollama support

Optional OpenAI and Amazon Bedrock configurations

An AI mentor that answers within the context of the current lesson

The curriculum progresses through:

LLM Foundations

Prompt Engineering

Structured Output and Validation

RAG and Embeddings

AI Memory

Agents and Tool Calling

Model Evaluation

AI Security

Production AI Systems

The complete LLM Foundations module is free: 15 lessons, hands-on labs and a starter project that runs locally with Ollama. No paid AI API or credit card is required.

EngineerPrep is intentionally focused: one structured path for working software engineers who want to understand how AI systems are designed, implemented and debugged in production—especially with Java and Spring Boot.

You can try it here:

https://engineerprep.io

I would genuinely appreciate feedback on three things:

Does the first lesson make the value clear quickly?

Is the lesson → incident → project structure useful?

What would prevent you from completing the free module?

I built this independently, so direct criticism is welcome. It will help me decide what to improve next.

Hi everyone. This is Satya , java developer from the past 18 years. Interested very much in AI stuff.

satya-anguluri — Tue, 14 Jul 2026 21:27:39 +0000

Capstead

satya-anguluri — Mon, 13 Jul 2026 16:36:02 +0000

I built Capstead: A Governance & Observability Layer for AI Capabilities in Spring Boot

As more teams integrate AI into their Spring Boot applications, I kept running into the same questions:

Which AI capabilities does our application expose?
Who owns each capability?
How much does each capability cost?
Which capabilities are failing or exceeding latency budgets?

Frameworks like Spring AI provide excellent model integrations and metrics, but I wanted visibility at the business capability level rather than just individual model calls.

That's why I built Capstead, an open-source governance and observability control plane for AI capabilities in Spring Boot.

With a simple @Capability annotation (or a bodyless @CapabilityClient interface), Capstead provides:

📋 Live capability catalog
💰 Per-capability cost attribution, token usage, latency, and success rate
🎯 Daily budget enforcement
🌳 Durable execution history with parent/child execution trees
🤖 MCP export so capabilities can be exposed as agent tools
🔌 Provider-neutral support (Spring AI, LangChain4j, custom SDKs, or any HTTP client)

The goal is to provide a governance layer on top of existing AI integrations without changing how developers build AI applications.

I'd love feedback from engineers building AI-enabled systems:

Is this a problem you've encountered?
What governance or observability features are you missing today?
What integrations would you like to see next?

GitHub: https://github.com/satya-anguluri/capstead

I'd really appreciate your thoughts and suggestions!