The Agent Retried the HTTP Call and Double-Charged the Card

#aiagents #testing #api #engineering

The test was flaky in the way payment tests always are: POST /charge sometimes took 820ms against a 500ms CI timeout. Fail one out of ten runs. The agent was asked to fix it.

It diagnosed the timeout, read the handler, and wrapped the upstream call in a retry loop. Two attempts, exponential backoff, a jitter. The test went green. The diff looked reasonable.

It forgot that the upstream call was the charge itself.

The block the agent retried

Here is roughly what the handler did, before the agent touched it:

async function charge(req) {
  const payment = await db.payments.create({ id: req.body.idempotency_key, status: 'pending' });
  const result = await gateway.charge(req.body.amount, req.body.payment_method);
  await db.payments.update(payment.id, { status: 'paid', external_id: result.id });
  return result;
}

The agent's retry looked like this:

async function charge(req) {
  const payment = await db.payments.create({ id: req.body.idempotency_key, status: 'pending' });
  const result = await retry(() => gateway.charge(req.body.amount, req.body.payment_method));
  await db.payments.update(payment.id, { status: 'paid', external_id: result.id });
  return result;
}

The test went green because on the slow run, the first attempt timed out client-side but the gateway had actually received the charge. The second attempt went through. Two charges, two rows in the gateway dashboard, one happy CI run.

The question the agent didn't ask

Before wrapping any block in a retry, the cheap human instinct is to ask one thing: is this block safe to replay?

For GET /invoice/42, yes. For a read, a query, a lookup — replay is free. For a POST /charges, a queue publish, a webhook dispatch, a sendEmail, a file upload — replay is the bug.

The agent doesn't know this unless the contract says so. It reads the code, sees a flaky network call, and reaches for the generic fix. The generic fix is correct for reads and catastrophic for writes.

The pattern repeats everywhere:

Retry around a webhook dispatch → the receiver gets two events.
Retry around a queue publish → two consumers process the same job.
Retry around a sendEmail → the customer gets two receipts.
Retry around a DB INSERT without a unique constraint → two rows.

Each one looks like a fix in the PR. Each one is a louder version of the original failure.

What the correct fix actually looked like

There are two legitimate fixes, and the agent reached for neither.

Option one: widen the timeout. The upstream SLA was 600ms. CI timeout was 500ms. The fix was to set the client timeout to 2000ms and leave the handler alone. A flaky test that fails at 820ms against a 500ms deadline is a test that has the wrong deadline.

Option two: make the call idempotent and retry. The gateway already accepted an Idempotency-Key header. The correct change was to send the client-generated idempotency_key on every attempt, so a retried charge maps to the same payment on the gateway side. That is a real fix. It requires knowing the gateway contract, reading the API, and threading the key through the client.

The agent picked a third, easier option: retry the side effect and hope the network was the only problem. It wasn't.

How to catch this in review

A few cheap gates I now look for in any agent-authored retry:

What does the block do? If it writes, charges, publishes, or sends, the retry needs an idempotency key or a dedup check. Point this out in review before merging.
What does the test assert? A green flaky test after a retry loop means the test is now exercising the retry, not the handler. Ask for a test that simulates a first-attempt timeout and a successful second attempt, and assert that the side effect fired exactly once.
Where is the key generated? If the idempotency key is generated inside the retry loop, every attempt gets a new key and the retry still double-charges. The key must be created upstream of the retry boundary.
What does the contract say? OpenAPI / upstream API docs are the source of truth for whether a POST is idempotent and which header carries the key. If the agent had read the spec before patching, it would have found Idempotency-Key documented in five minutes.

The tooling angle

This is the kind of drift that local-first tooling catches cheaply. When you keep the upstream OpenAPI spec as the source of truth and diff generated clients against it, a missing Idempotency-Key header shows up as a contract violation in review — not as a production double-charge six weeks later. We wound up building that flow into Powerduck because we kept having this exact conversation: the agent reads the code, not the contract, and the contract is where the idempotency rule lives.

The lesson isn't "don't let agents write retries." It's "before you retry a write, read the contract, generate the key on the client, and prove in a test that the side effect fires exactly once." Everything else is theater.

DEV Community