My AI Agent Charged a Customer Twice in 11 Seconds. The Bug Was Invisible in Every Test I Wrote.

The email landed at 7:42 on a Tuesday, subject line: charged twice?

One customer, one subscription, two identical charges on their card statement, eleven seconds apart. Same amount, same invoice number, same everything. They were polite about it, which somehow made it worse.

I run a small set of AI agents that handle the back office of my one-person business: invoicing, payment webhooks, provisioning, follow-up emails. It had been humming for months. I'd written tests. I'd watched the logs. And yet somewhere in that pipeline, a single payment had become two, and a real person had to email me to find out why.

This is the post-mortem. Not the cleaned-up version — the actual sequence, including the part where I'd built the exact failure mode myself and called it fault tolerance.

What I saw first

My first instinct was the obvious one: the payment provider must have sent the webhook twice. Duplicate webhooks happen. Providers retry on timeout, on network hiccups, on their own internal glitches. So I pulled the raw request log.

The provider had sent the payment.succeeded event three times.

• Attempt 1: 06:14:02 — my server was mid-deploy, returned a 502.
• Attempt 2: 06:14:09 — processed, charge recorded, customer provisioned.
• Attempt 3: 06:14:13 — processed again, second charge recorded.

Eleven seconds between the two successful charges. That matched the customer's statement exactly.

So the provider did retry. But that's not the bug. Retries are a fact of life in any system that talks over a network. The bug was that my code treated each webhook as a brand-new instruction instead of asking the only question that matters: have I already done this?

The actual root cause

My handler looked, in hindsight, embarrassingly reasonable:

@app.post("/webhooks/payment")
def handle_payment(event):
    if event["type"] == "payment.succeeded":
        invoice = get_invoice(event["invoice_id"])
        charge_customer(invoice)      # <- ran every time
        provision_account(invoice)
        send_receipt_email(invoice)
        return {"status": "ok"}

There was no record of which events I'd already handled. Every delivery was treated as fresh work. Attempt 2 and attempt 3 were different HTTP requests with the same payload, and my code had no way to tell them apart. So it did the work twice.

The uncomfortable part: I had deliberately made the handler forgiving. I'd added the retry-friendly response codes, I'd made sure it wouldn't crash on malformed input, I'd logged everything. I'd built for reliability and, in doing so, built a machine that happily repeats the most expensive thing you can repeat — taking someone's money.

Why every test passed

This is the part I keep coming back to, because it's the lesson that generalizes.

My tests all used unique, well-formed events. The test suite fired one webhook, asserted one charge, moved on. Nothing in the test environment ever delivered the same event twice, because nothing retried. There was no flaky network, no 502, no second attempt. The exact condition that causes the double-charge simply never occurs in a test that runs once and succeeds.

I had tested the happy path thoroughly and never once tested the one scenario distributed systems guarantee: the same message arriving more than once.

That gap has a name. In messaging terms, my pipeline offered at-least-once delivery (the event will definitely arrive, possibly more than once) but I had written the consumer as if it were exactly-once (each event arrives precisely one time). At-least-once is what you actually get from webhooks, queues, and retries. Exactly-once is what you have to build on top of it, on the consumer side. I'd assumed the delivery guarantee instead of engineering for it.

The fix

The fix is small enough to be almost insulting, which is the point.

processed = set()  # backed by a real store in production

@app.post("/webhooks/payment")
def handle_payment(event):
    event_id = event["id"]                 # provider-guaranteed unique per event
    if event_id in processed:
        return {"status": "already_seen"}  # ack, do nothing
    processed.add(event_id)

    if event["type"] == "payment.succeeded":
        invoice = get_invoice(event["invoice_id"])
        charge_customer(invoice)
        provision_account(invoice)
        send_receipt_email(invoice)
        return {"status": "ok"}

Three things matter here:

1. Use the provider's event ID, not the invoice ID. The event ID is unique per delivery attempt's underlying event — the provider guarantees attempts 2 and 3 share the event ID. Keying off it collapses all retries of the same event into one.
2. Record the ID before doing the work. If you record it after, a crash between "did the work" and "wrote the record" recreates the bug.
3. Acknowledge duplicates gracefully. Returning a success-shaped response for an already-seen event stops the provider from retrying again, which would just add noise.

In production the set is a durable store (a table with a unique constraint on event_id, or a Redis SET with a TTL) so it survives restarts. The unique constraint is the real hero — it makes the "have I done this?" check atomic, so two simultaneous retries can't both slip through.

I refunded the duplicate charge within the hour and sent the customer a genuinely embarrassed apology. They were gracious. I didn't deserve it.

The rule I now apply everywhere

One sentence: every consumer of an external event must be idempotent, because every external delivery is at-least-once.

Idempotent means running the same input twice produces the same result as running it once. It's the single most important property for any automation that touches the outside world — payments, emails, API calls, provisioning, anything with a side effect you can't take back.

The concrete checklist I now run before any agent or webhook handler ships:

• Find the natural dedupe key. Providers almost always give you one (event ID, idempotency key, message ID). If they don't, hash the payload.
• Check-and-record atomically, ideally with a unique constraint in a database rather than in-memory state.
• Do the side effect after the dedupe check, not before.
• Test the duplicate, not just the event. My test suite now fires the same webhook twice and asserts exactly one charge. That test would have caught this on day one.
• Make the expensive thing the guarded thing. If only one step has a real-world cost, that's the step the dedupe gate protects.

The honest part

I want to be straight about what I got wrong, because the fix makes it sound like I just forgot a line. I didn't forget a line. I had a wrong mental model. I treated retries as an edge case to be survived rather than the default to be designed for, and I wrote tests that confirmed my assumption instead of challenging it. The tests were green right up until a customer's bank statement wasn't.

I also hadn't reconciled. There was no end-of-day check comparing "charges I made" against "charges the provider recorded." A five-line reconciliation query would have flagged the duplicate that same evening instead of waiting for a human to notice. I've added it. It runs nightly and it's already caught two near-misses in other handlers.

If you run agents that do anything irreversible — charge a card, send an email, delete a row, call a paid API — assume today that the same instruction will arrive twice. Not as paranoia. As a spec.

Because somewhere out there, a provider is already retrying.

---

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund. (Enter the code in the discount field on the checkout page itself — it's a text box, not a URL parameter.)

Comments

[Marcus Kim]

Recording the provider event ID before charging closes the duplicate window, but it opens the opposite one: a crash after inserting event_id and before charge_customer means every retry gets acknowledged while the work remains incomplete. The unique constraint and duplicate-delivery test are solid; for a money-moving side effect, I'd pair them with a durable state machine or outbox, pass a stable idempotency key to the payment provider, and use the nightly reconciliation to catch stuck states. One timeline detail also needs tightening: 06:14:09 to 06:14:13 is four seconds, not eleven.