340 customers were charged twice. No one touched the code.
An e-commerce platform processes orders through a simple pipeline: EventBridge (serverless event bus, invokes Lambda asynchronously — Lambda retries up to 2 times on any thrown error) receives an order.placed event, routes it to a Lambda (serverless compute), which charges the payment and writes the confirmed order to DynamoDB (managed NoSQL database). The architecture ran in production for 4 months — 8,000 orders per day, zero duplicate charges, zero incidents.
The platform runs a promotional campaign. Traffic spikes 40× over 2 hours. 14,000 orders in one window.
Post-campaign, the payments team pulls the report. 340 orders show two charges. CloudWatch (AWS monitoring service) shows EventBridge delivered each event exactly once. The Lambda has no retry logic. No alerts fired during the campaign.
The engineer on-call checks the Lambda logs. Every invocation shows a successful payment charge. Then — on 340 of them — a DynamoDB write error at the very end. Then a second invocation of the same function, for the same order, a few seconds later.
The second invocation charges the card again.
No code changed. No bug was introduced. The system behaved exactly as designed.
What is the root cause?
A) EventBridge (serverless event bus, at-least-once delivery) delivered duplicate events during the spike — at-least-once delivery means duplicates increase under high throughput
B) Lambda (serverless compute, retries async invocations up to 2 times on any thrown error) retried failed invocations after the DynamoDB write threw — the payment charge ran again on retry because the function is not idempotent
C) DynamoDB (managed NoSQL database, can throttle writes under sudden traffic spikes) throttling caused a race condition between two concurrent Lambda invocations processing the same event
D) The payment provider has its own retry logic — the platform's architecture is correct, the provider caused the double charge
Answer in the comments.
Top comments (5)
The answer is B.
EventBridge (serverless event bus) invokes Lambda (serverless compute) asynchronously. When a Lambda function invoked asynchronously throws any error — for any reason — Lambda automatically retries the invocation up to 2 additional times. This is documented, default behavior. The retries are Lambda’s responsibility, not EventBridge’s. EventBridge did its job: it delivered the event once.
Here is the exact sequence for each of the 340 duplicate charges:
Lambda is invoked. It validates the order. It calls the payment provider — the charge succeeds. It then calls DynamoDB (managed NoSQL database) to write the confirmed order. Under the 40× traffic spike, DynamoDB is throttling writes and returns a ProvisionedThroughputExceededException. Lambda sees a thrown error, marks the invocation as failed, and schedules a retry.
On retry, Lambda runs the full function from the beginning. It calls the payment provider again. The payment provider has no record of a previous charge — the function passed no idempotency key. The card is charged a second time. The DynamoDB write succeeds on retry. The order appears in the database once. The card was charged twice.
Two fixes:
Option 1 — Pass a stable idempotency key to the payment provider on every charge request. Use order_id. If the provider receives a second charge request with the same key, it returns the result of the original charge without processing a new one.
Option 2 — Before calling the payment provider, check DynamoDB for an existing payment record keyed on order_id. If one exists, skip the charge.
Lambda retries are a feature, not a failure. Every Lambda function invoked asynchronously must be safe to run multiple times.
D — The payment provider received two separate, valid charge requests with no idempotency key. Honoring both is correct behavior on their part. The platform sent two requests. That is the platform’s bug to fix.
C — Lambda’s async event source mapping invokes the function once per event and only schedules a retry after the first invocation completes with an error. The two invocations are separated by seconds and strictly sequential. No concurrency, no race.
A — EventBridge does guarantee at-least-once delivery. But CloudWatch (AWS monitoring service) confirms each event was delivered exactly once. The pattern — successful payment, DynamoDB error, second invocation seconds later — is a Lambda retry signature, not an EventBridge duplicate. An EventBridge duplicate would produce two near-simultaneous invocations, not a sequential gap with an error in between.
Also, it would mean a lot to me if you could support my content and stay in touch 🙏
YouTube: youtube.com/@system-design-lab
LinkedIn: linkedin.com/in/joud-awad/
Medium Blog: joudwawad.medium.com/
Substack: joudawad.substack.com/