The Ledger That Charged Twice: S3 Lambda Idempotency

In the counting-houses of old Venice, a merchant's quiet terror was not the thief in the night. It was the honest scribe who wrote the same payment into the ledger twice. The gold had moved once; the book said twice. By the time anyone noticed, the customer was furious, the accounts would not balance, and no one could say when the phantom entry had crept in.

Venice has a fair claim to being where careful bookkeeping grew up. An early printed account of double-entry bookkeeping, Luca Pacioli's Summa de arithmetica, came off a Venetian press in 1494, and the whole point of the method was to make exactly this kind of mistake visible. Two columns, each entry mirrored by another, so that a stray line could not hide.

Six centuries later I rebuilt that same mistake on AWS, by accident, in a handful of lines of Python.

My counting-house is modern: a file lands in Amazon S3, a Lambda wakes and charges a customer, the result is written to a DynamoDB ledger. I placed one order. One. The ledger recorded two charges. Nothing crashed, no red ink in the logs, no bell rang. The second charge simply appeared, the way the scribe's phantom entry did, and the only witness was a number in a table that read 2 where it should have read 1.

This is the story of that phantom charge: why S3-triggered Lambdas conjure it, why the remedy everyone reaches for is itself half-cursed, and the bookkeeping discipline, older than computers, that gets you most of the way to banishing it (and the one gap it cannot close alone). Everything here runs on your own machine first, then on real AWS, and the whole counting-house is on GitHub.

If you only have a minute, here is the whole tale in four lines:

• An S3-triggered Lambda is an at-least-once system twice over.
• A handler that does a side effect and then errors will run that side effect again on the retry.
• The usual fix, an idempotency key, has a quiet failure mode of its own.
• Record your work as in progress, then done, and pass the key to whoever takes the money.

Two truths that summon the phantom

The double charge is not a bug in your logic. It is two documented AWS behaviours meeting at the worst possible moment.

Truth one: S3 event notifications are at-least-once. The S3 documentation says it without flinching: notifications are designed to be delivered at least once, and on rare occasions S3's own retry mechanism can produce a duplicate for the same object event. It recommends you account for that. Rare, non-deterministic, but real. The messenger may knock twice.

Truth two: Lambda retries failed async invocations. This is the one that actually bites. An S3 trigger invokes your function asynchronously, and async Lambda retries a failed invocation up to two more times by default (MaximumRetryAttempts, range 0 to 2). The first retry comes after about a minute, the second about two minutes after that. So if your handler does its work and then errors, on something as ordinary as a downstream timeout, Lambda calls it again (after about a minute, then about two more), and the work runs again.

Here is what that looks like on the clock:

The side effect runs before the error, so the retry repeats it. The ledger climbs to 2. The customer pays twice for one upload, and your logs are spotless, because from Lambda's vantage nothing is wrong; it is dutifully retrying a failure, exactly as designed.

What "at-least-once" really promises

It helps to be fair to the machinery, because at-least-once delivery is not a defect. It is a deliberate trade.

A system that delivers a message at most once can lose it. A system that delivers it exactly once needs the sender and the receiver to agree on something that, across a network, nobody can fully guarantee. A system that delivers it at least once never loses it, at the price of occasionally handing you the same message twice. For something like "a file arrived, go and process it," losing the event is far worse than seeing it twice, so the platform picks the second option and hands the duplicate-handling problem to you.

That is the whole contract. S3 and Lambda promise you will not miss the event. They do not promise you will see it once. Your handler has to be written for a world where the same event can arrive again, minutes later, in a different container, with no memory that it already ran.

The word for a handler that survives that world is idempotent: running it twice has the same effect as running it once. Setting a value to 5 is idempotent. Adding 1 to a counter is not. Charging a customer is, unless you do something about it, very much not.

Charging a card is only the most dramatic example. The same shape hides in any handler that sends an email, opens a support ticket, decrements stock, posts to a chat channel or starts a shipment. If repeating it would embarrass you, it needs an answer to the question this article is about.

The counting-house, in AWS terms

Here is the stack I used to watch the sin happen on real infrastructure. One SAM template, five moving parts:

• An S3 bucket, versioned, that fires on every ObjectCreated event.
• A Lambda function that reads two settings from its environment: HANDLER_MODE (which pattern to run) and FAIL_ONCE (whether to force the first attempt to fail).
• A DynamoDB table that holds both the charge counter and the idempotency records.
• An SQS queue, wired in as the function's on-failure destination, which is where an event goes once Lambda has run out of retries.
• The IAM policies that let the function talk to the table and the queue, and nothing else.

One honest footnote on that queue: once its retries are used up, Lambda simply discards the event unless you gave the function somewhere to put it. In the repo's template that somewhere is an SQS queue configured as an on-failure destination. I call it the DLQ throughout, because that is the job it does.

The naive scribe, in code

Here is the handler that re-creates the Venetian mistake. It charges, then (to force the failure deterministically for the demo) raises:

def naive_handler(event, table):
    charge(table)                       # the side effect: money moves
    raise RuntimeError("downstream failed; Lambda will retry")

Lambda retries. charge() runs on every attempt. One upload, two, sometimes three charges. This is the entire bug, and it is one of the most common in production serverless, precisely because it never announces itself.

Notice how ordinary it is. There is no race condition, no clever concurrency, no exotic failure. The handler did exactly what it says. The platform did exactly what it says. The bug lives in the gap between the two sentences.

Proving it before spending a coin

Before touching AWS, I built a local simulator so the whole tale runs offline. It models the one thing that matters, a side effect that runs, then an error, then a retry, and lets four handlers face it. It uses a tiny fake DynamoDB table that supports the one feature the fix depends on, a conditional write that fails if the item already exists, and it builds S3 events shaped exactly like the real ones, including the fields the idempotency keys rely on: the object key, eTag, versionId and sequencer.

Read the top block. The naive scribe double-charges. Three different fixes each hold the transient case to a single charge. Looks solved. Any of them would do. Hold that thought, because the bottom block is where one of those fixes quietly betrays you.

And because a phantom you cannot summon on command is just a ghost story, every row above is pinned by a test:

The remedy everyone reaches for

The standard cure for "my handler runs twice" is an idempotency key: a unique name for this exact piece of work, recorded with a conditional write so the second attempt recognises itself as a duplicate and stands down.

What makes a good key for an S3 event? Two solid designs, and the repo ships both:

• Object key + versionId. Unique and clean, but requires bucket versioning.
• A hash of object key + eTag + sequencer. Works without versioning.

A trap worth marking on the map: the sequencer field on an S3 event lets you order events for one object, but it does not deduplicate them. A genuine duplicate delivery of one event carries the same object key, versionId and sequencer, which is exactly why they make a good key. But the same object uploaded twice produces two distinct events with two sequencers. Deduplication is your ledger's job, not S3's.

The pattern is "claim the key, then do the work," using a DynamoDB conditional write that fails if the key already exists:

def claim_first(event, table, key_fn):
    idem_key = key_fn(object_of(event))
    try:
        table.put_item(
            Item={"pk": idem_key},
            ConditionExpression="attribute_not_exists(pk)",
        )
    except ConditionalCheckFailed:
        return            # already recorded, skip the charge
    charge(table)         # first time only

Run that against the transient failure and it holds: one charge, the retry safely skipped. Ship it?

Not yet. First, one more reason the conditional write is the right tool, and then turn the ledger to the next page.

Why one atomic write is the whole trick

You might be tempted to write the claim as two steps: read the table to see whether the key exists, then write it if it does not. Do not. Two duplicates that arrive at the same moment will both read "not there," both write, and both charge. The gap between the read and the write is exactly the size of the bug.

A conditional put_item closes that gap, because DynamoDB evaluates the condition and performs the write as one atomic operation on a single item. Exactly one of two concurrent callers wins. The loser gets a ConditionalCheckFailed and knows it came second.

No lock, no queue, no clock. That is also why the key lives in DynamoDB and not in the function's memory, a lesson I will come back to, because I learned it the hard way.

Choosing a key without cutting yourself

The key looks like a detail. It is where most of the quiet bugs live, so it is worth a few rules.

Derive it from the work, never from the attempt. A retry carries the same event, so anything computed from the event (the object key, versionId, eTag, sequencer) is identical on every attempt. Anything computed from the clock or a fresh random id is different every time, and a key that changes per attempt deduplicates nothing.

Do not mistake identical content for the same event. An eTag is derived from the object's content, so the same bytes uploaded twice can share one. That is why the second recipe mixes in the sequencer: the content may repeat, the event does not.

Mind what versioning off does. The versionId-based key falls back to the string null when the bucket is not versioned. Two uploads of the same file name then collide on one key, and the second, perfectly legitimate upload is skipped as a duplicate. If you cannot turn versioning on, use the other recipe.

Scope the key to the operation. If one upload triggers two side effects, say a charge and a receipt email, give each its own key prefix. Share a single key between them and whichever runs second will conclude it has already been done.

The curse in the cure

Look again at the bottom of that sim table, the poison scenario, where the charge itself fails on every attempt (a dead payment provider, say). Here the three patterns stop agreeing, and the disagreement is the heart of this piece. First, the same story attempt by attempt:

And here are the tallies, computed by the simulator rather than typed in by hand:

• naive triple-charges. Plainly broken.
• claim-first shows a count of 1, and that looks fine until you trace it. It claimed the key, charged the customer, then the operation failed and reported an error. The retry is deduped away. So the money moved, but the system believes the operation failed. Your ledger and your reality now disagree, and nothing raised a hand. In a counting-house, this is the subtler, costlier error: not a visible double entry, but a silent mismatch no auditor was told to look for.
• robust shows a count of 0. It refused to leave money in a half-finished state and left the record in_progress, so the failure ends up in a dead-letter queue where a human can see it.

Think about what each outcome costs you in practice. A double charge produces an angry customer and a refund. A silent mismatch produces a month-end reconciliation that does not reconcile, and a hunt through logs that say everything went fine. The first is loud and cheap. The second is quiet and expensive.

The robust discipline is claim as in-progress, do the work, then mark complete, the digital echo of a scribe who pencils an entry, waits for the gold to actually land, and only then inks it in:

def robust_handler(event, table):
    o = object_of(event)
    idem_key = key_a(o)

    record = table.get(idem_key)
    if record and record["status"] == "done":
        return                                   # already completed, skip

    if not record:
        try:
            table.put_item(                      # pencil it in: in_progress
                Item={"pk": idem_key, "status": "in_progress"},
                ConditionExpression="attribute_not_exists(pk)",
            )
        except ConditionalCheckFailed:
            return                               # a concurrent duplicate lost the race

    charge(table)                                # if this throws, record stays
                                                 # in_progress for a retry or the DLQ
    table.update_status(idem_key, "done")        # ink it in, only after success

A retry that finds an in_progress record may safely re-attempt; one that finds done stands down; a charge that keeps failing falls to the DLQ instead of being silently swallowed. That is the gap between "I added an idempotency key" and "I actually handled failure." Most write-ups stop at claim-first. The poison column is the reason you should not.

Three states, one small machine

If you squint, the robust handler is a tiny state machine with a record that moves one way:

Nothing, then in_progress, then done. Each arrow is a single conditional write. A retry that lands on in_progress loops back and tries the work again. A retry that lands on done stands down. A duplicate that loses the race on the first claim returns early. And when the retries finally run out, the event leaves the machine through the failure queue instead of vanishing.

The reason this works is that the record tells the truth about how far the work got, not merely that someone once looked at it. Claim-first records "seen." Robust records "seen, and finished." That one extra word is the whole difference.

What the table looks like after each scribe

If you open the DynamoDB table after a run, each pattern leaves its own handwriting, and reading it is the quickest way to understand what happened.

The naive scribe leaves one row: the counter for the account, with charges at 2. Nothing else. There is no trace of why it is 2, which is the problem.

The claim-first scribe leaves the counter plus one extra row per upload, keyed by the idempotency key and marked as seen. The key tells you which upload was claimed, but not whether the work behind it ever finished.

The robust scribe leaves the counter plus a record per upload that says in_progress or done. That single field is what lets you, or an alarm, find the half-finished entries. And when you force failures for a demo, you will also see small failflag rows, the markers that remember the first attempt already failed.

A ledger you can interrogate after the fact is half the value. The other half is the reconciliation below.

The three scribes, side by side

Here is everything above on one page. The first four rows are what the simulator and the code show. The last is a practical note on the shipped versions.

Look at the fourth row, because it never changes. None of the three closes the window between the charge and the commit on their own. I will come back to that.

What the tests pin down

I did not want any of this to rest on my reading of a table, so each row is pinned by a test that runs one handler through one scenario and asserts the charge count:

Two of those tests deserve a second look. test_naive_double_charges_on_transient_failure asserts the bug exists, on purpose, so the phantom can never be quietly "fixed" by a refactor. And test_claim_first_leaves_money_state_inconsistent_on_poison asserts the inconsistency on purpose, so the trap in the middle pattern is documented in code, not just in prose.

Taking the counting-house to real AWS

A simulator proves the logic. I wanted to watch a real Lambda commit the sin. So I deployed the whole pipeline as one SAM stack: a versioned S3 bucket, the Lambda, a DynamoDB table, a dead-letter queue, and the IAM to bind them. Two parameters drive the demo. HandlerMode picks the pattern (naive, safe or robust), and FailOnce forces the first attempt for each object to fail so that Lambda's retry actually fires.

The deploy prints the stack outputs: the bucket to upload to, the table that holds the ledger, and the failure queue. With the handler set to naive, I uploaded a single file and read the ledger with query_state.py:

Two charges, one upload, on real Lambda and real DynamoDB. Remember that the retry arrives after a delay, so the second charge shows up a little after the first. Then I flipped the same stack to the robust handler, uploaded a fresh file, and forced the same first-attempt failure:

One charge, and no phantom. I will be straight about how much that proves in a section below, because it proves less than the screenshot suggests.

A practical note if you run this yourself: use a fresh file name for every run. The failure flag and the idempotency record are both stored per object, so uploading the same key twice will not trigger the failure the second time.

The gotcha that caught me building the demo

Forcing that live double charge was not smooth, and the reason is worth more than the screenshot, because I fell into the exact trap the article warns about.

My first attempt to force the failure used an in-memory flag: "have I already failed for this object? No? Then fail once and remember it." On real Lambda it stubbornly charged only once, and I lost an hour to it.

The cause is the whole lesson in miniature: Lambda reuses warm execution environments. The retry ran in the same container as the first attempt, so my in-memory "I already failed" flag was still set, the handler skipped the failure, and it charged once. My own test harness had the precise bug the article is about, trusting in-memory state across invocations.

The fix was to move the flag into DynamoDB, where it survives across invocations and containers:

def fail_once(table, key):
    try:
        table.put_item(
            Item={"pk": f"failflag#{key}"},
            ConditionExpression="attribute_not_exists(pk)",
        )
    except ConditionalCheckFailed:
        return                 # flag already set: this is the retry, let it pass
    raise RuntimeError("forced first-attempt failure; Lambda will retry")

Only then did the retry genuinely fire and the phantom charge appear. "Do not trust in-memory state across Lambda invocations" is not a footnote in this story. It is the moral, and I managed to trip over it while staging the demonstration of it.

The same logic applies in reverse. A warm container is a convenience for performance, not a place to keep facts. Anything your handler needs to remember between invocations, including "I already did this," belongs in a store that every container can see.

What robust does not promise

Be careful with the word "fixed." In my demo the forced failure in the robust handler lands at the charge itself, so nothing has been charged yet when the retry arrives. That is the poison case, and robust handles it well: no money moves, and the record waits for a human.

The case it does not cover is the nastiest one. The charge lands, and then the function dies before it can mark the record done:

The retry finds in_progress, assumes the work never finished, and charges again. I checked this against the simulator by moving the failure to just after the charge, and the robust pattern double-charges there too. The repo has a small script, local/limits.py, that shows it:

No bookkeeping on your side of the wire can close that window, because the charge happens on someone else's side of it. The complete cure is to hand the same idempotency key to whatever takes the money, so the provider itself recognises the second request as a repeat. Many payment APIs accept one for exactly this reason. Claim, charge, commit is still the right shape for your own ledger: it stops silent drops and repeat work after a commit. It just cannot make an external side effect happen exactly once on its own.

In other words, there are two ledgers to keep honest. Yours, in DynamoDB, which the pattern above protects. And theirs, on the other side of the API call, which only a shared key can protect. Do both.

How long should the key remember?

A record that expires too soon is worse than no record, because the duplicate shows up after it is gone and gets treated as new. The record has to outlive the longest time a duplicate can still arrive.

For a function error, Lambda's retries happen within a few minutes. But for throttling and system errors, Lambda keeps trying for up to six hours by default, and events can sit in the async queue for a long time when the function lacks capacity. So a TTL measured in minutes is a trap.

A day is a sensible place to start, and then you tune it to your own risk. DynamoDB's TTL feature can expire the records for you, so the table does not grow forever.

How would you even notice?

The cruel part of a duplicate is that it is silent. Nothing fails, so nothing alerts. But it does leave footprints, if you know where to look:

The first is the most honest check: count what came in and count what you recorded, and ask why the numbers differ. The others are signals Lambda and your failure queue already give you, and an alarm on any of them is cheap.

Which scribe do you need?

Not every handler needs the full ceremony. Here is the road map I would follow:

A few notes on reading it. If the side effect can be made naturally idempotent, do that first and skip the machinery: overwriting a value or upserting a row by a stable id is safe to repeat. If you do need a key, claim-first is enough when losing the work on a failure is acceptable and you are watching for it. The moment losing the work would cost you something, switch to claim, work, commit. And the moment the side effect lives in someone else's system, send them the key too.

A safety net: reconcile

Even a well-built scribe deserves an auditor. The merchants of old closed the day by checking the book against the cash, and the digital version is a small scheduled job that compares what arrived with what was recorded: list the objects that landed, list the entries marked done, and flag anything that is missing, stuck at in_progress for too long, or counted twice.

It will not stop a duplicate, but it makes a duplicate loud. Idempotency prevents the mistake; reconciliation finds the ones that slip through, which, given the window we just drew, is the honest thing to plan for.

What I did not test

A write-up like this is only as good as its limits, so here are mine.

• I never saw a real S3 duplicate. In my demo, the duplicate comes from Lambda's retry. S3's own duplicates are documented as rare, and I did not reproduce one.
• I reasoned about concurrency from the semantics of the conditional write and drew it, but I did not load-test it.
• It ran in one region, one account, with tiny files.
• The fake table in the simulator mimics the conditional write, not DynamoDB's latency, throttling or any of its edge cases.
• The "charge" is a counter. No real payment provider was involved.

None of that changes the shape of the argument. It does mean you should treat the numbers as an illustration of the failure modes, not as a benchmark.

Run it yourself in five minutes

Everything in the first half runs offline, with no AWS account and no cost:

git clone https://github.com/mursalfk/idempotent-s3.git
cd idempotent-s3

python -m venv .venv
source .venv/bin/activate        # Windows (Git Bash): source .venv/Scripts/activate
pip install -e ".[dev]"

python -m local.sim              # the table: naive 2, the guarded handlers 1
python -m local.limits           # the case robust does not close: charge count 2
python -m pytest tests/ -v       # 7 passed

If you then want to watch the real thing, the README walks through the SAM deploy, the two parameters, and the cleanup. Delete the stack when you are done.

A few questions you may be asking

Does S3 really send duplicates? Rarely, and the documentation says so. The Lambda retry path is the one you will actually hit, which is why the demo uses it.

Why not put a queue between S3 and Lambda? It is a fine design, and an SQS standard queue is also at-least-once, so you still need an idempotent consumer. A FIFO queue deduplicates messages within a time window, which helps with duplicate deliveries but does not make your downstream charge exactly-once.

Why DynamoDB for the records? Because a conditional write is an atomic claim, and because it is a store every container can see. Set a TTL on the records so the table does not grow forever.

Should I hand-roll all this? Probably not in production. The idempotency utility in Powertools for AWS Lambda packages the same ideas. This repo exists to show you why those details matter, so that when you read the library's documentation, every option makes sense.

What does it cost to try? A handful of requests on pay-per-request DynamoDB, a few Lambda invocations and a few small S3 objects. It costs next to nothing, but delete the stack afterwards with sam delete.

What the ledger teaches

• An S3-triggered Lambda is an at-least-once system twice over: S3 may deliver an event more than once, and Lambda retries failures. A handler that is not idempotent will, in time, repeat a side effect.
• The dangerous version is silent. No crash, no error, just a second charge discovered in a support ticket weeks later.
• An idempotency key is necessary but not the whole discipline. Claim-first stops the double charge yet can move money on an operation that then fails. Claim, do the work, then commit, and let genuine failures reach the DLQ. Then pass the same key to whatever takes the money, because the window between the charge and the commit is still open.
• Do not trust in-memory state across invocations. Lambda reuses containers, so "I already did this" belongs in a store, never a variable. I learned it by shipping the bug into my own demo.
• The conditional write is the atom of the whole discipline. One atomic claim per piece of work, never a read followed by a write.
• Let the record live longer than any retry can, and let it say how far the work got, not merely that it was seen.
• Idempotency prevents the mistake; reconciliation finds the ones that slip through. Plan for both.

If you take only one thing from the exercise, take the checklist:

The whole counting-house is on GitHub: the local simulator so you can run it offline, all three handler patterns, the SAM template, and the tests that pin every row of that table. MIT licensed. Clone it and charge yourself twice in about a minute.

Repo: https://github.com/mursalfk/idempotent-s3

The old scribes closed a good entry the way they closed a good letter, with a seal pressed into warm wax. One entry, one seal, and nobody could press it twice without it showing. A handler that survives a retry is the same idea in a different medium. Enter it once, mark it done, and let the second knock on the door find the seal already set.