A lot of payment webhook handlers look reasonable on the surface. They validate the signature, parse the payload, wrap the database write in a try/catch, and log an error if anything goes wrong. It works fine in every test anyone runs against it. Then, months later, a customer opens a ticket saying they were charged for the same order twice, and nobody on the team can immediately explain how, because nothing in the code looks obviously broken.
The answer is almost always the same: the handler assumed every incoming request represented a new event. Nothing in a try/catch block checks whether this exact event has already been processed. It only catches errors that happen while processing it, which is a completely different problem.
Try/Catch Solves the Wrong Failure Mode
A try/catch block protects against exceptions: a malformed payload, a database connection drop, a null field where you expected a value. Those are real failure modes worth handling, and every handler should have them. But a duplicate webhook delivery isn't an exception. It's a perfectly valid, correctly formatted request that happens to represent something you already did. The handler runs successfully both times. There's no error to catch, because nothing failed. The code just does the same real-world action twice, cleanly, with no exception anywhere in the stack trace to point you at the problem.
This is why the bug is so easy to miss in code review. Reviewers scan for missing error handling, and this handler has plenty of it. The gap isn't in what happens when something goes wrong. It's in the assumption that every request is new.
Where the Duplicate Deliveries Actually Come From
Payment processors and most other webhook senders guarantee at-least-once delivery, not exactly-once, and they're explicit about this in their documentation because it's a deliberate design tradeoff, not an oversight. If your endpoint doesn't return a success status within their timeout window, or returns any error status, the sender will redeliver the same event, sometimes more than once. That timeout window is often short, in the 10 to 30 second range, which means anything that makes your handler briefly slow, a database migration running in the background, a cold start, elevated load, increases your odds of triggering a redelivery you'll then have to handle correctly.
None of this is misbehavior on the sender's side. Redelivery on timeout is the correct, documented response to an ambiguous outcome, since the sender genuinely doesn't know whether your handler processed the event or not. The responsibility for handling that ambiguity sits with your endpoint, not with the sender's retry policy.
What an Idempotency Table Actually Looks Like
The fix is a dedicated table, separate from your business data, with the webhook's event ID as a unique-constrained column. Most payment processors and webhook senders already assign a unique ID to every event, including redeliveries of the same logical event, so you don't need to generate your own key. Before doing any real work, check whether that event ID already exists in the table. If it does, return success immediately and stop. If it doesn't, insert the ID as the first write, then do the real processing.
The order matters. Inserting the event ID first, before the business logic runs, and relying on the unique constraint to reject a second concurrent insert, closes the race condition where two near-simultaneous redeliveries both check for the ID before either has recorded it. Checking for existence, then doing the work, then recording the ID afterward, leaves a window where two redeliveries arriving close together both pass the check and both do the real work.
A Minimal Schema
CREATE TABLE processed_webhook_events (
event_id TEXT PRIMARY KEY,
event_type TEXT NOT NULL,
processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
The primary key constraint does the heavy lifting here. An INSERT against this table either succeeds, meaning you're the first (and only) handler processing this event, or it fails with a constraint violation, meaning someone else already claimed it. Catching that specific constraint violation and treating it as "already handled, return success" rather than an application error is the one piece of exception handling that's actually relevant to this problem, as opposed to the general try/catch around the rest of the handler. This pattern works the same way on any relational database that enforces uniqueness atomically, PostgreSQL included, which is part of why it's such a portable fix across different tech stacks.
What This Actually Costs You
The objection teams sometimes raise is that this adds a new table, a new write, and a new failure surface to a handler that already works most of the time. That's true, and it's worth being honest about the tradeoff rather than pretending the fix is free. The counterpoint is that the cost is fixed and small, one row, one indexed lookup, while the cost of not having it scales with your traffic and your webhook sender's retry behavior, neither of which you control. A handler that processes a few hundred events a day might go months without a visible duplicate. One that processes tens of thousands a day, across a sender that occasionally has its own network issues, will see duplicates often enough that the missing table becomes a recurring support burden rather than a rare edge case.
Handling Slow Processing Without Missing the Timeout Window
Even with the idempotency table in place, a handler that does all its real work synchronously, updating the order, sending a confirmation email, syncing a CRM, before responding, is still vulnerable to triggering a redelivery through simple slowness. The fix is to separate the fast, synchronous part (validate the signature, check and record the event ID) from the slower work, which runs in a background job after the response has already gone out. This shrinks your acknowledgment window down to essentially one database write, which is difficult for anything short of a full outage to push past a sender's timeout.
Systems like Redis work well as the queue for that background job if you don't already have one, since the durability requirement is modest: you just need the job to survive a brief restart, not to guarantee delivery across a distributed cluster.
Testing This Properly
The test that matters here isn't "does the handler process a valid webhook correctly." It's "does the handler behave correctly when the exact same payload arrives twice, including when both arrive close enough together to race." A test that fires the same event synchronously, once, then again, checking that the second call is a no-op, covers the sequential case. A test that fires both concurrently and asserts exactly one insert succeeded covers the race condition, and it's the test most teams skip because it takes more setup than a sequential call.
Watching for the Table Itself Becoming a Bottleneck
Once the table is in place, it's worth keeping an eye on its growth and lookup performance, since it's now in the hot path of every incoming webhook. An index on the event ID column, which the primary key constraint already provides, keeps lookups fast well past the point most teams will ever reach in practice. Pruning records older than your sender's realistic redelivery window, most senders redeliver within a few days at most, keeps the table from growing indefinitely and keeps those lookups fast for years rather than months.
The Broader Pattern
This same table-plus-unique-constraint approach applies beyond payment webhooks specifically, to any endpoint where a client-controlled or sender-controlled identifier can stand in for "have I already done this." 137Foundry's engineering blog has a deeper walkthrough of the client-side version of this pattern, including how to scope idempotency keys, handle the conflict case where a key gets reused with a different payload, and design the retention window, in How to Design an Idempotency Key Strategy So Retried API Requests Never Double-Process. The webhook case is the same underlying mechanism applied to inbound events instead of outbound API calls, and once you've built it once, adding it to a second endpoint is a small amount of incremental work rather than a new design problem.
The Takeaway
A try/catch block and an idempotency table solve different problems, and a payment handler needs both. The try/catch protects against real failures during processing. The idempotency table protects against the far more common case where nothing failed at all, and the handler simply ran twice for the same event because that's exactly what at-least-once delivery, correctly implemented on the sender's side, is supposed to do under the conditions the specification allows for, as documented for systems like Apache Kafka that formalize the same delivery guarantee.
Top comments (0)