When the Claim Arrives Twice
The first time it happened in production, I stared at the logs for twenty minutes before I understood what I was looking at.
Two identical claims. Same member, same date of service, same procedure code, same everything. Processed twice. Not because anyone submitted the form twice on purpose. Because a network retry fired, the receiving system acknowledged too slowly, and the queue decided "maybe it didn't work" and tried again. Both went through. Both wrote records. Both triggered downstream workflows. The system was perfectly correct about each one individually, and catastrophically wrong in aggregate.
That moment is the reason I think about idempotency differently now than I did early in my career. Back then, it felt like a theoretical computer science concept. Now it keeps me from being paged at 2am. And it forced me to ask a question I did not expect to be so hard: how do you make a system safe when the same input can arrive more than once?
The Deceptive Simplicity of "Just Check First"
The obvious answer is: before you process a claim, check whether it already exists. If it does, skip it. Simple.
Except it isn't.
The check and the write are two separate operations. Between the moment you check and the moment you write, another process can sneak in. In a busy production environment, that gap is milliseconds, which sounds like nothing right up until you have a queue consuming messages with eight concurrent workers. Now the same claim can pass the "does this exist" check in two workers simultaneously, both conclude "no, it's new," and both proceed to write. You have not solved the problem. You have given it a race condition.
You can tighten the gap with locks. A pessimistic lock before the check prevents concurrent reads, which prevents the race. But now you have serialized your entire processing pipeline on a single bottleneck, and your throughput drops as fast as your retry budget climbs. The cure starts to feel like a disease.
There is another failure mode that does not get talked about enough: partial processing. A claim does not land in one atomic moment. It triggers a chain: validation, benefit lookup, adjudication, writing the determination, notifying downstream systems. If a failure happens at step four, you have three steps' worth of state in your database and a half-processed claim that will come back around and fail your "does this exist" check in confusing ways. The system is neither fully processed nor safely reprocessable. It is stuck.
The naive approach assumes the failure modes are simple and linear. Production systems are neither.
Setting the Savepoint Before You Act
The insight that changed my thinking came from asking a different question. Instead of "how do I know if I've already done this?" I started asking "how do I make it safe to do this twice?"
That shift matters. The first question tries to prevent duplication by detecting it early. The second accepts that duplication will happen and designs the processing to survive it gracefully.
The pattern begins at ingest, before any processing logic runs. Every claim arrives carrying its complete payload. Hashing that full payload through SHA-256 produces the same 64-character string every time the byte-identical message arrives, regardless of which system sent it, which retry attempt it represents, or when it comes through. The hash is not derived from a selection of identifying fields. It is derived from the complete, unmodified byte content of the payload as received. Any two transmissions that differ by even a single character produce different hashes and are treated as distinct claims. Any two transmissions that are byte-identical produce the same hash and are recognized as the same unit of work.
That ingest-time identity check is the foundation. With it in place, the processing sequence becomes: set a savepoint in the active transaction, then check whether this content hash already appears in the staging layer.
If the hash is new, the system writes the staging records fresh and proceeds. This is the common path.
If the hash already exists, the system does something that initially seems counterintuitive. It cleans up the existing staging records associated with that hash, then rewrites them from scratch using the payload it just received. Cleanup first, then rewrite, all within the boundary of the savepoint. The reason is specific: existing staging records for a given hash may be partial. A previous arrival of this claim may have failed partway through the staging phase, leaving some records written and others not. Returning that partial state downstream would propagate the error. Cleaning it up and rewriting it from the byte-identical payload in hand produces one complete, consistent staging state that downstream processing can act on reliably.
The savepoint is what makes this recoverable. If anything fails during cleanup or rewrite, the transaction rolls back to the savepoint marker. The database returns to its pre-attempt state. The claim goes back to the queue. The next arrival runs through the same sequence and starts from clean ground.
The concurrent-duplicate race closes at the database level. Two workers holding the same content hash, both checking at their respective savepoints and finding no existing record, will both proceed to write staging records. They will collide at the unique constraint on the hash column. One write wins and commits. One fails with a constraint violation. The losing worker rolls back to its savepoint and follows the same duplicate path: cleanup, then rewrite from the payload it holds. The caller gets one result. The database holds one consistent set of staging records. Downstream processing runs once.
What enforces the invariant is not an application-level lock. It is the unique constraint on the hash column, applied atomically by the database itself, paired with the savepoint that contains each worker's recovery within its own transaction boundary. Neither mechanism is sufficient alone. The constraint without the savepoint gives you collision detection with no clean recovery path. The savepoint without the constraint gives you contained transactions with no atomicity guarantee between concurrent writers. Together they make the staging layer idempotent by construction: any number of arrivals of a byte-identical payload will produce exactly one consistent set of staging records, regardless of timing or order.
The Second Arrival Is Not a Problem
The thing nobody tells you about idempotency is that it changes how you operate, not just how you build.
When you know a claim can arrive ten times and produce the same staging state every time, your relationship with retries changes entirely. You stop being afraid of them. You can tune retry policies aggressively because you know they cannot corrupt state. You can add observability tooling that replays events without worrying about side effects. You can restore from a backup, replay the queue, and walk away knowing the staging records will reflect exactly one consistent attempt per unique payload.
The oncall conversation changes too. An alert about a claim that "ran twice" becomes bounded. You check the content hash. You verify the staging records are in a consistent state. You confirm exactly one downstream workflow fired. You are not reconstructing damage. You are confirming what the system already resolved.
Go back to where this started. Two identical claims, both processed, both writing records, both triggering workflows. The system was correct about each one individually and catastrophically wrong in aggregate. What was missing was not detection, not better alerting, not more careful operators. It was a structural guarantee at the transaction level: compute the hash at ingest from the complete byte-identical payload, set the savepoint, clean up and rewrite the staging records within its boundary, enforce uniqueness atomically at insert time. The second arrival runs through exactly the same sequence. The staging layer ends in exactly the same state. Downstream processing sees one result, because there is only one result to see.
The claim can arrive twice. When the pattern holds, the second arrival is indistinguishable from a no-op. That is not a hope or a best-effort promise. It is what the savepoint, the unique constraint, and the cleanup-then-rewrite commit to together, on every arrival, every time.
Top comments (1)
Reframing from "how do I detect the duplicate" to "how do I make running this twice safe" is the part people skip, and the cleanup-then-rewrite inside the savepoint is the piece that makes it actually work — a partial staging state from the failed attempt is exactly the thing a naive existence check hands downstream as if it were finished.
One thing I'd watch: hashing the full byte payload means a semantically identical claim that arrives with different whitespace or a re-serialised timestamp gets a new hash and is treated as fresh work. If that ever bites you, the usual escape hatch is a second, business-level key (member + date of service + procedure code) validated alongside the content hash — the content hash proving "same bytes", the business key proving "same real-world claim". Which one is your alerting signal today?