The Real Problem Is Not Duplicate Requests
The naive backend implementation looks safe until two requests arrive at the same time. Both check whether an idempotency key exists. Both see nothing. Both create an order.
That is a time-of-check to time-of-use race. Checking the key in application code does not reserve it.
Postgres can arbitrate that race with a primary key on (user_id, key) and INSERT ... ON CONFLICT DO NOTHING. Among competing transactions that successfully commit, only one can claim that key.
But winning the claim is only the beginning.
If you commit the key and then create the order in a separate transaction, the process can crash between those steps. The key says processing, but there is no order to process. Every retry finds the same unfinished claim.
The fix is to commit the idempotency record and the order together. That protects order creation. Protecting what happens outside Postgres requires another guarantee.
High-Level Architecture
The flow starts with four components. The iOS client sends an order with an Idempotency-Key header. The Rust API opens a Postgres transaction, claims the key, creates the order, and links the two records. It commits before calling the matching engine.
Postgres holds the durable state for both the original request and its retries. Redis is unnecessary for correctness in this design.
If the API crashes before commit, neither record becomes committed, and a retry can try again. If it crashes after commit but before calling the matching engine, the order remains pending, and a background reconciler can recover it.
There is one more failure window: the matching engine accepts the order, but the API crashes before recording that result. Postgres still says pending. A reconciler might submit the order again.
To make that recovery safe, the matching engine must recognize a repeated submission using a stable identifier, such as the order ID. The database transaction prevents duplicate order creation; downstream deduplication prevents duplicate execution.
Designing the Table to Store Intent, Not Just Keys
An idempotency record needs to describe the request and its outcome. A useful schema includes:
- A primary key on
(user_id, key). - A request fingerprint for detecting changed payloads.
- The order ID created for that request.
- A processing status.
- The response status and body, once available.
- An expiration timestamp.
If the table serves multiple operations, include the operation in the key scope or explicitly require keys to be unique across those operations.
Define the fingerprint deliberately. Hashing raw JSON bytes treats changes in whitespace or field order as different requests. For an order API, hashing a normalized representation of the validated order fields may better capture the intent.
Persisting the response lets retries replay the original outcome without rebuilding it from an order whose state may have changed. Storing JSON in jsonb preserves its content, but does not promise the original byte representation.
The IETF Idempotency-Key draft recommends returning the result of a previously completed operation. It also recommends 422 when the same key is reused with a different payload and 409 when the original operation is still outstanding. These are draft recommendations, rather than a requirement for byte-identical response replay.
ACID in Practice
Atomicity: Claim and Order Must Commit Together
Begin a transaction and try to insert the idempotency record.
If the insert succeeds, create the order and link it to the claim within that same transaction. Commit both records together.
If the insert does not create a row, end that transaction and read the existing record in a fresh transaction. A competing insert can wait for the winning transaction to finish, so a conflict does not necessarily mean that the entire business operation has completed.
The existing record determines the next step:
- A different fingerprint returns
422. - A completed operation returns its stored result.
- An outstanding operation follows the API’s documented concurrent-request behavior.
For this design, that last case returns 409. An asynchronous API can instead define order acceptance as the completed operation, persist a 202 acceptance response alongside the order, and expose matching progress through a separate endpoint.
The important point is to choose a clear contract. “Order accepted” and “order matched” are different outcomes.
Consistency: A Key Identifies One Intent
Reusing the same key for a different payload must not silently return an unrelated order.
Store the fingerprint when claiming the key and compare it on every duplicate request. A view-model bug that reuses a key for a different symbol should produce an explicit error.
The idempotency key does not replace business constraints. Balance checks, valid order transitions, and other invariants still need their own enforcement.
Isolation: Let the Unique Index Arbitrate the Claim
Do not rely on SELECT followed by INSERT to decide who owns the request. Both transactions can observe that the key is absent.
Use the unique constraint to arbitrate ownership. Under PostgreSQL’s default Read Committed isolation, retrieve the existing record with a subsequent read after the conflicting insert finishes.
For later state transitions, use conditional updates or row locks where necessary. If both the API and a reconciler can process pending work, they also need coordination to avoid unnecessary concurrent submissions.
That coordination reduces overlap. The matching engine’s deduplication guarantee still handles retries after uncertain outcomes.
Durability: Persist the Outcome Before Responding
After the matching engine returns a definitive result, update the order and idempotency record together in a second transaction. Store the response, commit, and then respond to the client.
If the API crashes after that commit but before the client receives the response, a retry can retrieve the stored outcome.
If it crashes before recording the result, the original claim and order still exist. The retry does not need to create another order. Recovery must resolve the existing order’s outcome, either by querying the matching engine or by safely resubmitting the same order ID.
This guarantee assumes Postgres is configured for durable commits. ACID protects the records in Postgres; it cannot make an external call part of the same transaction.
Walking Through the Crash Windows
| Failure point | Durable state | Recovery |
|---|---|---|
| Before the first commit | No committed claim or order | Retry can claim and create |
| After commit, before submission | Claim and pending order | Reconciler submits the existing order |
| After engine acceptance, before recording the result | Claim and apparently pending order | Query the engine or resubmit with downstream deduplication |
| After recording the result, before responding | Completed record and stored response | Retry replays the result |
The third row is the easy one to miss. A timeout or crash does not tell you whether the external operation happened.
How Long to Keep the Key
Retention is part of the API contract.
Twenty-four hours might suit some order workflows. Other systems may need several days or longer. Choose the window based on client retries, offline behavior, recovery time, and the consequences of replaying an old request.
Deleting a key ends its deduplication protection. A late retry may then look like a new request.
Publish the expiration policy, and avoid deleting unresolved records merely because their original retry window has elapsed.
Closing
Mobile taught me to distrust the network. Backend taught me to distrust my own process crashing between two lines of code.
Idempotency is where those two distrusts meet.
A transaction makes the request’s intent durable. A stored response makes the outcome repeatable. Once execution crosses a service boundary, the receiving system must recognize that same intent too.
Next: Where COMMIT Is Not Enough — The Transactional Outbox
The diagram still contains a gap: we commit the order, then call the matching engine.
A reconciler can recover pending orders, but we can represent the intent to publish explicitly. Instead of relying on an immediate call after commit, insert an outbox event in the same transaction as the order and idempotency claim.
The transaction now commits three things together:
- The request’s idempotency record.
- The order.
- The durable intent to publish that order.
A separate relay reads committed outbox events, publishes them, and marks them as sent.
This moves the intent to publish inside the database transaction. The external publication still happens outside it.
If the relay publishes successfully and crashes before marking the event as sent, it will publish that event again. The outbox therefore supports at-least-once delivery; consumers must deduplicate using a stable event or order ID. Ordering also requires an explicit design.
In Part 3, I will implement the Postgres outbox and a Rust relay. I will use LISTEN/NOTIFY to wake the relay when work arrives, with startup and reconnect scans plus periodic recovery checks for outstanding work.
Notifications are a signal to look for work. The outbox table is the durable record of that work.
Then I will carry the same order ID through delivery and matching, so replaying an outbox event cannot execute the order twice.
Top comments (0)