TL;DR: "At-least-once delivery" really means "sometimes twice". In TxFlow, my Kafka payment orchestrator, five consumer groups process every payment event, and each one has to survive seeing the same event twice. I use two layers: a Redis key as a fast "have I seen this?" check, and a Postgres primary key as the actual guarantee for the one side effect that can never run twice, the wallet debit. This post is about the crash timing that makes both layers necessary.
Quick context
TxFlow is a payments workflow I built to learn Kafka properly. One POST /payment produces a single event on payments.initiated, and five independent consumer groups react to it: fraud, wallet, notify, audit and analytics. (I wrote about why I moved from SQS to Kafka for this.)
Consumers commit offsets manually, and only after one of three outcomes:
- the event was processed successfully,
- it was sent to the dead-letter queue, or
- it was skipped as a duplicate.
That's what gives at-least-once processing. It's also exactly what creates duplicates.
Where duplicates come from
The dangerous window is between "I did the work" and "I told Kafka I did the work":
poll event E
→ debit wallet (committed in Postgres)
→ 💥 process crashes / rebalance / network blip
→ offset never committed
restart → poll event E again → debit wallet again ❌
Kafka isn't misbehaving here. You get the same thing from producer retries, consumer-group rebalances, or someone replaying a topic. Any consumer with side effects has to be idempotent.
Layer 1: a Redis dedup key per consumer
Every consumer checks a key like this before doing any work:
wallet:processed:{event_id}
fraud:processed:{event_id}
notify:processed:{event_id}
...
Each key has a TTL (24 hours by default). It's cheap, and it catches the common cases like redelivery after a rebalance. Each consumer group has its own key namespace, because "the fraud consumer handled E" says nothing about whether the wallet consumer did.
Why Redis alone isn't enough
Here's a simplified sketch of the wallet consumer's order of operations:
def handle(event):
key = f"wallet:processed:{event['event_id']}"
if redis.exists(key):
return "duplicate"
debit_wallet(event) # Postgres transaction commits here
redis.set(key, 1, ex=86400) # ...and the process can die right before this line
return "processed"
If the process dies after the Postgres commit but before the Redis SET, the redelivered event passes the Redis check and debits the wallet again. You could flip the order and set the key first, but then a crash after setting the key and before the debit skips the payment forever. That's a lost payment instead of a double payment, which isn't better.
Two separate systems can't be updated atomically. So Redis can only ever be an optimisation. The guarantee has to live in the same transaction as the side effect.
Layer 2: the database is the guarantee
The wallet consumer records every event it has processed in the same Postgres transaction as the debit:
-- simplified
CREATE TABLE wallet_processed_events (
event_id uuid PRIMARY KEY,
processed_at timestamptz NOT NULL DEFAULT now()
);
def debit_wallet(conn, event):
with conn.transaction():
cur = conn.execute(
"INSERT INTO wallet_processed_events (event_id) VALUES (%s) ON CONFLICT DO NOTHING",
(event["event_id"],),
)
if cur.rowcount == 0:
return "duplicate" # already debited; nothing else runs
cur = conn.execute(
"UPDATE wallets SET balance = balance - %s WHERE user_id = %s AND balance >= %s",
(event["amount"], event["user_id"], event["amount"]),
)
if cur.rowcount == 0:
raise InsufficientFunds(event["user_id"]) # rolls back the marker too
return "processed"
The marker row and the debit commit together or not at all:
- Crash before commit: nothing happened, and the redelivery does the work.
-
Crash after commit: the redelivery hits the primary key,
rowcount == 0, and it's skipped.
There's no window in between. The Redis key still saves a database round trip for most duplicates. It just isn't what keeps the payment correct.
Not every consumer needs both layers
This is the part I find most interesting. The right level of protection depends on what a duplicate costs:
| Consumer | Cost of a duplicate | Protection |
|---|---|---|
| wallet | Money taken twice | Redis + DB primary key in the same transaction |
| audit | Duplicate log row | Redis dedup (append-only table) |
| notify | A second email | Redis dedup |
| analytics | Counter off by one | Redis dedup, no DLQ at all |
The analytics consumer is deliberately the "cheapest" one. It increments Redis counters and doesn't even get a dead-letter queue, because a slightly wrong dashboard number isn't worth an operational process.
The TTL is a real decision
A 24-hour TTL means the Redis layer "forgets" events older than a day. If I replayed the topic from the beginning:
- wallet would be fine, because the database marker never expires,
- analytics would double-count everything older than 24 hours.
For a demo dashboard, that's acceptable. In production, I'd either give the analytics consumer a database-backed marker or rebuild analytics from the topic with a fresh consumer group instead of replaying into the old counters.
Don't forget the producer side
Consumers aren't the only source of duplicates. A client retrying POST /payment after a timeout would create a new event with a new event_id, which gets past every dedup layer downstream. So the API requires a client-generated idempotency_key, stores it with a unique constraint in the outbox table, and returns 409 on reuse. Deduplication has to happen at every hop, starting from the first one.
Rule I took away: a dedup cache makes duplicates cheap. Only a constraint in the same transaction as the side effect makes them impossible.
Question for you: where do you keep your idempotency markers? In the same database as the side effect, or a separate store? And have you ever lost data because you set the dedup key before doing the work?
Top comments (0)