The Redelivery Trap: Building a Safe Transactional Inbox for Google Pub/Sub

Pub/Sub will deliver the same message twice. Not often, but reliably enough that if your consumer charges a client, it will eventually charge them twice.

In part one we built a Postgres-backed outbox that guarantees every event from BigQuery reaches Pub/Sub. It deliberately chose at-least-once delivery, so this part is the other half: making the consumer safe when the same message arrives more than once.

The naive pattern: "blind trust"

A typical GKE worker or Cloud Run service:

1. Listens to a Pub/Sub subscription
2. Pulls a message
3. Executes the business logic (updates database state, calls a payment gateway)
4. Acknowledges the broker

The failure mode

![The failure mode: the "blind trust" pattern]

If the network blips after the business logic completes but before the ack reaches Pub/Sub, the broker redelivers the same message, possibly to another pod. The database gets hit again, and this time it's a duplicate.

If losing a message means you forgot to charge a client $50k, processing it twice means you just double-billed them. Try explaining to compliance that it wasn't fraud, just a highly available network retry.

The solution: a transactional inbox

Instead of trusting the network, make the database the arbiter of truth: an inbox table with a unique constraint on the message's idempotency key.

Everything happens in one atomic database transaction:

1. The gatekeeper (deduplication): the worker tries to insert the message's key into the inbox table.
2. The execution (business logic): if the insert succeeds, the message is new. Run the business logic in the same transaction, commit, then ack. If the insert finds the key already there, skip the business logic and just ack, so the broker stops retrying.

Which ID to deduplicate on

It's tempting to use Pub/Sub's own msg.ID. That only catches broker redeliveries. Pub/Sub assigns a new message ID every time something is published, so if the outbox from part one publishes the same row twice — which at-least-once allows — the two copies arrive with different IDs and both get processed.

Deduplicate on a key the producer controls instead. The outbox already has one: its row ID. Publish it as a message attribute and use that.

The key piece: ON CONFLICT DO NOTHING

Pushing deduplication into Postgres removes race conditions across the whole GKE cluster. If two pods receive the same message at once, the second insert waits on the unique index until the first transaction finishes, then inserts nothing.

// The producer's idempotency key, set by the outbox as a message attribute.
// Don't use msg.ID: Pub/Sub assigns a new one on every publish.
key := msg.Attributes["outbox_id"]

// 1. Begin a single atomic database transaction.
tx, err := db.Begin()
if err != nil {
    return err
}
// Always defer a rollback so an early return can't leak the connection.
// If tx.Commit() succeeds later, this becomes a safe no-op.
defer tx.Rollback()

// 2. Try to record the key.
res, err := tx.Exec(`
    INSERT INTO transactional_inbox (idempotency_key, created_at)
    VALUES ($1, NOW())
    ON CONFLICT (idempotency_key) DO NOTHING;
`, key)
if err != nil {
    return err
}

rowsAffected, err := res.RowsAffected()
if err != nil {
    return err
}

// 3. Zero rows means this message was already processed.
// Still ack, so the broker stops redelivering it.
if rowsAffected == 0 {
    broker.Acknowledge(msg)
    return nil
}

// 4. Run the business logic inside the SAME transaction.
// if err := executeBusinessLogic(tx); err != nil {
//     return err // the deferred rollback undoes everything
// }

// 5. Commit, then ack.
if err := tx.Commit(); err != nil {
    return err
}

broker.Acknowledge(msg)
return nil

If the pod crashes between the commit and the ack, the message is redelivered, the insert finds the key, and the worker simply acks. Nothing runs twice.

Why not Pub/Sub's exactly-once delivery?

Pub/Sub can guarantee that a message you've successfully acknowledged won't be redelivered. That narrows the window but doesn't close it. If the transaction commits and then the ack fails or the ack deadline expires, the message still comes back. And it can't help with duplicates the producer created by publishing twice. The inbox handles both.

Why not a cache instead?

I often hear: "Why add database overhead? Just deduplicate in Redis."

Caches get evicted; ledgers are forever.

If the Redis cluster restarts or a key expires early, your line of defence is gone. Idempotency isn't a buzzword here. It's the firewall protecting your revenue.

When the monthly reconciliation runs, your CFO doesn't care about network partitions or broker limits. They care that the ledger balances. Anchoring deduplication in the same ACID database that holds the financial state guarantees each message is applied exactly once, and the client is billed correctly.