DEV Community: Anna Sbrodova

The Redelivery Trap: Building a Safe Transactional Inbox for Google Pub/Sub

Anna Sbrodova — Mon, 12 Oct 2026 13:57:48 +0000

Pub/Sub will deliver the same message twice. Not often, but reliably enough that if your consumer charges a client, it will eventually charge them twice.

In part one we built a Postgres-backed outbox that guarantees every event from BigQuery reaches Pub/Sub. It deliberately chose at-least-once delivery, so this part is the other half: making the consumer safe when the same message arrives more than once.

The naive pattern: "blind trust"

A typical GKE worker or Cloud Run service:

Listens to a Pub/Sub subscription
Pulls a message
Executes the business logic (updates database state, calls a payment gateway)
Acknowledges the broker

The failure mode

![The failure mode: the "blind trust" pattern]

If the network blips after the business logic completes but before the ack reaches Pub/Sub, the broker redelivers the same message, possibly to another pod. The database gets hit again, and this time it's a duplicate.

If losing a message means you forgot to charge a client $50k, processing it twice means you just double-billed them. Try explaining to compliance that it wasn't fraud, just a highly available network retry.

The solution: a transactional inbox

Instead of trusting the network, make the database the arbiter of truth: an inbox table with a unique constraint on the message's idempotency key.

Everything happens in one atomic database transaction:

The gatekeeper (deduplication): the worker tries to insert the message's key into the inbox table.
The execution (business logic): if the insert succeeds, the message is new. Run the business logic in the same transaction, commit, then ack. If the insert finds the key already there, skip the business logic and just ack, so the broker stops retrying.

Which ID to deduplicate on

It's tempting to use Pub/Sub's own msg.ID. That only catches broker redeliveries. Pub/Sub assigns a new message ID every time something is published, so if the outbox from part one publishes the same row twice — which at-least-once allows — the two copies arrive with different IDs and both get processed.

Deduplicate on a key the producer controls instead. The outbox already has one: its row ID. Publish it as a message attribute and use that.

The key piece: `ON CONFLICT DO NOTHING`

Pushing deduplication into Postgres removes race conditions across the whole GKE cluster. If two pods receive the same message at once, the second insert waits on the unique index until the first transaction finishes, then inserts nothing.

// The producer's idempotency key, set by the outbox as a message attribute.
// Don't use msg.ID: Pub/Sub assigns a new one on every publish.
key := msg.Attributes["outbox_id"]

// 1. Begin a single atomic database transaction.
tx, err := db.Begin()
if err != nil {
    return err
}
// Always defer a rollback so an early return can't leak the connection.
// If tx.Commit() succeeds later, this becomes a safe no-op.
defer tx.Rollback()

// 2. Try to record the key.
res, err := tx.Exec(`
    INSERT INTO transactional_inbox (idempotency_key, created_at)
    VALUES ($1, NOW())
    ON CONFLICT (idempotency_key) DO NOTHING;
`, key)
if err != nil {
    return err
}

rowsAffected, err := res.RowsAffected()
if err != nil {
    return err
}

// 3. Zero rows means this message was already processed.
// Still ack, so the broker stops redelivering it.
if rowsAffected == 0 {
    broker.Acknowledge(msg)
    return nil
}

// 4. Run the business logic inside the SAME transaction.
// if err := executeBusinessLogic(tx); err != nil {
//     return err // the deferred rollback undoes everything
// }

// 5. Commit, then ack.
if err := tx.Commit(); err != nil {
    return err
}

broker.Acknowledge(msg)
return nil

If the pod crashes between the commit and the ack, the message is redelivered, the insert finds the key, and the worker simply acks. Nothing runs twice.

Why not Pub/Sub's exactly-once delivery?

Pub/Sub can guarantee that a message you've successfully acknowledged won't be redelivered. That narrows the window but doesn't close it. If the transaction commits and then the ack fails or the ack deadline expires, the message still comes back. And it can't help with duplicates the producer created by publishing twice. The inbox handles both.

Why not a cache instead?

I often hear: "Why add database overhead? Just deduplicate in Redis."

Caches get evicted; ledgers are forever.

If the Redis cluster restarts or a key expires early, your line of defence is gone. Idempotency isn't a buzzword here. It's the firewall protecting your revenue.

When the monthly reconciliation runs, your CFO doesn't care about network partitions or broker limits. They care that the ledger balances. Anchoring deduplication in the same ACID database that holds the financial state guarantees each message is applied exactly once, and the client is billed correctly.

The Dual-Write Problem: Building a Scalable Transactional Outbox for Google Pub/Sub

Anna Sbrodova — Mon, 12 Oct 2026 12:57:23 +0000

If a service writes data and then publishes an event about it, there is a window where one succeeds and the other doesn't. That's the dual-write problem, and in a billing system it costs real money.

This is how we closed that window for a BigQuery → Pub/Sub pipeline, using a Postgres outbox and a generic Go dispatcher on GKE.

The setup

Imagine a billing platform. The source of truth is BigQuery, where the heavy analytical lifting happens, but downstream services need real-time updates through Google Pub/Sub to trigger invoices, emails and ledger updates.

The goal is simple: get data from BigQuery to Pub/Sub.

The naive pattern: "the direct shot"

Most teams start with a Cloud Run Job or a cron task:

Query BigQuery
Loop through the rows
Call pubsub.Publish()
Job finishes. Done.

The question that matters: what happens if the network flickers at step 3?

If the producer crashes, or the Pub/Sub API hiccups, you get a dual-write failure. The data was processed, but the message is gone. In a billing system, "we lost a few messages" means "we forgot to charge a client $50k."

Kafka and RabbitMQ users: the same architecture applies. The core challenge is distributed consistency, not the broker.

The solution: a transactional outbox

To sleep at night, we decoupled extraction from dispatching, with a Postgres-backed outbox as a reliable staging area.

Instead of one fragile step, there are two resilient ones:

The producer (Cloud Run Job) queries BigQuery and writes the serialized bytes into a pubsub_outbox table in Cloud SQL.
The dispatcher (GKE service) is a "dumb" pipe that picks up those bytes and makes sure they reach the broker.

Why not poll BigQuery directly? It's a FinOps problem

I'm often asked why the dispatcher doesn't just poll BigQuery. As a tech lead, I look at the bill.

BigQuery is built for analysis. On-demand pricing charges per terabyte scanned. Polling every 10 seconds to ask "is there new data?" burns budget for nothing.
Latency. BigQuery DML updates can take seconds. Postgres does it in milliseconds.
Concurrency. BigQuery has no SELECT ... FOR UPDATE SKIP LOCKED. Run two producer instances and they fight over the same rows.

The key piece: `SKIP LOCKED`

The Go dispatcher lets Postgres handle concurrency. Each pod claims a batch of rows that no other pod can see, so the dispatcher scales horizontally on GKE without two pods ever claiming the same row.

-- Fetch 100 messages, lock them so no one else sees them,
-- and skip anything already being handled by another pod.
UPDATE pubsub_outbox
SET
    lock_id = :worker_id,
    locked_at = NOW()
WHERE id IN (
    SELECT id FROM pubsub_outbox
    WHERE completed_at IS NULL
      AND topic_name = :target_topic
      AND (lock_id IS NULL OR next_retry_at < NOW())
    ORDER BY created_at ASC
    LIMIT 100
    FOR UPDATE SKIP LOCKED
)
RETURNING id, payload;

One thing worth being precise about: this gives at-least-once delivery, not exactly-once. If a pod publishes a message and crashes before marking the row complete, the row gets published again. That's the right trade-off for billing, since a duplicate is recoverable and a lost message isn't, but it means the consumer has to handle duplicates. That's part two of this series.

The generic advantage: infrastructure vs. logic

Because the payload is stored as BYTEA (raw bytes), the dispatcher becomes a universal delivery engine. It doesn't care where the data came from, only that it gets delivered.

Business logic, from any source:

Analytical sources: a scheduled Cloud Run Job queries BigQuery and writes large datasets into the outbox.
Operational sources: an upstream microservice (checkout, billing) inserts into the outbox as part of its own local database transaction, so the state change and the event are committed together.

Delivery infrastructure, zero-touch:

Nobody touches the dispatcher code. To support a new data stream, deploy the same Docker image to GKE with a new TOPIC_NAME environment variable.

The result: the team stops reinventing the reliability wheel. Every new data stream gets at-least-once delivery, exponential backoff and dead-letter handling out of the box.

Next in the series: the consumer side — what happens when Pub/Sub delivers the same message twice, and how a transactional inbox makes that safe.