You Probably Don't Need Kafka: A Durable Outbox and Job Queue in Plain Postgres

TL;DR — The moment you "save the order, then publish an event / send an email / call a downstream service," you have a dual write : two systems that must both commit with no transaction spanning them. Kafka doesn't fix that — it just gives you a second system to keep in sync. The fix is a transactional outbox : write the job into the same Postgres transaction as your business data, then drain it with a worker that dequeues using SELECT ... FOR UPDATE SKIP LOCKED. You get durable, at-least-once delivery with ordered, concurrent workers — and because delivery is at-least-once, your handlers must be idempotent. For a huge class of apps that is the entire messaging stack, and it's a table you already know how to back up.

The email that sends twice, or never

Here's the feature: when an order is placed, send the customer a confirmation. The obvious code:

await db.orders.insert(order); // 1. commit the order
await email.sendConfirmation(order); // 2. call the email provider

Two writes to two systems, and no transaction wraps both. So it fails in two directions, and both happen in production:

• Insert commits, then the process dies (deploy, OOM, the email API times out) → order exists, email never sends. Silent.
• Email sends, then the insert rolls back (a constraint trips on commit) → customer gets "your order is confirmed" for an order that doesn't exist. Worse than silent.

This is the same dual-write problem I hit moving order creation behind the payment webhook — two systems that must both commit, coordinated by nothing. And the instinct a lot of teams reach for here is: "add a message broker." Put Kafka (or SQS, or RabbitMQ) between the order and the email, publish an event, let a consumer send the mail.

But look at what that actually does to the failure above:

await db.orders.insert(order); // commit to Postgres
await kafka.publish('order.placed', order); // commit to Kafka — STILL a second system

You didn't remove the dual write. You moved it. Now the two systems that can disagree are Postgres and Kafka, and you've taken on a broker to run, monitor, upgrade, secure, and reason about — to solve a problem it doesn't actually solve. If the process dies between those two lines, the event is lost exactly like the email was.

The outbox: make the message part of the transaction

The insight is small and it's the whole thing: if the message lived in the same database as the business data, one transaction could commit both — atomically. No coordination problem, because there's only one system.

So you don't publish to a broker inside your request. You insert a row into an outbox table, in the same transaction as the order:

CREATE TABLE outbox (
  id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  job_type TEXT NOT NULL, -- 'send_confirmation', 'push_to_pos', ...
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending', -- pending | done | dead
  attempts INT NOT NULL DEFAULT 0,
  run_after TIMESTAMPTZ NOT NULL DEFAULT now(), -- for retry backoff
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- The index that makes the dequeue cheap: only ever scan runnable rows.
CREATE INDEX idx_outbox_runnable
  ON outbox (run_after)
  WHERE status = 'pending';


// The request handler. One transaction, two inserts, atomic.
await db.tx(async (t) => {
  const order = await t.orders.insert(orderRow);
  await t.outbox.insert({
    job_type: 'send_confirmation',
    payload: { orderId: order.id, customerId: order.customerId },
  });
});
// If this transaction commits, the job is durably queued.
// If it rolls back, there was no order AND no job. They can never disagree.

That's the trick. The message is now as durable and as consistent as the order itself, because it is the order's transaction. Nothing is published to anyone yet — a separate worker does that, on its own schedule, after the commit is real.

DUAL WRITE — two systems, no shared transaction, they drift
  [request] ──insert order──▶ [Postgres]
             └──publish event──▶ [Kafka] ✗ dies here → event lost

OUTBOX — one transaction owns both, a worker drains later
  [request] ──┐
                ├─(one tx)─▶ [Postgres: orders + outbox]
  order + job ──┘ │
                            [worker] ◀──┘ polls, sends, marks done
                                 │
                                 └──▶ email / POS / downstream

Draining the queue: FOR UPDATE SKIP LOCKED

Now the worker. You want several workers running for throughput, and you want them to never grab the same job. The naive "select pending, then update" has the classic race: two workers read the same row and both process it.

Postgres has a purpose-built answer, and it's the same primitive behind the atomic delivery dispatcher I wrote about: SELECT ... FOR UPDATE SKIP LOCKED. It says "lock the rows I'm taking, and skip any row another worker has already locked instead of blocking on it." Concurrent workers glide past each other and each pulls a disjoint batch:

async function drainBatch() {
  await db.tx(async (t) => {
    // Claim up to N runnable jobs; other workers skip these locked rows.
    const jobs = await t.query(`
      SELECT id, job_type, payload, attempts
      FROM outbox
      WHERE status = 'pending' AND run_after <= now()
      ORDER BY id
      FOR UPDATE SKIP LOCKED
      LIMIT 20
    `);

    for (const job of jobs) {
      try {
        await handlers[job.job_type](job.payload); // do the side effect
        await t.outbox.markDone(job.id); // status = 'done'
      } catch (err) {
        await backoffOrDie(t, job, err); // retry with delay, or park it
      }
    }
  });
}

Run drainBatch on an interval (or, better, LISTEN/NOTIFY to wake instantly on new rows and poll as a fallback). Run it in three processes and you have three concurrent, non-overlapping workers with zero extra infrastructure. The ORDER BY id gives you rough FIFO; the LIMIT bounds how much one worker holds at once.

Retry and backoff are just column math — no broker "redelivery" semantics to configure:

async function backoffOrDie(t, job, err) {
  const attempts = job.attempts + 1;
  if (attempts >= 8) {
    await t.outbox.update(job.id, { status: 'dead', attempts }); // dead-letter, alert on it
    return;
  }
  const delaySec = Math.min(2 ** attempts, 3600); // exponential, capped at 1h
  await t.outbox.update(job.id, {
    attempts,
    run_after: sql`now() + ${delaySec} * interval '1 second'`,
  });
}

The catch you must design for: at-least-once

Here's the honest part the "just use Postgres" posts gloss over. This queue is at-least-once , not exactly-once — and no queue, Kafka included, gives you true exactly-once delivery of a side effect. The reason is unavoidable: a worker can send the email, and then die before it writes status = 'done'. The row is still pending, so the next worker sends the email again.

You cannot close that window by being clever with ordering — "mark done, then send" just flips the failure to "marked done, never sent," which is worse. The window is fundamental. So you don't try to eliminate duplicates. You make duplicates harmless by requiring every handler to be idempotent:

• Sending email / push: dedupe on a natural key (order_id + 'confirmation') so the second send is a no-op.
• Calling a downstream API: pass an idempotency key so the provider collapses retries — the same discipline the payment webhook relies on.
• Writing to your own DB: a UNIQUE constraint or a conditional UPDATE ... WHERE status = 'pending' so a replay can't double-apply.

"Assume every job runs more than once, and make that fine" is the price of admission for at-least-once delivery. It's also, not coincidentally, exactly the discipline that makes a system resilient to any retry — client retries, load-balancer retries, a human clicking twice.

The safety net: reconciliation, not just delivery

The outbox guarantees a job is durably recorded. It does not, on its own, guarantee the downstream truth matches yours forever — a handler can succeed on your side and the downstream can still lose it, or a bug can park jobs as dead and no one notices. So the pattern isn't complete without a periodic reconciliation sweep that checks reality against your records:

// A cron that runs every few minutes — the backstop behind the queue.
async function reconcile() {
  // 1. Jobs that have been 'pending' far too long → the worker is wedged. Alert.
  // 2. Anything in 'dead' → surface it; a human or a fixed handler must resolve it.
  // 3. For integrations you can query back: ask the downstream
  // "what do you have that I don't, and vice versa?" and repair the delta.
}

This is the "scheduled job, not a lucky webhook" idea I closed the payment-webhook post promising to build — and it generalizes. At-least-once is a floor, not a ceiling: a message can still be dropped entirely by something outside your control. A reconciliation loop turns "we silently lost one" from an incident your customers report into a background task that self-heals. (I go deeper on this in Reconciliation, Not Sync.)

What actually broke in production

Four things bit me, and they're the four every outbox owner eventually meets:

1. Poison jobs wedged a worker. One malformed payload threw every time, and because I retried forever, that job's row got picked, failed, and re-queued in a hot loop that starved the batch. The fix is the dead state above: a hard attempt cap that parks the job and alerts, instead of retrying into infinity. A dead-letter you don't monitor is just a slower silence.

2. done rows piled up and the index bloated. The outbox table grew to millions of done rows and the runnable-rows query slowly got heavier. Two fixes: the partial index (WHERE status = 'pending') so the index only tracks live work regardless of table size, and a nightly job that deletes or archives done rows older than a few days. Treat the outbox as a spool, not a permanent log.

3. I reached for LIMIT without SKIP LOCKED first and workers serialized — each blocked waiting for the other's locked row instead of skipping it. Throughput collapsed to single-worker under contention. SKIP LOCKED is not optional; it's the entire reason this scales past one worker.

4. A slow handler held its transaction open too long. Because the claim-and-process ran inside one transaction, a handler that made a 30-second external call held row locks (and a connection) for 30 seconds. For slow side effects, claim the job in a short transaction (flip it to processing with a lease/locked_until timestamp), commit, then do the slow work outside the lock, and mark done in a second short transaction. Keep transactions that hold locks short — a lesson that shows up everywhere, including scaling the database itself under load.

When you actually do need Kafka

I'm not anti-Kafka; I'm anti-premature-Kafka. Reach for a real broker when you genuinely outgrow a table:

• Throughput in the hundreds-of-thousands of messages per second, sustained — past what one Postgres instance should be spooling.
• Fan-out to many independent consumer groups that each need their own replayable offset into a durable log.
• Event sourcing / stream processing where the log itself is the source of truth and you replay history to rebuild state.
• Cross-team decoupling where dozens of services subscribe and you want the broker as the contract boundary.

If you're sending a confirmation email, pushing an order to a POS, or kicking off a report — none of those are on that list. You have one app and one database. The outbox is less code, one fewer system at 3 a.m., and it's backed up by the same pg_dump you already run.

The one idea to take away

The dual-write problem doesn't get solved by adding a second system — it gets solved by making the message part of a transaction you already commit. Write the job into Postgres with your data, drain it with FOR UPDATE SKIP LOCKED, make every handler idempotent because delivery is at-least-once, and put a reconciliation sweep behind it. That's a durable, concurrent, retrying job queue in about forty lines and one table — and for most products, it's the whole messaging stack you were about to go operate a broker for.