DEV Community: Yury

The Other Half of the Dual-Write Problem: What Happens When a Job Finishes

Yury — Tue, 19 May 2026 13:00:00 +0000

In the first post of this series I talked about the dual-write problem on the producer side — the moment your app inserts a user and enqueues a "send welcome email" job, and one of those writes goes to Postgres while the other goes to Redis or Temporal. Two systems, no shared transaction, two ways to be inconsistent.

Queuert closes that gap by making startChain a row in the same transaction as the user insert. If the commit fails, the job never existed.

But the producer side is only half the story. The same dual-write hazard shows up again at the other end of the pipe — inside the attempt handler, when the job is actually running.

The Symmetric Problem

Same setup as the first post — Temporal activity, since that's the example we already have shared vocabulary for. The shape is the same in BullMQ, Sidekiq, Resque, SQS, anything where the queue's "completed" state lives outside your DB:

export async function chargePaymentActivity(orderId: number) {
  const order = await db.orders.findById(orderId);

  const { paymentId } = await stripe.charge(order.amount);

  await db.orders.update(order.id, { paymentId });

  return { paymentId };
  // ← Activity returns. The worker now reports completion to the
  //   Temporal cluster. Separate connection, separate transaction,
  //   separate system.
}

Four steps: read the order, charge Stripe, write payment_id back, return so Temporal records the activity as completed. Looks atomic. It isn't.

The seam is the last line. The DB update is one transaction against Postgres. The "mark this activity completed" write is a different transaction against the Temporal cluster's persistence layer, issued by the worker after the function returns. If the worker crashes between them — pod eviction, OOM, network blip, anything — Postgres has the payment_id, Temporal still thinks the activity is in flight, and the workflow re-runs the activity on the next attempt. Including the Stripe charge.

The fixes everyone reaches for are familiar: pass an idempotency key to Stripe, add a WHERE payment_id IS NULL to the update, maybe wrap the handler in a "did I already do this work?" guard. All of that is real, all of it works, and all of it is defensive plumbing that exists purely because the DB and the queue can't agree on whether the work happened.

This is the same dual-write problem, mirrored. Producer side: did the row and the job both get created? Consumer side: did the result write and the ack both land?

Why Most Libraries Don't Fix This

The honest answer is that most job libraries can't fix this, because the worker process and the library's completion write don't share a transaction with your handler. The queue's notion of "this job is done" and your DB's notion of "this work happened" live in different transactions, and the library's answer is to hand you idempotency as homework.

Receipts, one library at a time — expand the one you actually use:

BullMQ, Bee, Sidekiq, Resque, SQS, Cloud Tasks
Anything where the queue lives in Redis, RabbitMQ, or a managed broker. The worker does business work against your DB, then sends a separate ACK over the wire to the broker. Two systems, no shared transaction. The standard advice — "just make your handlers idempotent" — is real advice, but it's the library handing you back the dual-write problem and calling it a feature. You're now responsible for inventing an idempotency key for every write your handler performs.

Temporal
Temporal owns the truth, but in a third cluster. The activity finishes, reports success to the Temporal server, and that's a separate write from whatever your activity did to your application DB. The workflow engine reconciles the world by re-running activities until they look done — which is powerful, but it pushes you toward idempotency-by-default for every activity that touches state, and it's an entire service tier to operate.

graphile-worker
Postgres-native, which is the right substrate, but the task handler's withPgClient pulls a fresh pooled connection and the job's completion runs in a separate transaction issued by the worker pool after the task returns. Two transactions, no way to merge them. Better than Redis, but the dual-write surface is still there.

pg-boss
Worth a careful look, because its README markets "exactly-once job delivery" and it's easy to assume that means the consumer side is solved. It doesn't. The phrase refers to SKIP LOCKED on the fetch path — two workers can't claim the same row. The standard work() worker still runs your handler, returns, and then pg-boss calls complete() against its own pool in a separate transaction. If your handler commits domain writes and the worker crashes before complete() lands — or the job's lease expires (expireInSeconds, default 15 minutes) — the job is re-fetched and the handler runs again. Domain writes commit twice. pg-boss v12.17 added a { db } option that's accepted on send(), complete(), fetch(), and friends — so the primitives for atomic completion exist. But the work() worker loop doesn't surface them; to actually get a handler-tx-fused-with-completion-tx flow you have to opt out of work(), write your own fetch() + handler + complete(jobId, output, { db: tx }) loop, and re-implement the lease, retry, and backoff machinery yourself. Atomic consumer-side processing is possible, but it's a parallel API you build, not the shape of the supported worker.

Atomic Mode: When Dual-Write Never Happens

Queuert handlers come in two shapes. The simpler one — and the one most jobs should use — is atomic mode: the entire handler runs inside one transaction in your DB (Postgres or SQLite — Queuert supports both).

Here's reserve-inventory, which decrements stock, records the reservation, and queues the next step in the chain:

'reserve-inventory': {
  attemptHandler: async ({ job, complete }) =>
    complete(async ({ sql, continueWith }) => {
      const [item] = await sql`
        SELECT stock FROM items WHERE id = ${job.input.itemId} FOR UPDATE
      `;
      if (item.stock < 1) throw new Error("Out of stock");

      await sql`UPDATE items SET stock = stock - 1 WHERE id = ${job.input.itemId}`;
      await sql`
        INSERT INTO reservations (order_id, item_id)
        VALUES (${job.input.orderId}, ${job.input.itemId})
      `;

      return continueWith({
        typeName: "charge-payment",
        input: { orderId: job.input.orderId },
      });
    }),
}

One transaction wraps all of this:

The SELECT ... FOR UPDATE that locks the item row.
The UPDATE that decrements stock.
The INSERT into reservations.
The mark-as-completed write on the job row.
The insert of the charge-payment continuation.

If anything throws, nothing commits — the savepoint rolls back, the job stays pending, and the engine reschedules it with backoff. If the worker process dies mid-transaction, the DB rolls everything back on connection close. The next attempt starts from a clean slate.

There is no dual-write window because there is no second write. You never need an idempotency key for reserve-inventory, because the engine cannot leave the job half-done — Postgres won't let it. If your handlers are DB-bound (state machines, counters, ledger updates, workflow transitions), this is the mode you live in, and the consumer-side dual-write problem simply doesn't exist for your code.

Staged Mode: When You Have to Call Out

Atomic mode works because nothing in the handler escapes the transaction. The moment you need to call an external system — Stripe, SES, an internal microservice — that stops being true. You can't hold a database transaction open across a Stripe API call (the lock contention alone would melt your DB), and you don't want network failures rolling back DB work that's already valid.

That's what staged mode is for: read in one transaction, do the external work outside any transaction, write the result in a second transaction.

'charge-payment': {
  attemptHandler: async ({ job, prepare, complete }) => {
    const order = await prepare({ mode: "staged" }, async ({ sql }) => {
      const [row] = await sql`SELECT * FROM orders WHERE id = ${job.input.orderId}`;
      return row;
    });

    // External call — outside any transaction, must be idempotent.
    const { paymentId } = await stripe.charge(order.amount, {
      idempotencyKey: `order-${order.id}-attempt-${job.attempt}`,
    });

    return complete(async ({ sql, continueWith }) => {
      await sql`UPDATE orders SET payment_id = ${paymentId} WHERE id = ${order.id}`;
      return continueWith({
        typeName: "send-payment-receipt",
        input: { orderId: order.id, paymentId },
      });
    });
  },
}

The Stripe call still has to be idempotent — that's physics, not a library problem. You can't take back a network request, and any handler that calls an external system has to assume it might re-run after a retry. The idempotency key is unavoidable.

But everything that records what happened — the UPDATE, the job's transition to completed, the send-payment-receipt continuation — lands in one DB commit. There is no second system to ack. The job's "done" is a row update next to your UPDATE orders. Either both commit or neither does.

If the worker dies after Stripe returns but before complete finishes, the next attempt re-runs Stripe with the same idempotency key (Stripe returns the original charge), reads the order, sees payment_id is still null, and writes it. One source of truth, no reconciliation, no "did the ack make it back?" branch.

The thing staged mode buys you is this: even though the external call breaks the single-transaction property, the recording of its result stays atomic with everything downstream of it.

The Shape of the Guarantee

Put the two posts together and the picture is:

Producer side: "I created a user and a job." → One commit. Atomic.
Consumer side, atomic mode: "My job is DB-bound." → One commit. Reads, writes, completion, next chain step — all atomic. No idempotency keys needed.
Consumer side, staged mode: "My job calls an external system." → External call must be idempotent (always true, in every library). Recording the result is one commit, atomic with marking the job done and queuing the next step.

No Temporal cluster keeping a third ledger in sync. No outbox table to babysit. No "what if the ack didn't make it back?" branch in your retry logic. The job table lives next to your business tables in the same DB — Postgres or SQLite — and the database handles atomicity for both. That's the one thing relational databases are unambiguously good at.

The dual-write problem was never really about queues. It was about having two systems that needed to agree and no way to make them. Removing the second system — or, in pg-boss's case, opting into a wiring pattern that papers it over — is the only real fix. Queuert just makes it the default shape of the API rather than a feature you have to remember to turn on.

There's one more tier of side effect that doesn't fit either mode — the "fire after commit, but I'm fine if it gets dropped" kind. That's what transaction hooks are for, and they have an important non-guarantee worth understanding before you reach for them. That's the next post.

TypeScript-first job chains: end-to-end inference for background jobs

Yury — Mon, 04 May 2026 22:00:00 +0000

Most job queues forget your types at the queue boundary

A Promise chain in TypeScript is a thing of beauty:

const summary = await fetchUser(id)
  .then((user) => fetchOrders(user.id))
  .then((orders) => summarize(orders));
//        ^^^^^^ Order[]

Each step's input is typed by the previous step's output. Try to call fetchOrders(user.something_that_doesnt_exist) and the compiler stops you. Misuse the summary variable at the end and the compiler stops you.

Now the typical background-job version of the same thing:

await queue.add("fetch-user", { id });
// ... worker processes fetch-user, then enqueues fetch-orders ...
// ... worker processes fetch-orders, then enqueues summarize ...

The compiler sees string (the type name) and unknown (the payload). Whatever the fetch-user handler returned, nothing checks that the input to fetch-orders matches it. Renaming a field on the user payload silently breaks production.

This isn't a TypeScript limitation. It's a design choice in most job libraries: jobs are loose key-value messages, scheduled by name. The type system has nothing to anchor to.

I wanted job chains that worked more like Promise.then() — but persisted, retryable, and able to live across worker restarts. I wrote a library called Queuert that does this. Here's the type story.

Defining job types

You start by declaring the shape of every job type your application uses:

import { defineJobTypes } from "queuert";

const jobTypes = defineJobTypes<{
  "fetch-user": {
    entry: true;
    input: { userId: number };
    output: { id: number; email: string; name: string };
    continueWith: { typeName: "fetch-orders" };
  };
  "fetch-orders": {
    input: { userId: number };
    output: { orders: { id: number; total: number }[] };
    continueWith: { typeName: "summarize" };
  };
  "summarize": {
    input: { orders: { id: number; total: number }[] };
    output: { totalRevenue: number };
  };
}>();

Each entry has:

input — what the job receives.
output — what the handler returns.
continueWith — which job types this one can chain into next (optional; terminal jobs omit it).
entry: true — marks the type as one that can start a chain.

The whole declaration is a single TypeScript type literal. There's no codegen, no decorators, no separate schema language.

Type-checked chain creation

Starting a chain checks the entry type's input against the literal you pass:

client.startChain({
  // ...
  typeName: "fetch-user",
  input: { userId: 42 },
  //         ^^^^^^ must match { userId: number }
});

client.startChain({
  // ...
  typeName: "summarize",
  //         ^^^^^^^^^^^ Type '"summarize"' is not assignable
  //                     to type '"fetch-user"'
  // — only `entry: true` types are allowed at the start of a chain.
});

You can't start a chain at a non-entry type. You can't start it with the wrong input. Both errors are caught at compile time, before the job hits your queue table.

Type-checked continuations

Inside a handler, you call continueWith to chain into the next job. The library checks two things:

The typeName you pass must be one of the types declared in the current job's continueWith.
The input must match the next type's declared input.

"fetch-user": {
  attemptHandler: async ({ job, complete }) => {
    const user = await fetchUserFromDb(job.input.userId);

    return complete(async ({ continueWith }) =>
      continueWith({
        typeName: "fetch-orders",
        //         ^^^^^^^^^^^^ must be in fetch-user's `continueWith` declaration
        input: { userId: user.id },
        //         ^^^^^^ must match fetch-orders' declared input
      }),
    );
  },
},

Try to chain to a job type the current type didn't declare? Compile error. Pass the wrong input shape? Compile error.

This is the part that feels like Promise chains. The output of one job flows into the input of the next, type-checked at every step.

Branching

A chain can branch by listing multiple types in continueWith:

"trial-decision": {
  input: { subscriptionId: number };
  continueWith: { typeName: "convert-to-paid" | "expire-trial" };
};

In the handler, the compiler accepts either branch and forces you to pass the matching input for whichever you pick:

return complete(async ({ continueWith }) => {
  if (shouldConvert) {
    return continueWith({
      typeName: "convert-to-paid",
      input: { subscriptionId: job.input.subscriptionId },
    });
  }
  return continueWith({
    typeName: "expire-trial",
    input: { subscriptionId: job.input.subscriptionId },
  });
});

Each branch is type-checked independently against its target type's input.

Loops

A job can chain back to itself:

"charge-billing": {
  input: { subscriptionId: number; cycle: number };
  output: { finalCycle: number; totalCharged: number };
  continueWith: { typeName: "charge-billing" | "cancel-subscription" };
};

This is just a self-reference in the type system. The compiler resolves it cleanly. You can build retry loops, polling loops, recurring billing — anything where a job re-schedules itself based on its output and decides when to terminate.

Fan-in: typed blockers

Sometimes a job needs to wait for several other chains to complete before running. In Queuert that's a blockers declaration.

Variable-count blockers (e.g. "wait for any number of fetch-source chains"):

"aggregate-data": {
  entry: true;
  input: { reportId: string };
  output: { reportId: string; totalSources: number };
  blockers: [...{ typeName: "fetch-source" }[]];
};

Inside the handler, job.blockers is a typed array — each entry is a completed fetch-source job, with its output already inferred:

"aggregate-data": {
  attemptHandler: async ({ job, complete }) => {
    for (const blocker of job.blockers) {
      console.log(blocker.output.sourceId, blocker.output.data);
      //                  ^^^^^^^^^^^^^^^      ^^^^^^^^^^^^^^^
      //              from fetch-source's declared output
    }
    // ...
  },
};

Fixed-slot blockers (e.g. exactly one validate-user and one load-config) are also typed at the right index:

"perform-action": {
  blockers: [
    { typeName: "validate-user" },
    { typeName: "load-config" },
  ];
};

Now job.blockers[0].output is the validate-user output and job.blockers[1].output is the load-config output, with each type inferred at the right index.

What this prevents

A non-exhaustive list of bugs the compiler refuses to let you ship:

Renaming a field on a job's output without updating downstream handlers.
Chaining to a non-entry type as if it were an entry.
Chaining to a type that wasn't declared in continueWith.
Passing the wrong shape to a chain's input.
Reading a field off a blocker's output that doesn't exist.
Treating a fixed-slot blocker like an array of one type when it actually has heterogeneous slots.

In a typical Redis-backed queue, all of these become runtime errors — usually the kind that show up at 3am in your error tracker.

What it costs

Type-machinery this dense isn't free. Queuert publishes a type complexity benchmark: linear chains up to 100 types compile in ~1s, branched/blocker/loop graphs of similar size stay around the same, and merging 500 types from independent slices stays under 2.5s. There's a ceiling, but it's well above realistic application complexity.

The library also internally caches navigation maps and uses tail-recursive type forms to keep instantiation counts down — a 0.5.0 rewrite cut blocker-heavy navigation by ~86%. If you ever hit a ceiling, it's a tractable problem.

Try it

Queuert is MIT-licensed and pre-1.0. There's a companion post on the architectural side — why your database should be the source of truth for job state — that pairs with this one.

Repo: github.com/kvet/queuert
Docs: kvet.github.io/queuert

The dual-write problem (and a Postgres-native fix for Node.js background jobs)

Yury — Fri, 01 May 2026 18:08:32 +0000

What Temporal taught me about state

I've spent years building on Temporal — workflows-as-code, durable execution, the whole gospel. The thing Temporal got right, the thing that made it hard for me to go back to ordinary Redis-backed queues, is this: workflow state is durable on every step. It lives in a transactional database. There's no in-memory state that disappears on a crash, no message that's "in flight" without being persisted, no step that completes without its result being safely written.

But Temporal's durability story has a quiet caveat: it only protects state inside a workflow. The moment you have an application database alongside the workflow service — and most projects do — you're back to two systems.

Consider this signup handler:

import { Client } from "@temporalio/client";
import { sendWelcomeEmail } from "./workflows";

const temporal = new Client();

await db.transaction(async (tx) => {
  const user = await tx.users.create({ name: "Alice", email: "alice@example.com" });

  await temporal.workflow.start(sendWelcomeEmail, {
    taskQueue: "main",
    workflowId: `welcome-${user.id}`,
    args: [{ userId: user.id, email: user.email }],
  });
});

Innocent enough. Insert a user, start a workflow, life moves on.

Except sometimes Alice never gets the email. Other times Alice gets a welcome email pointing to a userId that doesn't exist. Why?

The dual-write problem

The application database and the workflow service are two different systems. The transaction in the snippet above is a Postgres transaction. temporal.workflow.start() writes to the Temporal cluster's own database. Neither system knows about the other.

So: tx.users.create() succeeds. workflow.start() succeeds. Then commit fails — constraint violation, network blip, anything. You now have a workflow running for a user that doesn't exist.

Reverse it: tx.users.create() succeeds. workflow.start() fails. You now have a user with no welcome workflow, and no record of the missing one.

Reverse it again: workflow.start() succeeds in transit but the connection drops before you get the handle back. The workflow is running. You retry, and (if your workflowId collides) you fail open or get two workflows.

This is the dual-write problem, and it's not unique to BullMQ-on-Redis — it shows up any time job/workflow state lives in a different system from your business state. BullMQ is just the most obvious case because Redis isn't transactional with your Postgres. Temporal hides it better but doesn't eliminate it.

Temporal's official answer is to make the workflow service the source of truth — keep workflow-relevant state inside Temporal, lean on Sagas for cross-service consistency, and treat your app DB as a read model fed by workflow events. That works, and it's the right answer for Temporal-native projects. But it's a real architectural commitment: a service tier with its own cluster, history/matching/frontend services, a Postgres or Cassandra of its own, and SDKs to keep in sync.

The standard workaround: transactional outbox

The textbook fix in the BullMQ-style world is the transactional outbox pattern. Instead of writing directly to your queue, you:

Insert a row into an outbox table inside the same transaction as your business write.
A separate process polls the outbox and forwards rows to your queue.
Once acknowledged, the outbox row is deleted (or marked sent).

This works. It also means you're now maintaining:

An outbox table and its schema migrations.
A poller process and its supervisor.
An idempotency layer (your queue can receive the same outbox row twice if the poller crashes between forwarding and marking sent).
Monitoring for outbox lag.
A retry policy.

For a small team shipping background jobs, that's a lot of infrastructure to maintain just to avoid orphaned data. And once you're maintaining it, you have to ask: what is Redis actually buying me here, that's worth this much glue code?

A smaller-shape question

Temporal's answer to the dual-write problem is "run the whole workflow service." That's the right answer when you need it — workflows-as-code, replay semantics, polyglot SDKs, a managed control plane, history that survives any single component dying. But it's also a service tier: separate cluster, history/matching/frontend services, a Postgres or Cassandra of its own, worker SDKs to keep in sync with the server.

For a lot of Node.js projects, that's overkill. You just want background jobs with the same durability guarantees Temporal taught you to expect — atomic with your business writes, no orphaned state, no in-flight messages that vanish on a crash.

What if that durability instinct could come in a smaller shape? What if the job queue was your existing Postgres — queue.add() becoming an INSERT into a jobs table inside your transaction?

That's the design behind Queuert, an open-source library I wrote to scratch exactly that itch.

What it looks like

Same signup handler, rewritten with Queuert (Kysely + Postgres in this example):

import { withTransactionHooks } from "queuert";

await withTransactionHooks(async (transactionHooks) =>
  db.transaction().execute(async (tx) => {
    const user = await tx
      .insertInto("users")
      .values({ name: "Alice", email: "alice@example.com" })
      .returningAll()
      .executeTakeFirstOrThrow();

    return client.startChain({
      db: tx,
      transactionHooks,
      typeName: "send-welcome-email",
      input: { userId: user.id, email: user.email },
    });
  }),
);

Two key things:

startChain takes the same transaction tx as the user insert. They're literally part of the same Postgres transaction. If the transaction rolls back, the job is never created. If the transaction commits, the job is created exactly once. No outbox needed.
transactionHooks defers side effects until after commit. Things like notifying workers (so they pick up the new job immediately) are buffered during the transaction and only fire if commit succeeds. If commit fails, the hooks are discarded.

That's the whole story for atomic consistency. No second system, no forwarder, no retry loop — just a BEGIN/COMMIT wrapped around your business write and an INSERT into the queuert_jobs table.

What you stop needing

Redis as a state store. You can still use Redis for low-latency wake-up notifications (Pub/Sub-style), but it's optional and stateless. Jobs are rows in your database. If Redis disappears, workers fall back to polling and nothing is lost.
An outbox table. The jobs table is the outbox.
A separate forwarder process. Workers query the jobs table directly with SELECT … FOR UPDATE SKIP LOCKED — a pattern Postgres has had since 9.5.
Two backup strategies, two replication setups, two monitoring stacks. One database, one set of operational tooling.

What you gain (besides the consistency)

If jobs are first-class rows in your database, a lot of things become easier:

You can JOIN on them. "Find all pending welcome emails for users created in the last hour" is a query, not a custom analytics pipeline.
You can use your existing audit logging. Every job state transition is a database write. If you already track changes via triggers or CDC, you get job history for free.
You can run integration tests against a real database. No mocking the queue. The job table is just another table.
Schema migrations stay in one place. Job table changes ride along with your application's migrations.

What this isn't

This isn't a Temporal replacement, and it isn't trying to be. If you need workflows-as-code, replay semantics, deterministic execution, polyglot SDKs, or a managed control plane — use Temporal. That's still what I reach for on bigger projects, and it's worth every operational dollar when the workflow surface justifies it.

It also isn't going to outscale dedicated queue infrastructure at the high end. Postgres handles a lot — the published benchmarks show ~21k chain creations per second batched and ~770 jobs/sec processed atomically on a Dockerized Postgres against a single worker — but if you're at the scale where a job queue is its own infrastructure tier with a dedicated team, dedicated tools probably make sense.

The target is the in-between: small-to-medium Node.js projects where Temporal is too heavy and BullMQ reintroduces the dual-write problem.

Try it

Queuert is MIT-licensed and pre-1.0. Postgres and SQLite adapters; in-process, Redis (incl. Cluster), NATS, and Postgres LISTEN/NOTIFY for the optional notify layer; an embeddable web dashboard; OpenTelemetry tracing across job chains.

Repo: github.com/kvet/queuert
Docs: kvet.github.io/queuert

A follow-up post on the TypeScript story — how job chains stay type-checked end-to-end across continueWith, branching, loops, and fan-in blockers — is up next.