Where async work goes to die, and how I made it observable

There's a particular kind of bug that only shows up on systems you didn't know you were running. A user clicks a button. A job is queued. The worker picks it up. The worker crashes or just hangs. The user refreshes. The dashboard says processing. The founder finds out a couple of days later in a support email.

The problem is not that queue-based work is fragile. It is that async work crosses invisible boundaries, and most stacks give you no way to inspect what happened at each one. The job is in flight. The job is done. The job is lost. Those three states collapse into the same status field, and the rest is hope.

I got tired of the hope. This post walks through how I rewired my async pipeline so each boundary has its own inspectable state. The architecture sits behind a small multimodal learning product I run solo.

The shape of the problem

Async work in a serverless stack has at least five boundaries, and most outages happen at the gaps between them:

• Acceptance boundary. Did the system actually take the work, or did the request return before the work was durably claimed?
• Queue delivery boundary. Did the message reach the worker, or did it get stuck, dropped, or duplicated?
• Persistence boundary. Once the worker processed the work, is the state durable and queryable?
• Recovery boundary. If a worker dies, does another worker notice, and can it take over safely?
• Authoritative status boundary. When something later asks did this finish?, is there one place that knows?

You can build a system that handles most of those well and still lose work. The last one is usually the one nobody writes down. I'll go boundary by boundary and show what I shipped at each one.

Acceptance: an atomic claim, not an OK response

The first rule I wrote myself: never let an HTTP request return success before the work is durably claimed. An OK response is an acknowledgment that the request was received. It is not a claim.

So the request lands in a Worker, and the Worker does one thing first. It opens a D1 transaction, inserts a task row with a claimed_at timestamp, and only then returns. If the row insert fails, the request fails. If it succeeds, the work has a home before the queue is touched.

// acceptance boundary: claim the row before responding (illustrative shape)
await db.batch([
  db.prepare(
    `INSERT INTO tasks (id, kind, status, claimed_at, attempts)
     VALUES (?, ?, 'claimed', ?, ?)`
  ).bind(taskId, kind, now, initialAttempts),
  db.prepare(
    `INSERT INTO task_events (id, task_id, kind, dedupe_key)
     VALUES (?, ?, 'enqueued', ?)`
  ).bind(eventId, taskId, idempotencyKey),
]);

The dedupe_key matters more than the row insert. If a client retries the same request (a webhook that fired twice, a UI that double-clicked, an upstream that timed out and replayed), the second attempt hits the unique constraint on task_events.dedupe_key and aborts cleanly. No duplicate work. No we sent two emails support tickets.

The boundary test is simple: pull the power on the Worker between the HTTP response and the queue publish. If the task is recoverable from D1 alone, the acceptance boundary is honest.

Queue delivery: one DLQ per logical queue

A managed queue is durable, but durable is not the same as always delivered. Retries run out. Messages expire. A poison payload can wedge a worker forever. If you only have one queue and one DLQ, you find out about the poison payload when the DLQ is full of unrelated retries.

I split the queue namespace by what the work is doing, not by who produced it. Each logical queue gets its own DLQ:

wrangler.toml: queue topology (illustrative shape)

[[queues.producers]]
queue = "email"
binding = "EMAIL_QUEUE"

[[queues.producers]]
queue = "ai"
binding = "AI_QUEUE"

[[queues.producers]]
queue = "webhook"
binding = "WEBHOOK_QUEUE"

[[queues.producers]]
queue = "github"
binding = "GITHUB_QUEUE"

[[queues.consumers]]
queue = "email"
dead_letter_queue = "email-dlq"
max_retries = retryBudget

[[queues.consumers]]
queue = "ai"
dead_letter_queue = "ai-dlq"
max_retries = retryBudget

webhook and github follow the same shape

When email-dlq starts filling, I know it is email-specific failure: bad templates, an upstream provider returning server errors, a malformed header. It is not a generic worker crash I have to chase. The DLQ is a signal, not a junk drawer. I drain it explicitly, and the act of draining is itself a D1 write so the inspection loop closes.

Persistence: queues transport work, D1 stores truth

The first version of this system kept status inside the queue message. That was a mistake. A queue message is a fact about now. Status is a fact about history. As soon as a worker needed to ask did this finish yesterday?, the queue was useless.

So the rule became: queues transport work, D1 stores truth. The schema is deliberately boring:

-- persistence boundary: task state in D1 (illustrative shape)
CREATE TABLE tasks (
  id TEXT PRIMARY KEY,
  kind TEXT NOT NULL,
  status TEXT NOT NULL,         -- claimed | running | done | failed | dead
  claimed_at INTEGER,
  started_at INTEGER,
  finished_at INTEGER,
  attempts INTEGER NOT NULL,
  last_error TEXT
);

CREATE INDEX tasks_status_idx ON tasks(status);
CREATE INDEX tasks_kind_status_idx ON tasks(kind, status);

The status column is intentionally an open enum. When I add a new pipeline stage I can introduce a new value without a migration. The attempts column is the recovery counter the next boundary needs.

What this buys me: a one-line SQL query tells me how many tasks are stuck in claimed or running past their expected duration. That is the dashboard. I don't need a tracing system, an APM, or a log search. The state is queryable, not inferred.

Recovery: heartbeats and a periodic reset

Workers die. That is the whole point of a queue. The question is what happens to the work they were holding when they died.

A heartbeat column on the task row is the simplest mechanism that actually works:

// worker: heartbeat while running (illustrative shape)
const heartbeat = setInterval(async () => {
  await db.prepare(
    `UPDATE tasks
     SET last_heartbeat_at = ?
     WHERE id = ? AND status = 'running'`
  ).bind(Date.now(), taskId).run();
}, heartbeatIntervalMs); // illustrative cadence

try {
  await doWork(task);
  await db.prepare(
    `UPDATE tasks SET status='done', finished_at=? WHERE id=?`
  ).bind(Date.now(), taskId).run();
} finally {
  clearInterval(heartbeat);
}

A separate recovery Worker runs on a schedule and asks one query:

-- recovery boundary: find rows whose worker stopped heartbeating (illustrative shape)
SELECT id FROM tasks
WHERE status = 'running'
  AND last_heartbeat_at < ?   -- now minus heartbeat window
ORDER BY last_heartbeat_at ASC
LIMIT ?;

Each of those rows gets reset to claimed with attempts = attempts + one. If attempts exceeds the retry budget, the row goes to dead and the DLQ catches the corresponding message. That is the orphan-task reset, and it is what stops a crashed worker from holding a job hostage.

The auth boundary sits on the wake-up side. When the pipeline calls back, the request carries a bearer token, a pipeline identity, an HMAC signature over the body, and a timestamp with a small replay window. The Dispatcher (the small scheduling service sitting between the main app and the pipeline) verifies all four before doing anything. The verification is cheap. The discipline is mandatory. A callback without HMAC and timestamp is a webhook-shaped security hole.

Authoritative status: the callback is a wake-up, not an answer

The last boundary is the one I almost skipped. The pipeline sends a callback when it is done. I was about to trust that callback as the source of truth.

I changed my mind. The callback is now treated as a wake-up signal, not a status update. The Dispatcher receives the callback, validates the auth, and immediately runs an authoritative query against the pipeline to ask is this job actually done?. Only then does it mark the task done in D1. If the authoritative query fails or times out, the task stays running and the polling fallback takes over.

// callback handler: never trust the callback alone (illustrative shape)
export async function handlePipelineCallback(req: Request) {
  const auth = await verifyPipelineAuth(req); // bearer + id + HMAC + ts
  if (!auth.ok) return new Response('unauthorized', { status: HttpStatus.Unauthorized });

  const { taskId } = await req.json();

  // wake the recovery loop
  await requeueStatusCheck(taskId);

  // query the pipeline authoritatively
  const status = await pipeline.statusFor(taskId);

  // mutate state only from the query result
  if (status.state === 'done') {
    await markDone(taskId);
  } else if (status.state === 'failed') {
    await markFailed(taskId, status.error);
  }
  // else: leave it running; polling will catch up
}

Polling is the fallback, not the strategy. It runs on a fixed cadence for any task in running past its expected duration, and it runs the same authoritative query. If the callback was lost, polling catches it. If polling fails, the next heartbeat timeout resets the task and the worker retries. Three independent paths reach the same authoritative answer.

What this design doesn't promise

This architecture does not give me a delivery guarantee. The managed queue gives at-least-once delivery; my DLQs catch the rest, but they don't auto-recover. If email-dlq fills up in the middle of the night, no one wakes me. I built alerts, not autonomy.

It also doesn't replace tests. The atomic claim is correct only because the D1 transaction is correct. The heartbeat reset is correct only because the SQL is correct. I have caught bugs in every one of these boundaries during refactors. The architecture makes those bugs visible. It does not make them impossible.

And none of this is novel. Atomic claim, idempotency keys, dead-letter queues, heartbeat recovery, callback-as-wake-up: these are patterns from the distributed-systems canon, transplanted into a Workers + D1 + Queues shape. The contribution isn't invention. It is the discipline of treating each boundary as something I can show the code for.

Why I bother

I am a solo founder. I don't have an SRE team. The only way my product survives contact with reality is if the architecture is something I can hold in my head and inspect late at night with one terminal open.

Splitting async work into five inspectable boundaries (acceptance, queue delivery, persistence, recovery, authoritative status) gave me that. Each one has a table, a queue, a query, or a config block I can point at. When something goes wrong, I don't ask what happened? I ask which boundary?. The answer narrows the search to a few hundred lines.

That is not a slogan. It is the reason this article exists at all. The code is the only proof I can offer without inventing a war story.

Comments

[Cubert Wang]

I’m documenting more of the systems behind this product as I build them—especially the parts that failed, changed, or looked simpler on paper than they were in production.

If you’re building with serverless queues, Workers, or async pipelines, follow me here. I’ll keep sharing the architecture decisions, implementation details, and mistakes that were actually useful.