Here's a bug that only shows up in production, usually under load, and never in your tests.
A user generates a track. The AI provider sends a "complete" webhook. Your handler saves the finished audio. Then — a few hundred milliseconds later — a second webhook arrives for the same job, this one a stale "first track ready" event that got delayed in the network. Your handler dutifully processes it and flips the task's status from complete back to generating. The user is now staring at a spinner for a song that finished a second ago.
Nobody wrote a bug. Every line did exactly what it says. The system is still wrong, because the webhook contract has three properties almost everyone forgets to design for. This post is about all three, using patterns from a Next.js app that runs AI music generation for real users.
Why "just save the result" is broken
When you offload a long AI job to a provider like Replicate, fal, you give it a callback URL and it webhooks you as the work progresses. That delivery has two guarantees you don't get to opt out of:
- At-least-once. The provider retries on any non-2xx response, on timeouts, and on network blips. It considers a webhook delivered only when it sees your 200 — so if your 200 is slow or lost, you get the same event again. Duplicates are the default, not the exception.
- No ordering. A single generation emits multiple callbacks (queued → text → first track → complete). They travel over the open internet. A later event can overtake an earlier one.
So your handler will be called with the same event more than once, and with events out of order. A handler that just inserts or overwrites on every call corrupts data under both conditions. To be correct it needs three properties at once: deduplication, ordering, and atomicity.
Property 1: deduplication — key on the provider's ID
The first instinct is an auto-increment primary key. Don't. If the same "complete" event arrives twice, you get two rows and, if you charge credits, two charges.
Key every write on the provider's own task ID, which is stable across retries:
// The provider's taskId is the identity of the work, not your row id.
const existing = await store.getByProviderTaskId(payload.taskId)
Now reprocessing the same event operates on the same record instead of creating a new one. Deduplication isn't a lookup table of "seen IDs" — it's choosing an identity that the retried event shares with the original. That single decision makes the rest of the handler tractable.
Property 2: ordering — model status as a forward-only state machine
A boolean isComplete can't express "this event is older than what I already know." A state machine can.
Give each state a rank and let events only move the task forward:
const RANK = { queued: 0, text: 1, first: 2, complete: 3, failed: 3 } as const
function isAdvance(current: Status, incoming: Status) {
return RANK[incoming] > RANK[current]
}
Now the stale "first track" (rank 2) arriving after "complete" (rank 3) is simply not an advance, so it's dropped. The out-of-order class of bugs disappears — not because you handle every ordering, but because you made backward transitions impossible by construction. Terminal states (complete, failed) are sinks: once reached, nothing moves the task again.
Property 3: atomicity — the read-then-write gap is a race
Here's the part almost every tutorial skips, and it's the reason the bug from the intro is so hard to reproduce. Even with deduplication and a state machine, this code is still wrong:
const task = await store.getByProviderTaskId(payload.taskId) // read
if (isAdvance(task.status, payload.status)) {
await store.update(task.id, { status: payload.status }) // write
}
Two webhooks for the same job can arrive within milliseconds and run concurrently. Both read status first. Both decide their event is an advance. Both write. The check-then-act is not atomic, so the guard you just built is a suggestion, not a lock. Serverless makes this worse: each webhook may land in a separate function instance, so an in-memory mutex protects nothing.
The fix is to make the database enforce the transition in a single statement, so the ordering check and the write happen atomically:
// Advance only if the stored rank is still lower — evaluated by the DB, once.
await db
.update(tasks)
.set({ status: incoming, updatedAt: Date.now() })
.where(and(
eq(tasks.providerTaskId, payload.taskId),
lt(tasks.statusRank, RANK[incoming]), // guard lives in the WHERE clause
))
If two callbacks race, the database serializes them. The first advances the row; the second finds statusRank already at or above its own value, matches zero rows, and no-ops. The guard now lives where concurrency is actually resolved. If your provider payloads carry a monotonic timestamp or sequence number, guard on that instead of a derived rank — same shape, fewer assumptions.
Putting it together
The whole handler is small once the three properties are in place:
export async function POST(req: Request) {
const raw = await req.text()
const payload = JSON.parse(raw)
// Property 1 + 2 + 3: identity + forward-only guard, atomic in one write.
const updated = await db
.update(tasks)
.set({ status: payload.status, statusRank: RANK[payload.status], result: payload.result })
.where(and(
eq(tasks.providerTaskId, payload.taskId),
lt(tasks.statusRank, RANK[payload.status]),
))
.returning()
// Ack fast. A duplicate/stale event updated nothing — that's success, not failure.
return Response.json({ ok: true })
}
Two things worth calling out. First, always return 2xx once you've durably accepted the event, even when it changed nothing — returning 500 on a harmless duplicate just triggers another retry on a system that's already busy. Second, do the heavy work (post-processing audio, generating cover art) after the atomic status write, keyed off the same task ID, so it also inherits the idempotency instead of racing on its own.
This is the pattern behind an AI music generator we build and run, where a single generation fans out into several ordered callbacks and every one of them is, from the network's point of view, allowed to arrive twice.
Takeaways
- Duplicates and reordering are the contract, not edge cases. Design for at-least-once, no-ordering delivery from line one.
- Identity is the provider's task ID. Key writes on it and reprocessing becomes a no-op instead of a duplicate row or double charge.
- A state machine beats a boolean. Rank your states and allow forward transitions only; stale events fail the guard automatically.
- The guard must live in the writing, not before it. Push the ordering check into the SQL WHERE clause so the database resolves concurrent callbacks — an if statement in your handler can't.
- Ack fast, work after. Return 2xx on durable acceptance; run side effects keyed on the same ID so they're idempotent too.
Get these three properties right and the flaky, unreproducible "sometimes the finished job goes back to loading" bug never gets written in the first place.
Top comments (0)