The Jobs That Never Came Back: How I Fixed Silent Failures in an Async Video Pipeline

Gabriella Andrews — Mon, 26 Oct 2026 07:58:32 +0000

The first ticket that scared me did not contain an error message. It contained a screenshot of a spinner.

A user had uploaded a photo, paid for a render, and waited. The page kept spinning. There was no failed job in my dashboard, no error in my logs, and no refund request, because from the user's side nothing had technically gone wrong. Nothing had happened at all. That is the worst kind of failure for a tool that charges per job: the system looks alive, the money is gone, and the user is left staring at a loading animation.

I run a small image-to-video tool. You upload a still photo, pick a motion style, and get back a short clip. The pipeline looks simple on a whiteboard: upload, queue, send to a rendering provider, wait for a webhook, deliver the file. In practice, every arrow between those boxes is a place where a job can die quietly. It took me about a month, and roughly 1,400 jobs, to learn all the ways. About 6% of jobs in that first month never reached the user in a usable form. Here is what was actually breaking, and the design that stopped it.

The pipeline, and where it lies to you

My first version tracked a job with two states: pending and done. The code called the provider, waited for the webhook, and flipped the job to done when the webhook arrived. If the webhook never arrived, the job stayed pending forever. My dashboard showed a healthy queue with a few slow jobs. The users saw a spinner that never stopped.

The fix started with admitting that "pending" was hiding four different situations. I now persist five states before I ever call the provider: queued, rendering, delivered, failed, and dead. The state row is written to the database first, and only then does the code call the provider. That order matters. If the process crashes between those two steps, I have a queued job with no provider call, and a sweeper can pick it up. In the old order, a crash left a provider-side render running with no record of it on my side at all.

Every state change is also timestamped. That one field turned out to be the cheapest debugging tool I own. A job stuck in rendering for nine hours is not a slow render. It is a dead job wearing a slow job's clothes.

Failure one: the provider just stops answering

Rendering providers time out. Sometimes the provider is down, sometimes a single render gets stuck on their side, sometimes the network drops at exactly the wrong moment. My first timeout was 45 seconds, set because that felt reasonable. It was not based on any measurement, and it showed: normal renders took much longer than that, so the timeout fired on healthy jobs, and my code marked them failed while the provider kept rendering them in the background.

I changed two things. First, I stopped treating a timeout as a failure. A timeout now means "unknown." The job goes back to a checking state where a separate worker asks the provider for the job's status instead of assuming the worst. Second, I set the timeout from real data: I looked at the distribution of render times, picked a value above the 99th percentile, and only then called it a timeout. Guessing at timeouts is how you build a system that fails its healthiest jobs.

Failure two: the webhook never arrives

The worst weekend I had came from my own deploy. I shipped a small change to the webhook endpoint on a Friday evening, and the new code returned a 500 error for a payload shape I had not tested. The provider retried a few times, then gave up. By Monday morning, 38 jobs were stuck in rendering. The renders had actually finished on the provider's side. My system simply never heard about them, and there was no code path that went looking.

That weekend taught me the rule I now consider non-negotiable: never let a webhook be the only way you learn a job finished. I run a sweeper every ten minutes that lists jobs stuck in rendering past their expected window and asks the provider directly for their status. The webhook is still the fast path, and most jobs still complete through it. But the sweeper is the path that catches everything the fast path drops, including deploys like mine. It also pages me differently: one stuck job is a curiosity, ten stuck jobs in an hour is an incident, and the alert now says which.

Failure three: the file arrives, but it is broken

This one took the longest to notice, because my code was marking these jobs delivered. The provider returned a file, the webhook arrived on time, and the user got a download link. The file just did not play. Sometimes it was truncated, sometimes it was a few frames of black, sometimes the container was corrupt. My system had verified that a file existed. It had never verified that the file was a video.

Now nothing is marked delivered until a verification step passes: the file has a plausible size, the container parses, and the duration matches what the job requested within a small margin. If any check fails, the job goes to retry, not to the user. This added a few seconds to delivery, and it killed an entire class of tickets that used to start with "the video is broken." Verifying the output feels like wasted work until the first time it catches a corrupt file before a user does.

Failure four: the job that delivered twice

Retries fixed the stuck jobs, and then created a new problem. A webhook arrived late, after my sweeper had already retried the job and delivered it. The provider's retry and my retry both completed, and the user was charged twice for one clip. The user noticed before I did, which is embarrassing in a specific way that stays with you.

The fix is an idempotency key, and it has to live in the right place. Each job gets one key at creation, and that key is stored with the job row, not generated fresh on each attempt. The charge, the provider call, and the delivery all check the key before acting. A second delivery attempt with the same key returns the first result instead of doing the work again. This is the kind of thing that sounds obvious in a blog post and is easy to skip when you are rushing to ship retries. Skipping it cost me real refunds and one very patient user who deserved better.

The credit ledger: hold, settle, refund

All of this sits on top of a credit system, and the credit system needed its own redesign. In the first version, credits were deducted when the job was created. A failed job meant a support ticket and a manual refund, which is fine at ten jobs a day and miserable at a hundred.

Now credits are held when a job is created, settled only when delivery is verified, and released automatically when a job reaches a terminal failed state. The ledger records every hold, settle, and release as its own row, so I can answer "where did my credits go" without guessing. The dead-letter queue ties into this: a job that exhausts its retries lands in the dead queue with its full history attached, and releasing its held credits is part of moving it there, not a separate manual step. Users stopped asking about credits at almost exactly the moment the ledger became boring and automatic, which is how I knew it was working.

What I got wrong along the way

My first fix for all of this was "retry everything." It reduced stuck jobs and increased duplicate charges, which is a trade I would not make again. Retries without idempotency are just a second chance to cause the same harm.

My second mistake was alerting on every failure. My phone buzzed all night for individual jobs that the retry loop would have fixed on its own, and I started ignoring the alerts, which is worse than having none. Now I alert on rates and on dead-letter growth, not on single jobs. A single failure is the system working as designed. A rising dead-letter queue is the system telling me something changed upstream.

The third mistake was treating the provider as reliable and my code as the risk. It is the other way around. Providers go down, webhooks get lost, deploys break endpoints. The pipeline has to assume every external call can vanish, and still give the user either a video or their credits back. That is the whole job, really: never leave a user with neither.

A short checklist, if you are building one of these

Write the state row before you call anyone. Give every job one idempotency key and keep it for the life of the job. Treat a timeout as "unknown," not "failed," and go ask the provider. Run a sweeper, because webhooks will fail on the worst weekend. Verify the output before you call it delivered. Hold credits, settle on verified delivery, refund automatically on terminal failure. And alert on the queue growing, not on each job dying.

In the month after I put this in place, user-visible failures fell from around 6% to under 1%, and most of what remained was jobs that genuinely could not render, which now fail loudly with a refund instead of spinning forever. The spinner is still there, but now it has a deadline, and behind it there is a system that either delivers or gives the money back. That is a much easier thing to explain to a user.

I build Image to Video tools for turning still photos into short clips, and most of what I know about async pipelines came from these failures, not from getting the design right the first time.

DEV Community: Gabriella Andrews