Video generation broke every assumption my request-handling code was built on. A job takes anywhere from forty seconds to eleven minutes. It fails a small but non-trivial fraction of the time for reasons unrelated to your input. It costs real money per attempt, so a careless retry is a charge, not a hiccup. And you're doing this across several providers, each with its own idea of what a job status API looks like.
This is not an HTTP problem. It's a distributed systems problem wearing an HTTP costume.
Own the job, not the request
The first architectural decision that matters: your job ID is not the provider's job ID. POST /generate returns 201 {"job_id": "job_7fc2a1", "status": "queued"}, and that ID is minted before you talk to any provider. That indirection buys you three things: retrying onto a different provider without the client noticing, surviving a provider that loses its own job ID, and a stable identity for logs and billing.
The state machine should be explicit and boring:
queued -> submitted -> running -> succeeded
-> failed(retryable) -> queued (bounded)
-> failed(terminal)
-> expired (no update past deadline)
expired earns its place. Providers drop jobs — not "return an error," just never update again. Without a deadline sweeper those rows sit in running forever while users watch a bar that will never finish. Every job gets an absolute deadline at submit time.
Idempotency: the expensive kind
Standard idempotency keys prevent duplicate work. Here a duplicate submission is a duplicate charge — a double-tap or a retried POST buys the same video twice. The key must derive from request content, not a client-generated UUID; a client retrying after a timeout may well generate a fresh one:
def idem_key(user_id, params):
canonical = json.dumps({
"prompt": params.prompt.strip(),
"model": params.model, "seed": params.seed,
"duration": params.duration,
"image_sha": params.image_sha, # image bytes, not its URL
}, sort_keys=True, separators=(",", ":"))
return sha256(f"{user_id}:{canonical}".encode()).hexdigest()
Two subtleties that bit me. Hash the input image bytes, not the URL — signed URLs carry expiring tokens, so the same image yields a different key every request. And deliberate re-rolls must not collide: a second take of the same prompt is not a duplicate. Scope idempotency to a short window (say 60 seconds) so double-taps collapse but an intentional re-roll goes through. Getting this wrong means users cannot re-roll, which for a generation product is worse than the occasional duplicate.
The claim-and-submit sequence must also be crash-safe. Claim the row (queued → submitting) in a transaction, submit outside it, then write back the provider reference. If the process dies in between you have a charged job with no reference — which is why you send the idempotency key to the provider too, so recovery re-attaches rather than re-bills.
Webhooks are an optimization; polling is the guarantee
Every provider offers webhooks. Trust none as your only path — they get lost, arrive out of order, arrive twice, and come from providers whose retry policy is "we tried once."
The pattern that has held up: poll as the baseline, treat webhooks as a latency optimization.
def next_poll_delay(job):
elapsed = now() - job.submitted_at
if elapsed < 30: return 3 # most fast jobs land here
if elapsed < 120: return 10
if elapsed < 600: return 30
return 60 # long tail, don't hammer
A webhook simply short-circuits the wait — it marks the job dirty and triggers an immediate poll rather than being trusted on its own. That means the webhook handler never writes terminal state from its payload; it says "something changed, go look." This one rule eliminates forgery concerns, out-of-order delivery, and double-crediting at a stroke, because the status endpoint is always the authority. Keep the handler fast: verify signature, mark dirty, return 200.
Retries when attempts cost money
Not all failures are equal, and the classification determines whether you may spend again:
| Failure | Retry? | Where |
|---|---|---|
| 429 / rate limited | yes | same provider, backoff + jitter |
| 5xx / timeout on submit | yes, guarded | same provider, with idem key |
failed, no reason given |
yes, once | prefer a different provider |
| Content policy rejection | no | terminal, surface to user |
| Malformed params / 400 | no | terminal, it's your bug |
| No update past deadline | yes | different provider |
Content policy rejection being terminal matters: retrying burns money and never succeeds, and cross-provider retry of policy-rejected content is exactly how accounts get suspended.
Cross-provider retry is also where credit accounting gets sharp. If one backend fails and you fall back to another at a different unit cost, the user was still quoted one price. My rule: quote and hold credits at submit; settle on success, with failed attempts releasing the hold. The cost variance is yours to absorb — that's the entire value proposition of a unified credit balance, and the accounting model behind hyper-frames. If you pass backend cost variance through to users, you have not abstracted anything.
Honest progress with dishonest ETAs
Providers report progress badly. Some give a percentage that sits at 5% then jumps to 100%. Some give nothing. Some give an ETA off by 4x.
Do not display a provider percentage directly — users read a stalled 30% as "broken" and hit cancel-and-retry, which costs you a generation. What works better:
- Phase labels over percentages. "Queued → Generating → Encoding → Ready" is four honest states. Nobody feels lied to by a phase name.
- Your own historical p50/p90 for this model and duration, not the provider's ETA. Show a range: "usually 2-4 min."
- Never let the bar go backwards or stall at 99%. Drive it from elapsed time against your own p50, approaching ~90% asymptotically and jumping only on real completion.
- Say when it's slow. Past p90, replace the estimate with "taking longer than usual — still running." That kills most of the "is it broken?" support load.
The principle underneath all of this: the provider is an untrusted, slow, occasionally amnesiac collaborator. Your database is the source of truth about what the user asked for and what they were charged. Everything the provider tells you is a hint to be verified — including, especially, its claims about success.
Top comments (0)