Orchestrating Slow, Unreliable Async Jobs Across Multiple Providers

leojxu — Sat, 25 Jul 2026 08:08:56 +0000

Video generation broke every assumption my request-handling code was built on. A job takes anywhere from forty seconds to eleven minutes. It fails a small but non-trivial fraction of the time for reasons unrelated to your input. It costs real money per attempt, so a careless retry is a charge, not a hiccup. And you're doing this across several providers, each with its own idea of what a job status API looks like.

This is not an HTTP problem. It's a distributed systems problem wearing an HTTP costume.

Own the job, not the request

The first architectural decision that matters: your job ID is not the provider's job ID. POST /generate returns 201 {"job_id": "job_7fc2a1", "status": "queued"}, and that ID is minted before you talk to any provider. That indirection buys you three things: retrying onto a different provider without the client noticing, surviving a provider that loses its own job ID, and a stable identity for logs and billing.

The state machine should be explicit and boring:

queued -> submitted -> running -> succeeded
                    -> failed(retryable) -> queued  (bounded)
                    -> failed(terminal)
                    -> expired  (no update past deadline)

expired earns its place. Providers drop jobs — not "return an error," just never update again. Without a deadline sweeper those rows sit in running forever while users watch a bar that will never finish. Every job gets an absolute deadline at submit time.

Idempotency: the expensive kind

Standard idempotency keys prevent duplicate work. Here a duplicate submission is a duplicate charge — a double-tap or a retried POST buys the same video twice. The key must derive from request content, not a client-generated UUID; a client retrying after a timeout may well generate a fresh one:

def idem_key(user_id, params):
    canonical = json.dumps({
        "prompt": params.prompt.strip(),
        "model": params.model, "seed": params.seed,
        "duration": params.duration,
        "image_sha": params.image_sha,   # image bytes, not its URL
    }, sort_keys=True, separators=(",", ":"))
    return sha256(f"{user_id}:{canonical}".encode()).hexdigest()

Two subtleties that bit me. Hash the input image bytes, not the URL — signed URLs carry expiring tokens, so the same image yields a different key every request. And deliberate re-rolls must not collide: a second take of the same prompt is not a duplicate. Scope idempotency to a short window (say 60 seconds) so double-taps collapse but an intentional re-roll goes through. Getting this wrong means users cannot re-roll, which for a generation product is worse than the occasional duplicate.

The claim-and-submit sequence must also be crash-safe. Claim the row (queued → submitting) in a transaction, submit outside it, then write back the provider reference. If the process dies in between you have a charged job with no reference — which is why you send the idempotency key to the provider too, so recovery re-attaches rather than re-bills.

Webhooks are an optimization; polling is the guarantee

Every provider offers webhooks. Trust none as your only path — they get lost, arrive out of order, arrive twice, and come from providers whose retry policy is "we tried once."

The pattern that has held up: poll as the baseline, treat webhooks as a latency optimization.

def next_poll_delay(job):
    elapsed = now() - job.submitted_at
    if elapsed < 30:  return 3    # most fast jobs land here
    if elapsed < 120: return 10
    if elapsed < 600: return 30
    return 60                     # long tail, don't hammer

A webhook simply short-circuits the wait — it marks the job dirty and triggers an immediate poll rather than being trusted on its own. That means the webhook handler never writes terminal state from its payload; it says "something changed, go look." This one rule eliminates forgery concerns, out-of-order delivery, and double-crediting at a stroke, because the status endpoint is always the authority. Keep the handler fast: verify signature, mark dirty, return 200.

Retries when attempts cost money

Not all failures are equal, and the classification determines whether you may spend again:

Failure	Retry?	Where
429 / rate limited	yes	same provider, backoff + jitter
5xx / timeout on submit	yes, guarded	same provider, with idem key
`failed`, no reason given	yes, once	prefer a different provider
Content policy rejection	no	terminal, surface to user
Malformed params / 400	no	terminal, it's your bug
No update past deadline	yes	different provider

Content policy rejection being terminal matters: retrying burns money and never succeeds, and cross-provider retry of policy-rejected content is exactly how accounts get suspended.

Cross-provider retry is also where credit accounting gets sharp. If one backend fails and you fall back to another at a different unit cost, the user was still quoted one price. My rule: quote and hold credits at submit; settle on success, with failed attempts releasing the hold. The cost variance is yours to absorb — that's the entire value proposition of a unified credit balance, and the accounting model behind hyper-frames. If you pass backend cost variance through to users, you have not abstracted anything.

Honest progress with dishonest ETAs

Providers report progress badly. Some give a percentage that sits at 5% then jumps to 100%. Some give nothing. Some give an ETA off by 4x.

Do not display a provider percentage directly — users read a stalled 30% as "broken" and hit cancel-and-retry, which costs you a generation. What works better:

Phase labels over percentages. "Queued → Generating → Encoding → Ready" is four honest states. Nobody feels lied to by a phase name.
Your own historical p50/p90 for this model and duration, not the provider's ETA. Show a range: "usually 2-4 min."
Never let the bar go backwards or stall at 99%. Drive it from elapsed time against your own p50, approaching ~90% asymptotically and jumping only on real completion.
Say when it's slow. Past p90, replace the estimate with "taking longer than usual — still running." That kills most of the "is it broken?" support load.

The principle underneath all of this: the provider is an untrusted, slow, occasionally amnesiac collaborator. Your database is the source of truth about what the user asked for and what they were charged. Everything the provider tells you is a hint to be verified — including, especially, its claims about success.

Routing Between Video Models Without Leaking the Abstraction

leojxu — Fri, 24 Jul 2026 19:02:37 +0000

If you build on a single video generation model, you will rewrite your integration when it gets deprecated. If you build on several, you spend your time writing adapters. Here is what I learned choosing the second option.

The parameters do not line up

Every provider has a different idea of the same concept:

Duration: some take seconds, some take frame counts, some only accept a fixed enum
Aspect ratio: "16:9" vs {width, height} vs a named preset
Seeds: supported, ignored, or silently non-deterministic
Image conditioning: URL, base64, or a pre-uploaded asset ID

A lowest-common-denominator interface is tempting and wrong — you lose the capabilities you are paying for. The alternative that held up: a common core plus a typed passthrough.

type VideoRequest = {
  prompt: string
  durationSec: number        // normalised, adapter converts
  aspect: '16:9' | '9:16' | '1:1'
  image?: ImageInput
  providerOptions?: Record<string, unknown>  // escape hatch, typed per adapter
}

The core covers what every provider supports. The escape hatch means using a provider-specific feature does not require redesigning the interface.

Async is the actual hard part

Video generation takes minutes, so every provider is asynchronous — and each has invented its own async:

Polling with a job ID
Webhook callbacks
Server-sent events
A queue position field that sometimes goes backwards

Normalising this matters more than normalising parameters. What worked: adapters expose a single poll(jobId): Promise<JobState> and internally hide whatever the provider does. JobState is a small discriminated union — queued | running | done | failed — with a normalised progress when available and undefined when not.

Do not fabricate progress. A spinner that lies is worse than a spinner that says "still working."

Failure modes deserve normalising too

Providers fail differently: content policy rejections, transient 5xx, quota exhaustion, silent truncation. Mapping these into a common taxonomy is what makes retry logic possible:

retryable — backoff and retry
policy — do not retry, surface to user
quota — do not retry, switch provider or fail loudly

Getting this wrong means retrying a policy rejection 5 times and burning credits on a request that will never succeed.

Credits across providers

If you expose one balance over several backends with different pricing, you need a normalisation table and you need to be honest that it is an approximation. I run this in production at HyperFrames — text or image in, routed across Kling, Veo, Wan and others, one balance.

Caveat

This abstraction is worth it above roughly three providers. Below that, adapters cost more than they save — just write the integrations directly and move on.

DEV Community: leojxu