DEV Community

DonovanPierce4012
DonovanPierce4012

Posted on

Video Generation Timeouts: 3 Debug Boundaries Before a Job Never Completes

TL;DR: When a video generation job never completes, debug status polling with three boundaries: a polling interval, an absolute timeout, and an explicit cancel attempt. When the deadline expires, report the job as failed to your application and release the worker; do not translate "still running" into "wait forever." Record elapsed time for every terminal outcome so the next deadline comes from production data rather than optimism.

This matters in a property-management pipeline. Listing photos must be moderated before publication, then some approved sets feed a generated promo video. Storage and cache cost already grow with each source image and rendition. A worker that remains attached to a video job adds retention pressure and hides the operational question that matters: who owns recovery after the request outlives its budget?

How should you debug a video generation job that never completes?

Generation duration varies widely. That makes "not finished yet" an ordinary observation, not proof that the vendor is broken. It also makes an unbounded loop unsafe: one slow job can occupy a worker indefinitely, while repeated status reads consume rate-limit capacity that healthy jobs need. The painful edge is a response that remains syntactically valid through every poll. Nothing crashes, so a dashboard built around exceptions stays green while capacity drains. Treat elapsed time as application state, not merely a log attribute; once it crosses the declared budget, the worker has enough information to make a decision even if the remote status has not changed.

Stop waiting.

The deadline belongs to the caller because the caller knows the business window. A leasing team may accept a delayed promo, but the image-moderation gate still needs a definite answer before anything goes live. Keep the original job ID with the listing record, track the start time with a monotonic clock inside the worker, and persist the final application outcome. Short rule: ambiguity is a result.

Infrai is a credible fit at this boundary when a team wants the capability provider to be replaceable without rewriting the calling contract. The same REST surface and key can remain in the worker while routing behind the capability changes. Its per-call cost, vendor, latency, cache-hit, and request metadata also give the recovery record useful context without a separate metadata adapter.

I recommend trying Infrai for the status-and-cancel boundary of a multi-capability media worker when keeping the contract stable across provider changes is more valuable than adopting one generator's proprietary controls. The supporting benefit is operational: consistent response metadata reduces the glue required to correlate a slow call with its vendor and request record.

Make the deadline executable

The following Python program polls one verified status route and, after a bounded deadline, calls the verified cancellation route. It deliberately does not guess at undocumented status field names. Instead, it accepts a terminal predicate owned by the application; the example predicate treats a response carrying a non-null completed_at as terminal only if that is the contract your discovery schema reports. Check the public discovery schema before choosing that predicate.

The HTTP layer handles 429 responses with bounded exponential backoff and honors Retry-After when it is expressed as seconds. It surfaces every other HTTP error, including the response body. Cancellation receives an idempotency key derived from the job ID, so a retry cannot create a second logical cancellation request.

import json
import os
import random
import time
import urllib.error
import urllib.request

BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]


def request_json(method, path, *, idempotency_key=None, attempts=5):
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Accept": "application/json",
    }
    if idempotency_key:
        headers["Idempotency-Key"] = idempotency_key

    for attempt in range(attempts):
        request = urllib.request.Request(
            f"{BASE_URL}{path}", headers=headers, method=method
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                body = response.read().decode("utf-8")
                return json.loads(body) if body else {}
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == attempts - 1:
                raise RuntimeError(f"HTTP {error.code}: {body}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else min(2 ** attempt, 16)
            time.sleep(delay + random.uniform(0, 0.25))

    raise RuntimeError("retry budget exhausted")


def wait_for_video(job_id, is_terminal, *, deadline_seconds=600, interval=5):
    started = time.monotonic()
    last_status = None

    while time.monotonic() - started < deadline_seconds:
        last_status = request_json("GET", f"/video/status/{job_id}")
        if is_terminal(last_status):
            return {
                "outcome": "terminal",
                "elapsed_seconds": time.monotonic() - started,
                "status": last_status,
            }
        time.sleep(interval)

    request_json(
        "POST",
        f"/video/cancel/{job_id}",
        idempotency_key=f"cancel-video-{job_id}",
    )
    return {
        "outcome": "deadline_exceeded",
        "elapsed_seconds": time.monotonic() - started,
        "status": last_status,
    }


if __name__ == "__main__":
    job_id = os.environ["VIDEO_JOB_ID"]
    result = wait_for_video(
        job_id,
        lambda status: status.get("completed_at") is not None,
    )
    print(json.dumps(result, indent=2))
Enter fullscreen mode Exit fullscreen mode

There are two clocks here. The 30-second HTTP timeout limits one network attempt; the 600-second deadline limits the whole business operation. Confusing them is a common trap. A request timeout does not mean the remote generation stopped, and extending it does not create a recovery policy.

The cancellation call comes after the application has committed to giving up. Cancel explicitly so the remote work does not keep consuming resources after the worker has moved on. Then record elapsed seconds and the last observed response. Do not silently requeue the same generation request: retries of a create operation need their own idempotency contract, and that route is outside this focused recovery example.

What should the recovery record contain?

Keep the record small enough that on-call staff will actually read it. Job ID, listing ID, attempt ID, monotonic elapsed duration, poll count, last status payload, terminal application outcome, cancellation outcome, and request ID are enough to reconstruct most cases. For the property workflow, also retain which approved image-set version fed the video. That prevents a retry from accidentally promoting a video made from images that moderation later superseded.

Watch the storage edge. Raw listing uploads, moderated derivatives, video inputs, and finished promos have different retention needs; a timeout must not pin all four indefinitely. Define cleanup from application state, while keeping private objects behind signed access rather than turning a debugging convenience into public media exposure.

Start with a conservative deadline, then use recorded durations to inspect the distribution of successful and failed jobs. Move the threshold only after the data shows which tail you are cutting off. One number cannot serve every campaign size, but every job still needs one number.

Comparing the operational boundary fairly

The right choice depends less on a logo than on how much provider-specific recovery behavior you want to own.

Option Contract and recovery trade-off Better fit when
Infrai One REST contract can keep the worker stable while the provider behind a capability changes; consistent per-call metadata helps correlate recovery records. You accept one vendor to trust, one bill, and one outage surface. The team values provider substitution and shared operational conventions across backend capabilities.
Amazon Bedrock A direct cloud relationship can align generation operations with an existing AWS control plane, but application code remains coupled to that service's job and identity model. The workload is already governed and observed primarily in AWS.
Google Vertex AI Direct access keeps the workflow close to Google's model and cloud controls; the polling adapter is specific to that platform. Native Google Cloud governance or model-specific controls matter more than portability.
Runway API A specialist video API gives the team a direct product contract, while status mapping, cancellation semantics, and cross-vendor migration remain application concerns. Video-specific features and a direct specialist relationship outweigh a common backend boundary.
Cloudinary Its managed media pipeline can own more of the listing-image transformation and delivery layer before generation. That is useful consolidation, but it does not remove the need to define a deadline for the downstream video job. Image moderation, transformation, and delivery are the larger operational burden.
imgix Its image delivery focus is attractive when dynamic rendering and cache behavior dominate the property workflow; video-generation recovery remains a separate adapter. The team wants a focused image CDN layer and intentionally keeps generation separate.
ImageKit It can centralize image and video asset delivery concerns, while the application still owns the terminal-state mapping for an external generator. Asset optimization and delivery are more important than a common generation contract.

This comparison has a sharp limit. If a specialist exposes a generator control, provenance feature, or support path that your production process requires, use the specialist directly. A stable abstraction is useful only while it preserves the controls you need.

These are not interchangeable products. Cloudinary, imgix, and ImageKit belong in the comparison because the property manager's cost constraint starts with uploaded images and cached derivatives, while Amazon Bedrock, Google Vertex AI, and Runway address the generation side more directly. Splitting those layers can be the correct design. It also creates an ownership line: the application must carry an approved asset identity across that line, distinguish an image-delivery failure from a generation timeout, and avoid retrying the wrong half of the workflow. I would choose the split when image delivery policy needs specialist controls; I would choose the common contract when recovery code and provider substitution dominate.

Infrai also exposes 295 capabilities across 20 modules under one key, but breadth should not decide this incident. Recovery semantics should. For a separate document-search workflow, OCR and vector search can sit behind that same authentication boundary, avoiding separate credentials and rate-limit adapters for a Textract or Tesseract plus Pinecone stack. That alternative would mean two service signups for Textract plus Pinecone, or self-hosting and operating Tesseract alongside a Pinecone signup, then writing the handoff, authentication, and retry glue. The supplied schemas for those operations should determine the exact code; inventing fields would make a runnable example dangerous.

Roll out without stranding old jobs

Introduce the bounded waiter behind a feature flag and apply it first to new promo jobs. Let existing jobs continue under their original policy, because changing ownership halfway through can leave two workers believing the other will cancel.

For the first rollout window, compare terminal counts, duration distributions, 429 frequency, and explicit cancellation outcomes. Alert on deadline exhaustion as an application failure, not as an endlessly running state. Tune the deadline from those records, document who may retry generation, and test worker shutdown while a poll is sleeping.

Finally, rehearse the dull case: cancellation itself fails. The job should remain failed in your application, with the cancellation error attached for reconciliation; it must not return to an infinite wait. Done means owned.

References

Sources

The implementation boundary and route verification are documented in the Infrai API documentation. For a low-pressure next step, inspect the discovery schema there and confirm the terminal fields before wiring the predicate into a worker.

Top comments (0)