Video generation jobs: Node.js polling, download, and own-bucket storage
Short answer: treat video generation as an asynchronous job, persist the job state in your database, and copy the finished bytes into a bucket you control before serving them from Express. The important design choice is the handoff boundary, not the polling interval.
Why a video request should become a durable job
An HTTP request that waits for an encoded video ties up a connection while a remote worker renders frames. That makes retries ambiguous: a client timeout does not tell you whether rendering stopped or only the response disappeared. A job record makes the outcome explicit.
I use three states: queued, running, and succeeded (with failed as a terminal state). The record stores the provider job identifier, an idempotency key, the object key we intend to use, and timestamps. The API that creates a job returns 202 Accepted; a separate status endpoint reads our record. This keeps Express responsible for a small, predictable transaction.
The object key should be derived from our job ID, never from a filename supplied by a customer. Keep the original request metadata beside the object, so a later retention task can remove both without guessing.
How should Node.js poll video status, then download and store it?
Polling is a control loop with a budget. Start at a few seconds, cap the delay, and add jitter so a fleet of workers does not wake up together. Stop after a deadline and mark the job for review; do not keep a request alive forever.
Here is the shape of the worker. The /jobs/{id} and /download paths are placeholders for the rendering service documented for your account; keep that adapter isolated so changing providers does not change the rest of the pipeline.
type RenderState = "queued" | "running" | "succeeded" | "failed";
type RenderJob = {
id: string;
remoteId: string;
state: RenderState;
objectKey: string;
};
async function waitForVideo(job: RenderJob, signal: AbortSignal): Promise<Uint8Array> {
const deadline = Date.now() + 15 * 60_000;
let delay = 2_000;
while (Date.now() < deadline) {
const statusResponse = await fetch(`https://render.example/jobs/${job.remoteId}`, { signal });
if (!statusResponse.ok) throw new Error(`status request failed: ${statusResponse.status}`);
const status = (await statusResponse.json()) as { state: RenderState; downloadUrl?: string };
if (status.state === "failed") throw new Error("rendering failed");
if (status.state === "succeeded" && status.downloadUrl) {
const videoResponse = await fetch(status.downloadUrl, { signal });
if (!videoResponse.ok) throw new Error(`download failed: ${videoResponse.status}`);
return new Uint8Array(await videoResponse.arrayBuffer());
}
const jitter = Math.floor(Math.random() * 500);
await new Promise((resolve) => setTimeout(resolve, delay + jitter));
delay = Math.min(Math.floor(delay * 1.5), 10_000);
}
throw new Error("render deadline exceeded");
}
The download is streamed in production rather than accumulated in memory. Check the declared content type and enforce a byte limit while writing. After the write completes, verify the object exists, then transition the database row to succeeded in one transaction. A repeated worker run should see the existing object key and return without creating a second copy.
The storage boundary decides your real cost
Keeping a renderer's temporary URL in your database is convenient, but it creates a hidden dependency on URL expiry and remote retention. Copying once into your own bucket gives you a stable key and lets lifecycle rules delete previews after, say, seven days while retaining approved clips longer. The trade-off is an extra download and upload, plus egress and storage charges that must be measured rather than hand-waved.
I track four numbers per job: rendered bytes, downloaded bytes, stored bytes, and wall-clock seconds. A small thumbnail may be cheap to store but expensive to regenerate if its source prompt is gone. A large master can have the opposite profile. Your mileage may vary, especially when the renderer uses variable bitrate encoding; measure a representative week before setting quotas.
Measure first.
| Decision | Useful default | Revisit when |
|---|---|---|
| Object key | videos/{jobId}/master.mp4 |
customers need immutable versions |
| Retention | short-lived previews, explicit approval for masters | support cases require replay |
| Poll deadline | bounded worker lease | renders regularly exceed the lease |
| Delivery | signed, short-lived read URL | a public campaign needs a CDN |
Use an idempotency key on job creation and a compare-and-set update on state transitions. A worker may crash after uploading bytes but before committing the database update. On restart, inspect the deterministic object key: if its checksum and size match the expected artifact, commit the state; otherwise resume or quarantine the partial object.
For example, imagine a worker that receives a success response at 14:03:12, starts a 240 MB upload, and loses its lease at 14:03:19. A second worker should not blindly create master-2.mp4. It checks the deterministic key, asks the bucket for the committed size and checksum, and either adopts that object or writes to a temporary key and retries. The database transition is conditional on the row still being running, so two workers cannot both publish the same job. This small amount of bookkeeping is less exciting than a new codec, but it is what keeps a retry from doubling storage and confusing a support agent who is trying to find the canonical clip.
Do not infer success from a 200 status alone. Validate the media type, size, and (where available) a checksum. Keep the remote job ID and a correlation ID in structured logs. Metrics should separate time spent waiting for rendering from time spent transferring bytes; otherwise a single latency number hides the expensive part.
There is a boundary here. This pattern is not suitable when users need frame-by-frame interactive feedback or when policy forbids copying media into a customer-managed bucket. In those cases, use a renderer with a streaming protocol or retain media in the approved managed system. Stick with a simpler synchronous endpoint when clips are tiny, deterministic, and completed within your request timeout.
Top comments (0)