For a large batch of gaming event photos, start with an asynchronous job, bounded workers, and a durable manifest of photo IDs. Publish progress from completed outputs, not from requests accepted. Cancellation should stop new work, abort active transfers where possible, and leave already published images intact. That is the smallest design that gives a UI an honest progress bar while keeping storage and cache growth visible.
TL;DR: Choose the processing boundary according to where originals live and how many distinct derivatives you actually need. A request that returns before the job completes needs a job ID, a status read, and an explicit cancel operation; an HTTP disconnect alone cannot define the fate of a background job.
| Approach | Pick this when | Progress and cancellation boundary | Storage and cache consequence |
|---|---|---|---|
| Eager, bounded batch | Each event needs a small, known set of sizes | Count verified derivatives; stop scheduling on cancel | Predictable objects per original, including sizes nobody views |
| On-demand derivatives | Views request many sizes but only a few are popular | Track ingestion separately from first-view transformations | Fewer stored outputs; more cache-key variants and first-view work |
| Hybrid | A thumbnail is needed immediately; larger views are uncertain | Finish the thumbnail job, then observe demand for other sizes | Fixed thumbnail footprint plus demand-driven variants |
How should an API process a large batch of event photos?
Pick eager processing when every gallery item needs the same thumbnail for a launch feed. Define the size and encoding policy once for the event, and count a photo as done only after its required output is available. This trades extra stored derivatives for a straightforward completeness test. The cost question is concrete: if there are 10,000 originals and two required outputs per original, plan for up to 20,000 derivative objects before retention or deduplication. That is arithmetic for planning, not a benchmark or a price quote.
It adds up fast.
Pick on-demand processing when the catalog contains plenty of photos that never get opened. Keep the original, generate only requested representations, and normalize allowed dimensions and formats so arbitrary request parameters cannot multiply cache keys. Here, an upload progress bar is not a transformation progress bar. There may be no meaningful batch-wide transformation percentage at all.
Pick a hybrid when a gaming event gallery must show every thumbnail promptly but full-size viewing is sparse. Complete and monitor the fixed thumbnail batch; treat larger renditions as a separate demand-driven path. The boundary matters operationally: a single percentage that mixes uploads, transformations, and cache fills tells the reader very little.
There are limits to each choice. Eager processing is not suitable when most photos are never viewed and originals must be retained anyway; on-demand processing is a poor fit when all thumbnails must be ready before the gallery opens. The hybrid approach incurs two distinct operational paths. If your team cannot monitor and retry both, a single bounded batch may be the better trade-off even if it stores more. Before selecting a boundary, record how many photos the event produces, which variants the gallery actually requests, when viewers arrive, and which originals can legally be removed; those observations decide whether a cold request or extra stored output is the more costly mistake. A batch of 10,000 photos with two required derivatives has a very different object count from the same batch with one required thumbnail and larger renditions generated only after a viewer asks for them. Keep that distinction visible on the dashboard, because a cache hit rate without the number of requested variants cannot explain which path is consuming storage.
What does cancellation actually stop?
Think of the job as a short path: manifest entry, queued photo, active transform, verified output. A cancel request closes the gate between queued and active. An abort signal can interrupt cooperative work already active. A verified output stays verified; deleting it is a separate retention decision. This distinction prevents a cancelled job from looking like a failed job and makes retries legible.
Cancellation has a boundary.
Give each manifest entry a stable ID and one terminal state: succeeded, failed, or skipped. Keep completed monotonic, with completed = succeeded + failed + skipped; report total from the frozen manifest. If a photo fails, show the failed count, not a stalled 99% bar. Mark a cancelled job terminal only after active workers settle, so the UI does not say "cancelled" while writes are still in flight. A request to cancel is an acknowledgment of intent, not proof that every downstream action was interrupted.
The implementation below covers the worker boundary. The caller persists the manifest and status, supplies a transform that respects AbortSignal, and persists each terminal result before exposing a new snapshot. On restart, requeue entries without a terminal result; use a stable output key so a retry can check whether the derivative was already written. Keep the job's cancel flag in shared durable state if more than one process can run workers.
type Photo = { id: string; source: string };
type Outcome = "succeeded" | "failed" | "skipped";
type Snapshot = {
total: number;
succeeded: number;
failed: number;
skipped: number;
cancelRequested: boolean;
};
async function processBatch(
photos: readonly Photo[],
concurrency: number,
signal: AbortSignal,
transform: (photo: Photo, signal: AbortSignal) => Promise<void>,
record: (id: string, outcome: Outcome) => Promise<void>,
publish: (snapshot: Snapshot) => Promise<void>,
): Promise<void> {
if (!Number.isInteger(concurrency) || concurrency < 1) {
throw new RangeError("concurrency must be a positive integer");
}
const snapshot: Snapshot = {
total: photos.length, succeeded: 0, failed: 0, skipped: 0,
cancelRequested: false,
};
let next = 0;
async function worker(): Promise<void> {
while (true) {
if (signal.aborted) break;
const index = next++;
if (index >= photos.length) break;
const photo = photos[index];
let outcome: Outcome;
try {
await transform(photo, signal);
outcome = "succeeded";
} catch {
outcome = signal.aborted ? "skipped" : "failed";
}
await record(photo.id, outcome);
snapshot[outcome]++;
await publish({ ...snapshot, cancelRequested: signal.aborted });
}
}
await Promise.all(Array.from(
{ length: Math.min(concurrency, photos.length) },
() => worker(),
));
snapshot.cancelRequested = signal.aborted;
await publish({ ...snapshot });
}
The example deliberately does not mark untouched entries as skipped. They remain queued in the manifest until the job coordinator records a final cancellation disposition. If record or publish fails, let the job stop and reconcile from persisted outcomes before resuming; an in-memory counter is not a recovery log. For multiple processes, claim work atomically in the durable store rather than sharing this local next counter.
Do not guess at progress.
What should the dashboard measure?
Start with three views: job state, worker behavior, and output footprint. Job state needs counts by terminal outcome plus queued and active counts; those numbers should reconcile with the manifest. Worker behavior needs time from queued to started, transformation duration, retry count, and cancellation drain time. Output footprint needs original bytes, derivative bytes by preset, and cache requests by normalized variant. Keep event ID and preset as bounded labels; avoid photo ID as a metrics label, and use a log field or trace attribute for individual photo diagnosis.
A useful alert is a job that remains active while completed counts stop changing and queued work remains. Another is a persistent rise in failed outputs for one preset. Both need a time window and an expected event schedule; a quiet job with no queued photos is not stuck. Measure cache hit ratio alongside cache-key cardinality, because a single overall hit rate can hide a growing tail of one-off dimensions. Validate a sample of decoded outputs before promotion, and deploy a new preset version under a distinct output key so older cached images are not silently mixed with new transforms.
That last check matters. Otherwise, a retry might report success against an older output while the current gallery expects a different encoding or size, and a dashboard built only around completed counts would miss the mismatch. Persist the preset version and output key beside the photo ID, then have verification check the object associated with that exact version. After a deployment, compare successful output counts by version and inspect a few decoded images from each one before increasing worker concurrency. Workers that finish quickly while output bytes or cache variants grow unexpectedly are doing work, but not necessarily the work the gallery needs. The decision to roll forward should use both job completion and the resulting image footprint.
Test the uncomfortable edges. Cancel while workers are active; restart after an output write but before its success record; submit duplicate photo IDs; request the same job twice. The expected result is stable counts and no ambiguous overwrite. Watch the bytes, too. A beautiful progress bar cannot explain an unexpected jump in stored derivatives.
Limits of this field guide
The best boundary depends on actual view distribution, acceptable first-view latency, and retention policy. Gather those measurements before expanding derivative presets. A local worker pool is enough to explain scheduling, but a production job that survives process restarts needs durable claims, retry limits, output verification, and authorization on both status reads and cancellation. Cancellation cannot reverse a write that already succeeded; handle removal under a separate lifecycle policy.
References
- https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types
- https://developer.mozilla.org/en-US/docs/Web/API/AbortController
- https://nodejs.org/api/globals.html#class-abortcontroller
- https://opentelemetry.io/docs/specs/semconv/general/metrics/
Top comments (0)