A bounded worker queue is the safest default for a catalogue import that generates responsive thumbnails. Submit individual jobs with stable identifiers, cap concurrency, and retry only failed work; choose a batch endpoint only when it exposes item-level outcomes and an equally precise retry path. The deciding constraint is recoverability under rate limits, not HTTP request count.
Short answer: A loop without a concurrency limit can turn one import into a traffic spike. One opaque batch can turn one bad image into an ambiguous replay. Put a durable queue between ingestion and transformation, treat each source image as its own state machine, and measure queue age plus outcomes. This also makes duplicate derivatives visible before they inflate storage and cache work.
Replace the request-count mental model
The tempting comparison is 1,000 calls versus one call containing 1,000 items. That counts envelopes. It does not describe how much decoding, resizing, encoding, storage, or cache invalidation the system performs. Those units exist behind either interface.
Envelopes lie.
Picture the pipeline: catalogue row enters; source image is validated; a job receives an idempotency key; workers create named variants; objects are committed; the row becomes ready. Metrics sit beside every arrow. A dead-letter path leaves the worker stage while preserving the original job identity.
Before, an import handler starts a promise for every image and waits for the array. A rate-limit response or network interruption is mixed into a large in-memory operation. After, the handler records desired work, a fixed worker pool drains it, and each item reaches a terminal state independently. An illustrative 1,000-item run can report 913 ready, 74 pending, and 13 failed without treating the whole submission as one success or failure. Those are example states, not benchmark results.
Good. Now alerts can answer a useful question: how old is the oldest unfinished item, and can the system resume it without creating another derivative?
That distinction matters.
A copyable bounded worker
This TypeScript sketch uses generic interfaces. The queue can be backed by any durable store, and transformation can be local or remote. The contract matters more than either implementation.
type Job = {
jobId: string;
sourceUrl: string;
variant: "card" | "detail" | "search";
};
type Result =
| { jobId: string; status: "ready"; objectKey: string }
| { jobId: string; status: "retry"; retryAfterMs: number }
| { jobId: string; status: "rejected"; reason: string };
interface Transformer {
create(job: Job, idempotencyKey: string): Promise<Result>;
}
interface JobStore {
mark(result: Result): Promise<void>;
}
async function runBounded(
jobs: Job[],
transformer: Transformer,
store: JobStore,
concurrency = 8,
): Promise<void> {
let cursor = 0;
async function worker(): Promise<void> {
while (cursor < jobs.length) {
const job = jobs[cursor++];
try {
const result = await transformer.create(
job,
`${job.jobId}:${job.variant}`,
);
await store.mark(result);
} catch {
await store.mark({
jobId: job.jobId,
status: "retry",
retryAfterMs: 1_000,
});
}
}
}
await Promise.all(
Array.from({ length: concurrency }, () => worker()),
);
}
Eight workers and a one-second retry marker are example configuration, not universal defaults. Start with the downstream contract and the catalogue's latency objective. Preserve explicit retry timing when it is available, and add randomized delay when many workers could otherwise wake together.
The idempotency key joins the logical image job and variant. A retry can refer to the same intended derivative. The storage layer also needs a deterministic object key or atomic commit rule, so repeated execution does not quietly create objects. Idempotent intent does not guarantee idempotent storage. Design both.
Storage is state.
Validate media from file content rather than trusting a filename, and decide which formats the pipeline accepts. The MDN image format guide provides browser-support and format details. Keep that choice separate from submission shape: changing a batch size cannot make a decoder understand an unsupported file.
What should the dashboard expose?
A request-rate graph is necessary, but it is a weak description of progress. An importer can remain below a limit while its queue grows for hours. It can also show few errors while repeatedly regenerating derivatives and paying the storage and cache cost again.
Requests are not progress.
Track accepted, started, ready, retryable, and rejected outcomes by import and variant. Add oldest-job age, queue depth, active workers, retry count, bytes written, and derivative objects created. Use low-cardinality reason codes such as decode_failed and rate_limited; keep job IDs in logs and traces rather than metric labels.
Alert on user-visible delay: oldest eligible job age exceeds the readiness objective while incoming work continues. A second alert can catch a sustained rise in terminal rejection ratio. Rate-limit responses are a cause signal. Queue age is the symptom editors and shoppers eventually feel.
Storage needs reconciliation too. Compare the manifest of expected variants with committed object keys. Count cache fills by variant and generation version. When a preset changes, treat the output as a deliberate new generation; retention and invalidation policy determine when old objects disappear.
When is a batch actually better?
A batch interface earns its place when it reduces coordination overhead while preserving per-item identity. Its response or status resource must distinguish accepted, rejected, and pending items. Retry semantics need the same granularity. Without those properties, partial failure becomes a guessing exercise.
Individual calls through a bounded queue are easier to reason about when items vary widely in size or processing time. One slow decode does not hold an entire envelope open, and backoff can apply to one item. The trade-off is more protocol overhead and more client-side state transitions.
An asynchronous batch may amortize connection and parsing overhead. Yet batch size becomes another control: larger envelopes can take longer to validate, consume more memory, and make timeouts more consequential. No fixed size is correct without measurements from the real workload.
The trade-off is deliberate. Individual calls spend more effort on coordination so that one item's state remains obvious. A batch spends less effort on submission, but its status model must return that visibility later. For a catalogue team, the better choice is the one that lets an operator point to every missing thumbnail and say whether it is queued, running, retryable, or permanently rejected. If the interface only says that a container was accepted, it has moved uncertainty rather than removed it; the team still needs a reconciliation record before it can safely retry or declare the import complete.
| Concern | Bounded individual jobs | Item-aware asynchronous batch |
|---|---|---|
| Partial failure | Naturally isolated | Safe with per-item results |
| Backoff | Per item | Per item or sub-batch if supported |
| Progress | Direct job counts | Requires durable status lookup |
| Bookkeeping | More transitions | Richer reconciliation |
| Duplicate storage | Controlled by item identity | Batch identity is insufficient |
Test with the real distribution of image dimensions and formats. Inspect queue age, retry amplification, bytes written, and time to a complete catalogue. Averages hide the outliers that occupy workers longest.
Doesn't batching solve the rate limit?
Only if the published limit is defined around batch submissions and the server does not apply a separate item or compute quota. Interface shape cannot prove that. Read the contract and capture returned limit signals.
Admission control still belongs in the client. A batch endpoint may accept work faster than processors finish it. If that queue is invisible, cap outstanding items and use a polling schedule that cannot create a second traffic spike. Backpressure survives the move from many envelopes to one.
Keep it bounded.
Can Promise.allSettled handle this? It improves result collection for a small, bounded set, but array length is not a concurrency policy, process memory is not a durable queue, and a restart erases the retry plan. It does not provide persistence, backoff, or admission control.
The practical rule is short. Pick the interface that makes unfinished work enumerable. Then limit how quickly workers claim it, persist every transition, and make repeated execution converge on the same derivative objects. That protects transformation capacity while keeping storage and cache growth explainable.
Top comments (0)