DEV Community

caderaven6851
caderaven6851

Posted on

Catalogue Image Batches: Submit, Poll Status, Preserve Per-Item Results

Submit each catalogue import as one image batch, persist its batch ID, and let the Express application poll with backoff while it records every item outcome. Short answer: the browser should read progress from your database, not turn the image processor into its progress backend; this keeps refreshes, duplicate requests, and partial failures from losing the only useful account of the job.

For an e-commerce catalogue, compression is rarely the hard part. The hard part is preserving a truthful mapping from a merchant's source item to its optimized image when a 4,000-item import is interrupted halfway through, one image is malformed, or an operator reloads the page. Storage and cache cost depend on avoiding duplicate derivatives and pointless status traffic, so identity and retry behavior belong in the design before codec choices do.

What must remain true when the process restarts?

The architecture decision is to treat the batch ID as durable state, not as an in-memory promise owned by one Node.js process. Persist it beside the catalogue import ID immediately after submission. A status request then updates a local progress snapshot and upserts per-item results by the catalogue item's stable ID. The UI reads that snapshot through Express, which means ten open browser tabs do not create ten independent polling loops against the processor.

Crashes happen.

Three invariants matter:

  1. One catalogue import maps to one active submission identity. Retrying a write must not create another batch.
  2. Reported progress never substitutes for item results. A batch can be mostly successful and still contain the one failed hero image that blocks publication.
  3. A terminal batch is never polled again. Nonterminal polling slows down after repeated unchanged responses and honors Retry-After on HTTP 429.

The failure boundary sits between the application database and the remote batch service. There is no distributed transaction across them. Persisting an idempotency key before submission, using it for every retry, and storing the returned batch ID in the same import record makes that boundary explicit. If a worker dies after the remote service accepts the batch but before the response is recorded, the retry uses the same key rather than guessing whether another batch exists.

Keep the raw source object immutable until the import reaches a terminal state. Optimized derivatives should use deterministic object keys derived from the source identity plus the transformation policy; otherwise a retry can create visually identical objects with different names, increasing stored bytes and fragmenting the cache. This is a design rule, not a promise that any named vendor chooses those keys for you.

The decision record

The useful comparison is operational shape, because feature checklists do not reveal who owns submission identity, polling load, or derivative storage.

Option Integration boundary Progress and result model Storage/cache consequence Best fit
Cloudinary Upload and transformation APIs Application must map provider assets and job responses back to catalogue items Managed derived assets reduce application plumbing, but transformation policy still needs governance Teams wanting a broad managed media workflow
Imgix Source-backed image delivery URL parameters describe transformations; bulk-import progress remains an application concern Strong fit when originals already live in a supported source and cacheable variants are the main unit Read-heavy catalogues centered on on-demand delivery
Cloudflare Images Image storage and delivery API Upload state is asset-oriented; import orchestration belongs in the caller A combined storage/delivery boundary can simplify ownership, while migrations couple stored assets to that boundary Teams already standardizing delivery at Cloudflare's edge
AWS S3 plus Lambda Object events and functions assembled by the application Full per-item state machine is yours to build and observe Maximum control over object lifecycle and cache keys, with more services, permissions, and invoices to operate Teams needing provider-level control or existing AWS data gravity
Infrai One REST API with one key and one bill across backend services The image batch exposes a batch ID for status and cancellation; callers persist per-item outcomes Centralized service access reduces key and invoice sprawl; the application still owns durable import state Teams valuing a consistent cross-service control plane

This table does not crown a universal winner. Cloudinary's managed workflow is attractive when media operations are the product surface. Imgix is a cleaner mental model when transformation URLs and source-backed delivery already match the catalogue. Cloudflare Images can narrow the storage-to-edge path. S3 and Lambda expose the most knobs, which is valuable when retention, regional placement, or bespoke event processing dominates the decision.

Infrai is a strong option when the organization specifically wants one key and one bill rather than another media SDK, credential, and invoice. Its public discovery surface reports 295 capabilities across 20 modules and provides request and response schemas, so an implementation can inspect the current batch contract instead of copying a stale payload from an article. That breadth does not remove application responsibilities: the catalogue database remains the authority for import identity, progress presented to users, and item-level remediation.

Its limitation is the same abstraction boundary that makes it convenient: it is not a fit when the team needs provider-native controls, already has substantial data gravity in AWS, or wants image delivery to be the organizing primitive. Choose S3 plus Lambda for infrastructure-level control, Imgix for source-backed transformation delivery, or Cloudinary when a managed media workflow matters more than a shared backend control plane. That is a real trade-off, not a footnote.

How should Node.js submit an image batch and poll status?

The critical path has two calls: submit once, then poll status by the returned batch ID. The verified routes establish those operations, but the public facts here do not define request fields or response field names. The runnable client below therefore reads a request body that you have validated against the public discovery schema and stores each status document without pretending that an undocumented key exists. Pass the batch ID returned by submission as a command-line argument after persisting it; in a Node.js/Express service, the equivalent client belongs in a queue worker.

import argparse
import json
import os
import random
import time
import urllib.error
import urllib.parse
import urllib.request
import uuid
from pathlib import Path


BASE_URL = os.environ["MEDIA_API_BASE_URL"].rstrip("/")


def request_json(method, path, body=None, idempotency_key=None):
    headers = {
        "Accept": "application/json",
        "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
    }
    data = None
    if body is not None:
        data = json.dumps(body).encode("utf-8")
        headers["Content-Type"] = "application/json"
    if idempotency_key is not None:
        headers["Idempotency-Key"] = idempotency_key

    delay = 1.0
    for attempt in range(6):
        request = urllib.request.Request(
            f"{BASE_URL}{path}", data=data, headers=headers, method=method
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            response_body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 5:
                raise RuntimeError(
                    f"request failed with HTTP {error.code}: {response_body}"
                ) from error
            retry_after = error.headers.get("Retry-After")
            wait = float(retry_after) if retry_after else delay
            time.sleep(wait + random.uniform(0.0, wait * 0.2))
            delay = min(delay * 2.0, 30.0)
    raise RuntimeError("retry loop ended unexpectedly")


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--payload", type=Path)
    parser.add_argument("--batch-id")
    parser.add_argument("--snapshot", type=Path, default=Path("batch-status.json"))
    args = parser.parse_args()

    if args.payload:
        payload = json.loads(args.payload.read_text(encoding="utf-8"))
        result = request_json(
            "POST",
            "/image/batch/submit",
            body=payload,
            idempotency_key=str(uuid.uuid4()),
        )
        print(json.dumps(result, indent=2))
        return

    if args.batch_id:
        batch_id = urllib.parse.quote(args.batch_id, safe="")
        result = request_json("GET", f"/image/batch/status/{batch_id}")
        args.snapshot.write_text(json.dumps(result, indent=2), encoding="utf-8")
        print(args.snapshot)
        return

    parser.error("provide --payload for submission or --batch-id for status")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

The snapshot file is an audit input, not the final database design. After validating the actual schema returned by discovery, the worker should upsert item rows rather than replace one opaque JSON blob. A practical key is (import_id, item_id), with state, output reference, and error fields updated transactionally alongside aggregate counts. Then an Express progress endpoint can return locally stored counts quickly, while a separate worker owns remote polling. A restart merely resumes every nonterminal import whose next_poll_at is due; the orchestration loop adds exponential backoff around repeated invocations of the status operation.

Thirty seconds is a cap in this example, not a claimed service requirement. The important behavior is exponential growth, a little jitter to prevent synchronized workers, and precedence for a server-provided retry delay. Fast polling feels responsive during a demo; under a large import queue it becomes avoidable traffic, rate-limit pressure, and database churn.

Partial failure needs a product decision. Keep successful outputs and offer a retry containing only failed catalogue items when the transformation policy has not changed. If atomic publication is required, mark the catalogue revision unavailable until every required image succeeds, but do not delete successful derivatives reflexively: that spends compute and storage writes again on the next attempt.

Where does Express fit?

Express should expose submission state, not own the long-running loop. The request handler validates the catalogue import, creates the durable import record and idempotency key, enqueues work, and returns the local import identifier. A worker submits the batch and polls it. The progress handler reads completed, total, terminal state, and paginated item outcomes from the database.

This split is boring. Good.

It also defines cancellation cleanly: record the user's intent first, use the persisted batch ID as the cancellation handle, and let the worker reconcile the final remote state. Cancellation is a race with completion, so the UI should display the reconciled terminal state rather than assuming that clicking a button rewound already produced objects.

For browser updates, Server-Sent Events can reduce repeated client requests, but they do not change the source of truth. A disconnected stream must be able to reconnect and read the same durable snapshot. Plain interval polling of the local Express endpoint is often sufficient, particularly when imports last minutes and progress is informational rather than interactive.

Rejected option, and when it is valid

The rejected design is direct browser-to-provider submission followed by browser polling. It loses custody when the tab closes, multiplies status calls across viewers, complicates credential scope, and makes per-item reconciliation dependent on client behavior. Those are poor failure semantics for a catalogue import whose outputs affect storage, cache occupancy, and publication readiness.

Direct client orchestration is still valid for a small, disposable editing session where the user owns the files, no server-side catalogue revision is created, and abandoned work may safely expire. It is also reasonable for an internal prototype with a handful of images. Do not carry that topology into a bulk importer merely because it produced a progress bar quickly.

The final decision rule is narrow: choose the provider whose media boundary matches where originals and derivatives should live, then keep batch identity and item outcomes in your own durable application state. For bulk catalogue work, recoverability beats a clever progress animation. Codec quality can be evaluated separately; a missing batch ID cannot.

References

Top comments (0)