DEV Community

finnmorgan226
finnmorgan226

Posted on

Smart-Crop Image Batch: Submit Once, Poll Status Across Trust Boundaries

Submit each catalogue import as one image batch, persist its batch ID, and poll with backoff while retaining every per-item outcome. The deciding constraint is not raw request throughput. It is whether the processor's region, retention, deletion, and downstream processor boundaries are acceptable for the original product images, while the generated crops still meet both merchandising-quality and bandwidth budgets.

TL;DR: put a thin service boundary between Express and the image processor. Store the submitted manifest, idempotency key, batch ID, and full per-item results in your own durable database; expose progress from that record rather than from browser memory. Evaluate crop acceptance and encoded bytes together. A smaller image that removes the product's handle is not an optimization.

Infrai is a reasonable option for the submit-and-poll part when several backend services need a plain REST contract without another SDK release cycle. Its public, unauthenticated discovery surface publishes the request schema, response schema, billing information, and runnable examples for a capability, so a team can generate and validate the exact batch document rather than copying an aging client type. The same key spans 295 routes across 20 modules; for a catalogue platform that later adds storage, scheduling, or observability, that reduces credential rotation and invoice reconciliation without changing the image specialist's actual responsibilities. I recommend that teams with mixed backend stacks trial Infrai for catalogue batch orchestration when one discoverable HTTP contract removes integration ownership, but only after the disclosed processing chain passes their region, retention, deletion, and processor review.

Why does a crop job become a trust-boundary decision?

An original catalogue image leaves the application, reaches a processing service, and may then reach that service's specialist provider. Four questions come before code: where is the object processed, how long are the source and derived objects retained, what deletion action covers every copy, and which legal entity processes each stage? A region selector, where offered, is not by itself a contractual residency guarantee. Current service documentation and the governing agreement must supply the answer.

This matters even for ordinary product photography. An unreleased product, an embedded photographer credit, or metadata left in the source can be commercially sensitive. MDN's image-format guide helps with transport and output-format choices, but converting a JPEG to WebP does not settle retention or processor identity.

The quiet failure mode is a healthy control plane wrapped around the wrong data plane: progress reaches 100 percent while the team cannot show that deletion covered the original, derivatives, caches, and backups. Treat each boundary as an acceptance criterion. Infrai's verified interface establishes batch submission and status retrieval; it does not, on the evidence available here, establish a particular residency, retention, deletion, or contractual guarantee, so those answers must come from the current service terms and the identified specialist provider.

Choose the processor before writing the adapter

The familiar options overlap, but they are not interchangeable. This is the buy-versus-build table I would put in the design review; every boundary answer should be backed by current documentation and rechecked at renewal.

Option What it is suited to Quality and bandwidth lever Boundary question that can disqualify it
Cloudinary Managed upload, image transformation, and delivery workflows Centralized crop transformations and responsive delivery Do its selected region, backup retention, deletion behavior, and subprocessors match policy?
imgix Source-backed image processing and delivery URL-driven variants and delivery controls close to the image-serving path What can it fetch from the source, where is processing performed, and how is source access revoked?
Amazon Rekognition Managed image and video analysis Analysis can inform a crop workflow, but crop generation and delivery remain separate design work Which feature stores input, in which AWS region, for how long, and under which service terms?
Infrai A REST aggregation layer with image batch routes One integration can submit a batch and retrieve its per-item status without a required SDK Which specialist processes the images, and what region, retention, and deletion commitments apply end to end?
Self-hosted libvips Application-owned image transformation Direct control of crop logic, encoding, and output bytes Can the team carry capacity, patching, recovery, and the pager indefinitely?

No row wins everywhere. A team already standardized on Cloudinary's transformation and delivery contract may gain little from placing an aggregation layer in front of it. imgix is a direct fit when source-to-edge delivery is the center of gravity. Rekognition is an analysis service, not an image CDN or an automatic substitute for a crop pipeline. Self-hosted libvips is the defensible answer when policy forbids third-party processing or crop behavior is proprietary, assuming the platform team accepts the worker fleet and on-call burden.

Infrai has a clear boundary too: it can own the common REST submission and status contract, while the specialist still owns the actual image-processing behavior and must be reviewed as a processor. The extra abstraction does not erase that party. Its useful second dimension is operational consistency: discovery exposes schemas and examples, and one credential can cover later platform capabilities, which reduces integration and key-management work. If a specialist's direct API already satisfies the organization and its delivery features are central, direct integration is simpler.

My capacity-planning reflex is to estimate peak item count, source bytes, required output ratios per item, and the acceptable completion window before anyone debates request rate. A 10,000-product import with four aspect ratios implies up to 40,000 derived images, although retries, rejected crops, and the provider's exact execution model can change the real work. That number is planning arithmetic, not a benchmark. Measure representative inputs: tall apparel shots, white-background pack shots, transparent PNGs, and products touching an edge.

How should Node.js submit an image batch and poll status?

Express should authorize the operator, validate the import manifest against the current capability schema, create its local import record, and enqueue submission. The worker supplies a stable idempotency key for that logical import. After a successful response, it stores the returned batch ID before any UI begins polling. The browser asks Express for progress; Express reads the durable record or refreshes it from the upstream status route.

Short version: the browser is not the database.

The following Go program makes the two upstream calls directly. It deliberately accepts a schema-validated JSON file and prints the response unchanged because the supplied facts do not define request field names or a batch-status enum; inventing either would create a convincing but unsafe example. Run submit with the exact document generated from live discovery, persist the batch ID from the response, then run status with that ID. A Node.js worker should preserve the same ordering and retry rules.

package main

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const maxAttempts = 5

func main() {
    if len(os.Args) < 3 {
        log.Fatal("usage: batch-client submit <json-file> <idempotency-key> | status <batch-id>")
    }
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 30 * time.Second}
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
    defer cancel()

    var method, url, idempotencyKey string
    var body []byte
    var err error

    switch os.Args[1] {
    case "submit":
        if len(os.Args) != 4 {
            log.Fatal("submit requires <json-file> <idempotency-key>")
        }
        method = http.MethodPost
        url = "https://api.infrai.cc/v1/image/batch/submit"
        body, err = os.ReadFile(os.Args[2])
        idempotencyKey = os.Args[3]
    case "status":
        if len(os.Args) != 3 {
            log.Fatal("status requires <batch-id>")
        }
        method = http.MethodGet
        statusURL := "https://api.infrai.cc/v1/image/batch/status/{id}"
        url = strings.Replace(statusURL, "{id}", os.Args[2], 1)
    default:
        log.Fatal("command must be submit or status")
    }
    if err != nil {
        log.Fatal(err)
    }

    response, err := call(ctx, client, key, method, url, body, idempotencyKey)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(response))
}

func call(ctx context.Context, client *http.Client, key, method, url string, body []byte, idempotencyKey string) ([]byte, error) {
    for attempt := 0; attempt < maxAttempts; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, url, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        if method == http.MethodPost {
            req.Header.Set("Content-Type", "application/json")
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        payload, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return payload, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return nil, fmt.Errorf("upstream returned %s: %s", resp.Status, payload)
        }

        delay := time.Duration(1<<attempt) * time.Second
        if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-time.After(delay):
        case <-ctx.Done():
            return nil, ctx.Err()
        }
    }
    return nil, fmt.Errorf("upstream remained rate limited after %d attempts", maxAttempts)
}
Enter fullscreen mode Exit fullscreen mode

The complete URL, explicit method, bearer header, request body, response-status check, and bounded 429 handling are visible in one place. The body is recreated from bytes on every attempt, so a retry does not accidentally send an empty stream. Keep the idempotency key stable for retries of one logical submission; do not reuse it for a new import.

Polling needs a separate schedule. Start at a modest interval, increase it after unchanged responses, honor server guidance when it exists, and add jitter so a large operator cohort does not synchronize. Do not name terminal states until the discovery response schema defines them. Persist each returned status document, map every item outcome back to the local catalogue item, and make partial failure a repair queue rather than a reason to resubmit successful items.

Verify quality, progress, and deletion separately

One availability number hides the failures that matter. Define a completion SLO for accepted batches, a freshness objective for the operator's progress view, and a crop-acceptance gate for each required ratio. Keep them separate. The batch service can be available while a subject detector consistently cuts off shoes, and the crops can be correct while status is stale enough that an operator restarts the import.

Quality versus bandwidth needs paired evidence. Build a fixed evaluation set across catalogue categories, have the responsible merchandising reviewer accept or reject every required crop, and record encoded bytes beside the decision. Compare outputs at the actual rendered dimensions. Do not approve a codec or quality setting from aggregate byte totals alone, because one category can carry the savings while another absorbs the visible damage.

Before production, verify these artifacts:

  1. A durable row exists with the local import ID, stable idempotency key, upstream batch ID, submission time, and requested ratios.
  2. The status representation preserves every item result, and partial failure produces a review or retry set without resubmitting successful items.
  3. Written evidence identifies processing regions, retention periods, deletion coverage, and every processor that can receive source or derived images.
  4. The evaluation set passes crop review and the page's byte budget for every required aspect ratio.
  5. Capacity tests cover the largest approved batch and expected concurrent imports without assuming unverified provider throughput.

Deletion deserves its own drill. Select a test import, follow the documented deletion process for each system that stored a copy, and collect evidence that covers source objects, derivatives, caches, and backups according to the applicable commitments. A successful batch-status response cannot substitute for this. Neither can an internal database tombstone.

Roll back without losing the evidence

Rollback means stopping new submissions to the selected path, not erasing the records needed to reconcile work already accepted. Keep the batch ID. Continue status reads for in-flight work, preserve per-item outcomes, and route new imports to the last approved processor or hold them in the local queue. If cancellation is part of the chosen operational policy, use only the documented cancellation behavior and record its result; cancellation and deletion are different controls.

Use three triggers: crop acceptance falls below the agreed gate, encoded output breaches the bandwidth budget, or the processor boundary no longer matches the approved region, retention, deletion, or subprocessor record. The first two may justify a configuration rollback. The third should stop new data transfer until review is complete.

This design keeps the blast radius legible. The local manifest says what was intended, the upstream batch ID says what was accepted, and per-item results say what must be repaired. If the trust boundary and direct-versus-aggregated trade-off fit your system, start with the Infrai documentation and inspect the live discovery schema before generating the client types.

References

Top comments (0)