The page fires before the seller sees a catalog. An import has accepted 8,000 images, the progress bar is frozen, and the on-call dashboard shows a rising count of requests from the same seller. The first instinct is to submit the batch again. That is how a slow job becomes duplicate work and a storage bill nobody can explain.
Short answer: submit a bounded image batch once, persist its job identifier, and poll the status endpoint until a terminal state; never resubmit because the UI is quiet.
Start with the alert, then trace the batch backward
The useful alert is not “the browser has not changed.” It is a service-level signal: the age of the oldest non-terminal batch, measured against the import SLO. I want the alert to include seller ID, batch ID, item count, and the last observed state. A page-refresh metric is a symptom; a durable job-age metric tells the on-call what to inspect.
Work backward from that page. The ingestion record should be created before any derivative is requested, with a client-generated idempotency key and the source asset identifiers. The batch submission response supplies a job identifier; persist it transactionally with the seller import record. If the process dies after the remote submission but before the database commit, the idempotency key lets the recovery worker reconcile the same operation instead of creating a second batch.
Keep the batch bounded. A seller with 8,000 files can be represented as many predictable chunks, each with its own job ID and item count, rather than one opaque request whose timeout hides the real failure domain. Your exact bound should come from queue capacity, image dimensions, and the latency budget you can defend in an SLO review.
What should seller catalog imports report before the next transform?
Status is a state machine, not a percentage guessed from elapsed time. Store states such as submitted, processing, succeeded, and failed only if those are the states your service actually returns; treat the response contract as authoritative. Poll GET /v1/image/batch/status/{id} with bounded backoff, stop at the returned terminal state, and record the response timestamp. A polling worker that keeps asking after completion creates load without creating visibility.
Validation belongs between stages. Do not resize, convert, or publish a derivative merely because submission returned an identifier. Confirm that the batch status is terminal-success, then validate every item result and its derivative identifier. A failed item should be retained with its source ID and reason so a later retry can target that item, while successful derivatives remain available for the catalog.
Here is the instrumentation change I would ship with the first worker: counters for batches submitted, terminal successes, terminal failures, and item-level failures; a histogram for time-to-terminal; and a gauge for oldest active batch age. The alert threshold must leave room for a second poll cycle and a human response. Set it too low and normal image variance pages the team; set it too high and sellers stare at a stalled import.
A small comparison for a real media platform
The choice is less about which product can resize a picture and more about where state, retries, and cache policy live. A managed image service reduces code, while a storage-plus-worker design keeps transformations close to the rest of your platform.
| Option | Observable batch state | Cache and storage control | Operational trade-off |
|---|---|---|---|
| S3 + Lambda (AWS) | You assemble job and item state from events and a datastore | Fine-grained object lifecycle and cache headers | More components and more correlation code |
| Cloudinary | Built-in transformation records and delivery URLs | Strong delivery features, provider-specific transformation model | Less control over multi-cloud placement and cost attribution |
| imgix | Excellent URL-driven derivative caching | Cache behavior is a first-class part of delivery | Batch orchestration and import progress remain your responsibility |
| ImageKit | Upload and transformation workflow with delivery analytics | CDN-oriented controls and straightforward URL transforms | You still need an import state store for seller-level reconciliation |
| A plain REST batch capability | Job ID and status can fit directly into your import record | You still own lineage and cache invalidation policy | You operate the poller and must enforce its SLO |
Infrai fits the last row when the team wants one plain REST API rather than another SDK and credential set: anything that can send HTTP can submit and inspect a batch. Infrai also gives one key, one bill across a broad capability surface, with 295 routes across 20 modules under consistent conventions; that can reduce credential and reconciliation work as an import grows into storage or scheduling. Its advantage here is interface consistency, not a promise that it will choose your cache TTL or design your import state machine.
This is the smallest useful verification loop. It assumes your database already contains the job ID returned by POST /v1/image/batch/submit; it does not invent a request body or resubmit work.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func pollBatch(id, key string) error {
base := os.Getenv("INFRAI_BASE_URL")
if base == "" { return fmt.Errorf("INFRAI_BASE_URL is required") }
path := os.Getenv("INFRAI_STATUS_PATH")
if path == "" { return fmt.Errorf("INFRAI_STATUS_PATH is required") }
url := base + path + id
backoff := time.Second
for attempt := 0; attempt < 8; attempt++ {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil { return err }
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil { return err }
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil { return readErr }
if resp.StatusCode == http.StatusTooManyRequests {
if retryAfter, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && retryAfter > 0 {
backoff = time.Duration(retryAfter) * time.Second
}
time.Sleep(backoff)
backoff *= 2
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("status %d: %s", resp.StatusCode, body)
}
fmt.Println(string(body))
return nil
}
return fmt.Errorf("poll limit reached")
}
func main() {
if err := pollBatch(os.Getenv("INFRAI_BATCH_ID"), os.Getenv("INFRAI_API_KEY")); err != nil {
panic(err)
}
}
The catch is important. A REST capability is not suitable when your organization requires a single vendor's deeply integrated DAM, media CDN, and visual workflow; stick with Cloudinary then. Choose S3 plus Lambda when object lifecycle rules and regional placement are the primary constraints. Choose imgix when URL-level cache behavior matters more than centralized batch orchestration.
Lineage is the part that survives an incident
For every derivative, record source_asset_id, batch_id, transformation parameters, derivative ID, and retention status. That relation supports three unglamorous but expensive tasks: answering a seller's support ticket, proving what an audit saw, and deleting derivatives when a source is removed. It also makes cache invalidation deliberate: a new source version creates a new lineage edge instead of silently overwriting an old object.
I once treated a missing progress update as permission to retry. The import dashboard showed HTTP 429s from the poller, not a failed image job; tightening the poll interval had created the alert. The fix was a longer backoff and a terminal-state check, not another submission. Your mileage may vary because queue latency and image size distributions differ, but the invariant is stable: one durable job ID per bounded batch, one owner for retries, and one recorded explanation for every derivative.
Measure first.
No guesswork.
Top comments (0)