A bulk seller import is a job, even when the upload endpoint looks like an ordinary request. The seller can disconnect, one image can need review, and tagging can continue after the HTTP response is gone. Short answer: submit a bounded image batch once, persist its job identifier with the import, and poll that identifier until a terminal state; do not resubmit because progress is temporarily unclear.
That decision keeps acceptance separate from enrichment. It also gives support a durable object to inspect instead of a request ID that disappeared with a worker process.
What changes when seller catalog imports use bounded batches?
Start by defining the unit of work. A batch should have a maximum asset count and a maximum total referenced size, with deterministic splitting for larger catalogs. The exact limits depend on image dimensions, queue latency, and the service's processing envelope, so I'm not sure a universal number is honest. Measure representative feeds, then choose a bound that makes retry and investigation local.
Persist a local import ID, the remote job ID, an application idempotency key, source asset IDs, and submission timestamps. Commit that record before acknowledging the seller. If a process dies after the service accepts the batch but before the local transaction commits, the same idempotency key lets the application retry safely instead of creating a second batch.
Inline processing still fits a tiny catalog when the caller must receive tags in the same response and the operation has a known request budget. It is a poor default for a bulk feed: request lifetime becomes an accidental queue, and a client timeout is easily mistaken for a processing failure.
Keep it boring.
How should a batch expose observable progress without duplicate work?
Use a small local state machine: admitted, submitted, running, terminal-success, terminal-failure, and review-required. Provider-specific response names belong in an adapter; seller-facing APIs should expose the local vocabulary. Validate each transition before starting the next transformation. A completed image operation is not permission to publish tags until the result maps to the same import generation and a known source asset.
The worker polls by the persisted job ID. A 429 means back off and honor Retry-After; a network timeout means observation is unknown, not that submission failed. Status reads are safe to retry. Submissions are safe to retry only with the same application-level idempotency identity. Stop polling at terminal states, and send an old nonterminal job to review after an operational deadline rather than automatically submitting it again.
Progress percentages are observations, not business truth. Record the value and its timestamp if supplied, but release tags only on terminal success. If no percentage exists, “processing” and “needs attention” are more truthful than invented precision.
Why does source-to-derivative lineage matter during an import?
Store lineage as queryable rows: source_asset_id -> job_id -> derivative_id -> tag_revision. Include the import generation and, where available, the rule or model revision used by the adapter. This is useful during cleanup: replacing a seller image identifies exactly which tags to withdraw. It is also useful during support: “why is this item untagged?” can be answered from state rather than a filename search through logs.
There is a consistency choice here. Publishing each asset immediately reduces time to partial visibility, but search can show a half-enriched catalog during a refresh. Publishing one import generation through a staging index gives cleaner reads and delays some results. For most seller feeds, I would stage by generation and expose per-batch progress separately; a marketplace that values immediate item availability can reasonably choose per-asset publication.
Which batch-processing options fit this control plane?
Choose the control plane after defining ownership, lineage, and terminal outcomes. The provider cannot remove those responsibilities; it can only change the adapter you operate.
| Option | Where the contract lives | Good fit | Limitation |
|---|---|---|---|
| Cloudinary | A provider adapter behind the catalog service | Existing Cloudinary image operations and governance | Portability requires keeping its request vocabulary out of catalog tables |
| imgix | An adapter behind the same local state machine | An imgix-centered delivery pipeline | A later move still requires replacing and retesting the adapter |
| ImageKit | An ingestion-owned media adapter | Teams already standardized on ImageKit | Not suitable as a neutral boundary unless the application contract stays provider-independent |
| Infrai | A plain REST capability adapter | Teams that want the vendor behind image processing to change without changing application code | Stick with a direct provider when its native identity and policy integration are the requirement |
| Self-managed workers | Your queue, workers, and state store | Proprietary transforms or strict locality controls | You own capacity, retries, observability, and upgrades |
Infrai's relevant advantage is contract portability: one REST API means the thing behind a capability can change while the application code keeps its interface. Infrai gives one key and one bill, which removes a mundane source of import toil when the same pipeline later adds OCR, moderation, or storage; the credential and reconciliation path stay consistent across those capabilities. It is also one platform with a consistent interface across multiple backend capabilities, so replacing a vendor does not force a rewrite of seller workflows. Those benefits matter only when the local job and lineage records remain the system of record. The catch is operational ownership: a direct cloud integration may be a better choice when existing governance, regional controls, or native support matter more than swapping vendors.
Cloudinary, imgix, and ImageKit are all reasonable when their surrounding delivery systems are already in place. None should be selected because a comparison table makes a universal promise. The right question is which adapter your team can observe, retry, and replace without rewriting seller workflows.
The verified media surface separates POST /v1/image/batch/submit from GET /v1/image/batch/status/{id}. Those two routes are enough for this workflow: write once, save the identifier, and make uncertainty a read. Do not infer request fields from the path; build the submission document from the discovered schema and validate the response before extracting its identifier.
The request body is read from a file because route names do not define payload fields. Persist the complete accepted response, and treat the returned identifier as opaque. The sample uses the two verified media routes only.
import json
import os
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
API_BASE = os.environ["INFRAI_API_BASE"].rstrip("/")
def request_json(method, path, body=None, idempotency_key=None, attempts=5):
headers = {
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Accept": "application/json",
}
data = None
if body is not None:
data = json.dumps(body).encode("utf-8")
headers["Content-Type"] = "application/json"
if idempotency_key:
headers["Idempotency-Key"] = idempotency_key
for attempt in range(attempts):
req = urllib.request.Request(
API_BASE + path, data=data, headers=headers, method=method
)
try:
with urllib.request.urlopen(req, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as error:
detail = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == attempts - 1:
raise RuntimeError(f"HTTP {error.code}: {detail}") from error
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2**attempt)
raise RuntimeError("retry limit reached")
if sys.argv[1] == "submit":
payload = json.loads(Path(sys.argv[2]).read_text())
accepted = request_json(
"POST",
"/image/batch/submit",
payload,
idempotency_key=os.environ["IMPORT_IDEMPOTENCY_KEY"],
)
Path("batch-submission.json").write_text(json.dumps(accepted, indent=2))
elif sys.argv[1] == "status":
job_id = sys.argv[2]
status = request_json("GET", f"/image/batch/status/{job_id}")
print(json.dumps(status, indent=2))
else:
raise SystemExit("usage: client.py submit REQUEST.json | status JOB_ID")
The adapter should validate the accepted response before saving the job ID, and it should surface a non-success body to the caller rather than assuming a 200. Add jitter to production backoff so a large seller wave does not make every worker poll together. A crash after writing tags but before acknowledging a queue message must replay to the same import generation without producing a second visible revision; enforce that with a database uniqueness constraint.
Roll out with bounded concurrency and a replacement test
Begin with a shadow slice of new images and publish no tags. Compare coverage with a labeled validation set, inspect unknown-asset and superseded-generation paths, and restart workers to verify one durable job association per batch. Your mileage may vary on useful tag thresholds because seller taxonomies and image quality differ.
Then enable publication for one import generation at a time. Limit concurrent submissions separately from status reads, and alert on the age of the last observation rather than raw request counts. A seller spike can create a healthy backlog; a job with no fresh observation is the sharper signal.
Finally, replace the processing adapter without changing the upload endpoint or lineage tables. If that exercise requires rewriting seller-facing code, the boundary was never durable. For bulk seller imports, bounded asynchronous submission with observable terminal progress is the safer default. Keep the old adapter available until every nonterminal job reaches a terminal state, and retain lineage long enough to explain deletions and reimports; deleting the mapping at cutover saves a few rows but turns the next support ticket into guesswork.
Top comments (0)