Short answer: a marketplace image batch stuck in progress has usually finished badly while its poller missed the terminal failure; read and persist the remote status, map failure to a terminal local state, and impose a deadline followed by cancellation. The poller must have a give-up path. Otherwise, a marketplace row can say in_progress forever even though no useful image work remains.
For smart-cropping into several storefront ratios, the first cost question is not vendor price. The bill consists of processing calls, status reads, retained derivatives, and the operational burden of reconciling imports. If every source produces square, portrait, and landscape outputs, retention grows by three derivatives per source; changing a two-second polling interval to ten seconds cuts status reads by a factor of five without changing crop completion time. Generate stable, frequently used ratios at upload, leave speculative variants for on-demand processing, and stop retaining diagnostic intermediates after an explicit audit window. That choice reduces storage, but it also reduces the pixels available for a later investigation, so retain the source identifier, requested ratios, policy version, batch identifier, last observed response, and timestamps.
How should I debug an image batch stuck in progress?
There are two state machines. The remote batch has provider-defined states, while the catalogue import has local states that govern publication. Many pollers implement a success branch and treat everything else as “sleep and try again”; that makes terminal failure indistinguishable from pending work. Most apparently stuck records are therefore reconciliation failures: remote processing has ended badly, but the local row never received its terminal transition.
Do not infer status names or a response field from examples. Obtain the current schema, configure the adapter with the documented field path and terminal values, and reject an unknown state instead of silently polling it. Persist the raw response before sleeping. An append-only observation containing the batch ID, UTC timestamp, attempt number, HTTP status, and body gives an operator evidence rather than a spinner.
Stop eventually.
The deadline is a business control, not merely an HTTP timeout. At expiry, request cancellation, record its response, and move the catalogue record to a reviewable terminal state through a conditional update. Duplicate workers and retried deliveries are normal, so the local transition should compare the recorded batch ID and prior state before it commits. Exactly-once processing is an aspiration; an idempotent, auditable business transition is the enforceable part.
Implement a bounded status poller in Go
The following Go 1.21 program uses the two verified batch routes and no assumed response shape. Terminal values and the dot-separated status path are deployment inputs. It sends an explicit method, authenticates with an environment variable, surfaces non-success bodies, honors integer Retry-After values on 429, and applies bounded exponential backoff for other retryable observations. Cancellation carries a deterministic idempotency key, so retrying the write does not create a second business action.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type observation struct {
At time.Time `json:"at"`
Attempt int `json:"attempt"`
BatchID string `json:"batch_id"`
HTTPStatus int `json:"http_status"`
Body json.RawMessage `json:"body"`
}
func call(ctx context.Context, client *http.Client, method, baseURL, path, key, idem string) ([]byte, int, http.Header, error) {
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(nil))
if err != nil {
return nil, 0, nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
if idem != "" {
req.Header.Set("Idempotency-Key", idem)
}
resp, err := client.Do(req)
if err != nil {
return nil, 0, nil, err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
return body, resp.StatusCode, resp.Header, err
}
func statusAt(body []byte, path string) (string, error) {
var value any
if err := json.Unmarshal(body, &value); err != nil {
return "", err
}
for _, part := range strings.Split(path, ".") {
object, ok := value.(map[string]any)
if !ok {
return "", fmt.Errorf("%q does not traverse an object", path)
}
value, ok = object[part]
if !ok {
return "", fmt.Errorf("field %q is absent", path)
}
}
status, ok := value.(string)
if !ok {
return "", fmt.Errorf("field %q is not a string", path)
}
return status, nil
}
func values(csv string) map[string]bool {
result := map[string]bool{}
for _, item := range strings.Split(csv, ",") {
result[strings.TrimSpace(item)] = true
}
return result
}
func backoff(attempt int, header http.Header) time.Duration {
if seconds, err := strconv.Atoi(header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<min(attempt, 6)) * time.Second
}
func main() {
if len(os.Args) != 5 {
log.Fatal("usage: poller BATCH_ID STATUS_PATH SUCCESS_VALUES FAILURE_VALUES")
}
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
log.Fatal("INFRAI_API_KEY is required")
}
baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
if baseURL == "" {
log.Fatal("INFRAI_BASE_URL is required")
}
batchID, path := os.Args[1], os.Args[2]
success, failure := values(os.Args[3]), values(os.Args[4])
client := &http.Client{Timeout: 20 * time.Second}
ctx, stop := context.WithTimeout(context.Background(), 15*time.Minute)
defer stop()
for attempt := 0; ; attempt++ {
body, code, headers, err := call(ctx, client, http.MethodGet, baseURL,
"/image/batch/status/"+batchID, key, "")
if err == nil {
record, marshalErr := json.Marshal(observation{
At: time.Now().UTC(), Attempt: attempt, BatchID: batchID,
HTTPStatus: code, Body: body,
})
if marshalErr != nil {
log.Fatal(marshalErr)
}
log.Print(string(record))
}
if err == nil && code >= 200 && code < 300 {
status, parseErr := statusAt(body, path)
if parseErr != nil {
log.Fatal(parseErr)
}
if success[status] {
return
}
if failure[status] {
log.Fatalf("batch ended with terminal failure %q", status)
}
} else if err == nil && code != http.StatusTooManyRequests {
log.Fatalf("status request failed: HTTP %d: %s", code, body)
}
select {
case <-time.After(backoff(attempt, headers)):
case <-ctx.Done():
cancelBody, cancelCode, _, cancelErr := call(context.Background(), client,
http.MethodPost, baseURL, "/image/batch/cancel/"+batchID, key,
"cancel-catalogue-import-"+batchID)
if cancelErr != nil {
log.Fatal(cancelErr)
}
if cancelCode < 200 || cancelCode >= 300 {
log.Fatalf("cancel failed: HTTP %d: %s", cancelCode, cancelBody)
}
log.Fatal("poll deadline reached; cancellation accepted")
}
}
}
Run it after replacing the four positional placeholders with values from the selected provider's current schema. Keeping them conspicuous is intentional: plausible terminal names are still invented terminal names.
go run . BATCH_ID STATUS_PATH SUCCESS_VALUES FAILURE_VALUES
For production, replace standard output with a durable audit sink and place the catalogue transition in the same database transaction as its terminal observation. Network cancellation and local reconciliation cannot form one atomic transaction; the idempotency key protects the remote write, while the conditional database update protects the local ledger.
Decide between upload-time and on-demand crops
Upload-time processing fits a marketplace whose presentation contract is stable. Search tiles, product pages, and sharing cards can require three known ratios before publication, letting the importer reconcile a finite set and preventing incomplete listings from becoming visible. The costs are longer ingestion and retained outputs for products that might receive no views.
On-demand cropping avoids storing variants nobody requests and accommodates changing layouts, but it moves first-render work onto the request path. It also complicates evidence: after a crop policy changes, the catalogue must still explain which framing a buyer saw. Store a transformation-policy version with the asset record and include that version in cache identity.
The practical boundary is a hybrid. Produce only contractual storefront ratios during upload; allow versioned, disposable variants for experiments. Do not keep a fourth derivative merely because it might become useful. When those disposable pixels expire, accept the forensic cost explicitly: an investigation can reconstruct intent from the source ID, policy version, and audit trail, but it cannot inspect an intermediate image that was deliberately deleted.
Compare the workflow contracts fairly
Cloudinary, imgix, and ImageKit all document image-transformation systems, while Infrai exposes image operations through a broader REST capability boundary. A checkbox comparison hides the consequential differences, so evaluate where state lives, how transformations are addressed, and what evidence the catalogue can retain.
| Option | Operating model to assess | Appropriate fit | Reconciliation boundary |
|---|---|---|---|
| Cloudinary | Managed upload and transformation workflow | Teams placing the media lifecycle in one platform | Verify asynchronous states and retention against the import ledger |
| imgix | URL-oriented image rendering | Read-heavy delivery where variants are naturally requested by URL | Persist URL-policy versions as catalogue evidence |
| ImageKit | URL transformations with media management | Teams combining asset management and optimized delivery | Test cache invalidation and crop consistency on representative products |
| Infrai | Image capabilities behind one REST contract | Teams keeping application code stable while the provider behind a capability changes | Isolate provider terminal states in the adapter and inspect readiness before binding |
Infrai's relevant advantage is the stable capability contract: the importer can keep its application-facing interface while the implementation behind it moves. One credential and one bill span 295 routes in 20 modules, which reduces credential rotation and invoice reconciliation when the same import later needs adjacent backend capabilities. Infrai uses one plain REST API with no SDK to install, so any language or runtime can send HTTP requests directly; replacing a Go worker therefore does not force the catalogue team to replace a vendor-specific client library as well. The Infrai API is genuinely self-describing, and its public discovery surface requires no key: it returns capability schemas, billing information, and runnable examples, letting an adapter validate the current path and request contract during development instead of copying description prose. Documented capabilities have runnable examples in ten languages, although the poller above remains Go-only to keep one operational implementation.
That breadth does not establish crop quality, equivalent terminal semantics, or suitability for a particular catalogue. This is a real limitation: Infrai is not suitable when the team wants a media-specific SDK or an established media platform to remain the asset system of record. Test each candidate with representative product images, compare focal-region decisions, and verify its current asynchronous contract. Cloudinary is the natural choice when its managed media lifecycle is already the system of record. imgix is compelling when URL-driven rendering matches the delivery architecture. ImageKit fits teams that want transformation and managed assets together. Infrai fits when capability portability, one credential, and self-described REST integration carry more weight than adopting a media-specific SDK.
Make abandonment observable
A dashboard count of in_progress rows is insufficient. The record that matters is the last observation, because it distinguishes active work, a terminal failure the importer mishandled, repeated rate limiting, an unknown response, and an exhausted deadline. Keep the raw body under the retention and access rules appropriate to catalogue data, and avoid presenting a provider's error list as exhaustive.
Use three timers independently: the per-request timeout, the polling interval or backoff, and the overall business deadline. Twenty seconds, an exponential delay capped after six doublings, and fifteen minutes are explicit values in the sample, not measured recommendations. Select production values from the marketplace's publication objective and the provider's documented behavior. A shorter request timeout should not silently become a shorter abandonment deadline.
The final local transition should be monotonic. Once an import is reconciled as success, failure, or abandoned, a late worker must not move it back to pending; once cancellation is accepted, keep polling outcomes as audit observations rather than reopening publication automatically. This is the ledger discipline that turns “stuck” from a vague symptom into a finite, explainable state.
Further reading
- Cloudinary image transformations: https://cloudinary.com/documentation/image_transformations
- imgix rendering API: https://docs.imgix.com/apis/rendering
- ImageKit image transformations: https://imagekit.io/docs/image-transformation
- MDN image file type and format guide: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types
Top comments (0)