Short answer: treat in_progress as a lease, not a fact about worker health; record heartbeats, expose a small set of terminal states, and move a batch to abandoned after a bounded deadline instead of polling forever.
For a fintech catalogue import that smart-crops each source image into several aspect ratios, the important question is not which image library wins a feature list. It is who owns the state transition when a worker disappears between two writes. My rule is to let workers claim work with a lease, let clients poll a read model, and make the server—not the browser—decide when an attempt is terminal.
That boundary keeps a stale browser from reviving a job and gives support staff an explainable answer. A batch can be queued, in_progress, succeeded, failed, or abandoned; cancel_requested is an instruction, not a terminal result. Keep the vocabulary boring. Boring state is auditable state.
How should Node.js debug a stuck image batch with status polling?
Start with one batch timeline, not another retry. For every transition, persist batch_id, item_id, attempt, state, updated_at, lease_expires_at, and an error category that contains no customer secrets. Then compare the poll response with the timeline. If updated_at advances while the count does not, the worker is making partial progress or repeatedly failing one item. If neither timestamp advances and the lease is expired, the claim is stale. If the lease is fresh, a fast client timeout is noise, not proof of a stuck batch. I also keep the raw transition sequence for at least the retention period of the import record: a count alone cannot show that item 17 failed twice, item 18 was reclaimed, and item 19 finished after the browser disconnected. That sequence lets an operator replay the decision without guessing which poll response arrived first, and it makes a disputed catalogue image traceable to a source digest and attempt number rather than to a mutable dashboard label.
Stop polling forever.
Polling should be conditional and bounded. A client can send If-None-Match with the last representation, increase its interval from 1 second to 10 seconds, and stop after a server-provided deadline. Returning 304 Not Modified saves bandwidth, but it does not change the state machine. The server still needs a durable record of the last meaningful event.
I use a 15-minute lease for ordinary crops only as a starting policy; image dimensions, queue latency, and the slowest decoder should determine the real value. I'm not sure a single timeout is portable across teams. Measure p95 processing time and add enough margin for one retry, then review the number when the catalogue mix changes.
Decision record: lease-based workers or blind status polling?
The two techniques solve different problems. A lease controls ownership of work. Polling controls how a reader learns about work. Replacing the first with the second creates the familiar forever-spinner: the browser sees in_progress, but no component can prove whether a worker still owns the item.
| Choice | What it guarantees | Failure it exposes | Suitable boundary |
|---|---|---|---|
| Lease-based worker claims | One active owner until expiry; another worker can reclaim stale work | Duplicate work after a timeout unless writes are idempotent | Batch processors with retries and worker churn |
| Blind status polling | A simple read loop for clients | No evidence that progress is alive; high request volume | Small internal tools with a hard server deadline |
| Event-driven updates | Low-latency progress without frequent reads | Delivery gaps require a later reconciliation read | User interfaces that already operate a message channel |
| Synchronous request | Immediate success or failure for one item | Request timeout can hide a still-running operation | Tiny images with a measured sub-second path |
The rejected option is infinite polling. It looks kind to users, yet it converts an infrastructure decision into a UI superstition. A browser cannot distinguish a slow decoder, a lost worker, and a database write that committed after the response was dropped. Give it a deadline and a terminal response instead.
Critical path: claim, crop, and commit an idempotent result
The worker below is deliberately storage-agnostic. It claims an item with a compare-and-set update, verifies that its lease is still valid before committing each derivative, and uses a deterministic output key. The image operation is represented by crop_to_ratio; the same contract works behind a local process or a remote service.
from datetime import datetime, timedelta, timezone
from hashlib import sha256
LEASE = timedelta(minutes=15)
def utcnow():
return datetime.now(timezone.utc)
def output_key(source_digest, ratio):
safe_ratio = ratio.replace(":", "x")
return f"derivatives/{source_digest}/{safe_ratio}.jpg"
def process_item(db, object_store, item_id, ratios):
now = utcnow()
lease_until = now + LEASE
claimed = db.compare_and_set(
"items",
key=item_id,
expected_states=("queued", "retryable"),
values={"state": "in_progress", "lease_expires_at": lease_until,
"updated_at": now},
)
if not claimed:
return "not_claimed"
source = db.read_source(item_id)
source_digest = sha256(source.bytes).hexdigest()
try:
for ratio in ratios:
if utcnow() >= lease_until:
db.transition(item_id, "retryable", "lease_expired")
return "retryable"
derivative = crop_to_ratio(source.bytes, ratio)
object_store.put_if_absent(
output_key(source_digest, ratio), derivative, content_type="image/jpeg"
)
db.transition(item_id, "succeeded", None)
return "succeeded"
except ValueError:
db.transition(item_id, "failed", "invalid_image")
return "failed"
except TimeoutError:
db.transition(item_id, "retryable", "dependency_timeout")
return "retryable"
put_if_absent is the important detail, even if its implementation differs by object store. A retry must not publish a half-written derivative under a new random name. The database transition also needs a guard: a reaper may mark an expired item retryable while an old worker is still returning. The old worker's late succeeded write must fail its state predicate.
For catalogue imports, aggregate status from item rows rather than incrementing a counter in the browser. A batch is succeeded when every required ratio for every item has a committed manifest. It is failed when a non-retryable item exists. It is abandoned when the batch deadline passes and remaining items cannot be safely reclaimed. Those rules make a partial crop visible instead of presenting a green check beside missing formats.
Terminal states, give-up paths, and operator evidence
Define the give-up path before production. At the batch deadline, stop accepting new claims, mark unclaimed work abandoned, and let an operator requeue selected items after checking the event timeline. For an item that exceeded three attempts, keep the source object and failure category, but do not silently loop. A manual requeue should create a new attempt number while retaining the original batch identifier.
The status endpoint should expose counts and the oldest updated_at, plus a reason for a terminal state. It should not expose stack traces, access tokens, or source filenames that contain account data. Log a correlation id on every claim, crop, write, and transition; emit metrics for lease expirations, retryable failures, terminal failures, and poll requests returning unchanged representations.
One short rule helps on call: a fresh lease is working, an expired lease is reclaimable, and a terminal state is never polled back into life.
The catch is that leases add clock and cleanup work. If database clocks drift, a worker can be reclaimed too early; use a single time authority or a generous margin. This design is not suitable when the crop operation must be exactly once with an external side effect that cannot be deduplicated. In that case, use a coordinator or an idempotency protocol at the side-effect boundary. Stick with synchronous processing when the measured operation is tiny and a user genuinely needs the pixels in the same response.
Top comments (0)