TL;DR: To process hundreds of event photos with progress, keep the Node.js or Express upload request short: persist one batch manifest, then run every photo as an independently retryable job. The same pattern applies to the logistics system below, where product photos need background removal. Run that work at upload when every downstream consumer needs the same cutout; defer it until request time only when variants depend on the destination. Page on stalled useful work, not merely on a large queue.
The page reads batch stalled: 0 completions in 10m. The batch contains 480 product photos, 311 are complete, 12 are actively leased, and 157 are waiting. That is enough context for the on-call to distinguish slow work from dead work without opening a dashboard and guessing which line moved. The question I ask is narrower: what page fired, and does it identify an action?
A single request that loops over hundreds of images cannot answer that well. Its process can disappear after photo 311, its caller may time out while useful work continues, and its progress counter vanishes with the heap. Split acceptance from execution. Return a batch identifier quickly, keep the manifest and each photo's state in durable storage, and let bounded workers claim individual items. This has a real limitation: it adds a durable state machine, lease recovery, and cleanup work, so it is not a fit for a tiny batch that comfortably finishes inside the request deadline. In that case, synchronous processing is easier to operate and easier to explain.
How should Node.js process hundreds of event photos in one batch?
Work backward from the stalled-batch page. Queue depth by itself should not page: a healthy upload spike also makes the queue deep. Error count alone is weak for the opposite reason; workers can stop claiming jobs and produce no errors at all. The earlier signal is the age of the oldest eligible item, paired with the rate of completed items and the number of live worker leases.
No completion, no progress.
Instrument state transitions, not just function calls. Record queued_at, started_at, and finished_at for every photo; increment counters for transitions to succeeded, retryable, and failed; and expose the oldest queued age per workload class. A page can then require two conditions, such as oldest eligible work older than a chosen service threshold and zero completions over a second window. Those values are policy, not universal constants. Derive them from the delivery promise and measured processing distribution.
The progress response should summarize total, queued, running, succeeded, and failed. Include a monotonically increasing revision or an updated_at value so clients can avoid repainting unchanged state. Do not equate 100 percent attempted with success; a finished batch can contain permanent failures that require action.
That distinction matters.
The smallest durable shape
The HTTP layer needs three operations: create a batch, fetch its progress, and retry failed items under an explicit policy. The worker talks to generic storage and removal interfaces. Authentication, multipart limits, image decoding, and the segmentation algorithm remain separate concerns.
package batch
import (
"context"
"errors"
"sync"
)
type Job struct {
ID, SourceKey, OutputKey string
Attempts int
}
type Store interface {
Claim(context.Context, string) (Job, error)
Succeed(context.Context, Job) error
Retry(context.Context, Job, error) error
Fail(context.Context, Job, error) error
}
type Remover interface {
RemoveBackground(context.Context, string, string) error
}
var ErrNoWork = errors.New("no eligible work")
func Run(ctx context.Context, batchID string, workers, maxAttempts int, s Store, r Remover) error {
if workers < 1 || maxAttempts < 1 {
return errors.New("workers and maxAttempts must be positive")
}
ctx, cancel := context.WithCancel(ctx)
defer cancel()
errCh := make(chan error, workers)
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for {
job, err := s.Claim(ctx, batchID)
if errors.Is(err, ErrNoWork) { return }
if err != nil { errCh <- err; return }
err = r.RemoveBackground(ctx, job.SourceKey, job.OutputKey)
if err == nil {
err = s.Succeed(ctx, job)
} else if job.Attempts < maxAttempts {
err = s.Retry(ctx, job, err)
} else {
err = s.Fail(ctx, job, err)
}
if err != nil { errCh <- err; return }
}
}()
}
done := make(chan struct{})
go func() { wg.Wait(); close(done) }()
select {
case <-done:
return nil
case err := <-errCh:
cancel(); <-done; return err
case <-ctx.Done():
<-done; return ctx.Err()
}
}
Claiming must be atomic and leases must expire, otherwise two workers can write the same output or an interrupted worker can strand a photo forever. Output keys should be deterministic and writes idempotent. A retry then converges on the same object instead of manufacturing duplicates. Keep the original upload immutable; a later algorithm change can regenerate derivatives without another photo shoot.
This uses a fixed worker count because unlimited goroutines merely relocate backpressure to memory, storage, or the image processor. Start with a conservative bound, measure completion latency and resource saturation, then change one limit at a time. Fast is useful. Recoverable is mandatory.
The trade-off is operational weight. A durable queue is the wrong choice when the workload is small, bounded, and safe to repeat as a whole; a plain synchronous handler has fewer states and fewer cleanup paths. Once batches can outlive the caller, however, pretending the request is the job does not remove those states, it only hides them in timeouts and partial output. The proposed worker pool also does not make CPU-heavy pixel work free: concurrency above the available compute can increase latency, and a remote remover can impose a different bottleneck. The useful limit is therefore discovered under representative images and failure injection, not copied from an example constant.
Upload time or on demand?
Choose upload-time processing when the cutout is part of the catalog's definition of ready. Operators get deterministic validation, downstream exports see one stable asset, and failures surface before a product enters another workflow. The cost is paid even for images nobody later requests, and ingestion waits unless readiness is modeled asynchronously.
Choose on-demand processing when destinations need different crops, backgrounds, or formats, or when only a small fraction of uploads are viewed. It avoids speculative work but moves latency and failure into the read path. Cache derivatives by source version plus transformation parameters, and prevent a burst of identical requests from creating identical jobs.
A mixed policy often fits logistics: create the canonical transparent asset after upload, then derive destination-specific dimensions on demand. Preserve the source format and inspect it before decoding. Browser and pipeline support differs among image formats, while metadata and animation can affect what the decoder yields; the MDN image format guide is a useful compatibility reference. Format conversion should be explicit, not an accidental side effect of background removal.
| Decision signal | Process at upload | Process on demand |
|---|---|---|
| Every downstream flow requires a cutout | Strong fit | Adds read-path risk |
| Transform varies by destination | May create unused variants | Strong fit with caching |
| Failure must block catalog readiness | Strong fit | Detects the problem late |
| Original is rarely requested | Work may be wasted | Defers work until demand |
Prove recovery before tuning speed
Test with a manifest larger than one worker can finish during a lease interval. Kill a worker after it writes an output but before it records success; the replacement must reclaim the job, recognize or overwrite the deterministic output, and mark exactly one terminal result. Repeat with a corrupt file, an unsupported format, a storage timeout, and coordinator cancellation. These are state-machine tests, not screenshots of a progress bar.
Load tests should vary file size and dimensions rather than repeating one friendly sample. Reject inputs beyond documented limits before expensive decoding, and separate permanent validation failures from transient dependency errors. A poison image must consume a finite retry budget. Once exhausted, it becomes failed, the batch continues, and progress identifies the item that needs replacement.
Roll workers independently from the upload API, drain leases during shutdown, and keep state transitions compatible across a rolling release. Track batch completion latency, item attempt count, oldest eligible age, lease expirations, and terminal failure ratio. Dashboards can help during investigation, but none of them is a page definition.
The threshold has a bill. Set the stalled-work window too low and normal variation wakes someone who can do nothing; repeated false positives teach the rotation to distrust the channel. Set it too high and a catalog batch misses its operational deadline in silence. Review actual pages: which signal crossed first, what action resolved it, and would the proposed earlier signal have changed the outcome? If there is no action, remove the page or turn it into a ticket.
Further reading
- MDN, Image file type and format guide: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types
Top comments (0)