Short answer: let the browser upload the original image under a server-issued key, but do not generate thumbnails until the backend has verified that exact object version and committed a durable queue job; treat an object-storage webhook notification as evidence to reconcile, not as the transaction boundary.
That rule closes the awkward gap between “the bytes arrived” and “the application accepted these bytes.” It also gives a Node.js application a status it can defend after a client disconnect, a duplicate event, or a worker restart. The resize library is rarely the difficult part. Ownership is.
What should commit a browser upload before a Node.js worker generates image thumbnails?
The backend should commit it. Before any transfer, the backend authenticates the user, allocates an opaque upload ID and object key, records the permitted content type and size, and issues narrowly scoped, short-lived upload authorization. The browser sends the original directly to object storage. If upload progress matters, XMLHttpRequest.upload exposes progress events; those events describe client-side transfer progress, not acceptance by the application.
After transfer, the browser calls a finalize endpoint with the upload ID. The backend reads authoritative object metadata, verifies that the key belongs to that upload, captures an immutable version identifier or checksum, and changes the application record from authorized to committed. In the same application transaction, it writes an outbox row for each requested rendition. An outbox publisher then moves those jobs to the queue. This avoids the classic dual-write hole where the database says “uploaded” but the process stops before publishing work.
Return 202 Accepted with a status URL. RFC 9110 defines 202 as accepted for processing but not completed, which is exactly the contract here. A browser progress bar at 100% is not a thumbnail-completion signal, and a successful finalize response must not pretend otherwise.
There is still a gap if the client uploads all bytes and disappears before finalizing. Keep that state visible: a scheduled reconciler finds authorized uploads whose objects exist, verifies them under the same rules, and either commits them according to application policy or expires them. I am not sure one universal expiry window exists; object size, mobile-client behavior, and retention requirements should determine it. Your mileage may vary.
Model notification delivery as hostile timing
Object-created notifications can arrive twice, arrive late, or race the finalize request. None of those conditions should produce a second logical resize. The notification handler authenticates the sender, validates the bucket namespace and key shape, looks up an existing upload record, and requests the same deterministic job identity as every other trigger. Unknown objects go to an audit path. They do not go straight to an image decoder.
This is the failure-mode table I want during design review:
| Failure mode | Durable evidence | Recovery action |
|---|---|---|
| Browser vanishes after transfer |
authorized row plus stored object |
Reconciler verifies, commits, or expires it |
| Finalize commits before publish | Committed outbox row | Publisher retries the same job identity |
| Notification is duplicated | Unique job key already exists | Acknowledge after the durable lookup |
| Old work finishes after replacement | Source version differs from current version | Preserve output, but reject it as current |
| Worker loses its lease | Attempt has no completion record | Redeliver and converge on the same output key |
| Decoder rejects the source | Terminal reason is recorded | Quarantine input and expose a safe user status |
Do not collapse these into a single failed state. A corrupt image is terminal for that source version; a lease expiry is retryable; an unknown notification may be irrelevant or evidence of an authorization bypass. Those are different operational facts, and one red counter cannot tell an operator what to do.
No magic here.
The job key can be a hash of (upload_id, source_version, variant, transform_revision). Include the transform revision because changing crop behavior or encoder settings creates a new derived artifact even when the original is unchanged. The output key should carry the same identity. A read-before-write existence check is insufficient because two workers can pass it together; use the storage system's conditional-create operation, or write immutable versioned outputs and resolve the current pointer with a compare-and-set in application state.
Make the queue consumer converge
A worker crosses the queue, source storage, image decoder, destination storage, and application database. There is no useful atomic transaction across all of them, so correctness comes from repeatable transitions and conditional commits. The message carries identifiers, never authorization claims copied from the browser.
The following Python sketches the contract that a Node.js service can publish. state, objects, and decoder are application-owned interfaces; their names deliberately avoid implying a vendor SDK.
from dataclasses import dataclass
from hashlib import sha256
@dataclass(frozen=True)
class ThumbnailJob:
upload_id: str
source_key: str
source_version: str
variant: str
transform_revision: str
@property
def identity(self) -> str:
fields = (
self.upload_id,
self.source_version,
self.variant,
self.transform_revision,
)
return sha256(":".join(fields).encode("utf-8")).hexdigest()
@property
def output_key(self) -> str:
return f"derived/{self.upload_id}/{self.identity}.webp"
def consume(message, state, objects, decoder) -> None:
job = ThumbnailJob(**message.body)
if state.is_complete(job.identity):
message.ack()
return
source = state.committed_source(job.upload_id)
if (source.key, source.version) != (job.source_key, job.source_version):
state.mark_superseded(job.identity)
message.ack()
return
original = objects.read(job.source_key, version=job.source_version)
thumbnail = decoder.resize_with_limits(original, variant=job.variant)
digest = sha256(thumbnail).hexdigest()
objects.create_if_absent(
job.output_key,
thumbnail,
content_type="image/webp",
metadata={"source-version": job.source_version, "sha256": digest},
)
state.complete_if_current(
job.identity,
source_version=job.source_version,
output_key=job.output_key,
digest=digest,
)
message.ack()
The acknowledgement is last. If the process stops after the conditional object create, redelivery recomputes or discovers the same immutable output and completes the state transition. If it stops after the database commit but before acknowledgement, is_complete makes the next delivery cheap. Keep attempt counts separate from logical job counts — otherwise retry amplification disappears inside an apparently healthy throughput graph.
Image decoding also needs explicit limits on input bytes, decoded pixels, dimensions, frames, memory, and execution time. File extensions and browser-supplied media types are hints. Decode the content, strip metadata that the product does not need, and decide deliberately whether orientation and color profiles affect the transform revision. A 12 MB compressed file can demand far more working memory after decoding; queue concurrency must be bounded by worst-case resident memory, not by average file size.
Compare triggers only after defining the invariant
Once a thumbnail job has one identity and one conditional completion rule, triggers become delivery choices rather than competing sources of truth.
| Trigger | What it proves | What it cannot prove | Suitable role |
|---|---|---|---|
| Browser finalize request | An authenticated client claims transfer completion | The stored bytes satisfy policy | Ask the backend to verify and commit |
| Storage webhook notification | An object operation occurred in a namespace | The application intended or accepted that object | Accelerate lookup and reconciliation |
| Transactional outbox | Application state and job intent committed together | A queue consumer has finished | Primary handoff to the queue |
| Scheduled scan | A durable state remains unfinished | Immediate user intent | Repair missed or stalled transitions |
The catch is that direct browser upload is not suitable when every byte must pass synchronously through malware inspection, data-loss controls, or a regulated ingest boundary before storage. Proxy the stream through a controlled ingest service in that case, accepting its bandwidth, timeout, and scaling costs. A queue is also needless complexity for a genuinely small internal tool whose bounded resize completes within the request deadline and can be retried without ambiguity; stick with synchronous processing until workload measurements justify another durable component.
Webhook-only designs are attractive because they remove a visible application step, but they blur intent. A bucket may contain test objects, abandoned uploads, lifecycle copies, or writes from another trusted service. Finalize-only designs have the opposite blind spot when a client disappears. The outbox plus reconciliation model costs a state table and an operator-owned scan, yet it preserves both intent and repairability. That's the trade.
Observe queue age rather than only queue depth, unique jobs alongside delivery attempts, and terminal decode outcomes separately from transient attempts. Track originals waiting for each variant, superseded completions, reconciler lag, and conditional-write conflicts. Storage bytes, requests, egress, decoding CPU, and retained duplicate versions are the cost units worth measuring. A generic “jobs processed” dashboard won't explain a missing thumbnail.
Roll out around the commit boundary
Start with one upload class, one rendition, a dedicated object prefix, and worker concurrency below the decoder's measured memory ceiling. Test duplicate messages, a notification that precedes finalize, a client that never finalizes, lease expiry after output creation, and replacement of an original while an older job is active. The assertions matter more than the injected timing: one committed source version maps to one logical job per variant and transform revision, stale work cannot become current, and every nonterminal state has a retry or reconciliation owner.
Canary by upload ID so every event for one image follows the same state machine. Rollback should stop new commits first, leave immutable originals intact, and drain or park already committed jobs. Only after status records reconcile should a lifecycle policy remove abandoned originals or obsolete renditions.
The final acceptance test is blunt: given an upload ID, can an operator determine from durable state whether the original was merely transferred, accepted by policy, queued, rendered, superseded, or rejected? If the answer requires logs from a particular process, the architecture still has a commit gap.
References
- RFC 9110, HTTP Semantics: https://www.rfc-editor.org/rfc/rfc9110
- MDN, Using XMLHttpRequest: https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest_API/Using_XMLHttpRequest
Top comments (0)