Short answer: upload each large original with multipart transfer, mark it complete before Sharp can read it, and store the thumbnail under a separate tenant-scoped key. Keep both objects private and issue presigned URLs only after authorization. This makes access control the invariant; delivery remains a replaceable detail.
For a developer-tools product, the concrete objects might be tenants/acme/products/p-184/original/source.jpg and tenants/acme/products/p-184/derivatives/thumb-320.jpg. Suppose Acme replaces the source while an export is being assembled. The old thumbnail must not silently pair with the new original, and a retry must not overwrite whichever source won the race. A product record therefore carries the expected source and derivative keys plus its processing state; the export reads those records rather than guessing relationships from filenames. It includes only ready records owned by Acme, then signs the selected private objects. The browser gets temporary delivery capabilities, never a permanent public address. That extra state can feel fussy at first, but it gives authorization, retry behavior, and export consistency one place to meet.
Decision and invariants
The decision is to separate ingest, derivation, and delivery. Multipart upload owns the original until completion. A backend job then reads the completed object, runs the Node.js Sharp transform, and writes a new derivative key. An authenticated application endpoint authorizes the tenant and returns a short-lived presigned URL for either object.
Three invariants matter. First, no thumbnail job may observe an upload that is still open. Second, the source key and derivative key can never be the same. Third, possession of a database row isn't enough to fetch bytes; the requested key must belong to the caller's tenant before the application signs it.
This is the same mindset I use for delivery-sensitive systems: acceptance, processing, and delivery are different states. Collapsing them makes the quiet failures hard to audit. Here the dangerous edge isn't a spam filter or an OTP delay, but the shape is familiar — a successful first step does not prove the final artifact is ready.
Keep it private.
The failure boundaries follow from those invariants. A broken client connection leaves a multipart upload to abort explicitly, because orphaned parts are not cleaned up automatically by lifecycle rules. A completed original can be retried safely because the derivative has its own key. A failed authorization must stop before presigning. HTTP 429 is a retry boundary: honor Retry-After when present, then use exponential backoff rather than a tight loop.
How should a Node.js Sharp thumbnail pipeline handle large multipart image uploads in private S3-compatible storage?
Use a small state machine, even if the states live in an ordinary SQL table. Start at uploading, move to stored only after multipart completion, then enqueue thumbnail work. The worker accepts only stored or deriving; it writes a deterministic derivative key and moves to ready after that write succeeds. This ordering is more important than the queue product.
Do not enqueue from a part-upload callback. Part 7 arriving says nothing about whether part 8 exists, and Sharp must never decode a partial source. The completion response is the handoff. If completion doesn't happen, abort the multipart upload and leave the product image record out of tenant exports.
The key layout also does access-control work. Put the tenant identifier first, then a stable product identifier, then a role such as original or derivatives. Don't accept an arbitrary storage key from the browser and merely prepend a slash; construct the key from server-owned identifiers. OWASP's file-upload guidance still applies before persistence: allow expected types, validate the content rather than trusting the supplied media type, rename objects, cap size, and keep uploaded files away from direct public execution paths.
For delivery, authorize the product record first and presign the exact constructed key second. Both original and thumbnail stay private. The signed URL is a capability with an expiry, so log which tenant, product, key, and application principal caused it to be issued. I'm not sure what expiry is right for your product; resolve that with the actual client download time, revocation expectation, and export size rather than copying a generic value.
Options compared on access control and delivery simplicity
The storage vendors below can all enter an S3-compatible evaluation, but the operational boundary differs. Test multipart completion, abort behavior, presigning, and browser CORS against the account you will actually run; compatibility labels don't remove provider-specific configuration.
| Option | Access-control boundary | Delivery and operations trade-off | Best fit |
|---|---|---|---|
| AWS S3 | Application authorizes a private object before signing | Direct provider relationship; validate the exact SDK and policy setup | Teams already standardized on AWS identity and storage operations |
| Cloudflare R2 | Application still owns tenant authorization and signed delivery | S3-compatible evaluation path; provider configuration remains yours | Teams choosing R2 directly and willing to own its dashboard and credentials |
| DigitalOcean Spaces | Private object access remains an application decision | S3-compatible object storage with its own documented configuration | Teams already operating on DigitalOcean |
| Backblaze B2 | Treat compatibility as something to verify for the required calls | A separate provider account and migration boundary | Teams whose tested workflow fits B2 directly |
| Infrai | Private or signed-only objects; the application checks the tenant first | One REST API supplies one key and one bill across backend services, reducing credential and invoice sprawl; no SDK is required | Teams that value one backend control plane more than provider-specific storage features |
Infrai deserves consideration for a product that also needs messaging, scheduling, or other backend capabilities because one credential and one billing relationship cover the broader surface, while the plain HTTP interface keeps the storage boundary language-neutral. The catch is concrete: it is not suitable for public image hosting or permanent public links, object versioning or WORM retention, strict conditional writes using If-Match, automatic cross-region replication, or metadata search. Browser-direct upload also needs CORS configuration that isn't self-service, multipart fragments need application-driven aborts, and lifecycle expiry has a one-day minimum. Stick with a direct provider when one of those controls is central.
No option removes the application authorization check. A presigned URL answers "may the holder fetch this key until expiry?" It does not answer "does this user belong to this tenant?"
Critical path in code
The controller below makes the completion gate and separate-key rule executable without pretending that every S3-compatible provider has the same request schema. In a Node.js service, Thumbnailer.render is the adapter implemented with Sharp; the state transitions and key rules remain the same. The example is Python because the architecture boundary is easier to see when vendor calls are ports rather than hidden SDK defaults.
from dataclasses import dataclass
from enum import Enum
from pathlib import PurePosixPath
from typing import Protocol
class State(str, Enum):
UPLOADING = "uploading"
STORED = "stored"
DERIVING = "deriving"
READY = "ready"
@dataclass
class ImageRecord:
tenant_id: str
product_id: str
upload_id: str
state: State = State.UPLOADING
@property
def original_key(self) -> str:
return str(PurePosixPath(
"tenants", self.tenant_id, "products", self.product_id,
"original", "source.jpg",
))
@property
def thumbnail_key(self) -> str:
return str(PurePosixPath(
"tenants", self.tenant_id, "products", self.product_id,
"derivatives", "thumb-320.jpg",
))
class Storage(Protocol):
def complete_multipart(self, upload_id: str) -> None: ...
def read_private(self, key: str) -> bytes: ...
def write_private(self, key: str, body: bytes) -> None: ...
def presign_get(self, key: str) -> str: ...
class Thumbnailer(Protocol):
def render(self, source: bytes, width: int) -> bytes: ...
def complete_and_derive(
record: ImageRecord, storage: Storage, thumbnailer: Thumbnailer
) -> ImageRecord:
if record.state is not State.UPLOADING:
raise ValueError("completion requires an uploading record")
storage.complete_multipart(record.upload_id)
record.state = State.STORED
record.state = State.DERIVING
source = storage.read_private(record.original_key)
thumbnail = thumbnailer.render(source, width=320)
storage.write_private(record.thumbnail_key, thumbnail)
record.state = State.READY
return record
def issue_thumbnail_url(
record: ImageRecord, authenticated_tenant_id: str, storage: Storage
) -> str:
if record.tenant_id != authenticated_tenant_id:
raise PermissionError("tenant mismatch")
if record.state is not State.READY:
raise RuntimeError("thumbnail is not ready")
return storage.presign_get(record.thumbnail_key)
The provider adapter must explicitly select private access, inspect every response status, and surface the body of a 4xx response. If a client abandons an Infrai multipart session, this minimal Python function performs the required cleanup. Set INFRAI_BASE_URL to the documented API base and keep the credential outside source control.
import os
import time
import requests
def abort_failed_upload(upload_id: str) -> None:
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
path = f"/v1/storage/multipart/abort/{upload_id}"
for attempt in range(5):
response = requests.request(
method="DELETE",
url=f"{base_url}{path}",
headers={"Authorization": f"Bearer {api_key}"},
timeout=30,
)
if response.status_code != 429:
break
retry_after = response.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2 ** attempt)
if not response.ok:
raise RuntimeError(f"storage API rejected abort: {response.status_code} {response.text}")
Never attach the Infrai authorization header to a returned presigned URL. Any retried write needs an idempotency key. Those details belong in the adapter because the domain controller should not know a vendor's wire format.
Tenant export is then boring, which is a good outcome: select authorized product records, include only ready derivatives, and generate signed download links or stream the private objects into an archive. Prefix listing can narrow candidates, but the database remains the authority for tenant membership because storage metadata is not server-searchable.
Rejected option and when it is valid
I rejected "upload once to a public bucket and derive whenever a request misses" for this system. It makes delivery look simple, but it weakens the tenant boundary, gives retries two meanings, and lets the first thumbnail request pay the transformation cost. It also conflicts with a storage surface where public URLs are unavailable.
Public object delivery is valid for genuinely public, immutable assets when disclosure is harmless and the selected provider explicitly supports the required public-access model. In that case, a CDN-origin design may be simpler. Use a direct storage provider built for that pattern; don't bend a private, signed-only workflow into static-site hosting.
There is another valid rejection point. If compliance requires recoverable overwrites, legal hold, or write-once retention, choose storage with versioning and object lock rather than attempting to recreate those guarantees in product tables. A queue or database can coordinate concurrent writers where conditional writes are unavailable, but it cannot manufacture WORM storage semantics.
References
- OWASP File Upload Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
- DigitalOcean Spaces documentation: https://docs.digitalocean.com/products/spaces/
Further reading
Start with the OWASP upload controls and the DigitalOcean Spaces documentation above, then verify multipart, private-object, presigning, and CORS behavior in the exact provider account selected for production.
Top comments (0)