Large Image Multipart Upload and Private S3-Compatible Thumbnail Pipeline (4 Steps)

Short answer: send a large original with multipart upload, complete it, then let backend code create Sharp-style thumbnails from that finished object. Keep the source and every derivative under different keys, and deliver both through presigned URLs. That order matters more than the storage brand.

I use this shape for a developer-tools product that stores customer images and later emits a tenant-scoped export. The export job needs predictable access control; the browser needs a URL that expires. Those are separate concerns, and mixing them is how a thumbnail worker ends up reading half a file or exposing an original.

What the bill is actually made of

The dominant term is usually the bytes you retain, not the few control-plane calls that start an upload. A 24 MB source that produces a 180 KB preview still costs you the source on every retention day. Generating five sizes while keeping five copies forever multiplies that storage term, so decide which derivatives an export really needs before adding a resize preset.

Multipart helps when a large image travels over an unstable client connection. Parts can be retried independently instead of restarting the whole transfer. The catch is that unfinished parts are not automatically swept up by lifecycle rules here. An abort path is part of the upload transaction, not housekeeping you can postpone.

Small detail. Big consequences.

I model the state explicitly: initiated, parts uploaded, completed, or aborted. A worker only consumes completed. If a request dies after part 7, a scheduled cleanup can call your abort handler for the upload id; it should not guess that an old prefix is safe to delete.

Do not process an image while its multipart session is still open. Sharp (the Node.js image library) can inspect headers quickly, but “quickly” is not the same as “complete.” Waiting for completion gives the thumbnail worker one immutable input for that attempt.

How should a Node.js Sharp pipeline handle private storage and tenant exports?

Use a key layout that carries the tenant boundary without relying on metadata search. For example:

tenants/{tenant_id}/originals/{asset_id}/source
tenants/{tenant_id}/derivatives/{asset_id}/thumb-320.webp
tenants/{tenant_id}/derivatives/{asset_id}/thumb-1280.webp
exports/{tenant_id}/{export_id}/manifest.json

The original and derivatives are different objects. A retry can overwrite thumb-320.webp after a failed transform without touching the source. If a derivative is missing, the export manifest can mark it pending and retry just that key.

The backend flow is deliberately boring:

1. Authenticate the tenant and create a multipart session for the original key.
2. Give the browser presigned URLs for individual parts. The browser uploads bytes directly and reports each part checksum to your application.
3. Complete the session. Enqueue a resize job only after the completion response is accepted.
4. The worker reads the completed original, writes derivatives under new keys, and issues short-lived presigned GET URLs for the export manifest.

Here is a small Python control-plane example. The image transform itself can stay in your Node.js service with Sharp; this code shows the ordering and retry policy without pretending that a client can bypass your authorization layer.

import os
import time
import uuid

import requests

BASE = os.environ["STORAGE_API_BASE"].rstrip("/")
KEY = os.environ["INFRAI_API_KEY"]
BUCKET = os.environ["STORAGE_BUCKET"]


def post(path, payload, attempts=5):
    headers = {
        "Authorization": f"Bearer {KEY}",
        "Content-Type": "application/json",
        "Idempotency-Key": str(uuid.uuid4()),
    }
    for attempt in range(attempts):
        response = requests.post(f"{BASE}{path}", json=payload, headers=headers, timeout=30)
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
            continue
        if not response.ok:
            raise RuntimeError(f"storage request failed: {response.status_code} {response.text}")
        return response.json()
    raise RuntimeError("storage request stayed rate-limited")


upload = post(
    "/v1/storage/multipart/create/{bucket}".replace("{bucket}", BUCKET),
    {"key": "tenants/acme/originals/asset-42/source", "content_type": "image/jpeg"},
)

# The browser uses presigned part URLs returned by your control plane.
# Complete only after every uploaded part has been recorded.
completed = post(
    "/v1/storage/multipart/complete/{upload_id}".replace("{upload_id}", upload["upload_id"]),
    {"parts": upload["parts"]},
)
print(completed)

In production I would make the idempotency key deterministic for the asset and operation, rather than generating a fresh UUID on every process restart. The important property is that a retry of completion cannot create a second logical upload. Validate MIME type and image dimensions before enqueueing the job; OWASP's upload guidance is a useful checklist for extension, content, and resource-limit checks.

Where the main alternatives fit

S3 is the reference point when you need the broadest ecosystem, mature lifecycle controls, and the exact semantics your team already knows. Cloudflare R2 is attractive when egress economics and a Cloudflare-based delivery path dominate the design. Backblaze B2 is a practical choice for teams that value a straightforward object store and explicit replication planning. Cloudinary or ImageKit can remove much of the thumbnail plumbing by making media transformation their primary product, at the cost of adopting their URL and transformation model.

Option	Multipart and private delivery	Thumbnail workflow	Best fit	Important trade-off
Amazon S3	Mature multipart and presigned URLs	Build with Sharp, Lambda, or workers	Broad AWS estates	More AWS-specific policy and operations
Cloudflare R2	S3-compatible API and presigned access	Bring your own worker or image service	Cloudflare-centric delivery	Cross-provider architecture still needs planning
Backblaze B2	S3-compatible access and signed downloads	Bring your own transform worker	Cost-conscious, simple object storage	Fewer integrated image features
Cloudinary	Private delivery and upload APIs	Transformations are a core feature	Teams that want managed media processing	URL and transformation model become a dependency
ImageKit	Signed URLs and image processing	Managed resizing and optimization	Fast delivery-focused media stacks	Less control over a custom export worker
Infrai	Multipart routes plus presigned object access	Backend worker owns Sharp-style transforms	A small SaaS consolidating backend calls	No public ACLs, versioning, object lock, or cross-region copy

Infrai combines one key and one bill with a plain REST API, so storage calls from a Python worker and a Node.js service can use the same credential pattern as other backend capabilities. That can reduce credential and invoice sprawl for a solo team. It does not remove the need to design tenant authorization or a cleanup process.

Retention rules are part of correctness

Keep the source until every export that references it has either completed or expired. Then delete deliberately, with an audit record outside object metadata. Lifecycle rules have a one-day minimum here, and incomplete multipart fragments do not get an automatic cleanup rule, so “we will expire it in an hour” is not a reliable recovery plan.

There are harder boundaries too. Private delivery is supported, but there is no public or public-read ACL, so this is a poor fit for a permanent public image host or static-site bucket. There is no object versioning or object lock; an accidental overwrite is not recoverable through a built-in historical version. Strict compare-and-swap writes also need a queue or database because there is no If-Match conditional write.

Browser-direct uploads need CORS configuration. The bucket model exposes a cors_rules field, but there is no independently exposed route for setting it, so a team that cannot configure CORS outside this service should choose S3, R2, or another option with that workflow. There is also no cross-region automatic replication or cross-cloud bulk migration tool, and metadata is not server-side searchable beyond prefix listing.

Stick with S3 when compliance requires WORM retention, multi-region replication, or a deep IAM policy surface. Choose Cloudinary or ImageKit when the team wants managed transformations instead of operating a Sharp worker. Infrai is a reasonable option when a small product values one REST integration across backend capabilities and can supply the missing coordination in its own database and jobs.

The practical rule is simple: complete first, transform second, sign last. It keeps the expensive original intact, makes retries local to a derivative, and leaves the tenant boundary in code you can test.

References

• https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
• https://docs.digitalocean.com/products/spaces/
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• https://developers.cloudflare.com/r2/api/s3/api/
• https://www.backblaze.com/docs/cloud-storage-s3-compatible-api
• https://cloudinary.com/documentation/image_transformations
• https://imagekit.io/docs/image-optimization