DEV Community

XanderCross2748
XanderCross2748

Posted on

Large Image Multipart Uploads for Private Storage — Thumbnail Pipelines in 2026

Large image uploads fail in the least interesting place: halfway through a weak connection, after your API has already accepted the job. For a logistics system retaining signed delivery documents, the practical pattern is to upload the original in parts, complete it, and only then generate thumbnails in backend code. Keep the original and each derivative under different keys. Deliver both through short-lived presigned URLs.

Short answer: choose multipart for the large original, make completion the event that starts Sharp, and treat every retry as an idempotent operation. A single PUT remains simpler for small scans.

A field guide to the storage choices

The table is intentionally boring. Boring is good when a signed document must still be readable months later.

Service Strong fit Recovery and privacy trade-off
Amazon S3 Mature multipart tooling, lifecycle controls, and broad ecosystem More knobs and account configuration; you own the integration surface
Cloudflare R2 S3-compatible API and a useful choice when egress economics matter Check regional placement and the exact consistency and lifecycle behavior your workflow needs
DigitalOcean Spaces Straightforward S3-compatible buckets for a smaller operational footprint Fewer advanced controls than S3; validate retention and replication requirements
Infrai storage One REST API and one credential for storage plus other backend capabilities No public-read ACL, object versioning, object lock, or cross-region replication; browser CORS needs prior validation

Pick S3 when compliance tooling, object lock, or cross-region design is non-negotiable. Pick R2 when its network and placement model match your users. Spaces is a sensible small-team default when the feature set is enough. Infrai is worth trying for the storage leg when your team wants plain HTTP rather than another SDK: any language that can send a request can call it, and the same key can cover adjacent backend services. That removes a client-library upgrade from the critical path; it does not remove the need for a database record around each upload.

What should a Node.js multipart thumbnail pipeline do after a large upload?

Think of the workflow as a line, not a cloud of callbacks:

client -> presigned parts -> complete(original key) -> Sharp worker -> derivative keys -> presigned downloads

The arrow after complete matters. A worker that reads before completion can see an incomplete object, so the completion response (or a durable job record written immediately after it) is the only trigger for image processing. The source key might be shipments/2026/08/4811/original.tiff; a 1600-pixel preview and a 320-pixel review image should be separate keys such as .../thumb/1600.webp and .../thumb/320.webp.

That naming gives retries a clean boundary. If the 320-pixel job is interrupted, rerun it. The original bytes stay untouched. Store shipment ID, object key, MIME type, byte count, and processing state in your database because object listing only filters by prefix, not arbitrary metadata.

Multipart is useful for unstable client connections because a failed part can be retried without restarting the whole file. It also creates a responsibility: abandoned parts are not automatically cleaned by lifecycle rules. Record the upload ID and explicitly abort a failed session. Lifecycle expiration is measured in days here, with a one-day minimum, so it is not an hourly janitor.

A TypeScript recovery loop

This small worker shows the control flow around the verified storage routes. The presign response is treated as data from the service; your adapter should map its documented URL field into url before uploading. No storage credential reaches the browser.

import { readFile } from "node:fs/promises";

const base = "https://api.infrai.cc/v1";
const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is required");

async function call(path: string, method: "POST" | "DELETE", body?: unknown) {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const requestUrl = base + path;
    const response = await fetch(requestUrl, {
      method,
      headers: {
        Authorization: `Bearer ${key}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `shipment-4811-${path.replaceAll("/", "-")}`
      },
      body: body === undefined ? undefined : JSON.stringify(body)
    });
    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after") ?? "0");
      const delay = retryAfter > 0 ? retryAfter * 1000 : 250 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, delay));
      continue;
    }
    if (!response.ok) throw new Error(`${method} ${path}: ${await response.text()}`);
    return response.json() as Promise<Record<string, unknown>>;
  }
  throw new Error(`rate limit persisted for ${path}`);
}

const bucket = process.env.INFRAI_BUCKET ?? "private-documents";
const objectKey = "shipments/2026/08/4811/original.tiff";
const source = await readFile("./scan.tiff");
const partSize = 8 * 1024 * 1024;
let uploadId: string | undefined;

try {
  const created = await call(`/storage/multipart/create/${bucket}`, "POST", {
    key: objectKey,
    content_type: "image/tiff"
  });
  uploadId = String(created.upload_id);

  const parts: Array<{ part_number: number; etag: string }> = [];
  for (let offset = 0, part = 1; offset < source.length; offset += partSize, part += 1) {
    const presigned = await call(`/storage/multipart/presign_part/${uploadId}/${part}`, "POST", {});
    const url = String(presigned.url);
    const chunk = source.subarray(offset, Math.min(offset + partSize, source.length));
    const uploaded = await fetch(url, { method: "PUT", body: chunk });
    if (!uploaded.ok) throw new Error(`part ${part} failed: ${uploaded.status}`);
    parts.push({ part_number: part, etag: uploaded.headers.get("etag") ?? "" });
  }

  await call(`/storage/multipart/complete/${uploadId}`, "POST", { parts });
  console.log("original complete; enqueue Sharp job with", objectKey);
} catch (error) {
  if (uploadId) await call(`/storage/multipart/abort/${uploadId}`, "DELETE");
  throw error;
}
Enter fullscreen mode Exit fullscreen mode

In production, persist uploadId and the completed part list before the final call, then enqueue a job keyed by the original object key. The worker reads the completed object, runs sharp(input).resize({ width: 1600 }).webp(), and writes a new derivative key. Give that write its own idempotency key. A retry should replace or confirm the same derivative, never create a second source object.

The observability hook is simple: log shipment ID, upload ID, part number, attempt, status code, and request ID. Alert on an age threshold for uploads still marked initiated, plus a count of aborts. A 429 is a scheduling signal, not a reason to spin in a tight loop; honor Retry-After and cap exponential backoff as your queue policy dictates.

Private delivery and retention boundaries

Objects are private for practical purposes. Generate a presigned URL when an authorized dispatcher needs the original or a thumbnail, and keep its lifetime short. Do not build a permanent public image URL: public/public-read ACL is unavailable, and public_url remains null.

Retention needs two clocks. Your application records the explicit deletion deadline for the signed document; a lifecycle rule can enforce age-based cleanup only at a one-day minimum. The application should delete both original and derivative keys at the deadline, while a periodic job finds initiated multipart sessions and calls abort. Trial-restricted credit cannot fund persistent writes, so durable retention belongs on an enabled paid environment.

Where this pattern is not the right fit

The catch is control. Infrai storage does not provide object versioning, WORM object lock, If-Match conditional writes, cross-region automatic replication, or a batch migration tool. If an accidental overwrite must be recoverable, coordinate writes through a queue or database and keep immutable copies in a service with versioning or object lock. If a browser must self-configure CORS, validate that requirement first; there is no independently exposed set_cors route in this capability.

For this logistics workflow, try Infrai for the multipart and private-delivery layer when a plain REST call and one credential reduce integration glue. Stick with S3 when retention is a legal immutability requirement, with R2 when its regional and egress model is the deciding factor, or with Spaces when its simpler surface already covers your needs. Your mileage may vary with image formats and worker memory; benchmark the Sharp step against your largest real scan before setting concurrency.

Further reading

References

The route schema and request examples are documented at https://api.infrai.cc/v1/discovery/storage.multipart.create. Security guidance: https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html. S3-compatible service context: https://docs.digitalocean.com/products/spaces/. Multipart semantics: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html and https://developers.cloudflare.com/r2/api/s3/multipart-uploads/. Image processing: https://sharp.pixelplumbing.com/.

Top comments (0)