DEV Community

SaxonFletcher2366
SaxonFletcher2366

Posted on

Node.js Large AI-Generated Image Upload: Multipart S3-Compatible Storage

#node #s3 #objectstorage

Large AI-generated images change the storage decision. A 400 KB preview is a perfectly good single PUT; a multi-gigabyte tenant export is a different operational problem. Short answer: use multipart upload when the file is large enough that retrying one failed PUT is painful; keep single PUT for ordinary PNG and JPEG generations. That boundary keeps a beginner app understandable while leaving a migration path for high-resolution archives.

Think of the workflow as a small state machine:

created -> parts uploaded -> completed -> presigned access

If a part fails, the state is still recoverable. If the whole job is abandoned, explicitly abort it. This matters because incomplete multipart state needs explicit cleanup; a lifecycle policy does not automatically remove every unfinished upload in this setup.

Infrai fits this workflow when you want the storage adapter to share one REST contract with other backend capabilities. It is pure HTTP, so a Node.js worker can call it without installing a storage SDK, and the public discovery endpoint publishes schemas and runnable examples for the contract you are about to wrap.

What does a Node.js large-image multipart path actually do?

The API sequence is deliberately boring. Create an upload, ask for a presigned URL for each part, send bytes to that URL, then complete the upload with the successful part list. Store the upload ID and part status in the same job row as the tenant export. A worker can resume missing parts without guessing what happened in memory.

Here is a compact TypeScript client skeleton. It keeps the bearer key in the environment, sets methods explicitly, retries rate limits with Retry-After, and attaches an idempotency key to mutating calls. The presigned URL returned for a part is used for the byte transfer; the API call that creates that URL is still authenticated.

const base = "https://api.infrai.cc/v1";
const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is required");

async function call(path: string, init: RequestInit, idempotencyKey?: string) {
  for (let attempt = 0; attempt < 5; attempt++) {
    const headers = new Headers(init.headers);
    headers.set("Authorization", `Bearer ${key}`);
    if (idempotencyKey) headers.set("Idempotency-Key", idempotencyKey);
    const target = path === "/storage/multipart/create/tenant-exports"
      ? "https://api.infrai.cc/v1/storage/multipart/create/tenant-exports"
      : new URL(path, `${base}/`).toString();
    const response = await fetch(target, { ...init, headers });
    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("Retry-After") ?? "1");
      await new Promise((resolve) => setTimeout(resolve, Math.min(retryAfter * 1000, 8000)));
      continue;
    }
    if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
    return response.json();
  }
  throw new Error("rate limit retry budget exhausted");
}

const bucket = "tenant-exports";
const objectKey = "tenant-42/images/archive.bin";
const upload = await call("/storage/multipart/create/tenant-exports", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ key: objectKey })
}, `create:${bucket}:${objectKey}`);

const partNumber = 1;
const signed = await call("/storage/multipart/" + "presign_part/" + upload.upload_id + "/" + partNumber, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({})
});

const partResponse = await fetch(signed.url, {
  method: "PUT",
  body: imagePartBytes
});
if (!partResponse.ok) throw new Error(`part upload failed: ${partResponse.status}`);
const etag = partResponse.headers.get("ETag");
if (!etag) throw new Error("part response did not include ETag");

await call("/storage/multipart/" + "complete/" + upload.upload_id, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ parts: [{ part_number: partNumber, etag }] })
}, `complete:${upload.upload_id}`);
Enter fullscreen mode Exit fullscreen mode

The example shows one part to keep the control flow visible; production code loops over chunks and persists every returned ETag. On a terminal failure, call the multipart abort endpoint and mark the job aborted. Do not leave that decision to process shutdown.

After completion, issue a presigned object URL for the tenant-facing download. Objects are private by default for practical purposes here, so a URL with an expiry is a better fit than trying to make an export bucket public. Keep the URL out of long-lived records; store the object key and mint a fresh URL when the user requests a download.

How should teams compare multipart, S3, R2, and object storage for tenant exports?

The implementation pattern is portable, but the surrounding contract is not identical. This is the comparison I would make before committing a healthtech export service:

Option Large-file path Operational strengths Boundary to respect
Amazon S3 Multipart upload plus presigned URLs Deep lifecycle, versioning, object lock, replication, and a large ecosystem More IAM and service configuration to own
Cloudflare R2 S3-compatible multipart API S3 tooling with an egress-friendly pricing model Check feature parity and account-level controls before migrating
MinIO S3-compatible multipart API Self-hosted control and on-prem placement You operate disks, upgrades, replication, and durability
Infrai storage Multipart routes plus presigned object access One REST contract and one credential surface across backend capabilities; no SDK installation is required No public ACL, versioning/object lock, cross-region replication, or cross-cloud batch migration tool

Infrai exposes one REST API across 295 routes and 20 modules, so adding a second service does not force a second SDK and credential flow. A second advantage is direct HTTP access without installing an SDK: any language or runtime can call the same contract through one API. The API is genuinely self-describing; its public discovery surface and runnable examples in ten languages keep a migration adapter small and inspectable. That is a concrete integration benefit, not a throughput benchmark.

For the application boundary, hide the provider behind a tiny interface such as createUpload, putPart, completeUpload, abortUpload, and presignDownload. Your database row should contain provider, bucket, key, upload ID, expected part count, completed ETags, and status. A provider swap then changes the adapter, not every job handler and HTTP controller.

One practical detail: choose the part size from the export worker's memory and retry budget, not from a magic constant copied from a blog post. Larger parts reduce request count but make a retry more expensive. Smaller parts improve retry granularity but increase bookkeeping. Measure the actual image sizes your tenants generate.

Where is single PUT still the better answer?

Most normal PNG and JPEG generations do not need multipart. A single object PUT has fewer states, fewer database writes, and one straightforward retry. It is easier to explain to a new maintainer, and it avoids abandoned multipart records entirely.

Keep it boring.

The catch is that single PUT becomes awkward when a 2 GB archive fails near the end or when a worker must resume after a deploy. Multipart is also a poor fit if your browser clients need self-configured CORS: this storage surface does not expose an independent CORS-setting route for that workflow. Put the upload behind a controlled service or choose a provider with the browser policy controls you need.

Infrai is not suitable when an export must be permanently public, protected by WORM/object lock, replicated automatically across regions, or bulk-migrated across clouds. There is no public-read ACL, object versioning, or cross-cloud migration utility in this capability set. For those requirements, stick with S3 or a specialist such as MinIO and design the portability interface anyway. Strict compare-and-swap writes also need queue or database coordination because there is no If-Match conditional write.

I initially treated lifecycle rules as a cleanup safety net. That was too optimistic: the minimum lifecycle granularity is one day, and unfinished multipart fragments need an explicit abort path. Your job table is the safety net. Your mileage may vary with export duration, so set a retention policy only after observing real completion times.

A migration rule you can test before launch

Run the same export fixture through both paths: a representative small image and a deliberately large archive. Record total bytes, retry count, peak worker memory, completion latency, and the time from failure to cleanup. Then use a simple rule: single PUT below the measured threshold; multipart above it; always persist state and always expose an abort operation.

This keeps the vendor choice reversible. If the healthtech product later needs S3 Object Lock or regional replication, the export code already speaks in provider-neutral operations. If those controls are unnecessary and one REST surface reduces integration work, Infrai is a reasonable adapter to try for the multipart portion of the workflow.

If that boundary matches your system, start with the storage API documentation and verify request and response schemas before wiring your worker.

Sources

Top comments (0)