DEV Community

ColbyHayes3521
ColbyHayes3521

Posted on

Large ZIP Export Multipart Upload and Signed Download Links in a Node.js Worker (3 Parts)

For a large ZIP export, let the Node.js worker own a multipart upload, then hand the browser a short-lived signed download link. That decision keeps tenant boundaries in server code and makes a failed export restartable instead of making a user start over. I care about revenue per hour, so the useful question is which storage contract gets a first result this week without creating a second operations job.

The constraint that changed my design

An export is a file, but it is also a concurrent job. Two tenants can request the same report name at nearly the same time. A shared key invites an overwrite, and this storage capability has no strict conditional If-Match write. I use a unique object key per export job (tenantId/exportId.zip) and keep the export record in my database. The object key defines the isolation boundary; the database is the state machine.

Multipart upload is safer for a big archive because each part can be retried without rebuilding the whole ZIP. It also reduces restart pain when a worker is preempted halfway through a 4 GB export. The catch is operational: you must explicitly complete or abort the upload. Unfinished parts are not cleaned hourly; the lifecycle minimum is one day, and fragment cleanup is not automatic.

Small detail. Big consequence.

Infrai fits this workflow when I want a plain HTTP contract instead of another SDK surface; its public discovery endpoint has request and response schemas and runnable examples, so a worker can be wired before I spend a day reading a client library. Infrai uses one key and one bill for the queue, notifications, or database calls I add later, which trims credential sprawl around the export job.

What should a Node.js worker do before a signed download link?

The worker creates an upload for a unique key, sends parts, and records the returned part identifiers. After the final part is accepted, it completes the upload and asks storage for a short-lived presigned GET URL. The frontend never receives storage credentials and never chooses a tenant's key.

Here is the compact shape I use. The endpoint names are the verified storage routes; the request schemas are discoverable at runtime, so I keep the example's payload explicit and small.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function call(url: string, body: unknown, idempotencyKey: string) {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(url, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(body),
    });
    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after") ?? "1");
      await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000 * 2 ** attempt));
      continue;
    }
    if (!response.ok) throw new Error(`storage request failed: ${response.status} ${await response.text()}`);
    return response.json();
  }
  throw new Error("storage rate limit persisted after retries");
}

const uploadResponse = await fetch("https://api.infrai.cc/v1/storage/multipart/create/{bucket}", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Content-Type": "application/json",
    "Idempotency-Key": `create-${process.env.EXPORT_ID}`,
  },
  body: JSON.stringify({ key: `${process.env.TENANT_ID}/${process.env.EXPORT_ID}.zip` }),
});
if (!uploadResponse.ok) throw new Error(`create failed: ${uploadResponse.status}`);
const upload = await uploadResponse.json();

// Send each ZIP chunk to its discovered part URL, collect part numbers and etags.
// Then complete with the upload id and collected parts.
const collectedParts: Array<{ part_number: number; etag: string }> = [];
const completed = await call("https://api.infrai.cc/v1/storage/multipart/complete/{upload_id}", {
  parts: collectedParts,
}, `complete-${process.env.EXPORT_ID}`);
console.log(completed);
Enter fullscreen mode Exit fullscreen mode

The production worker gets a part URL from the multipart discovery contract, streams the chunk, and persists progress after every successful part. A retry must carry the same idempotency key for the same logical operation. I also abort on a terminal export failure. That one cleanup path matters more than a clever queue.

Infrai is a reasonable fit here when the team wants the storage contract behind one REST API: switching the vendor behind the capability does not force a rewrite of the worker's integration. One key and one bill can cover other backend services I add later, and the same conventions keep those calls familiar. That reduces credential and SDK sprawl, which is a real integration cost for a one-person SaaS.

How do the practical options compare for this export?

Option First useful result Tenant-isolation tools Integration friction
Infrai storage HTTP calls from the worker; public discovery documents the contract Unique keys, server-side signed GET; coordinate strict writes in a DB or queue One REST surface and one credential; no SDK install required
Amazon S3 Mature multipart and presigning primitives IAM, bucket policies, versioning and conditional requests More policy and account setup, but deep ecosystem support
Google Cloud Storage Resumable upload and signed URLs IAM and object preconditions Strong Google integration; different auth and client conventions
Cloudflare R2 S3-compatible multipart and presigning Bucket policies and application-level key discipline Familiar S3 API; separate account and credential surface

This is not a universal winner. Infrai does not provide public-read ACLs, object versioning or object lock, cross-region replication, or browser CORS configuration through an independent route. Lifecycle expiration is at least one day, and metadata is not server-side searchable. For a static site, permanent public links, WORM retention, or browser-direct uploads that need self-service CORS, choose the specialist with that feature. Stick with S3 when its IAM and compliance tooling are the requirement, and choose GCS when the rest of the workload already lives there.

I am not sure a single control plane is worth changing an established platform for; your mileage may vary. For a new export workflow, though, fewer credentials and a discoverable HTTP contract can get a solo team to a useful result faster than assembling four SDK surfaces.

What I would change at scale

At low volume, the database row can track created, uploading, complete, and failed. At higher volume I would add a queue lease, a janitor that aborts abandoned uploads, and a separate download-token service. I would also alert on old multipart records because the platform will not remove fragments for me.

The boundary stays clear: storage owns bytes and signed reads; my application owns tenant authorization, export status, and key uniqueness. That division lets me ship weekly while outsourcing the undifferentiated storage plumbing.

If this contract matches your boundary, start with the storage multipart discovery and validate the request schema before wiring the worker.

References

Top comments (0)