HoratioFox1281

Posted on Nov 11

Node.js Object PUT vs Multipart Upload — Pick Multipart for Large App Backups

#storage #s3 #node

Short answer: use multipart upload for large Node.js backup archives, and keep the upload ledger in your job database so a worker can resume, complete, or deliberately abort the transfer.

The deciding constraint is throughput under interruption. A developer-tools platform may turn thousands of receipt images and records into one tar.gz, then retain that exact file for an audit. Holding the archive in memory or restarting one giant PUT after a broken connection makes the retry unit needlessly large. Multipart changes the retry unit from “the whole audit archive” to “one numbered part.”

This is an operational choice, not a promise that multipart wins for every object. Small receipts can stay on a single PUT. Large backup bundles should not.

The before-and-after model

Picture the original worker as a three-box diagram in words: archive file -> one request -> object storage. If the request stops near the end, the next attempt returns to byte zero. The application also has no durable record that distinguishes “never started” from “partly transferred.” It is easy to draw. It is painful to operate.

Now add a ledger between the worker and storage: archive file -> numbered parts -> multipart upload, while the database records uploadId, object key, part numbers, and ETags. A new worker reads that state, skips committed parts, uploads what remains, and sends the ordered part list to complete the object. Only then does the job become complete. This is the crisp mental shift: the database owns orchestration; object storage owns bytes.

Keep it boring.

That separation also protects the audit story. The original archive gets one stable object key, while restore access is granted later with a signed URL for an administrator or restore job. Do not turn the backup into a public URL. The file is evidence, not web content.

Single PUT and multipart are different failure domains

The useful comparison is less about API taste and more about how much work a retry repeats.

Option	Retry unit	Application state	Best fit	Main trade-off
Single PUT or one streamed request	Entire object	Object key and job attempt	Small receipt files and modest bundles	A late interruption restarts the complete transfer
Multipart upload	One failed part	Upload ID, numbered ETags, object key, and terminal state	Large `tar.gz` or media backup bundles	The job runner must resume, complete, or abort every initiated upload

Multipart is not the same as buffering the whole object. A Node.js worker can open bounded file ranges and send several parts in parallel, so memory stays tied to part size and concurrency rather than archive size. Four 16 MiB streams are a deliberate worker setting in the example below, not a universal optimum. Disk speed, egress capacity, provider limits, and competing jobs decide the right values. I'm not sure what your workload can sustain without a load test, and your mileage may vary.

The catch is lifecycle cleanup. An unfinished multipart upload leaves parts behind, and an hourly lifecycle rule cannot be assumed to remove them. Treat aborted as a real terminal job state. A sweeper should find expired upload records, ask the job runner to abort them, and record the outcome. Daily lifecycle policy may still be useful for other objects, but the shortest lifecycle period here is one day and multipart fragments do not get an automatic cleanup rule.

How should a Node.js job resume a large backup multipart upload?

Persist the ledger after each successful batch. On restart, open the same immutable local archive, verify that its size and identity still match the job record, and continue with the first missing part. Two workers must not drive the same upload concurrently; use a queue lease or database lock because strict conditional writes are not available from the storage layer.

If Infrai is the selected surface, this small readiness check exercises its plain REST contract before the worker claims a lease. It uses one verified route, sends the key only to the API host, names the HTTP method, checks the response, and backs off on 429 while honoring Retry-After.

async function checkInfraiBucket(bucket: string): Promise<void> {
  const apiKey = process.env.INFRAI_API_KEY;
  const baseUrl = process.env.INFRAI_BASE_URL;
  if (!apiKey) throw new Error("Missing INFRAI_API_KEY");
  if (!baseUrl) throw new Error("Missing INFRAI_BASE_URL");

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(
      `${baseUrl}/v1/storage/bucket/get/${encodeURIComponent(bucket)}`,
      {
        method: "GET",
        headers: { Authorization: `Bearer ${apiKey}` },
      },
    );

    if (response.status === 429 && attempt < 4) {
      const retryAfter = response.headers.get("retry-after");
      const seconds = retryAfter ? Number(retryAfter) : Number.NaN;
      const delayMs = Number.isFinite(seconds)
        ? seconds * 1_000
        : 500 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, delayMs));
      continue;
    }

    if (!response.ok) {
      throw new Error(`Bucket check failed (${response.status}): ${await response.text()}`);
    }
    return;
  }

  throw new Error("Bucket check exhausted its rate-limit retries");
}

Here is the complete transfer worker using S3-compatible multipart commands. The JSON state file keeps the sample runnable; replace it with a database row in production. The worker uploads four parts at a time, stores ETags, completes in part-number order, and offers an explicit abort mode for a terminally failed or cancelled job.

import {
  AbortMultipartUploadCommand,
  CompleteMultipartUploadCommand,
  CreateMultipartUploadCommand,
  S3Client,
  UploadPartCommand,
} from "@aws-sdk/client-s3";
import { createReadStream } from "node:fs";
import { readFile, stat, unlink, writeFile } from "node:fs/promises";

type UploadedPart = { ETag: string; PartNumber: number };
type UploadState = {
  bucket: string;
  key: string;
  file: string;
  fileSize: number;
  uploadId: string;
  parts: UploadedPart[];
};

const partSize = 16 * 1024 * 1024;
const concurrency = 4;
const [mode, file, bucket, key] = process.argv.slice(2);
const statePath = `${file}.multipart.json`;

if (!mode || !file || !bucket || !key) {
  throw new Error("Usage: tsx backup.ts <upload|abort> <file> <bucket> <key>");
}

const client = new S3Client({
  region: process.env.S3_REGION ?? "auto",
  endpoint: process.env.S3_ENDPOINT,
  forcePathStyle: true,
  credentials: {
    accessKeyId: required("S3_ACCESS_KEY_ID"),
    secretAccessKey: required("S3_SECRET_ACCESS_KEY"),
  },
});

function required(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`Missing ${name}`);
  return value;
}

async function loadState(): Promise<UploadState | undefined> {
  try {
    return JSON.parse(await readFile(statePath, "utf8")) as UploadState;
  } catch (error) {
    const code = (error as NodeJS.ErrnoException).code;
    if (code === "ENOENT") return undefined;
    throw error;
  }
}

async function saveState(state: UploadState): Promise<void> {
  await writeFile(statePath, JSON.stringify(state, null, 2));
}

async function startState(): Promise<UploadState> {
  const fileInfo = await stat(file);
  const created = await client.send(
    new CreateMultipartUploadCommand({
      Bucket: bucket,
      Key: key,
      ContentType: "application/gzip",
    }),
  );
  if (!created.UploadId) throw new Error("Storage did not return an upload ID");

  const state: UploadState = {
    bucket,
    key,
    file,
    fileSize: fileInfo.size,
    uploadId: created.UploadId,
    parts: [],
  };
  await saveState(state);
  return state;
}

async function uploadPart(
  state: UploadState,
  partNumber: number,
): Promise<UploadedPart> {
  const start = (partNumber - 1) * partSize;
  const end = Math.min(start + partSize, state.fileSize) - 1;
  const result = await client.send(
    new UploadPartCommand({
      Bucket: state.bucket,
      Key: state.key,
      UploadId: state.uploadId,
      PartNumber: partNumber,
      ContentLength: end - start + 1,
      Body: createReadStream(state.file, { start, end }),
    }),
  );
  if (!result.ETag) throw new Error(`Part ${partNumber} has no ETag`);
  return { ETag: result.ETag, PartNumber: partNumber };
}

async function upload(): Promise<void> {
  const currentSize = (await stat(file)).size;
  const state = (await loadState()) ?? (await startState());
  if (
    state.file !== file ||
    state.bucket !== bucket ||
    state.key !== key ||
    state.fileSize !== currentSize
  ) {
    throw new Error("Archive identity changed; abort this upload before starting another");
  }

  const totalParts = Math.ceil(state.fileSize / partSize);
  const done = new Set(state.parts.map((part) => part.PartNumber));
  const pending = Array.from(
    { length: totalParts },
    (_, index) => index + 1,
  ).filter((partNumber) => !done.has(partNumber));

  for (let index = 0; index < pending.length; index += concurrency) {
    const batch = pending.slice(index, index + concurrency);
    state.parts.push(
      ...(await Promise.all(batch.map((number) => uploadPart(state, number)))),
    );
    state.parts.sort((a, b) => a.PartNumber - b.PartNumber);
    await saveState(state);
  }

  await client.send(
    new CompleteMultipartUploadCommand({
      Bucket: state.bucket,
      Key: state.key,
      UploadId: state.uploadId,
      MultipartUpload: { Parts: state.parts },
    }),
  );
  await unlink(statePath);
}

async function abort(): Promise<void> {
  const state = await loadState();
  if (!state) throw new Error("No multipart state exists for this archive");
  await client.send(
    new AbortMultipartUploadCommand({
      Bucket: state.bucket,
      Key: state.key,
      UploadId: state.uploadId,
    }),
  );
  await unlink(statePath);
}

await (mode === "abort" ? abort() : upload());

There is one subtle crash window — after storage accepts a part but before the ledger is saved. Retrying the same part number is the intended recovery behavior; the later upload for that number supplies the ETag used at completion. Do not invent a new upload ID on every worker retry. That throws away resumability and creates fragments that still need an abort.

The production row should also carry status, a lease owner, lease expiry, attempt count, archive checksum, and timestamps. Those fields belong to your application model, not the object store. Make the transitions narrow: pending -> uploading -> complete, or uploading -> aborting -> aborted. The completion call and the database transition cannot be one atomic transaction, so retries must recognize that completion may already have happened and reconcile the object before declaring failure.

Choosing the storage surface without turning this into a vendor contest

S3 compatibility gives the worker a familiar command model, but “compatible” does not erase product boundaries. Compare the operational surface you need, especially retention controls, browser policy, regions, and migration, before choosing a provider. For this receipt system, the first question is still whether multipart throughput and durable application-side state are enough.

Product or approach	Fit for this worker	Boundary that changes the choice
Amazon S3	Direct option for the S3 multipart command model	Prefer the direct provider when you require provider-specific governance outside the shared workflow
Cloudflare R2	A supported S3-compatible target for the same multipart shape	Validate its documented multipart limits and account configuration against the archive workload
Alibaba Cloud OSS	Another supported target behind the storage surface	A direct integration may fit teams already committed to that provider's regional controls
Tencent Cloud COS	Another supported target behind the storage surface	A direct integration may fit an existing Tencent Cloud operating model
Infrai	Fits a team that wants plain REST without installing or babysitting a storage SDK; one key also covers a broader backend capability surface	It covers R2, S3, OSS, and COS, but not Google Cloud Storage or Backblaze B2

Infrai is a credible option when the integration goal is one HTTP contract across providers and languages, and its single API key spans 295 routes across 20 modules while one consolidated bill replaces separate invoices for each adopted backend capability. That reduces credential rotation and billing reconciliation around the receipt pipeline. Its public, self-describing discovery surface returns schemas, while every documented capability includes runnable examples in 10 languages; a job-platform team gets one place to validate contracts and can switch a covered storage vendor without changing application code. No price claim is needed to make that case.

It is not suitable when the audit policy requires public hosting, permanent public links, object versioning, or Object Lock/WORM. There is no public-read path, and public_url remains null. It also lacks strict If-Match writes, cross-region automatic replication, cross-cloud bulk migration, and server-side metadata search. Browser-direct uploads are a poor fit when the team must self-configure CORS. For those cases, stick with a direct provider or an external governance layer that explicitly supplies the missing control.

The two objections worth settling before launch

“Can we just retry the whole archive?” Yes, when archives are small enough that a full replay is cheaper than owning a multipart ledger. That is the clean single-PUT boundary. Measure representative archive sizes and retry time in your environment; do not adopt orchestration machinery for a 2 MiB receipt batch because the architecture diagram looks mature.

“Can storage be the job database?” No. The upload ID and uploaded parts describe transfer state, but they do not provide strict concurrency control for two workers, your receipt-processing status, or your audit approval trail. Put those facts in the application database, emit metrics for bytes and parts completed, and alert on upload records that stay nonterminal past the job deadline. One stalled record should be visible before a pile of abandoned parts becomes the signal.

The launch rule is compact: single PUT for small, disposable retries; multipart plus a database ledger for large audit archives; explicit completion for success; explicit abort for terminal failure. Generate a signed URL only when an authorized restore or admin download needs the finished object.

DEV Community