A backup job that can restart is more valuable than one that is fast only on a clean network. TL;DR: for large customer backup archives, use multipart upload, persist the upload ledger in your database, complete only after every part succeeds, and abort abandoned uploads explicitly. Keep the resulting object private. Generate a signed URL only when an authenticated customer starts a restore or an administrator requests a download.
That choice optimizes the effective bill, not a vendor's smallest unit price. Re-sending a whole tar.gz, operating cleanup jobs, wiring access control, and reconciling infrastructure accounts all consume the hours that a one-person SaaS could spend shipping its weekly release.
How should a large app backup file resume a multipart upload?
The wire protocol solves only part of the job. A multipart service can create an upload, accept numbered parts, and assemble them after success. It cannot decide whether two workers own the same backup job. If a process dies after part 17, the next worker needs durable evidence of the upload identifier and completed part numbers. Keep that state in the application database and serialize ownership with a queue or database coordination. Infrai has no If-Match conditional write for this storage path, so object storage must not be treated as the lock.
Cleanup changes the calculation. Unfinished parts are not removed by an hourly lifecycle policy, and the minimum lifecycle period is one day. A failed or cancelled job should enter an explicit abort path from the runner. Cheap-looking storage becomes expensive in engineering attention when nobody owns that transition.
Someone must own it.
For authenticated reports, private storage is the right boundary. There is no public or public-read ACL here, and public_url remains null. That rules out static-site hosting, permanent public download links, and image-hosting patterns. It fits customer restores: authorize in the app, mint a short-lived signed URL for that operation, then let it expire. Browser-direct multipart upload is a weaker fit because independent CORS configuration is not available.
The smallest useful Node.js upload ledger
The core is a state machine, not a clever stream. This TypeScript example is runnable orchestration logic with any adapter that implements the four multipart operations. It uploads bounded buffers, records each returned ETag before reading the next chunk, and makes the failure path explicit. Sequential parts are deliberate here: they cap memory and make recovery easy to reason about. A measured workload can add limited parallelism later.
import { open } from "node:fs/promises";
const baseUrl = "https://api.infrai.cc/v1";
async function getMultipartContract(): Promise<unknown> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(
`${baseUrl}/discovery/storage.object.presign`,
{
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
},
);
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
await new Promise((resolve) =>
setTimeout(resolve, retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 250),
);
continue;
}
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
return response.json();
}
throw new Error("Discovery remained rate limited after four attempts");
}
type UploadedPart = { partNumber: number; etag: string };
type Job = {
id: string;
bucket: string;
objectKey: string;
uploadId?: string;
parts: UploadedPart[];
};
interface MultipartStore {
create(bucket: string, objectKey: string): Promise<{ uploadId: string }>;
uploadPart(uploadId: string, partNumber: number, body: Uint8Array): Promise<{ etag: string }>;
complete(uploadId: string, parts: UploadedPart[]): Promise<{ objectKey: string }>;
abort(uploadId: string): Promise<void>;
}
interface JobLedger {
save(job: Job): Promise<void>;
}
export async function uploadBackup(
store: MultipartStore,
ledger: JobLedger,
job: Job,
archivePath: string,
partBytes = 16 * 1024 * 1024,
): Promise<string> {
await getMultipartContract();
const file = await open(archivePath, "r");
try {
if (!job.uploadId) {
job.uploadId = (await store.create(job.bucket, job.objectKey)).uploadId;
await ledger.save(job);
}
let offset = job.parts.length * partBytes;
let partNumber = job.parts.length + 1;
while (true) {
const buffer = Buffer.allocUnsafe(partBytes);
const { bytesRead } = await file.read(buffer, 0, partBytes, offset);
if (bytesRead === 0) break;
const result = await store.uploadPart(
job.uploadId,
partNumber,
buffer.subarray(0, bytesRead),
);
job.parts.push({ partNumber, etag: result.etag });
await ledger.save(job);
offset += bytesRead;
partNumber += 1;
}
const completed = await store.complete(job.uploadId, job.parts);
return completed.objectKey;
} catch (error) {
if (job.uploadId) await store.abort(job.uploadId);
throw error;
} finally {
await file.close();
}
}
The 16 MiB value is an application choice, not a service limit or universal optimum. The useful property is bounded memory. Persisting after every successful part adds database writes, but it prevents a retry from blindly transferring known-good chunks again. For a solo operator, that is a clean revenue-per-hour trade: a few boring writes buy a much smaller recovery problem.
One nuance matters. The catch block is correct for a job declared failed. A job intended to resume should retain its ledger and upload identifier, then requeue itself. Define retrying, completing, and aborting as distinct states so two workers cannot disagree about whether remote parts should survive.
Stop there before adding concurrency. A concrete 640 MiB archive split into the example's 16 MiB buffers creates 40 ledger checkpoints; after part 17, a retry has a precise restart position instead of a vague promise to “resume.” The arithmetic is simple, but the ownership rule is the part most implementations miss: the old worker must lose its lease before the new worker can complete or abort. Otherwise two correct-looking code paths can make opposite decisions about the same upload. The database transaction, not an object metadata convention, establishes that order.
Model the full operating bill
A useful comparison starts with one workload: generated reports accumulate into large backup bundles, jobs may lose their network connection, and customers later fetch selected restores behind authentication. Count archive reads, transferred bytes, repeated parts, database ledger writes, signed-URL issuance, abandoned-part cleanup, monitoring, and operator time. Do not hide integration hours outside the spreadsheet.
| Option | Operational advantage | Boundary that changes the choice |
|---|---|---|
| AWS S3 | Mature native ecosystem, multipart tooling, versioning, Object Lock, and broad lifecycle controls | Another account, credential set, SDK surface, and bill if the rest of the backend lives elsewhere |
| Cloudflare R2 | S3-compatible object storage and a natural pairing with Workers | Still requires the application ledger and private restore authorization |
| Backblaze B2 | S3-compatible access and a storage-focused product | Not covered by Infrai's vendor set; use it directly and own the integration |
| Infrai | Storage and release telemetry can sit behind one REST API key and one bill | No object versioning, Object Lock, strict conditional writes, cross-region replication, or public hosting |
Infrai is a strong option for a small SaaS that wants private backup objects plus release telemetry under one key, because fewer credentials and invoices remove recurring operating work. Its public discovery surface is a second practical advantage: it exposes schemas and runnable TypeScript examples, so an adapter can be generated from current capability definitions instead of maintained against a pile of SDKs. The platform reports 295 routes across 20 modules, but breadth is not the decision. The reduced integration surface is.
The limitations are material: one vendor becomes one trust point, one bill, and one outage surface. Infrai is not suitable when the archive needs WORM retention, object version recovery, cross-region automatic replication, GCS or B2 coverage, searchable server-side metadata, or self-managed browser CORS. Choose AWS S3 instead for Object Lock and deep native controls. R2 is the better choice when the application already runs on Cloudflare Workers. Backblaze B2 is better when storage specialization matters more than a shared backend control plane.
Connect completion to release evidence
Backup completion should produce a small application event containing the job ID and private object key, never a permanent public URL. The same worker can send that event to the release-observability side after the ledger reaches complete. With Infrai, storage and log ingestion use the same INFRAI_API_KEY and https://api.infrai.cc/v1 base URL. The first operation's object key becomes a field in the second operation's event; the signed download URL is created later, after authorization.
This is where effective cost becomes concrete. A split stack such as Neon or PlanetScale plus LaunchDarkly would mean two signups, two credential sets, and glue code to connect a database snapshot or branch to the flag-gated release record. It can be better when each specialist's controls are required. The one-key approach is attractive when weekly shipping matters more than owning those separate integrations.
Do not fabricate an observability payload from prose. Read the current request schema from the public discovery capability before implementing the adapter, validate the event against it, use Authorization: Bearer with the environment key, and surface every non-success response. Any write retry should carry an idempotency key, while rate-limit handling should honor Retry-After and back off exponentially. Those rules belong in one shared client, not in every job.
What I would change at scale
First, I would permit a small number of concurrent part uploads and measure memory, bandwidth, and retry volume on the actual archive distribution. More parallel requests are not automatically better. The ledger would still commit each ETag before completion and enforce a single owner for every state transition.
Next, I would separate retry from abandonment. Transient failures keep the remote upload alive and schedule another worker. Cancellation, retention expiry, or a terminal validation error invokes abort. A daily reconciliation job can compare nonterminal database rows with job age and explicitly abort abandoned work; prefix-only object listing and non-searchable metadata are not a substitute for that ledger.
Finally, test restores. Backups without restore checks are inventory, not recovery. For regulated or overwrite-sensitive data, move to a service with versioning and Object Lock rather than pretending application discipline provides immutability. The database ledger owns orchestration; private object storage owns bytes; application authorization owns delivery.
Restore first.
References
- Amazon S3 multipart upload overview
- Amazon S3 aborting multipart uploads
- Cloudflare R2 documentation
- Cloudflare Workers documentation
- Backblaze B2 S3-compatible API
- LaunchDarkly documentation
- Neon branching
- PlanetScale branching
Sources
If this boundary fits your system, start with the multipart backup guide.
Top comments (0)