Short answer: for a marketplace that accepts private uploads, treat a backup as a verifiable object plus a restore contract. Stream the database dump and archive to private object storage, record an immutable manifest, and issue short-lived presigned downloads only after authorization. The throughput decision is about multipart transfer and retry behavior, not about which dashboard has the nicest button.
Start with the restore contract
A cron job that creates a ZIP and gets a 200 response has not proved that the backup is usable. A restore needs an object key, byte length, checksum, creation time, schema version, and the list of included tenant or marketplace partitions. Keep that manifest beside the archive, and write a separate record in the application database so an operator can find the latest complete set without listing a bucket.
The private-upload path should have a different namespace from backups. For example, uploads/{tenant_id}/{asset_id} and backups/{date}/{run_id}/data.zip make accidental cross-use harder to miss in logs. Encryption at rest is a baseline; authorization still belongs in your service because a presigned URL is a bearer capability.
One rule saves painful nights: a backup is complete only after a read-after-write verification. Check the returned length and checksum, then read a bounded range or the full object from a worker account. A successful upload request alone can hide a truncated stream, a wrong content type, or a manifest that was written before the archive finished.
Verify twice.
How should a Node.js cron upload a database dump zip to object storage?
The implementation can stay boring. Run the database-native dump, pipe it into a compressor, and send the resulting stream with a multipart-capable client. Node.js cron should limit concurrency so four large jobs do not consume the same disk and network budget as customer traffic. If your dump is 18 GB, a 64 MB part size means about 288 parts; that is a planning input for memory, retry windows, and provider limits.
Here is a deliberately generic Python sketch of the worker contract. The storage SDK is an interchangeable adapter; the important pieces are the manifest and the verify step.
from dataclasses import dataclass
from hashlib import sha256
from datetime import datetime, timezone
@dataclass
class BackupManifest:
run_id: str
object_key: str
bytes: int
sha256: str
schema_version: str
created_at: str
def stream_backup(source, bucket, key, put_object, head_object):
digest = sha256()
size = 0
with source as chunks:
for chunk in chunks:
digest.update(chunk)
size += len(chunk)
put_object.write_part(bucket=bucket, key=key, data=chunk)
remote = head_object(bucket=bucket, key=key)
if remote["content_length"] != size:
raise ValueError("backup length mismatch")
return BackupManifest(
run_id="run-2026-09-09T010000Z",
object_key=key,
bytes=size,
sha256=digest.hexdigest(),
schema_version="orders-v7",
created_at=datetime.now(timezone.utc).isoformat(),
)
The production adapter should use bounded retries with jitter, abort incomplete multipart uploads, and emit one metric for each part retry. Do not retry a complete-part commit blindly: make the operation idempotent with a run ID, or you can create two manifests for one schedule. I once traced a restore delay to a cron process that kept the ZIP on an almost-full ephemeral volume; streaming removed that disk-pressure edge case.
That failure was quiet. The scheduler reported success, the database row existed, and the object listing looked normal until a restore worker compared the manifest's byte count. The fix was procedural as much as technical: the worker now closes the dump stream, finalizes the multipart upload, verifies metadata from a separate credential, and writes the “complete” state last. If any stage is interrupted, the run remains explicitly incomplete and the next cron cycle gets a fresh run ID. This ordering also gives on-call staff a useful answer at 02:00: there is either a complete object with a checksum, or there is no backup to restore.
No guesswork.
What fails first at marketplace scale?
Large-file throughput is usually constrained by the slowest shared resource. Database dump CPU, compressor CPU, egress bandwidth, object-store part limits, and the worker's file descriptors all compete. Measure them separately. A single end-to-end timer tells you only that the job was slow.
The failure modes are predictable:
| Failure | Detection | Design response |
|---|---|---|
| Dump succeeds, upload is partial | byte/checksum mismatch | abort the multipart upload and mark the run incomplete |
| Presigned link leaks | unexpected download principal or volume | short expiry, authorization check, audit event |
| Retention deletes the only good copy | restore drill finds a missing generation | keep overlapping generations and test lifecycle rules |
| Tenant data crosses a boundary | manifest has an unexpected tenant ID | partition keys, policy tests, and a deny-by-default reader |
Lifecycle rules are useful for stale temporary parts and old generations, but deletion is a data-governance action. AWS documents lifecycle transitions and expiration as rules evaluated by the service, so your runbook should allow for timing rather than assuming a deletion happens at an exact minute. Firebase Cloud Storage documentation makes a similar security point in a different product context: access control is separate from object retention.
Keep an offline restore fixture under version control: a tiny database dump, one representative upload, and its manifest. It catches schema drift and ZIP layout changes before a six-hour restore drill does. Your mileage may vary on compression ratio; it depends heavily on whether uploads are already compressed media.
Comparing compatible backends without choosing a winner
S3-compatible services, self-hosted object stores, and hyperscale buckets can all implement private objects and presigned requests, but their operational edges differ. AWS S3 has mature lifecycle and multipart controls; Cloudflare R2 removes some egress charges but changes the cost profile of reads; MinIO gives teams direct control of hardware and upgrades. Those are constraints, not a ranking.
Ask each candidate for documented limits on part count, minimum part size, checksum algorithms, lifecycle timing, replication, and audit-log retention. Then run the same restore fixture through each endpoint. A backend that wins a synthetic upload benchmark can still lose when a presigned URL must be denied after a user is suspended, or when a region cannot meet your recovery point objective.
Benchmarks are clues, not contracts.
The catch is that a self-hosted option is not suitable when your team cannot operate disks, replication, and key rotation. Stick with a managed bucket when the operational burden matters more than placement control; choose self-hosting when data residency or predictable hardware access is the hard requirement. Either way, keep the adapter narrow so changing the backend does not rewrite backup semantics.
Roll out with a reversible restore drill
Ship the manifest schema first, then dual-write one scheduled run while the existing backup remains authoritative. Compare checksums and restore duration for at least seven runs, including a deliberately interrupted upload. Only after the new path passes a full restore should the retention policy remove older copies.
During the drill, revoke the operator's session, expire a presigned URL, and attempt a cross-tenant key. These are cheap tests with expensive production consequences. Record recovery point and recovery time objectives next to the run, not in a separate wiki page that will drift.
The final decision is small: select the backend that meets throughput, isolation, retention, and restore-time targets with an operating model your team can sustain. The archive format is an implementation detail; the contract is what keeps app data recoverable.
Top comments (0)