DEV Community

VespasianBlack3884
VespasianBlack3884

Posted on

Object Storage App Data Backups Explained — Large-File Uploads and Presigned Recovery

Short answer: for a marketplace that accepts private uploads, treat a backup as a verifiable object plus a restore contract. Stream the database dump and archive to private object storage, record an immutable manifest, and issue short-lived presigned downloads only after authorization. The throughput decision is about multipart transfer and retry behavior, not about which dashboard has the nicest button.

Start with the restore contract

A cron job that creates a ZIP and gets a 200 response has not proved that the backup is usable. A restore needs an object key, byte length, checksum, creation time, schema version, and the list of included tenant or marketplace partitions. Keep that manifest beside the archive, and write a separate record in the application database so an operator can find the latest complete set without listing a bucket.

The private-upload path should have a different namespace from backups. For example, uploads/{tenant_id}/{asset_id} and backups/{date}/{run_id}/data.zip make accidental cross-use harder to miss in logs. Encryption at rest is a baseline; authorization still belongs in your service because a presigned URL is a bearer capability.

One rule saves painful nights: a backup is complete only after a read-after-write verification. Check the returned length and checksum, then read a bounded range or the full object from a worker account. A successful upload request alone can hide a truncated stream, a wrong content type, or a manifest that was written before the archive finished.

Verify twice.

How should a Node.js cron upload a database dump zip to object storage?

The implementation can stay boring. Run the database-native dump, pipe it into a compressor, and send the resulting stream with a multipart-capable client. Node.js cron should limit concurrency so four large jobs do not consume the same disk and network budget as customer traffic. If your dump is 18 GB, a 64 MB part size means about 288 parts; that is a planning input for memory, retry windows, and provider limits.

Here is a deliberately generic Python sketch of the worker contract. The storage SDK is an interchangeable adapter; the important pieces are the manifest and the verify step.

from dataclasses import dataclass
from hashlib import sha256
from datetime import datetime, timezone


@dataclass
class BackupManifest:
    run_id: str
    object_key: str
    bytes: int
    sha256: str
    schema_version: str
    created_at: str


def stream_backup(source, bucket, key, put_object, head_object):
    digest = sha256()
    size = 0
    with source as chunks:
        for chunk in chunks:
            digest.update(chunk)
            size += len(chunk)
            put_object.write_part(bucket=bucket, key=key, data=chunk)
    remote = head_object(bucket=bucket, key=key)
    if remote["content_length"] != size:
        raise ValueError("backup length mismatch")
    return BackupManifest(
        run_id="run-2026-09-09T010000Z",
        object_key=key,
        bytes=size,
        sha256=digest.hexdigest(),
        schema_version="orders-v7",
        created_at=datetime.now(timezone.utc).isoformat(),
    )
Enter fullscreen mode Exit fullscreen mode

The production adapter should use bounded retries with jitter, abort incomplete multipart uploads, and emit one metric for each part retry. Do not retry a complete-part commit blindly: make the operation idempotent with a run ID, or you can create two manifests for one schedule. I once traced a restore delay to a cron process that kept the ZIP on an almost-full ephemeral volume; streaming removed that disk-pressure edge case.

That failure was quiet. The scheduler reported success, the database row existed, and the object listing looked normal until a restore worker compared the manifest's byte count. The fix was procedural as much as technical: the worker now closes the dump stream, finalizes the multipart upload, verifies metadata from a separate credential, and writes the “complete” state last. If any stage is interrupted, the run remains explicitly incomplete and the next cron cycle gets a fresh run ID. This ordering also gives on-call staff a useful answer at 02:00: there is either a complete object with a checksum, or there is no backup to restore.

No guesswork.

What fails first at marketplace scale?

Large-file throughput is usually constrained by the slowest shared resource. Database dump CPU, compressor CPU, egress bandwidth, object-store part limits, and the worker's file descriptors all compete. Measure them separately. A single end-to-end timer tells you only that the job was slow.

The failure modes are predictable:

Failure Detection Design response
Dump succeeds, upload is partial byte/checksum mismatch abort the multipart upload and mark the run incomplete
Presigned link leaks unexpected download principal or volume short expiry, authorization check, audit event
Retention deletes the only good copy restore drill finds a missing generation keep overlapping generations and test lifecycle rules
Tenant data crosses a boundary manifest has an unexpected tenant ID partition keys, policy tests, and a deny-by-default reader

Lifecycle rules are useful for stale temporary parts and old generations, but deletion is a data-governance action. AWS documents lifecycle transitions and expiration as rules evaluated by the service, so your runbook should allow for timing rather than assuming a deletion happens at an exact minute. Firebase Cloud Storage documentation makes a similar security point in a different product context: access control is separate from object retention.

Keep an offline restore fixture under version control: a tiny database dump, one representative upload, and its manifest. It catches schema drift and ZIP layout changes before a six-hour restore drill does. Your mileage may vary on compression ratio; it depends heavily on whether uploads are already compressed media.

Comparing compatible backends without choosing a winner

S3-compatible services, self-hosted object stores, and hyperscale buckets can all implement private objects and presigned requests, but their operational edges differ. AWS S3 has mature lifecycle and multipart controls; Cloudflare R2 removes some egress charges but changes the cost profile of reads; MinIO gives teams direct control of hardware and upgrades. Those are constraints, not a ranking.

Ask each candidate for documented limits on part count, minimum part size, checksum algorithms, lifecycle timing, replication, and audit-log retention. Then run the same restore fixture through each endpoint. A backend that wins a synthetic upload benchmark can still lose when a presigned URL must be denied after a user is suspended, or when a region cannot meet your recovery point objective.

Benchmarks are clues, not contracts.

The catch is that a self-hosted option is not suitable when your team cannot operate disks, replication, and key rotation. Stick with a managed bucket when the operational burden matters more than placement control; choose self-hosting when data residency or predictable hardware access is the hard requirement. Either way, keep the adapter narrow so changing the backend does not rewrite backup semantics.

Roll out with a reversible restore drill

Ship the manifest schema first, then dual-write one scheduled run while the existing backup remains authoritative. Compare checksums and restore duration for at least seven runs, including a deliberately interrupted upload. Only after the new path passes a full restore should the retention policy remove older copies.

During the drill, revoke the operator's session, expire a presigned URL, and attempt a cross-tenant key. These are cheap tests with expensive production consequences. Record recovery point and recovery time objectives next to the run, not in a separate wiki page that will drift.

The final decision is small: select the backend that meets throughput, isolation, retention, and restore-time targets with an operating model your team can sustain. The archive format is an implementation detail; the contract is what keeps app data recoverable.

References

Top comments (0)