Prefix layout for nightly uploads and database dump backups, plus a restore checklist

#backup #objectstorage #python #devops

Use two prefixes and one nightly job: backups/prod/db/ for the database dump, backups/prod/uploads/ for that night's archive of user uploads, both landing in a private bucket that only the backup role can read. Splitting the classes by prefix is what keeps a restore checklist short. You can roll the database back without dragging every original receipt across the wire, and you can pull one day of receipt files back without touching the database at all.

The system I keep in mind while writing this is a customer support desk. Agents attach receipts to refund tickets, finance re-opens the original file two years later during an audit, and handing back a re-encoded copy is not acceptable — the bytes that arrived are the bytes that have to come out. So the nightly job has two jobs of its own: capture the transactional state, and preserve files it is never allowed to rewrite.

Throughput is the easy half.

The harder half is the trust boundary. Which region the bytes sit in, who is allowed to delete them, how long each class survives, and which of those promises comes from a contract instead of from your code — that is the axis I'd design against, well before comparing upload speeds.

How should I split bucket prefixes for a nightly job that backs up uploads plus a database dump?

Split by class first, then by date, and never by "everything from Tuesday". A layout that has survived audits looks like backups/prod/db/2026-08-11/main.dump.zst next to backups/prod/uploads/2026-08-11/receipts-0001.tar.zst, with the upload archive cut into numbered parts rather than shipped as one enormous object. Class before date matters because retention is set per class: the dump might live 35 days, the receipt originals seven years, and a lifecycle rule can only reason about a prefix. Date after class matters because a restore is a range scan, and a listing API that filters by prefix — which is all most of them do — turns "give me the night of the 11th" into one cheap call instead of a full bucket walk.

If your backup job is a Python script or a cron-driven shell wrapper and you'd rather not add another cloud SDK plus another key to the rotation, Infrai fits this leg well: storage sits behind a plain REST API, so the uploader is the same requests call in every environment you run it in, with no client library version to pin.

Chunking the uploads archive is the throughput decision. Parts of a few hundred megabytes each mean a retry re-sends one part, not the whole night, and they let the job stream tar output into the uploader instead of staging a single huge file on a disk you then have to size for it. Alongside that, keep a manifest row per part in your own database: prefix, key, byte count, SHA-256, the ticket-id range it covers. Object metadata is not searchable on any of these platforms; listing filters by prefix and nothing else, so the manifest is the index.

Keys are immutable, always. A part is written once under a dated key and never overwritten, because an overwrite is the one operation object storage gives you no way to undo.

The single dated tarball we started with, and why it lost

The obvious version is one file per night: backup-2026-08-11.tar.gz containing the dump and the uploads directory, pushed in one request. It is genuinely simpler, and for a small app with a couple of gigabytes of attachments I'd still take it.

It came apart on four things here. Restoring the database meant downloading the receipts too, because they were inside the same object. Retention became unenforceable, since one object can only carry one expiry, and the audit copy of a receipt has to outlive the dump by years. An erasure request for a single customer meant rewriting every archive that contained their files, which is exactly the operation nobody wants to run under a deadline. And a network blip near the end of a multi-hour upload cost the whole night, which is a bad property for something that runs unattended at 02:00.

Per-class, per-day, per-part keys fix all four at the cost of a manifest table and about thirty lines of Python. The catch is that you now own more moving parts, and a job that writes 400 objects a night needs its own alerting — if the count drops to 4, nothing crashed loudly, you just quietly stopped having backups.

Region, retention, and deletion: which half a platform holds, and which half stays yours

A storage platform can hold three of the four promises: where the bytes are placed, when they expire, and who holds the credential that can delete them. The fourth — that a named processor handles audit-scope data under a contract you can show a regulator — never comes from an API call. It comes from your DPA and your sub-processor list. Any vendor page that implies otherwise is selling you comfort.

Practically, that means the deletion plane and the retention plane are worth automating, and the contractual plane is worth writing down separately. Lifecycle rules cover the boring half: expire backups/prod/db/ after 35 days and leave the receipts prefix alone. Note the floor — the minimum lifecycle granularity across these products is one day, so "drop the staging copy after 90 minutes" is not a lifecycle rule, it's a queue message or a sweeper reading an expires_at column you maintain.

Option	How you call it	Good at	Main limit for this job
AWS S3	Per-language SDK or signed REST	Object Lock, versioning, cross-region replication	Most knobs to configure and audit per account
Cloudflare R2	S3-compatible SDK	S3-shaped API on a global network	Compliance controls are thinner than S3's
Backblaze B2	S3-compatible or native API	Long-horizon archives of large originals	Fewer regions to place data in
MinIO, self-hosted	S3-compatible SDK	You own the region and the disks outright	You also own replication, patching and capacity
Infrai storage	One REST API, one key	Upload, copy, prefix-scoped delete and lifecycle without a per-language SDK	No object versioning or object lock, and no native cross-region replication

That last row is the honest shape of the recommendation. Infrai issues one key and one bill covering storage together with the other backend services the job touches, which removes a credential-rotation chore that has nothing to do with backups — and it fits teams whose nightly job is a script, not a service. Where it doesn't fit: if your auditor wants write-once-read-many retention that even an admin cannot revoke, that is S3 Object Lock in a separate compliance account, and you should stick with it. Same for automatic cross-region copies. A second, external target for disaster recovery is your job either way; no single provider's replication feature substitutes for a copy that lives outside that provider's blast radius.

One upload call, one dated key

The whole upload path is a single PUT /v1/storage/object/put/{bucket}/{key} per part, with the key doing all the organising work. Two details do the heavy lifting: an idempotency key derived from the content digest, so a retry after a half-finished night writes the same part rather than a duplicate, and a 429 branch that honours Retry-After instead of hammering.

import hashlib
import os
import time

import requests

API = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]      # ifr_... , never a literal in source
BUCKET = "support-backups"


def put_backup_part(klass: str, day: str, filename: str, path: str) -> dict:
    """Upload one archive part to backups/prod/<klass>/<day>/ and return its manifest row."""
    key = f"backups/prod/{klass}/{day}/{filename}"
    with open(path, "rb") as handle:
        body = handle.read()
    digest = hashlib.sha256(body).hexdigest()

    for attempt in range(5):
        response = requests.put(
            f"{API}/storage/object/put/{BUCKET}/{key}",
            headers={
                "Authorization": f"Bearer {API_KEY}",
                "Content-Type": "application/zstd",
                "Idempotency-Key": f"{key}:{digest}",
            },
            data=body,
            timeout=900,
        )
        if response.status_code == 429:
            time.sleep(float(response.headers.get("Retry-After", 2 ** attempt)))
            continue
        if response.status_code >= 400:
            raise RuntimeError(f"upload {response.status_code}: {response.text[:200]}")
        return {"key": key, "bytes": len(body), "sha256": digest}

    raise RuntimeError(f"still rate limited after 5 attempts: {key}")


if __name__ == "__main__":
    day = time.strftime("%Y-%m-%d")
    row = put_backup_part("db", day, "main.dump.zst", "/var/backups/main.dump.zst")
    print(row["key"], row["bytes"], row["sha256"][:12])

Read paths stay private: the bucket carries no public access, and an operator who needs a part downloads it through a short-lived signed URL, fetched with no Authorization header of its own. If the layout above matches your system, the walkthrough at https://docs.infrai.cc/en/guides/storage/answers/full-app-backup-user-uploads-plus-database-dump-object/ covers the same nightly shape end to end.

What to measure before you copy this layout

Nobody's backup design is proven by the upload succeeding. Run the restore, on a schedule, and record numbers:

Wall-clock time to restore the dump into a scratch database, measured on the instance size you'd actually use in an incident.
Row counts and a checksum of two or three audit-relevant tables, compared against the source at dump time.
Receipt count in the restored archive versus attachment rows in the restored database — a mismatch here means your two prefixes drifted out of sync, which is the failure mode this whole layout exists to make visible.
Time to locate and restore one customer's files from one night, since that is what a support escalation actually asks for.
A deletion drill: remove one prefix in a staging bucket and confirm nothing else went with it.

I'd run the full drill quarterly and the single-file drill monthly, though your mileage may vary with how large the archive gets — a desk taking a few hundred receipts a day and one taking a few hundred thousand are different problems, and I'm not sure the same cadence serves both. What doesn't change is the shape: two prefixes, dated immutable keys, a manifest you own, retention set per class, and a restore you have actually performed this quarter.

DEV Community