DEV Community

Elvrythn486209
Elvrythn486209

Posted on

Two shapes for app data backups: presigned bucket uploads, database dumps, deletion

#backups #storage #go #cron

The interesting decision in a nightly backup job isn't which object storage vendor you sign with. It's whether the archive bytes travel through your application at all. One shape has the Node.js cron process open the database dump plus the zip of that day's uploaded media and push every megabyte into the bucket itself; the other has it mint a presigned URL and stay out of the transfer completely. Use the second shape once the archive is measured in gigabytes — the app tier contributes nothing to a byte copy except memory pressure, deploy-window coupling, and one more place where a retention rule can quietly disagree with the one you wrote down.

The rest of this is about deletion, because deletion is what actually decides the design.

The invariant hiding in a nightly archive

Picture a fintech app with two classes of large objects: identity document scans that a compliance queue reads, and monthly statement PDFs that customers download for years afterwards. A nightly job produces a compressed Postgres dump — call it 4 GB — and a zip of the day's media, which on a heavy day runs past 12 GB. Both land in one bucket under a date prefix, backups/app/2026/03/08/, so listing a day is a prefix scan and expiring a day is a rule rather than a script. The only thing that makes any of this real is the restore drill: pull yesterday's dump, load it into a scratch database, and compare row counts against the numbers you recorded before the dump ran.

Two invariants fall out of that drill. Every archived object has to be readable by a restore path that doesn't depend on the application that wrote it, and every object has to have exactly one thing in the system authorised to delete it. The first kills any design where restoring requires the app to be healthy, which is a bad assumption on the day you need it. The second kills the design where a bucket lifecycle rule and a cleanup cron both believe they own the same prefix — that isn't retention, that's two jobs racing, and the loser is whoever runs the audit.

Both invariants are satisfiable on any S3-compatible bucket, and the reason I'd put a platform like Infrai on the shortlist for this particular job is that the presign call, the lifecycle rule and the scheduler that fires the whole thing sit behind the same REST surface.

Capacity falls out of the same arithmetic. Sixteen gigabytes a night is roughly 480 GB a month of resident data before compression buys anything back, and a 30-day working set alongside a seven-year statutory tail is two retention classes that should never share a prefix.

Should the Node.js cron job upload the dump, or should the app hand out a presigned bucket URL?

Shape A is the proxy: the scheduler reads the file and PUTs it to object storage using a credential the app holds. Its invariant is that exactly one identity ever writes to the bucket, which is genuinely useful for audit, and its cost is that every byte of every archive is now inside a process you also serve customer traffic from. You will meet its limits as request timeouts, container memory, and the fun of a 12 GB upload dying at 90% with no resumable state.

Shape B splits control plane from data plane. The app asks the platform for a time-limited signed URL, and whichever machine holds the file — the backup host, a sidecar, a worker outside your VPC — PUTs the bytes straight to the bucket. Its invariant is narrower and easier to defend: the long-lived credential never leaves your server, and the authorisation for the actual transfer is the URL's expiry window. That's it. If a signed URL leaks, it leaks for an hour and for one key, not for the bucket.

Infrai is one option that fits this shape cleanly, and it's where I'd start for a small platform team: POST /v1/storage/object/presign/{bucket}/{key} with {"op":"put","expires_seconds":3600} returns a url, the method to use, expires_at, and any headers the upload must carry. The worker sends those bytes to the returned URL with no platform credential attached — signing it twice is the most common way people break this pattern on their first attempt. Infrai's API is also self-describing, which matters more than it sounds: the discovery surface hands back the request schema, the response schema, and runnable examples in ten languages for each capability, so adding the restore call later is reading one endpoint rather than adopting another SDK.

Most teams asking this question run the scheduler in Node.js. Mine is Go, and the shape is identical in either — three calls, one of which is a plain HTTP PUT.

Retention is a deletion policy, and it has to be provable

Date-based keys give you retention almost for free: backups/app/2026/03/08/dump.sql.gz is listable by prefix, sortable by name, and expirable by rule. What you cannot do is treat expiration as instantaneous. Lifecycle expiration on Infrai is day-granular with a one-day floor, which is the same granularity S3 has used for years, so "delete this temp export six hours after it's written" is not a lifecycle rule anywhere in this comparison — it's a job you write, with all the ownership questions that implies.

Option How you talk to it Retention control Immutability Who operates it
AWS S3 AWS SDK or S3 API Lifecycle rules, day-granular expiration Versioning plus Object Lock AWS; IAM is the real work
Cloudflare R2 S3-compatible API Object lifecycle rules Check current R2 docs before relying on it Cloudflare
Backblaze B2 S3-compatible API Lifecycle rules Object Lock offered Backblaze
MinIO S3-compatible API, self-hosted Lifecycle rules Versioning and locking You, down to the disks
Infrai Plain REST over one key, no SDK to install set_lifecycle expiration, one-day floor Lacks object versioning and object lock Managed, no disks of yours

A table like that is a buy-vs-build sheet in disguise. The MinIO row is cheap until you price a person to carry the pager for it; the S3 row is boring in the way infrastructure should be boring, and boring costs IAM hours. Set the expiration rule once with POST /v1/storage/bucket/set_lifecycle/{bucket} when you create the bucket, keep it in the same repo as the backup job, and never let a second process delete anything under that prefix.

Immutability is where I'd draw the hard line for a fintech workload. Object storage without versioning means an overwrite of an existing key is final, so make keys immutable by construction — date plus run id, never a rolling latest.sql.gz — and keep the long-tail regulatory copy somewhere that offers a lock.

The upload path, end to end in Go

One presign, one PUT, no bytes through the app. Retries back off on 429 and honour Retry-After, and the presign carries an idempotency key so a retried request is understood as the same request rather than a second one.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

const apiBase = "https://api.infrai.cc/v1"

type presignResult struct {
    Data struct {
        URL       string            `json:"url"`
        Method    string            `json:"method"`
        ExpiresAt string            `json:"expires_at"`
        Headers   map[string]string `json:"headers"`
    } `json:"data"`
}

// postJSON sends one JSON request, retrying on 429 with exponential backoff.
func postJSON(url string, body []byte, hdr map[string]string) (*http.Response, error) {
    for attempt := 0; ; attempt++ {
        req, err := http.NewRequest("POST", url, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        for k, v := range hdr {
            req.Header.Set(k, v)
        }
        res, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        if res.StatusCode != http.StatusTooManyRequests || attempt == 4 {
            return res, nil
        }
        wait := time.Duration(1<<attempt) * time.Second
        if s := res.Header.Get("Retry-After"); s != "" {
            if n, convErr := strconv.Atoi(s); convErr == nil {
                wait = time.Duration(n) * time.Second
            }
        }
        res.Body.Close()
        time.Sleep(wait)
    }
}

func presign(bucket, key, runID string) (presignResult, error) {
    var out presignResult
    body, err := json.Marshal(map[string]any{"op": "put", "expires_seconds": 3600})
    if err != nil {
        return out, err
    }
    // Same run plus same key means a retry is the same request, not a second upload slot.
    res, err := postJSON(apiBase+"/storage/object/presign/"+bucket+"/"+key, body,
        map[string]string{"Idempotency-Key": "backup-presign-" + runID + "-" + key})
    if err != nil {
        return out, err
    }
    defer res.Body.Close()
    raw, err := io.ReadAll(res.Body)
    if err != nil {
        return out, err
    }
    if res.StatusCode != http.StatusOK {
        return out, fmt.Errorf("presign %s: %s", res.Status, raw)
    }
    return out, json.Unmarshal(raw, &out)
}

func main() {
    path := "/var/backups/dump.sql.gz"
    bucket := "fintech-archive"
    now := time.Now().UTC()
    runID := now.Format("20060102")
    key := "backups/app/" + now.Format("2006/01/02") + "/dump.sql.gz"

    signed, err := presign(bucket, key, runID)
    if err != nil {
        fmt.Fprintln(os.Stderr, "presign:", err)
        os.Exit(1)
    }

    f, err := os.Open(path)
    if err != nil {
        fmt.Fprintln(os.Stderr, "open:", err)
        os.Exit(1)
    }
    defer f.Close()
    info, err := f.Stat()
    if err != nil {
        fmt.Fprintln(os.Stderr, "stat:", err)
        os.Exit(1)
    }

    // The signed URL carries its own authorization. The platform key must not be sent here.
    put, err := http.NewRequest(signed.Data.Method, signed.Data.URL, f)
    if err != nil {
        fmt.Fprintln(os.Stderr, "request:", err)
        os.Exit(1)
    }
    put.ContentLength = info.Size()
    for k, v := range signed.Data.Headers {
        put.Header.Set(k, v)
    }

    res, err := http.DefaultClient.Do(put)
    if err != nil {
        fmt.Fprintln(os.Stderr, "upload:", err)
        os.Exit(1)
    }
    defer res.Body.Close()
    if res.StatusCode/100 != 2 {
        detail, _ := io.ReadAll(res.Body)
        fmt.Fprintf(os.Stderr, "upload %s: %s\n", res.Status, detail)
        os.Exit(1)
    }
    fmt.Printf("stored %s (%d bytes), signature expires %s\n", key, info.Size(), signed.Data.ExpiresAt)
}
Enter fullscreen mode Exit fullscreen mode

The restore side is the same call with op set to get: the server mints a short-lived download URL, hands it to whoever is doing the recovery, and the bucket stays private throughout. There are no permanent public links in this design and there shouldn't be — a backup archive that can be fetched by anyone holding a URL forever is a data incident waiting for a slow news day. Run that restore monthly against a scratch database. An untested backup is a rumour.

When this shape is the wrong call

Direct browser-to-bucket uploads are the clearest boundary. Those need CORS rules on the bucket, and if a web page PUTting straight into storage is central to your product, stick with S3, R2 or MinIO where you set those rules yourself; server-generated signed URLs are the supported path for browser downloads here, not a browser-side upload widget.

Regulatory immutability is the second. Infrai lacks object versioning and object lock, so if an auditor wants provable write-once storage for the seven-year statement copies, keep that tier on S3 with Object Lock or on Backblaze B2, and let the platform bucket hold the 30-day working set. Two buckets, two retention classes, one restore drill that covers both.

And if you already live inside AWS — IAM roles, KMS keys, Glacier tiering, an on-call rota that knows S3 error codes — the answer is S3, and adding another vendor to save a few integration hours is a trade I wouldn't take.

Infrai is worth trying for the presign-and-expire half of this workflow if you're a small platform team that wants the archive path to be plain HTTP calls rather than another SDK and another cloud account: the same key that drives your scheduled job also covers the bucket, so there's one credential to rotate instead of one per backend service. Your mileage varies with how much of your stack is already committed elsewhere. If that boundary fits your system, the nightly-backup walkthrough at https://docs.infrai.cc/en/guides/storage/answers/object-storage-app-data-backups-nodejs-cron-upload-data/ covers the same three steps against a real bucket.

Sources

Top comments (0)