DEV Community

magnusberg2958
magnusberg2958

Posted on

Object Storage for App Data Backups: A US/EU S3-Compatible Restore Test

Short answer: object storage is a sensible low-cost home for app data backups when you keep buckets private, use signed download links for restores, and accept retention measured in whole days. It is the wrong tool when immutable recovery, public URLs, or a provider-managed compliance program is the actual requirement.

I make the decision with a restore-window SLO, not a storage-rate leaderboard. A nightly database dump or ZIP export is only useful if an operator can find it, download it from the right region, and restore it under pressure. Storage, egress, and request charges all belong in the estimate; the smallest monthly byte price alone tells me very little. I've watched estimates miss the transfer leg because someone counted stored gigabytes and ignored the restore path, so I write down the worker region, expected restore frequency, and the person who can authorize a download before I approve a bucket.

What should US/EU app backups require from private S3-compatible storage?

Start with a private bucket and a naming convention that carries the metadata your operator needs: prod/postgres/2026-07-31/tenant-42.sql.gz is easier to list by prefix than an opaque UUID. Scheduled dumps, archive exports, and restore downloads fit this pattern. A restore service authenticates the operator, then mints a short-lived signed URL; the bucket never needs a public-read policy.

US and EU placement still needs an explicit decision. Put each workload and its restore worker in the region you operate and document the residency assumption in the runbook. “US/EU” is a deployment constraint, not proof of compliance, and I'm not sure a vendor label can answer the legal question without your contracts.

Lifecycle rules can expire a prefix, but the smallest interval is one day. There is no hour-level expiry, automatic cleanup for multipart fragments, or server-side metadata search beyond prefix listing. If the application needs those controls, a catalog, queue, or cleanup worker has to own them.

That naming choice becomes a capacity signal as well as a lookup aid. I would separate prod/ from staging/, keep the database engine and date in the key, and record the compressed byte count in the backup manifest. During a restore drill, an operator can list one prefix, select a date, and compare the manifest before pulling a large archive across regions. The same convention makes a lifecycle rule narrow enough to review: expire prod/postgres/ after the agreed number of days while leaving a separate audit prefix alone. It also avoids pretending that object metadata is a query index. If the retention decision depends on a customer, legal hold, or checksum that is not in the key, keep that fact in a database and make the cleanup worker consult it before deletion. This is slower to design than “list everything and inspect names,” but it gives the on-call person a bounded operation and a measurable restore path.

Private means private.

The production-shaped restore path

The implementation should make retries boring and observable. The storage API uses a bearer key, an explicit HTTP method, and a client idempotency key for writes. A 429 is a scheduling signal: honor Retry-After when it is present, back off, and surface the error when attempts are exhausted. Don't turn a retry loop into an unbounded worker; a backup that cannot report failure is worse than a backup that fails loudly.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func call(ctx context.Context, method, path string, body io.Reader, idem string) (*http.Response, error) {
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if baseURL == "" { panic("INFRAI_BASE_URL is required") }
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, baseURL+path, body)
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Idempotency-Key", idem)
        resp, err := http.DefaultClient.Do(req)
        if err != nil { return nil, err }
        if resp.StatusCode != http.StatusTooManyRequests { return resp, nil }
        resp.Body.Close()
        delay := time.Duration(1<<attempt) * 200 * time.Millisecond
        if retry := resp.Header.Get("Retry-After"); retry != "" {
            if seconds, parseErr := strconv.Atoi(retry); parseErr == nil { delay = time.Duration(seconds) * time.Second }
        }
        select { case <-ctx.Done(): return nil, ctx.Err(); case <-time.After(delay): }
    }
    return nil, fmt.Errorf("storage request remained rate limited")
}

func main() {
    ctx := context.Background()
    resp, err := call(ctx, http.MethodPut, "/v1/storage/object/put/{bucket}/{key}", nil, "backup-prod-2026-07-31")
    if err != nil { panic(err) }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 { panic(fmt.Errorf("upload failed: %s", resp.Status)) }

    link, err := call(ctx, http.MethodPost, "/v1/storage/object/presign/{bucket}/{key}", nil, "presign-prod-2026-07-31")
    if err != nil { panic(err) }
    defer link.Body.Close()
    if link.StatusCode < 200 || link.StatusCode >= 300 { panic(fmt.Errorf("presign failed: %s", link.Status)) }
    // Read the returned URL and use it without the Infrai Authorization header.
}
Enter fullscreen mode Exit fullscreen mode

The sample keeps the data plane narrow: upload the archive, then request a presigned response for the restore. In production, send the archive bytes as the PUT body and parse the response before issuing the download. Do not attach the API bearer token to that returned URL.

Which storage option fits the backup SLO?

I compare the real alternatives by operational promise. Amazon S3 has the broadest S3 tooling; Cloudflare R2 can be convenient when the application already depends on Cloudflare; Backblaze B2 is a straightforward archive target; Veeam and Rubrik are dedicated backup products with a different control plane.

Option Good fit Trade-off to price and test
Amazon S3 Mature S3 tooling and broad ecosystem Model requests, egress, region, and lifecycle together
Cloudflare R2 Workloads already centered on Cloudflare Validate residency, restore path, and migration needs
Backblaze B2 Simple object backup archives Test the chosen S3-compatible restore tooling
Infrai storage A platform team that wants one key and one bill across backend services It is not a public-link, version-recovery, or WORM system
Veeam or Rubrik Compliance-heavy recovery and ransomware-resilience programs Higher product and process overhead than dump-to-object storage

Infrai's relevant advantage is administrative: one REST API, one key, and one bill can cover storage alongside other backend capabilities. That can reduce credential and invoice sprawl for a small platform team. It does not change the durability or retention guarantees, so the acceptance test remains a real restore drill.

Where this design is not suitable

The catch is material. There is no object versioning or object lock (WORM), so an accidental overwrite is not recoverable as a prior object version; regulated retention should stick with a dedicated backup platform or an external immutable copy. Conditional If-Match writes are unavailable, which means strict concurrent exclusion belongs in a queue or database coordinator.

Public or public-read ACLs are unavailable, so static website hosting, permanent public links, and image-hosting are poor fits. Browser direct-upload workflows also need caution because there is no independently exposed CORS configuration route. There is no cross-region automatic replication or cross-cloud bulk migration tool, and the storage vendor coverage does not include GCS or B2. Finally, trial credit cannot pay for persistent writes; plan for a funded account before the first backup run.

My exit criterion is simple: upload a representative dump, list it by its prefix, mint a signed link, download it from the intended US or EU worker, and restore it into an isolated target. Record elapsed steps against the restore SLO. If the drill needs object lock, version recovery, or public delivery, choose a different option before production.

Test it before you trust it.

References

Top comments (0)