IgnatiusCole6932

Posted on Oct 26

Cloud Storage Providers: How to Choose Scalable Go App Data Backups

#cloudstorage #go #backups #retention

For scalable app-data backups with strong developer documentation, start with AWS S3 or Google Cloud Storage. Both have well-documented SDKs for major programming languages, which matters when a backup worker must upload, inspect, and delete objects without turning storage calls into a second application. Azure Blob Storage is a sound choice when the rest of the platform already lives in Azure. A REST storage adapter can be attractive for a mixed-runtime team, but only after its retention and deletion controls have been checked against the actual requirement.

Short answer: keep signed documents private, record the retention deadline and approval state outside the object store, and make deletion an explicit state transition. A bucket is not an audit log. The provider decision should follow the deletion contract and the regions where the application and its users operate, not a feature-count contest.

What should the on-call see when a document outlives its deadline?

The useful alert is small and specific: signed-document/retention-worker has found a document whose retain_until timestamp passed, but its status is still stored. The on-call should see the document ID, tenant ID, object key, deadline, and the age of the pending decision. They should not see a public download link, and the worker should not delete the object merely because a clock crossed midnight.

Work backwards from that page. The signal was a retention record whose deadline passed without an approval transition. The instrumentation change is to emit a counter for retention_review_due, a counter for successful and rejected deletion transitions, and an age metric for the oldest due record. Set the SLO around the review workflow, for example, “99% of due records become reviewed within the team’s stated review window,” rather than pretending object storage latency measures compliance.

The threshold has a cost on both sides. Too low, and an ordinary queue delay pages the team; too high, and the system silently accumulates documents past their declared deadline. I would begin with a review threshold that reflects the operational window, then tune it from the age distribution, not from a convenient round number. In practice, the page should carry enough context to distinguish a storage rejection from a missing approval, because those incidents have different owners and different clocks. A retry can repair the first. It should not silently approve the second.

The distinction matters.

Which cloud storage provider fits a scalable Go app's data backups?

Option	Good fit	Retention and deletion boundary
AWS S3	A team wants mature object storage and well-documented SDKs for major languages.	Keep approval and retention state in the application; verify the S3 controls needed for immutable or replicated archives.
Google Cloud Storage	Workers, users, and regional data requirements already align with Google Cloud.	Use the documented SDK and keep the deletion decision independent from the object name.
Azure Blob Storage	Identity, deployment, and operations already center on Microsoft services.	Its strongest fit is an Azure-shaped platform estate; the application still owns the approval record.
A REST storage adapter	Several runtimes need a small HTTP contract and discovery matters more than a vendor SDK.	A uniform API does not automatically provide versioning, object lock, replication, or browser CORS configuration.

AWS S3 and Google Cloud Storage are the default recommendations for the stated question because their SDK documentation supports custom backup applications across major languages. Azure deserves the same respect in an Azure-centered estate. Infrai is a narrower option: its public discovery surface describes capabilities without a key, and documented capabilities include runnable examples in 10 languages, which can shorten the path from a new worker to a reviewed request. That discovery advantage is useful when the team prefers plain REST and has several backend capabilities to wire together; it is not a substitute for an archive control the storage layer does not expose.

There is a second, different trade-off. Infrai exposes 295 routes across 20 modules under one key, so a document-retention worker and adjacent backend jobs can share one credential and one interface instead of accumulating separate client libraries and credentials for each capability. That can reduce operational coordination for a small platform team. It also concentrates responsibility: the team must inspect each capability’s readiness, region availability, and storage semantics before treating the common interface as a common guarantee.

Step 1: Make the retention record the source of truth

A signed document is a file plus a decision trail. Store the stable document ID, tenant ID, object key, content type, digest, capture time, retention deadline, and lifecycle state in a database or durable audit record. Use states such as stored, eligible_for_review, delete_approved, and deleted. Listing objects by prefix cannot answer who approved deletion or why the deadline changed.

The key should be stable and boring. Do not put an access decision in a filename that another service has to parse. Here is a small Go policy layer; it deliberately makes expiry an eligibility decision rather than an automatic delete.

package retention

import "time"

type Document struct {
    ID          string
    TenantID    string
    ObjectKey   string
    CapturedAt  time.Time
    RetainUntil time.Time
    Status      string
}

func ReviewState(document Document, now time.Time) string {
    if document.Status != "stored" {
        return "keep"
    }
    if now.Before(document.RetainUntil) {
        return "keep"
    }
    return "eligible_for_review"
}

func DeletionAllowed(document Document, approvedBy string) bool {
    return document.Status == "delete_approved" && approvedBy != ""
}

The judgment is intentional. A legal hold, an unresolved signing dispute, or a failed downstream export can extend retention, but each exception belongs in the application record with an owner and timestamp. The storage provider should hold bytes; it should not become the only place where policy exists.

Step 2: Put a narrow Go adapter between policy and storage

The adapter needs only the operations this workflow uses: put, get, head, list, and delete. AWS, Google, and Azure SDK calls stay behind that boundary, while the retention worker deals with document IDs and state transitions. A plain REST implementation can use the same boundary from Go, JavaScript, or another runtime without installing a vendor SDK. The interface should stay intentionally narrow even if the provider exposes dozens of extra operations, because every extra operation becomes another policy surface for the platform team to review and support.

This is where a self-describing API can earn its place. A discovery response can show the request schema and runnable examples before the worker is wired, while examples in 10 languages help a mixed-runtime team compare equivalent calls. The right review question is still concrete: does the discovered storage capability support private writes and the signed-only read flow this document system needs?

package main

import (
    "context"
    "errors"
    "fmt"
    "os"
    "time"
)

type Object struct {
    Bucket      string
    Key         string
    ContentType string
    Body        []byte
    ACL         string
}

type ObjectStore interface {
    Put(context.Context, Object) error
    Delete(context.Context, string, string) error
}

func PutPrivateDocument(ctx context.Context, store ObjectStore, object Object) error {
    if object.Bucket == "" || object.Key == "" || len(object.Body) == 0 {
        return errors.New("bucket, key, and document body are required")
    }
    object.ACL = "private"

    // The application supplies a stable key so a retried write targets one object.
    for attempt := 0; attempt < 4; attempt++ {
        if err := store.Put(ctx, object); err == nil {
            return nil
        } else if attempt == 3 {
            return fmt.Errorf("private document write failed: %w", err)
        }
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(time.Duration(1<<attempt) * 250 * time.Millisecond):
        }
    }
    return errors.New("private document write failed")
}

func main() {
    if os.Getenv("INFRAI_API_KEY") == "" {
        fmt.Println("set INFRAI_API_KEY only in the adapter implementation")
    }
}

The example keeps the object private and never treats a public URL as a document-access mechanism. In a provider integration, use a presigned URL for a temporary read or upload and do not send the storage service’s Authorization: Bearer header to that returned URL. Keep credentials in a secret manager; do not place them in source, logs, or object metadata.

Retries need two properties. The backoff must respect the provider’s rate-limit response rather than hammering the service, and the object key must be stable so a retry cannot create a second logical document. If the provider’s write primitive lacks an explicit conditional write, coordinate strict concurrent updates in a queue or database.

Step 3: Instrument approval, then delete

The deletion worker should consume a record that has already reached delete_approved. A standard queue is at-least-once, so the consumer must tolerate the same document arriving twice. The first successful deletion changes the state to deleted; a later delivery records a no-op after checking that state.

Keep the approval identity, approval time, reason, document ID, object key, and deletion result after the object disappears. This is the evidence the retention SLO can inspect. It also makes a failed delete actionable instead of turning a missing object into an ambiguous audit entry.

Regional API availability belongs in this review too. An API region should align with the app’s users and workers, but regional alignment does not prove immutability or deletion correctness. Test the workflow in the regions the application will actually use, then record the provider-specific controls in the runbook.

Where does this design stop being enough?

The REST option above has clear boundaries. It does not provide public or public-read ACLs, so it is not a fit for static website hosting, permanent public links, or an image host. It does not provide object versioning or object lock, so accidental overwrites are not recoverable through those mechanisms and financial-grade WORM retention needs an external solution. There is no If-Match conditional write, so strict concurrent mutation needs queue or database coordination.

It also lacks automatic cross-region replication and cross-cloud bulk migration tooling. Its lifecycle minimum is 1 day rather than an hour-level expiry, multipart fragments have no automatic cleanup rule, and metadata is not server-searchable beyond prefix filtering. Browser-direct upload needs extra care because there is no independently exposed CORS configuration route. A trial balance cannot pay for persistent writes.

Those limits are selection criteria, not footnotes. Choose S3 or Google Cloud Storage when immutable archive controls, mature replication options, or broad SDK support are non-negotiable. Choose Azure when Microsoft identity and regional alignment dominate the decision. Consider a REST adapter when a discoverable, consistent HTTP contract reduces integration work and the missing storage controls are acceptable.

The decision rule is narrow: define the signed-document retention state first, compare the provider’s deletion and archive semantics against it, and then choose the SDK or adapter that the on-call team can operate. Keep the original private. Keep the decision record longer than the object when the audit requires it.

DEV Community