LiamFoster1844

Posted on Oct 28

How to Govern Browser Direct-to-Storage File Upload Service in Go (30-Day Presigned URLs)

#browser #upload #presignedurls #retention

A browser upload is easy to demonstrate and surprisingly easy to mis-govern. The operational constraint that changes the answer is the deletion deadline: a presigned URL can authorize a write for a short window, but it does not prove that the resulting object will disappear on day 30. I would make the service choice secondary to an expiry model that the application can observe, reconcile, and audit.

The URL is not the retention policy.

The failure chain is easy to miss. A client requests an upload, abandons it, and later retries with the same business identifier. A completion handler sees an object and marks the record accepted, even though the object was uploaded under a key generated for the earlier attempt. Thirty days later, the deletion worker removes the accepted key while the orphan remains. The fix is a server-owned immutable attempt identifier, an explicit completion state, and a reconciliation query for objects that have no live metadata row. This is why I keep the object key, acceptance timestamp, digest, and deletion deadline together: each field answers a different question during an incident, and none can be safely reconstructed from a browser request.

TL;DR: issue a narrowly scoped upload URL from your API, keep the deadline in application metadata, verify the completed object, and run deletion as an idempotent workflow with a measurable SLO. Then compare storage services on lifecycle semantics, conditional operations, auditability, regional requirements, and on-call burden.

The deadline changes the design

Consider a developer-tools product that retains signed release documents for exactly 30 days after acceptance. The browser sends bytes directly to object storage; the API must never trust the browser to choose the object key, content type, or retention date. The deadline is a business rule, while the signed URL is only a temporary capability. Confusing those two is the failure I would design around first.

A useful record is small:

type Document struct {
    ID             string
    ObjectKey      string
    AcceptedAt     time.Time
    DeleteAfter    time.Time
    UploadExpires  time.Time
    SHA256         string
    DeletedAt      *time.Time
}

The upload URL should expire in minutes, not 30 days. The object deadline should be calculated by the API after acceptance, stored durably, and treated as the source of truth for reconciliation. Object-storage lifecycle rules are still useful as a second line, but they are not a substitute for an application record when the requirement is an explicit deadline.

The production scenario is bounded on purpose: one document, one generated key, one completion callback, and one deletion job. That keeps the invariant testable. If a retry arrives after deletion, it must not recreate the document under the same key; if a completion callback arrives twice, it must not extend retention.

What should a presigned upload URL be allowed to do?

Only enough to complete one upload. In practice that means a server-generated key, a short expiry, an expected byte limit enforced before completion, and a content type treated as a hint until the stored object is inspected. CORS decides which browser origins may make the request; it does not authenticate the caller or enforce deletion.

The API can create an upload intent before it signs anything:

package upload

import (
    "errors"
    "fmt"
    "strings"
    "time"
)

var ErrInvalidUpload = errors.New("invalid upload intent")

func ValidateIntent(name, contentType string, size int64) error {
    if name == "" || strings.ContainsAny(name, "/\\") {
        return fmt.Errorf("%w: invalid client name", ErrInvalidUpload)
    }
    if size < 1 || size > 25<<20 {
        return fmt.Errorf("%w: size outside policy", ErrInvalidUpload)
    }
    if contentType != "application/pdf" {
        return fmt.Errorf("%w: unsupported content type", ErrInvalidUpload)
    }
    return nil
}

func Deadlines(now time.Time) (uploadExpires, deleteAfter time.Time) {
    return now.Add(10 * time.Minute), now.Add(30 * 24 * time.Hour)
}

The size ceiling is an example policy, not a universal limit. The important part is that it is explicit and tested. A browser can send a different header, truncate a body, or abandon an upload; completion must inspect the object and reconcile it with the intent record.

CORS belongs in the same review because a correctly signed URL can still fail in the browser if the storage endpoint does not allow the requesting origin and method. Configure the smallest origin, method, and request-header set that the application needs, then test the actual browser preflight path.

How do I make deletion an SLO instead of a hope?

Define the deletion SLO as a time bound after delete_after, with a policy for exceptions. For example, a document may be considered compliant when its object is absent within 15 minutes of the deadline, while the metadata row remains as a tombstone for audit. That distinction matters: “the user cannot download it” is weaker than “the object has been deleted.”

A worker should claim due records, issue an idempotent delete, verify the resulting state where the storage API supports that check, and record the attempt. Keep the state transition transactional around the claim, not around the remote delete; distributed transactions will not make the storage call atomic with your database.

package retention

import (
    "context"
    "fmt"
    "time"
)

type ObjectStore interface {
    Delete(ctx context.Context, key string) error
    Exists(ctx context.Context, key string) (bool, error)
}

func DeleteDue(ctx context.Context, store ObjectStore, key string, now, deadline time.Time) error {
    if now.Before(deadline) {
        return fmt.Errorf("object is not due")
    }
    if err := store.Delete(ctx, key); err != nil {
        return fmt.Errorf("delete %q: %w", key, err)
    }
    exists, err := store.Exists(ctx, key)
    if err != nil {
        return fmt.Errorf("verify deletion %q: %w", key, err)
    }
    if exists {
        return fmt.Errorf("object %q still exists", key)
    }
    return nil
}

This path needs metrics for due objects, oldest overdue age, delete attempts, verification failures, and callback-to-completion latency. Alert on the age of the oldest overdue document, not merely on worker process health. A green worker with a poisoned queue is still a retention incident.

The exception is legal hold or an active investigation. Model that explicitly with an immutable reason, actor, timestamp, and next review time. Do not let a missing flag silently convert a temporary hold into indefinite retention.

Which storage boundary fits the retention workflow?

The practical comparison is a buy-vs-build decision about control surfaces, not a leaderboard. A managed object store usually removes storage hardware and much of the durability machinery from the platform team's roadmap; self-hosting can offer more control but transfers upgrades, capacity planning, failure recovery, and security patching to the on-call rotation. The right boundary depends on the SLO and the team's ability to operate it.

Boundary	Signed browser access	Retention controls to verify	Operational trade-off
Amazon S3	Presigned URLs	Lifecycle expiration, object versioning behavior, delete markers, and audit events	Broad ecosystem; details such as versioned deletes require deliberate testing
Google Cloud Storage	Signed URLs	Lifecycle rules, retention policy, holds, and generation-aware writes	Strong policy surface; test how holds interact with deletion jobs
Azure Blob Storage	Shared access signatures	Lifecycle management, immutability policy, and delete behavior	Fine-grained SAS design; token scope and stored access policy choices add review work
Self-hosted S3-compatible storage	Depends on the implementation	Exact lifecycle, replication, and audit guarantees must be verified from the deployed version	More control and more capacity, upgrade, and incident ownership

Those rows are starting points, not equivalence claims. The acceptance test should exercise the exact deployment: issue a URL, upload from a real browser origin, reject an expired or wrong-key request, complete the object, apply a hold, delete after release, and inspect audit records. A feature checkbox is not evidence for a deletion SLO.

I would also price the operational surface without making price the argument: request volume, retained bytes during the deadline, replication, egress during review, audit-log retention, and the staff time needed to operate exceptions. A lower storage line item can be the wrong choice if it creates an unowned reconciliation job.

The decision rule I would ship

Choose the boundary that can demonstrate all of these in a staging account or a controlled self-hosted test: a browser upload with restrictive CORS, a token that cannot write outside its generated key, a completion check that detects a mismatched object, a deletion job with retries and tombstones, and an audit trail that answers who created, held, released, and deleted the document.

Keep the application deadline authoritative. Use storage lifecycle behavior as defense in depth, and test versioning, holds, replicas, and failed deletes as separate cases. The presigned URL solves a narrow authorization problem; the retention workflow owns the document's life.

DEV Community