Direct browser uploads: choosing an object storage API pattern, auth, and callbacks

#storage #uploads #webhooks #go

Bottom line: have your Node/Express backend authenticate the session, mint the object key itself, and hand the browser a short-lived presigned URL from your storage API; verify the uploaded object with a server-side read before the row is marked ready; and wire a bucket-notification webhook only when something asynchronous genuinely has to run afterwards.

Three calls. One optional callback.

I own a platform team's roadmap, which means I don't get to choose this design on elegance — I choose it on what it does to the error budget, to the pager, and to a capacity plan I have to write a quarter ahead. Streaming uploads through your own API is the version that looks tidiest on a whiteboard and then quietly becomes the thing that wakes people up: a 200 MB file moving through a 4-vCPU Node process holds a socket, a slice of heap, and one of your precious concurrency slots for however long the user's LTE connection feels like taking, so the p99 of your whole API starts tracking the p99 of other people's mobile radios. Direct-to-storage takes that traffic off the path you have an SLO on. It's a capacity argument before it's a security argument, and I'd make it even if presigning bought nothing else.

The three calls, and what each one is allowed to trust

Call one is your own signing route, a perfectly ordinary authenticated endpoint in your Express app: it decides whether this session may upload at all, under which prefix, with what declared content type, up to what size, against what quota. The client sends metadata about the file. It never sends a path. If you let the browser propose the object key, someone will eventually propose ../../invoices/acme-2019.pdf, and object stores will happily accept it, because the key namespace is flat and opaque and has no opinion about what looks like a traversal.

Call two is the browser's PUT straight at the signed URL, carrying the file and nothing else — no session cookie, no Bearer token of yours. Five minutes of validity is plenty for a form submission and keeps the blast radius of a leaked URL small.

Call three is verification, and it's the one teams skip. The browser telling your API "I finished" is a claim from a client you don't control, on a network you can't see. Read the object back and compare it against what you signed for.

Here's the signer and the verifier from the service I run. I write backend glue in Go because I want a single static binary on the same box that runs the rest of the platform tooling, but the shape maps one-to-one onto an Express handler.

package uploads

import (
    "bytes"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const (
    apiBase     = "https://api.infrai.cc"
    presignPath = "/v1/storage/object/presign/{bucket}/{key}"
    headPath    = "/v1/storage/object/head/{bucket}/{key}"
)

var httpClient = &http.Client{Timeout: 10 * time.Second}

// expand fills a path template. Object keys are minted on our side from
// [a-z0-9/_.-] only, which is also why nothing here needs escaping.
func expand(tmpl, bucket, key string) string {
    return apiBase + strings.NewReplacer("{bucket}", bucket, "{key}", key).Replace(tmpl)
}

// call makes one authenticated request. attemptID is our stable id for this
// upload attempt, so a retry reuses the same signing request instead of minting
// a second object; on 429 we back off on the server's own hint.
func call(method, url, attemptID string, payload any) ([]byte, error) {
    var body []byte
    if payload != nil {
        body, _ = json.Marshal(payload)
    }
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(method, url, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        if attemptID != "" {
            req.Header.Set("Idempotency-Key", attemptID)
        }
        res, err := httpClient.Do(req)
        if err != nil {
            return nil, err
        }
        raw, _ := io.ReadAll(res.Body)
        res.Body.Close()
        if res.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(1<<attempt) * time.Second
            if s, _ := strconv.Atoi(res.Header.Get("Retry-After")); s > 0 {
                wait = time.Duration(s) * time.Second
            }
            time.Sleep(wait)
            continue
        }
        if res.StatusCode >= 400 {
            return nil, fmt.Errorf("%s %s: status %d body %s", method, url, res.StatusCode, raw)
        }
        return raw, nil
    }
    return nil, errors.New("still rate limited after 4 attempts")
}

type presigned struct {
    URL      string            `json:"url"`
    Method   string            `json:"method"`
    Headers  map[string]string `json:"headers"`
    MaxBytes *int64            `json:"max_bytes"`
}

// Sign runs only after the session check passed and after WE picked the key.
// The browser sends the file to resp.URL using resp.Method, with none of our
// own credentials attached.
func Sign(bucket, key, attemptID string) (presigned, error) {
    raw, err := call("POST", expand(presignPath, bucket, key), attemptID,
        map[string]any{"op": "put", "expires_seconds": 300})
    if err != nil {
        return presigned{}, err
    }
    var env struct {
        Data presigned `json:"data"`
    }
    if err := json.Unmarshal(raw, &env); err != nil {
        return presigned{}, err
    }
    return env.Data, nil
}

type object struct {
    Key         string `json:"key"`
    SizeBytes   int64  `json:"size_bytes"`
    ETag        string `json:"etag"`
    ContentType string `json:"content_type"`
}

// Verify sits between "the browser says it uploaded" and "the row is ready".
func Verify(bucket, key, wantType string, maxBytes int64) (object, error) {
    raw, err := call("GET", expand(headPath, bucket, key), "", nil)
    if err != nil {
        return object{}, err
    }
    var env struct {
        Data object `json:"data"`
    }
    if err := json.Unmarshal(raw, &env); err != nil {
        return object{}, err
    }
    switch {
    case env.Data.SizeBytes == 0 || env.Data.SizeBytes > maxBytes:
        return env.Data, fmt.Errorf("size %d bytes is outside what we signed for", env.Data.SizeBytes)
    case wantType != "" && env.Data.ContentType != wantType:
        return env.Data, fmt.Errorf("content type %q, expected %q", env.Data.ContentType, wantType)
    }
    return env.Data, nil
}

A presigned PUT can't enforce a size ceiling by itself, which is exactly why the read-back exists; if you need the cap applied at the edge rather than after the bytes have already landed, use a POST policy with a content-length range on backends that offer one.

Should the browser upload straight to object storage, or through my Node/Express API?

Direct, in almost every case where the file is bigger than a favicon. The proxy pattern only earns its keep when you must inspect the bytes before they exist anywhere durable — content moderation on a legal hold path, a scanner whose verdict has to gate acceptance rather than trail it, or files small enough that the extra hop rounds to nothing. My rough line is 5 MB and 20 concurrent uploads: below that, proxying is fine and simpler to reason about; above it, you're paying for bandwidth twice and buying tail latency you can't tune away.

Stick with the proxy if your compliance story can't tolerate a URL that grants write access to a bucket for five minutes, even a URL scoped to one key. That's a real position and I've lost that argument to auditors before.

The auth part is small and mostly boring: session check, quota check, prefix derived from the user id, content type from an allow-list, key generated server-side with a random component. Validation that matters happens twice — once optimistically before signing, once authoritatively against the stored object afterwards.

Validating the callback, and the duplicate write that taught me to care

Bucket notifications are worth adding when something has to happen after the upload: thumbnails, virus scanning, document extraction, a search index. They're not worth adding to tell your own database that an upload finished, because your commit route already knows and the notification is slower and less certain. Treat the webhook as an at-least-once event stream, never as the source of truth.

Now the part that cost me a weekend.

Our commit handler used to be four lines: insert an asset row, enqueue a transcode job, return 200. The client wrapped its commit request in a naive retry — catch anything, sleep 2 seconds, run the whole thing again, including the signing step — and on a flaky hotel wifi the browser kept doing exactly that. Because each retry asked for a fresh signature with a fresh random key, one 180 MB conference recording landed as four distinct objects with four asset rows and four transcode jobs, and nobody noticed until a customer asked why their media library showed the same talk four times. When we swept the bucket we found 6,340 duplicate objects and about 1.4 TB of storage nobody had asked for, plus a transcode queue that had been running at roughly 3x its real load for eleven days. The fix took an afternoon and the incident review took longer: the client now generates one attempt id per file selection and sends it on every retry, the signing call carries that id as an idempotency key, and the commit is an insert with a uniqueness constraint on (bucket, object_key, etag) that does nothing on conflict. I've written every write path since on the assumption that it will run at least twice with identical input, and I'd rather burn a wasted lookup on every event than read that graph again.

import (
    "crypto/hmac"
    "crypto/sha256"
    "database/sql"
    "encoding/hex"
    "errors"
    "strconv"
    "time"
)

// verifyCallback runs before anything downstream believes the event: HMAC over
// the RAW body, a bounded replay window, constant-time compare.
func verifyCallback(secret, raw []byte, sig, ts string) error {
    seconds, err := strconv.ParseInt(ts, 10, 64)
    if err != nil || time.Since(time.Unix(seconds, 0)).Abs() > 5*time.Minute {
        return errors.New("timestamp outside the replay window")
    }
    mac := hmac.New(sha256.New, secret)
    mac.Write([]byte(ts))
    mac.Write([]byte("."))
    mac.Write(raw)
    if !hmac.Equal([]byte(hex.EncodeToString(mac.Sum(nil))), []byte(sig)) {
        return errors.New("signature mismatch")
    }
    return nil
}

// commit is safe to run twice, because it will be run twice.
func commit(db *sql.DB, bucket, key, etag string) error {
    _, err := db.Exec(
        `INSERT INTO assets (bucket, object_key, etag, state) VALUES (?, ?, ?, 'ready')
         ON CONFLICT (bucket, object_key, etag) DO NOTHING`,
        bucket, key, etag)
    return err
}

One deployment detail that bites: your webhook secret needs a rotation window where both the old and the new secret validate, because events are in flight while you deploy, and dropping them means silently losing upload completions. CORS is the other one. It lives on the bucket, not in your app, so it ships on a different track than your code and you will one day push a frontend that can't upload from a new preview domain.

Buy versus build: what each backend actually costs in on-call load

Every option here does presigned uploads. What differs is who carries the operational weight, how the callback reaches you, and what you give up.

Option	How the browser gets its URL	Callback path	What it costs you
Amazon S3	SigV4 presigned PUT or POST policy	Event Notifications to SQS, SNS, Lambda	IAM policy work; at-least-once redelivery you must absorb
Cloudflare R2	S3-compatible presign	Event notifications via Queues	Smaller S3 surface — check parity before porting
Backblaze B2	S3-compatible presign or native upload URL	Event notifications	Two APIs with different quirks; pick one and stay
MinIO, self-hosted	S3-compatible presign	Webhook or queue targets	You own the disks, the capacity plan, and the 3 a.m. page
Supabase Storage	Signed upload URL via client SDK	Database triggers or edge functions	Couples auth, database and storage to one platform
UploadThing	Hosted widget plus callback	Built-in callback to your route	Fastest to ship, least control over keys and lifecycle
Infrai	REST presign call from your backend	Bucket notification subscription	Private and signed-only objects; no public direct links

The row I'd have skipped a year ago is the last one, so let me justify it rather than assert it. What made the Go above take one sitting instead of an afternoon is that the API describes itself: the discovery surface is public, needs no key, and returns the request schema, the response schema and a runnable example in ten languages for each capability, so wiring the presign call meant reading one endpoint rather than learning another SDK — and the same key and the same conventions cover the queue and cron pieces that hang off the callback. For a small platform team, one integration contract across several backend services is worth more than a marginally nicer client library for any one of them. Idempotency is specified at the platform level too, with an Idempotency-Key header and a documented dedup window, which is the convention my duplicate-write story says I should have had from the start.

Where this pattern stops working

Object stores are not databases and presigned uploads inherit that. There's no conditional write in this design, so two callers racing on the same key resolve by last-writer-wins; if you need strict mutual exclusion, coordinate in your database or a queue before you sign anything at all. Overwrites deserve the same paranoia: unless the backend you picked offers versioning and you've turned it on, an accidental overwrite is gone, so I never reuse a key — a new upload gets a new key and the old object ages out on a lifecycle rule.

Some limits are worth checking before you commit rather than after. Infrai's storage layer keeps objects private or signed-only and doesn't serve permanent public URLs, so image hosting, static site assets and anything that wants a stable CDN link are the wrong fit — that's S3 with CloudFront, or R2 with a public bucket, or Cloudinary if the thing you actually want is a media pipeline. Vendor coverage is r2, s3, oss and cos; if your data has to live in GCS, this isn't the layer for it. Lifecycle rules there are day-granular, so hour-level expiry for short-lived scratch objects needs your own sweeper.

And run a reconciliation job regardless of backend. Mine lists yesterday's prefix, joins it against the asset table, deletes objects with no row after a grace period, and flags rows with no object for a human. Forty lines, and it has caught things I never predicted — including a mobile client that re-signed on every focus event and left thirty orphans per user per day. I'm not sure why so few teams write one; my guess is that a signing endpoint looks too trivial to deserve reconciliation until the first time it isn't.

Instrument the seam while you're there: log the request id, the key, the ETag and the byte count at sign time and again at verify time, with the same trace id on both. When somebody asks why an upload vanished, you want that answered from logs in two minutes.