DEV Community

BeckettHayes6821
BeckettHayes6821

Posted on

Customer File Retention: Object Storage Links and Presigned URL 403 Signature Mismatch

Short answer: keep the export private, preserve one canonical object key, and debug the signed request as a tuple of method, path, query, credentials, and time before changing CORS or making the object public. For a customer download, a 403 is usually useful evidence: the storage service did not accept the request that arrived.

This matters in a B2B SaaS product because an export link is part of the customer workflow, while the file often has a retention and deletion policy behind it. A link that works in a developer browser but fails for a customer after a queue delay is an availability problem. A link that exposes a deleted export is a policy problem. Treat both as one design.

The incident lesson: verify the signed request, not the URL

I would bound the first investigation to one export record and one failed request. Capture the object key as stored, a non-reversible key hash, the signing method, the signing scope, the issuer timestamp, the requested lifetime, the storage request ID, and the final request path. Do not log the complete presigned URL: its query string is temporary authorization.

The invariant is narrow: the bytes used to sign the bucket, key, method, query parameters, and validity window must describe the bytes in the request that reaches object storage. A 403 signature mismatch is the expected result when an intermediate layer decodes and rebuilds a path, changes a space representation, appends a query parameter after signing, or sends HEAD where GET was signed. The URL can look perfectly plausible.

Three quick checks usually separate the branches. First, compare the exact path before signing with the path at the storage boundary. Second, compare the signing service's clock with a trusted time source and leave useful lifetime after the message is delivered, rather than expiring the link while it sits in a notification queue. Third, verify the key and signing credential selected for this tenant and environment. Wrong key encoding and the wrong credential can produce the same opaque status code.

For example, a report named renewals + churn.csv can cross four representations: the database value, the signer input, the encoded URL path, and the proxy's access-log path. If one layer treats + as a space, the object may exist and the user's permissions may be correct, yet the signature is being checked against different bytes. Record each representation in a redacted diagnostic event, then compare them in order. That is slower than swapping a policy setting, but it leaves a testable explanation and a regression case.

Don't start by changing ACLs.

Measure it.

What makes a presigned download URL return 403 after a signature mismatch?

Make the application own a canonical key string and pass it through the same boundary for metadata checks and signing. Do not use a browser-decoded path as the database value. Do not normalize the signed URL after the signer returns it. The example below is deliberately a small policy layer; the provider SDK or HTTP client remains behind it, and the storage provider's documented signer is responsible for the actual signature.

package main

import (
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "fmt"
    "net/http"
    "time"
)

var errInvalidLink = errors.New("invalid download link policy")

// DownloadPolicy contains inputs that must remain stable across verification and signing.
type DownloadPolicy struct {
    Method      string
    ObjectKey   string
    IssuedAt    time.Time
    Lifetime    time.Duration
    MaxLifetime time.Duration
}

func (p DownloadPolicy) Validate(now time.Time) error {
    if p.Method != http.MethodGet || p.ObjectKey == "" {
        return errInvalidLink
    }
    if p.IssuedAt.IsZero() || p.Lifetime <= 0 || p.Lifetime > p.MaxLifetime {
        return errInvalidLink
    }
    if now.Before(p.IssuedAt) || now.After(p.IssuedAt.Add(p.Lifetime)) {
        return errInvalidLink
    }
    return nil
}

func keyFingerprint(key string) string {
    sum := sha256.Sum256([]byte(key))
    return hex.EncodeToString(sum[:])[:16]
}

func prepareDownload(p DownloadPolicy, now time.Time) (string, error) {
    if err := p.Validate(now); err != nil {
        return "", err
    }
    // The signer must receive this exact key; do not decode or rebuild it later.
    return fmt.Sprintf("method=%s key-fingerprint=%s expires=%s",
        p.Method, keyFingerprint(p.ObjectKey), p.IssuedAt.Add(p.Lifetime).UTC().Format(time.RFC3339),
    ), nil
}

func main() {
    now := time.Now().UTC()
    policy := DownloadPolicy{
        Method:      http.MethodGet,
        ObjectKey:   "accounts/42/exports/2026-08/report.csv",
        IssuedAt:    now,
        Lifetime:    15 * time.Minute,
        MaxLifetime: 30 * time.Minute,
    }
    description, err := prepareDownload(policy, now)
    if err != nil {
        panic(err)
    }
    fmt.Println(description)
}
Enter fullscreen mode Exit fullscreen mode

This does not prove that the object exists or that a provider accepts the request. It establishes the inputs your service can observe and test. In a Node.js implementation, apply the same rule around the official signer: store the key once, avoid string concatenation around encoded paths, and test the final method and URL without rewriting them in a middleware layer. Your mileage may vary with a proxy that canonicalizes paths; its access log and the storage request log are the two records that resolve that uncertainty.

Which failure mode is actually consuming the download SLO?

A useful runbook classifies failures before assigning an owner:

Observation First hypothesis Evidence to collect
403 with a signature-mismatch message Signed bytes differ from received bytes Canonical method, path, query, key fingerprint, request ID
403 after a long queue or email delay Validity window has ended Issued-at time, expiry, trusted server time, delivery delay
403 only for names containing spaces or non-ASCII text Encoding changed at a boundary Stored key bytes, encoded path before and after the proxy
Browser reports a CORS block while a server client gets a response Browser policy, not necessarily storage authorization Browser network details and the CORS response headers
403 for one tenant or environment Credential, scope, bucket, or key selection Redacted identity, tenant mapping, region or scope, object metadata

CORS deserves its own branch. It controls whether browser JavaScript may read a cross-origin response; it does not repair a bad signature. MDN's CORS reference is useful here because the browser console can obscure the difference between a policy failure and a response that storage already rejected. Test the URL with a controlled server-side client as well as in the browser, and keep the original storage status and request ID.

The operational target is an SLO for link issuance and a separate SLO for completed downloads. Count presign latency, metadata-check latency, and failed issuance separately from object transfer time. Capacity planning should model peak exports, retry amplification, notification delay, and the number of customers who click near expiry. Averages are poor protection against a month-end batch.

Buy, wrap, or operate the storage contract?

The decision is about ownership, not the existence of a presigned URL. A direct provider integration gives the platform team the provider's contract and tooling; an internal wrapper gives product services a stable interface but leaves the platform team responsible for translating semantics; a self-hosted gateway gives maximum control and puts clocks, credentials, capacity, and the SLO on your pager.

Approach Good fit Trade-off
Direct provider API The organization has committed to one provider and its signing contract Product or platform code absorbs provider-specific migration work
Internal storage adapter Several products need one application-facing contract The adapter becomes a platform product with compatibility and test obligations
Self-hosted gateway Policy requires control over credentials and the request path The team owns scaling, patching, signing correctness, and incident response
Managed abstraction The team values a single HTTP contract across supported backends Provider coverage, retention semantics, and export features may not match the requirement

The catch is that a managed abstraction is unsuitable when you need provider-specific features, immutable retention guarantees, public non-expiring assets, or a backend outside its supported set. The direct option is the better fit when those requirements are contractual. The same caution applies to an internal gateway: it is not automatically simpler just because the product teams see fewer SDK calls.

There is no universal expiry value.

Retention changes the decision. Use unique export keys when regeneration must not replace evidence, record deletion eligibility separately from link expiry, and make deletion idempotent. A presigned URL expiring does not delete the object. Conversely, deleting an object does not make every already-delivered URL disappear from a customer's inbox; your authorization and retention model must account for that gap.

Make the fix repeatable

Add a test corpus containing spaces, plus signs, Unicode, nested prefixes, and empty or deleted records. Generate a link, preserve it byte-for-byte, and exercise the exact signed method through the same proxy path used in production. Add a clock-skew test with a controlled issuer clock. Redact query credentials in logs while retaining a request ID and key fingerprint.

During an incident, change one boundary at a time. Compare the stored key to the signer input, the signer output to the client input, and the client request to the storage log. That sequence keeps a 403 from becoming a permission broadening exercise.

References

Top comments (0)