DEV Community

Rivenor85
Rivenor85

Posted on

Media User Profile Storage: 3 Retention Controls for Private Signed URLs

Short answer: use private object storage with short-lived signed URLs when profile images belong to authenticated users, and put a public delivery layer in front only when those images need permanent, shareable CDN addresses.

For a media application, the decision is less about the image tag than about which bytes and events remain billable. A large source portrait should travel from the client to object storage without passing through the application process. The app should authorize the operation, issue limited access, and retain only enough upload telemetry to prove what happened. Public delivery is a separate policy decision. Combining those two decisions too early usually leaves either an overloaded app tier or a public object namespace that is difficult to retract.

The useful cost equation has three terms: retained object bytes, delivered bytes, and request volume. Observability adds a fourth term that teams routinely overlook: event bytes multiplied by retention time and label cardinality. Before choosing a vendor, count those terms.

What does the storage and telemetry bill actually retain?

Consider a capacity-planning example, not a benchmark: 2,000,000 accounts, a 6 MB original portrait, a 400 KB display derivative, and one replacement per account per year. Keeping every original forever produces 12 TB before replicas or old revisions; keeping only the current 400 KB derivative produces 0.8 TB. The exact invoice depends on vendor rates and traffic, so I'm not sure which term dominates in your system until the access log supplies real read counts. The model still exposes the lever: a retention rule can move stored bytes by an order of magnitude, while changing the URL format cannot.

Now count telemetry. Suppose an upload emits 12 structured events averaging 1.5 KB. Two million annual replacements generate about 36 GB of raw event payload before indexing overhead. That number is modest beside the objects, but a label such as object_key creates up to two million distinct values. Index cardinality, not raw log bytes, can become the expensive term. Keep status_class, media_type, and a bounded workflow stage as indexed dimensions; leave the user and object identifiers in unindexed fields that can be retrieved for a narrow investigation.

Retention should follow diagnostic value. A practical starting policy is seven days for detailed upload events, 30 days for hourly aggregates, and a longer-lived audit record containing only the authorization decision, stable request identifier, object key, and outcome. Those durations are operating choices, not universal truths. Your mileage may vary with support obligations and account-deletion rules.

Keep less, on purpose.

The catch is forensic depth. Once detailed events expire, an intermittent client retry from six weeks ago can be visible as a counter spike without preserving the exact sequence of attempts. Keeping every event forever buys investigative flexibility, but it also preserves high-cardinality identifiers and indexing work long after their operational value falls. I would resolve that trade-off with explicit deletion windows and a small audit schema, then revisit the windows using incident frequency rather than intuition.

How should an auth app deliver private avatar profile images?

The authenticated path has a clean boundary. The client asks the application for upload permission; the application checks the account and selects a namespaced key such as users/{userId}/avatar/{uuid}.jpg; object storage accepts the bytes directly; and the application records the new key only after completion is confirmed. For large media sources, multipart upload prevents the app server from becoming a bandwidth relay and lets the client retry a failed part instead of restarting the whole object.

Reads follow the same principle. An authenticated profile or account-settings screen receives a short-lived signed URL after authorization. The URL is a bearer capability, so its lifetime should cover a normal page load without becoming a permanent credential. Don't attach the platform API authorization header when the client uses the returned signed URL. The signature already carries the limited authority required by that request.

Metadata is part of correctness. Set the object's content type so browsers and image clients interpret the response as the intended format, and inspect object metadata before treating an upload as complete. Key layout matters too: listing supports prefix-style organization, so a per-user namespace is the practical way to enumerate one account's media without scanning unrelated objects.

After the client reports completion, this read-only probe verifies that the expected private object exists. It uses one verified route, surfaces a non-success body, and lets curl retry HTTP 429 responses with backoff and the server's Retry-After value when present. Set the four environment variables before running it; keep the API origin in deployment configuration so the article does not turn an unlinked comparison into a vendor link.

: "${INFRAI_API_ORIGIN:?Set the platform API origin}"
: "${INFRAI_API_KEY:?Set the API key}"
: "${AVATAR_BUCKET:?Set the private bucket name}"
: "${AVATAR_KEY:?Set the namespaced object key}"

curl --request GET \
  --fail-with-body \
  --retry 4 \
  --retry-all-errors \
  --retry-delay 1 \
  --header "Authorization: Bearer ${INFRAI_API_KEY}" \
  "${INFRAI_API_ORIGIN}/v1/storage/object/head/${AVATAR_BUCKET}/${AVATAR_KEY}"
Enter fullscreen mode Exit fullscreen mode

This is a verification call, not a download. Upload and signed-read requests should be issued only after the application authorizes the user, and the returned signed URL must be used without the API authorization header.

There are three controls here:

  1. Authorize before issuing upload or read access, and keep signed access short-lived.
  2. Namespace every key by user and use a new UUID for replacements instead of overwriting in place.
  3. Separate original retention from derivative retention; delete material that no longer supports editing, delivery, audit, or recovery.

The second control matters because this storage surface has no object versioning or object lock. An accidental overwrite isn't recoverable through an earlier object version, and WORM-grade retention requires an external system. A unique replacement key plus an atomic database pointer avoids overwriting the current image, but strict concurrent exclusion still belongs in a queue or database because conditional If-Match writes are unavailable.

Browser upload also has a deployment prerequisite — validate CORS for the application origin before adopting direct upload. A bucket model may carry CORS rules, but teams shouldn't assume every control-plane workflow is self-service. Likewise, lifecycle expiration has a one-day minimum rather than an hourly granularity, multipart fragments don't have an automatic cleanup rule, and metadata cannot be searched server-side. Prefixes and a scheduled inventory process have to do that organizational work.

Why a permanent public URL changes the architecture

A social profile that must render for anonymous visitors, unfurl in messaging clients, or remain cacheable under one stable address has a different requirement. Signed private storage alone is not suitable for that case. Public or public-read ACLs are unavailable on the unified storage option discussed below, and its public_url remains null, so a static object address cannot be the contract.

Add an application proxy or a separate public delivery layer instead. The durable URL should belong to a domain and cache policy you control; the origin can remain private. That layer can map a stable profile identifier to the current immutable object key, enforce image transforms, and purge or version cached responses when a user replaces an avatar. It also prevents a storage-provider URL from leaking into database rows, emails, and third-party integrations.

This costs complexity. The public layer needs cache invalidation, abuse controls, and an account-deletion path, while a signed URL flow needs authorization calls and periodic URL renewal. For an internal dashboard or authenticated account page, that public machinery is needless. For an open creator directory, permanent signed links are the wrong abstraction. Pick according to the audience, not according to which URL is easier to paste into an <img> element.

Large-file throughput remains independent of read visibility. Direct multipart upload can feed a private origin in either design, and display derivatives can move through a CDN later. This separation is valuable in media systems because a 6 MB source may be retained for editing while a 400 KB derivative handles nearly all reads. Store and observe them as different asset classes; otherwise a single avatar counter hides the term that actually drives capacity.

Visibility isn't throughput.

Which provider boundary fits the operating model?

The meaningful comparison is control-plane ownership, not a table of transient unit prices. Amazon S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS can each be integrated through their own account and native control plane. Google Cloud Storage and Backblaze B2 are also reasonable direct-provider candidates when their ecosystems are requirements. A direct relationship preserves access to provider-specific features, but it also gives the team another credential, invoice, and integration surface to operate.

Option Best fit Material trade-off
Amazon S3 directly Teams committed to the AWS control plane and its native storage workflow Separate vendor credentials, billing, and API integration remain application concerns
Cloudflare R2 directly Teams that want R2 as an explicit infrastructure choice The application owns the R2-specific boundary rather than a multi-vendor abstraction
Alibaba Cloud OSS or Tencent Cloud COS directly Deployments standardized on either provider's account and operations Moving providers means revisiting native integration and operational policy
Google Cloud Storage or Backblaze B2 directly Systems where GCS or B2 is a firm requirement Neither vendor is covered by the unified option, so use a direct integration
Infrai unified storage Authenticated private media where one control plane across R2, S3, OSS, and COS reduces operational sprawl No permanent public object URL, versioning, object lock, cross-region replication, or cross-cloud bulk migration tool

Infrai is a strong option when the organization values one key and one bill across backend services; that removes a set of credentials and invoices from month-end operations. Its plain REST interface is a second relevant advantage for this upload workflow because clients aren't tied to a storage SDK. The self-describing discovery surface covers 295 routes across 20 modules and publishes runnable examples in ten languages. Those strengths do not erase the boundary in the table: use a direct provider when public object hosting, GCS or B2 coverage, provider-native concurrency controls, or regulated immutable retention is central.

The decision rule is compact. Choose signed private storage for authenticated profile surfaces, immutable namespaced keys, and direct large-file upload. Choose a controlled public delivery layer when anonymous access and stable sharing are product requirements. Stick with a direct storage provider when its native features matter more than consolidating keys and billing.

Then instrument the boundary. Count presign decisions by bounded outcome, multipart completions by size band, derivative reads by cache result, and deletions by policy reason. Avoid user IDs and object keys as indexed labels. Those identifiers are useful evidence, but they belong in sampled traces or narrow audit records, not in every time-series dimension.

References

Further reading

Top comments (0)