YukiKobayashi880

Posted on Nov 9

Saving User Documents in Object Storage: Retention Boundaries for Marketplace Media

#storage #architecture #marketplace #backend

Short answer: For saving user documents, the Supabase Storage versus Firebase Storage comparison should end with the same rule: treat storage as a byte-transfer mechanism, keep ownership and expiry in application records, and reconcile private objects with an idempotent worker.

The decisive difference between storage options for user documents is rarely the upload call. It is who can prove that an object should still exist, which system is allowed to delete it, and what happens when a deletion request arrives while a download is in progress. For a marketplace handling large listing videos, inspection photos, and seller documents, direct-to-object-storage uploads keep media bytes out of the application, but they also make retention a distributed-systems problem.

A sound design gives every object an owner, an expiry policy, and a deletion state that can be audited independently of the storage provider. The provider comparison comes after that design. Otherwise, a convenient SDK quietly becomes the retention policy.

Deletion is a business event.

What should happen when a document expires?

Start with a document record in the marketplace database. It should contain an opaque object key, the seller or listing owner, the document purpose, the creation time, the retention class, and a lifecycle state such as pending, active, quarantined, deleted, or delete_failed. The object key is an identifier, not a public URL.

That distinction matters because a signed download URL has a lifetime, while the underlying object may have a much longer or shorter lifetime. HTTP caching also has independent behavior: Cache-Control controls cache reuse and revalidation, and a cache directive does not itself delete the origin object. The browser, a CDN, the database, and the object store each observe a different part of the lifecycle.

For example, a seller's identity document might be retained until an account review is complete, while a listing draft's temporary video can be removed after seven days. Those are product-policy decisions, not storage defaults. I would represent them as named retention classes, because putting a raw number in a controller makes later policy changes hard to find and harder to audit.

The deletion workflow is deliberately boring: mark the record eligible, stop issuing new download authorizations, enqueue an idempotent job, delete by object key, record the result, and reconcile objects with records periodically. A failed delete is a state, not a successful delete with a cheerful log message. A missing object can be treated as already deleted only when the application has enough evidence to distinguish it from a stale or malformed key.

Why does direct upload change the retention design?

When the application proxies a large file, it can see the request body, enforce its own timeout, and decide whether to persist metadata after the transfer. Direct upload removes that byte path, which is usually the right performance boundary for marketplace media. It also means the browser or mobile client talks to storage using temporary authorization, while the application remains responsible for deciding what the upload means.

The safe sequence is to create an upload intent first. The intent binds an object key to the authenticated seller, an expected content type, a maximum size, and an expiration time. The client uploads to that key. A worker or callback verifies the object metadata and promotes the database record from pending to active. An uncompleted intent is garbage, not a valid document.

Then reconcile.

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from uuid import uuid4

@dataclass(frozen=True)
class UploadIntent:
    object_key: str
    owner_id: str
    purpose: str
    expires_at: datetime
    max_bytes: int

def create_upload_intent(owner_id: str, purpose: str, max_bytes: int) -> UploadIntent:
    now = datetime.now(timezone.utc)
    return UploadIntent(
        object_key=f"private/{owner_id}/{uuid4()}",
        owner_id=owner_id,
        purpose=purpose,
        expires_at=now + timedelta(minutes=15),
        max_bytes=max_bytes,
    )

The fifteen-minute window above is an example policy value, not a universal recommendation. The useful property is bounded authorization. If the client stalls, retries, or abandons the page, the intent eventually stops being valid without requiring the application to inspect a huge body.

Do not infer ownership from a client-supplied filename. Filenames are presentation data and can collide, contain confusing Unicode, or expose personal information. Generate the key on the server, keep it private, and store the original filename separately if the user needs to see it.

Names lie.

There is no shortcut here.

Which storage boundary should govern saving user documents?

The familiar comparison between Supabase Storage and Firebase Storage is incomplete unless the deletion boundary is explicit. Both can support an application that stores user documents, but their surrounding data model and authorization workflow lead teams toward different operational shapes. A third option, a general-purpose object store such as Amazon S3, makes the boundary more visible because the application must assemble more of the policy itself. These are engineering differences, not a ranking.

Choice	Useful boundary	Retention and deletion question	Main trade-off
Supabase Storage	Object storage paired with database-oriented access policies	Can the database row and object lifecycle be reconciled when a listing is removed?	Relational context still requires an explicit deletion worker and orphan scan
Firebase Storage	Object storage commonly authorized through identity and security rules	Does the rule authorize only the intended path, and who performs delayed cleanup?	Client-centric authorization is productive, but policy and cleanup must align with database state
Amazon S3	General object-storage primitives with lifecycle and versioning capabilities	What are the exact delete-marker, version, retention, and replication semantics?	Broad control increases the policy details the team must own and test

The table is not a feature scorecard. It is a reminder to ask where the retention decision lives. A storage rule that denies a read does not prove that the object has been deleted. A lifecycle rule that expires current objects does not automatically explain old versions, exports, or application records.

For a marketplace, path authorization deserves attention. A rule that allows access to seller_id/* may still be too broad if the same seller can view private review material belonging to another listing. Bind authorization to a database relationship or a short-lived server-issued capability, and test another seller, an unowned listing, an expired intent, and a deleted record.

Provider object semantics must be written down before implementation. Ask whether overwrites create versions, whether deletion is immediate or represented by a marker, how a missing object is reported, and what a listing of objects can prove. These answers affect reconciliation; they should come from current documentation and a small integration test.

This is the part teams tend to postpone because the upload demo is already working. That delay is expensive: once real keys have escaped into logs, exports, support tickets, and background jobs, changing the ownership model becomes a migration rather than a schema change. Give the object key a stable owner at creation time, and make every later operation present that ownership as a checked fact.

Which failure modes deserve a test before launch?

Retention bugs are usually boring at the API boundary and expensive in the queue behind it. Test an upload that completes without finalization, a duplicated finalize message, a deletion retry after success, a seller removal while a signed URL remains valid, a database delete followed by a storage timeout, a replacement upload, and an object with no database record.

Idempotency is the practical answer. Finalization should tolerate duplicate messages. Deletion should tolerate a missing object according to a documented policy. Reconciliation should be safe to run again tomorrow. If a timeout leaves the worker unable to tell whether deletion happened, record uncertainty and retry with bounded backoff.

Observability should expose counts and age, not just request success: pending intents past expiry, active records without objects, objects without records, deletion outcomes, and the oldest unresolved deletion. Keep object keys out of ordinary user-facing logs when they reveal account structure; use a correlation identifier instead.

The cache layer needs a separate test. A private document response should have directives consistent with its authorization model, and a response that must not be reused should say so explicitly. The MDN reference for Cache-Control distinguishes request and response directives and their caching implications. None of those directives replaces an origin deletion job.

A rollout that keeps the blast radius small

Begin with one retention class and one non-sensitive document type. Create upload intents, keep objects private, and make finalization observable before adding automatic deletion. Run a dry-run reconciler that reports candidates without deleting them; compare its results with database records for several days. This reveals abandoned uploads and stale rows while recovery is easy.

Then enable deletion for newly created objects only. Preserve an audit record containing the reason, policy version, object reference, attempt count, and final outcome. Do not use the object store as the sole audit database, because provider-side lifecycle actions may lack the business context support and compliance teams need.

My decision rule is narrow: choose the storage option whose authorization model and object semantics your team can explain in a deletion incident, then place retention policy in application-owned records and reconciliation code. The provider can move bytes and enforce coarse boundaries. It should not be the only place where the business decides what may remain.

DEV Community