DEV Community

mT41Gzp73rc6
mT41Gzp73rc6

Posted on

Avatar Metadata Retention: Deletion Proof Beyond Prefix-Scoped Object Listings

Short answer: keep avatar metadata and retention state in a queryable database, and treat object storage listing by prefix as a reconciliation input rather than a search engine. For a B2B SaaS product-image service that creates tenant-scoped exports, this split makes deletion provable even when the bucket can only list keys under a prefix.

The constraint is easy to miss. A key such as tenants/acme/avatars/u_42/original.webp tells you where bytes live; it does not reliably answer “which objects are image/webp and tagged profile?” Content type and tags belong in the metadata record you control. The object remains the payload.

What should a tenant avatar export query before object listing?

Start with the business query. An export request might ask for active avatars uploaded by tenant acme during a date window, restricted to image/webp, and labeled profile. SQL can filter those fields, enforce tenant ownership, and return a stable set of object keys. Only then should a worker fetch bytes or issue short-lived download URLs.

It is tempting to treat a prefix as if it were an index. An export can look fine, then include a deleted avatar because a listing was cached for a few minutes. That is a data-lifecycle mistake, not a storage outage. The repair is a metadata row with deleted_at, an object generation, and an export snapshot ID.

Use a deliberately boring record:

Field Purpose Deletion implication
tenant_id, avatar_id Isolation and identity Every delete is scoped and auditable
object_key, generation Exact payload reference Prevents deleting a newer replacement
content_type, tags Search and policy filters No full bucket scan for metadata
created_at, deleted_at Retention clock Makes eligibility deterministic
export_id, state Snapshot membership A running export can be cancelled safely

The object key can still be prefix-friendly. Put tenant and lifecycle state in the key for operational scans, but do not pretend that key order encodes every predicate. Tags are useful at the object boundary for policy checks; they are a poor substitute for a tenant-aware query index when a request spans many tags or time ranges.

Keep bytes dumb.

How can content type, tags, and prefix limits shape deletion evidence?

Deletion needs two independent facts: the database says an object is eligible, and storage confirms the exact version is gone. Record both events under one request ID. A successful DELETE response is an HTTP result, not proof that a later list will immediately be empty; cache behavior and listing consistency rules still deserve a test against your chosen store.

Here is the control flow I use in a worker. It never deletes by a broad prefix, and it refuses to act when the generation it read is no longer current.

from dataclasses import dataclass
from datetime import datetime, timezone


@dataclass(frozen=True)
class AvatarObject:
    tenant_id: str
    avatar_id: str
    object_key: str
    generation: str
    deleted_at: datetime | None


def deletion_candidates(rows: list[AvatarObject], now: datetime) -> list[AvatarObject]:
    return [
        row for row in rows
        if row.deleted_at is not None and row.deleted_at <= now
    ]


def delete_one(store, row: AvatarObject, request_id: str) -> None:
    # The storage adapter must use an exact-generation precondition.
    store.delete(
        key=row.object_key,
        generation=row.generation,
        request_id=request_id,
    )
Enter fullscreen mode Exit fullscreen mode

The adapter should emit delete_started, delete_confirmed, and delete_rejected events with the tenant, object key hash, generation, policy version, and retention deadline. Do not log the image URL or user-supplied tag values verbatim; those strings often contain emails or internal ticket numbers. Keep the audit event longer than the bytes, but document that split in the retention policy.

A prefix listing still has a job. Run it as a reconciler that finds unreferenced keys, stale multipart remnants, or objects whose metadata row was never committed. In a dry run, the useful output is a manifest of keys with no matching row; some belong to uploads whose database transaction has not committed yet, some are test fixtures, and the rest need an owner. That is why reconciliation should quarantine a mismatch first, attach the first-seen timestamp and source prefix, and require an explicit policy decision. Deleting every key not seen in SQL is dangerous during an in-flight upload or a replica delay, and a retrying worker can make the same orphan appear in several scans unless the reconciler stores its own observation ID.

Which storage boundaries are unsuitable for avatar search and exports?

Prefix-only listing is unsuitable as the primary metadata search when customers can define arbitrary tags, when exports must be repeatable, or when deletion has legal or contractual deadlines. Choose a database-backed index and a queue in those cases. Keep a direct listing workflow for small maintenance jobs where the prefix is the actual policy boundary and the operator can tolerate an eventually reconciled result.

The catch is operational ownership. A richer metadata index adds migrations, backfills, and privacy review. A storage-only design has fewer tables but pushes query semantics into brittle scans and makes “deleted” hard to prove. Your mileage may vary with very large tenants; partition metadata by tenant and time, and rate-limit export workers so one account cannot monopolize listing or download bandwidth.

The comparison is about boundaries, not brand names:

Design Good fit Limitation to state plainly
SQL metadata plus object payloads Tag search, repeatable exports, deletion audits Requires schema ownership and backfill jobs
Object tags plus prefix listing Simple policy checks and small buckets Cross-tag queries and history are awkward
Search index plus object storage Fast faceted discovery at scale Index lag must be reconciled before deletion

Do not use object count as the success metric. Measure export snapshot drift, deletion lag, orphan count, and the percentage of deletes with a matching generation. A five-minute dashboard gap can be acceptable; an unbounded orphan queue is not.

A rollout that makes retention testable

First, dual-write metadata for new uploads while the existing path remains authoritative. Backfill in tenant-sized batches, recording the source prefix and a checksum for every discovered key. Then run exports in shadow mode: produce a manifest and compare it with the current result without sending it to a customer. I would fail the rollout on any cross-tenant key, even if the total count matches.

Next, enable deletion for one tenant with a short test retention window. Kill a worker after the database transaction, after the storage delete, and before the audit event. The retry must converge on the same generation and request ID; a replacement avatar must survive because its generation differs. This is the edge case that usually exposes an over-broad delete(prefix) helper.

Finally, publish the contract to support and compliance: what “deleted” means, how long audit metadata remains, and how a legal hold pauses eligibility. Keep the API surface small. The policy is the product; the bucket is an implementation detail.

References

Top comments (0)