DEV Community

zanesterling7589
zanesterling7589

Posted on

Deletion Deadlines and Tenant Isolation in Private Object Storage for Signed Uploads

#objectstorage #gdpr #architecture #security

Use one private bucket per tenant, and let the explicit deletion deadline — not the upload flow — decide the rest of your object storage layout. Browser-direct uploads with short-lived signed URLs are the easy half of this architecture; the hard half is being able to say, on a specific calendar date, that a file is gone from every place your platform ever put it.

The system I have in mind is a document vault for game studios: e-signed contractor NDAs, tournament waivers, and guardian consent forms for players under 16. A studio admin uploads the PDF straight from the browser, the app users who may read it afterwards are four or five people inside that one studio, and every document arrives with a retention clock that ends on a date rather than "eventually".

That date is the design driver, and it is the reason I don't treat tenant isolation as a later refactor.

How should a beginner design private file upload and signed download for app users in the US and EU?

Split the work into a control plane and a data plane, and keep every policy decision in the control plane. The app authenticates the studio admin, resolves which tenant they belong to, allocates an opaque key inside that tenant's bucket, writes a pending row that already carries the deletion date, and only then asks storage to sign a short-lived upload URL. The browser sends the bytes directly to that URL. Your server never touches the file body, which is the entire point of the pattern.

Download is the same decision made again, later: authenticate, re-check ownership, re-check state, then mint a fresh signed URL with a lifetime measured in seconds. A signed URL is a bearer capability. Anyone holding it can use it until it expires, no matter who they are, so it belongs nowhere durable.

Beginners usually get that skeleton right and skip the two unglamorous parts — what the grant is scoped to, and what happens to the object after nobody needs it. OWASP's file upload guidance covers the first gap: validate the real file type instead of trusting the declared content type, rename the file to something you generated, cap the size, and re-apply authorization on every retrieval rather than treating a successful upload as proof of anything. The secrets guidance covers the second: the long-lived storage credential stays server-side, never in the browser bundle, and never in a config file shipped inside a desktop game launcher.

For the US/EU question specifically, two GDPR articles do most of the architectural work. Storage limitation (Article 5) says personal data is kept in identifiable form no longer than necessary, which is what turns a vague retention policy into a delete_after column. The right to erasure (Article 17) says a subject can ask for deletion before that date, which means your delete path has to work on demand and not only on a schedule. I'm not going to pretend an architecture diagram makes you compliant — residency, processor contracts, and lawful basis are legal questions — but a layout where deletion is a bucket-scoped operation is far easier to evidence than one where it's a LIKE 'tenant-7/%' query someone has to get right.

What an explicit deletion deadline demands from the storage layout

Start with the invariant: the database is authoritative for identity and lifecycle, and object storage holds bytes that a row points at. Every document row carries a tenant id, an opaque key, a state, a delete_after date, and a legal-hold flag. If a row can exist without an object, or an object without a row, your deletion deadline is already unenforceable, because the thing you scan on the deadline is the database and the thing that actually holds personal data is the bucket.

Now the failure modes, which are the part I actually care about.

Orphan objects are the common one. An upload grant is issued, the browser transfer completes, the callback that flips the row to available is lost, and a cleanup job later deletes the pending row. The PDF stays in the bucket with no row pointing at it, no deletion date attached to it, and no dashboard counting it — a signed waiver for a 15-year-old player sitting in storage indefinitely because a state machine dropped one transition. The fix is a reconciler that lists the bucket and compares against the table in both directions, plus a metric for objects older than the longest retention class. Bucket-per-tenant makes that listing cheap and bounded; a shared bucket with fifty thousand tenants makes it a paging exercise you'll be tempted to skip.

Prefix confusion is the one that bites teams who chose a shared bucket. Grants scoped to a prefix like tenant-1/ will happily match tenant-12/ unless every scope terminates with the delimiter and every key generator refuses to emit a tenant id that is a prefix of another. Numeric tenant ids make this worse. It's a one-line mistake with a cross-tenant blast radius, and it doesn't show up in tests unless someone writes the negative test where tenant A asks for tenant B's key and expects a 403.

Leaked download links deserve their own paragraph in a gaming context, because game clients are unusually chatty: crash reporters attach recent HTTP activity, support workflows encourage players to paste URLs into Discord, and community tooling scrapes anything that looks like a link. Keep download grants at a minute or less, keep them out of analytics events and application logs, and accept that the short lifetime is the only control you have once the URL has left your process.

Region drift is quieter. A control plane with a single default bucket will silently place an EU studio's consent forms in a US region the first time someone adds a tenant through a script that skips the region mapping, and you'll discover it during a data-mapping exercise months later. Making the bucket a required lookup — not a defaulted parameter — turns that into a KeyError at tenant creation, which is where you want it.

Two more boundaries, stated plainly. A legal hold has to outrank the deletion deadline, so the deletion query filters on it rather than the deletion job checking it as an afterthought; getting that precedence backwards destroys evidence. And deletion isn't finished when the object disappears from the bucket: backups, snapshots, and any cross-region copy carry their own clocks, so the deadline you promise a customer is really the deadline of the longest-lived copy. That's the trade-off nobody puts in the sales deck.

Three isolation layouts, compared

The comparison worth having is not about vendors — object storage APIs converge on the same primitives — but about how much work each layout leaves for the deletion deadline.

Layout Deleting on the deadline Blast radius of a leaked grant Where it fits
Shared bucket, tenant prefixes Per-object job; lifecycle rules are coarse and shared across every tenant Prefix-scoped grant; a missing delimiter exposes a neighbouring tenant Few tenants, one uniform retention class
Bucket per tenant Bucket-scoped lifecycle plus a listing small enough to reconcile nightly One tenant, and the bucket name lands in the audit trail Signed documents whose retention differs per contract
Bucket per tenant per region Same, and residency becomes a property of the bucket instead of a config flag One tenant inside one jurisdiction US and EU customers under one control plane

My decision rule for this workload is simple enough to state in one line: if the retention clock differs per tenant, the bucket should differ per tenant too. The catch is real, though — bucket-per-tenant means provisioning and CORS configuration become part of tenant onboarding, buckets often carry per-account limits, and a platform with a hundred thousand small tenants will hit those limits long before it hits a storage bill worth optimising. At that scale the shared bucket comes back and you pay for isolation in code review instead.

The critical path in Python

The code below is the control plane, not the storage adapter. The signing call is behind a two-method protocol on purpose: that boundary is the only provider-specific surface in the whole design, and keeping it that thin is what lets you migrate later without touching authorization logic.

from __future__ import annotations

from dataclasses import dataclass
from datetime import date, timedelta
from typing import Protocol
from uuid import uuid4

RETENTION = {
    "contractor_nda": timedelta(days=730),
    "tournament_waiver": timedelta(days=365),
    "guardian_consent": timedelta(days=1095),
}


@dataclass(frozen=True)
class Document:
    doc_id: str
    tenant_id: str
    bucket: str
    key: str
    kind: str
    state: str
    delete_after: date
    legal_hold: bool = False


class GrantSigner(Protocol):
    """The only provider-specific surface. Two methods, no more."""

    def sign_put(self, bucket: str, key: str, ttl_seconds: int) -> str: ...

    def sign_get(self, bucket: str, key: str, ttl_seconds: int) -> str: ...


class DocumentVault:
    def __init__(self, signer: GrantSigner, buckets: dict[str, str]) -> None:
        self.signer = signer
        self.buckets = buckets          # tenant_id -> its own private bucket
        self.docs: dict[str, Document] = {}

    def begin_upload(self, tenant_id: str, kind: str, signed_on: date) -> tuple[str, str]:
        bucket = self.buckets[tenant_id]   # KeyError here beats a defaulted bucket
        doc = Document(
            doc_id=str(uuid4()),
            tenant_id=tenant_id,
            bucket=bucket,
            key=f"{uuid4()}.pdf",          # opaque: no player name, no studio slug
            kind=kind,
            state="pending",
            delete_after=signed_on + RETENTION[kind],
        )
        self.docs[doc.doc_id] = doc
        return doc.doc_id, self.signer.sign_put(bucket, doc.key, ttl_seconds=300)

    def download_url(self, doc_id: str, tenant_id: str, today: date) -> str:
        doc = self.docs[doc_id]
        if doc.tenant_id != tenant_id:     # tenant check before state, always
            raise PermissionError("cross-tenant read")
        if doc.state != "available" or today > doc.delete_after:
            raise LookupError("document is not retrievable")
        return self.signer.sign_get(doc.bucket, doc.key, ttl_seconds=60)

    def due_for_deletion(self, today: date) -> list[Document]:
        return [d for d in self.docs.values()
                if today > d.delete_after and not d.legal_hold]
Enter fullscreen mode Exit fullscreen mode

Three details carry most of the weight. The deletion date is computed at upload time from the document kind, so the deadline is data rather than a policy someone has to remember. The bucket comes from a required lookup. And due_for_deletion filters the legal hold in the query, which is the precedence rule from the previous section written down where it executes.

The upload grant lives for 5 minutes and the download grant for 60 seconds, which are starting points rather than measurements; your mileage may vary with slow mobile uploads of scanned waivers, and the honest way to pick them is to look at the p99 transfer duration you actually observe rather than at a number in an article. Test the negative path first: a request from tenant B for tenant A's document id must raise before any signing happens, and that test should fail loudly if someone reorders those two checks.

Two things this design deliberately doesn't do. It doesn't attempt versioning or conditional writes, so a retried upload gets a brand-new key and a new row rather than overwriting a mutable one; if you need WORM-style immutability for regulated retention, that's an object-lock feature you should demand from the storage layer instead of emulating in application code. And it doesn't proxy bytes, so any scanning you owe — malware, content policy, PII detection in uploaded chat logs — happens asynchronously after the object lands, with the document staying in pending until it passes.

The layout I rejected, and when it is still right

I rejected the single shared bucket with tenant prefixes, which is the default in most tutorials and the design I'd have shipped a few years ago without arguing. It's cheaper to provision, it needs no per-tenant onboarding step, and it keeps CORS configuration in one place. It also puts every tenant's personal data behind one delimiter convention, turns deletion into a scan, and gives you a single lifecycle namespace to express retention rules that were never uniform to begin with.

Stick with it anyway when tenants are numerous and tiny, when retention genuinely is one global number, or when the storage account's bucket quota makes per-tenant buckets impossible. In that case, buy back the isolation elsewhere: opaque keys with a UUID per tenant, grants that always terminate the prefix with a delimiter, an automated test per release that proves cross-tenant reads are refused, and a deletion job that reconciles both directions instead of trusting the table.

The deadline is the contract you're actually signing with your customers, in 2026 as much as five years ago. Design the storage layout so that meeting it is a listing and a delete, not an investigation.

References

Top comments (0)