DEV Community

MiloHastings5316
MiloHastings5316

Posted on

Signed Leases, Hard Delete Dates: Object Storage Upload and Download for Tenant Files

#objectstorage #fileupload #gdpr #architecture

Use short-lived signed URLs in both directions — one that lets the browser put bytes straight into a private bucket, one that lets an authorized reader pull them back — and keep the deletion deadline in your own database instead of in a bucket rule. The deciding constraint for a property management app is not upload throughput. It's that a countersigned lease has a date on which it must stop existing, and object storage has no idea what a tenancy is.

Application owns the clock. Storage owns the bytes.

The system here is ordinary. Landlords upload countersigned leases and move-out inspection reports, tenants upload ID scans, and every one of those documents arrives with a retention rule stapled to it: a lease is kept for the tenancy plus whatever the local limitation period demands, while an ID scan taken for a one-time identity check should be gone in weeks. EU tenants' files land in an EU bucket and US tenants' files in a US one, because "where does this document physically live" is a question you will eventually have to answer in writing — GDPR Art. 5(1)(e) makes storage limitation a principle, and Art. 17 makes erasure something you have to be able to execute on demand. The beginner version of this architecture fails in a predictable place. Upload works on the first try, signed download works on the first try, and the deletion — the part nobody demos — turns out eighteen months later to have never run at all.

What a deletion deadline does to the upload path

Three invariants, written down before anyone touches a bucket policy.

  1. No object is readable without a fresh authorization decision from the application. A signed download URL is the output of that decision, never a replacement for it.
  2. Every object has a database row: owner, jurisdiction, opaque key, state (pending, ready, purged), and delete_after. Storage lists by prefix; it cannot answer "which documents passed their deadline last night."
  3. Nothing may outlive delete_after, and "nothing" includes noncurrent versions, cross-region replicas, CDN copies, and the access log line that captured the full signed URL.

That third invariant is where otherwise sensible designs quietly break their promise. Turn on versioning and a delete becomes a delete marker — the payload sits there as a noncurrent version, perfectly readable to anyone who can list versions, until a separate expiration rule removes it. Set up replication to a second region for durability and the story gets worse, since version-level deletes are not propagated to the destination bucket and delete marker replication is off unless you switch it on, so the copy you made for resilience becomes the copy that survives the erasure request. Signed URLs leak in their own way: the signature travels in the query string, which means it lands in load balancer logs, in browser history, and in whatever support ticket the tenant pastes it into.

Orphans are the quiet one. If the browser's upload succeeds and the callback into your API is lost — closed laptop, dropped connection, a proxy that gave up at 30 seconds — you are holding bytes with no row, therefore no deadline, therefore an object that is invisible to the process meant to destroy it.

So write the row first, before you hand out the grant, and treat the upload as the second phase of something you already committed to.

How should a beginner wire private uploads and signed downloads for app users?

Four steps, and the order matters more than the code.

  1. The application authorizes the actor against the lease, records a document row in state pending with an opaque key and a computed delete_after, then asks storage for an upload grant scoped to that one key, that one content type, and a size ceiling.
  2. The browser posts the file directly to storage with the returned fields. Your long-lived credentials never leave the server — that's the whole point of a grant, and it's the rule OWASP's secrets guidance keeps repeating.
  3. The browser tells your API the upload finished. The API verifies the object exists and matches the declared size, then flips the row to ready. A pending row older than the grant window is swept, not trusted.
  4. On every later read, the application repeats the authorization check and mints a download URL with a lifetime measured in minutes, plus a response content-disposition so the PDF downloads under a human filename instead of the UUID you stored it as.

Validation belongs in step 3, not step 2, because the browser's declared content type is a hint and nothing more. OWASP's file upload guidance is blunt about this: check the actual bytes, generate your own filename, keep user content off the origin that serves your application. For lease PDFs I'd add one property-specific check — reject anything that isn't a PDF outright rather than trying to sanitize it, since the business has no reason to accept a lease as a ZIP.

Delivery simplicity versus who gets to check the tenant

The axis that actually decides this design is where the access check lives relative to the bytes.

Delivery path Where the tenant is checked What you accept in exchange
Signed PUT/GET straight to the bucket In your app, once, when the URL is minted The URL is a bearer token in a query string; it can't be revoked before it expires, and it will appear in logs
Stream through the application In your app, on every request, every byte You own the bandwidth, the timeouts and the memory profile; revocation and per-read audit become trivial
CDN in front, signed tokens at the edge Split: your app mints, the edge enforces A second cache to purge on deadline day, and a second place a document can outlive delete_after

Signed URLs win for a document that a tenant opens twice a year, which is most of them. Streaming through the app wins when a document is under legal hold and every single read has to be attributable to a named user at a timestamp — you cannot get that from a URL that was handed out once and replayed silently. The edge option is the one I'm most skeptical of here; a cache is a copy, and a copy with its own eviction policy is a copy that isn't governed by your delete clock.

One more limit worth knowing before you pick expiry values: a SigV4 signed URL cannot outlive seven days, and if you sign with temporary session credentials it dies when the session does. That ceiling is fine for downloads. It's a problem if someone proposes emailing a "permanent" link to a landlord.

The critical path, plus the sweep nobody writes first

Grant issuance, with the row committed before the browser is told anything:

import os
import uuid
from datetime import datetime, timedelta, timezone

import boto3

storage = boto3.client("s3", endpoint_url=os.environ["STORAGE_ENDPOINT"])

MAX_BYTES = 25 * 1024 * 1024
GRANT_SECONDS = 300
LEASE_RETENTION = timedelta(days=730)   # tenancy end + 24 months


def issue_upload_grant(db, actor, lease_id):
    lease = db.leases.get(lease_id)
    authorize(actor, lease)                       # ownership decision happens once, here
    bucket = bucket_for(lease.jurisdiction)       # eu-tenant-docs / us-tenant-docs
    key = f"{lease.id}/{uuid.uuid4()}.pdf"

    db.documents.insert(
        lease_id=lease.id, bucket=bucket, key=key, state="pending",
        delete_after=lease.ends_on + LEASE_RETENTION,
        created_at=datetime.now(timezone.utc),
    )

    return storage.generate_presigned_post(
        Bucket=bucket,
        Key=key,
        Fields={"Content-Type": "application/pdf"},
        Conditions=[
            {"Content-Type": "application/pdf"},
            ["content-length-range", 1, MAX_BYTES],
        ],
        ExpiresIn=GRANT_SECONDS,
    )
Enter fullscreen mode Exit fullscreen mode

The reconciler is the half that gets skipped, and it's the half a regulator asks about:

from botocore.exceptions import ClientError

GONE = ("404", "NoSuchKey")


def purge_due_documents(db, now):
    for doc in db.documents.due_for_deletion(now, limit=500):
        listing = storage.list_object_versions(Bucket=doc.bucket, Prefix=doc.key)
        targets = [
            {"Key": v["Key"], "VersionId": v["VersionId"]}
            for v in listing.get("Versions", []) + listing.get("DeleteMarkers", [])
            if v["Key"] == doc.key
        ]
        if targets:
            storage.delete_objects(Bucket=doc.bucket, Delete={"Objects": targets})

        try:
            storage.head_object(Bucket=doc.bucket, Key=doc.key)
        except ClientError as err:
            if err.response["Error"]["Code"] not in GONE:
                raise
        else:
            raise RetentionError(f"{doc.key} still readable after purge")

        db.documents.tombstone(doc.id, purged_at=now)   # keep the row, drop the payload
Enter fullscreen mode Exit fullscreen mode

Two details in there carry most of the weight. Deleting every version by id — rather than issuing a plain delete and getting a delete marker — is what makes the bytes actually go away, and the head_object call afterwards is what turns "we ran a job" into "we verified the object is gone." Keep the row as a tombstone with purged_at; the evidence that you deleted something on time is more useful than the document ever was. Run the same sweep against every bucket holding a copy, because a version-level delete in one region does not travel to the other.

Three things make this operable rather than merely correct. Test it with a frozen clock: seed a document, jump the clock past delete_after, run the sweep, assert the head request comes back gone — deletion is one of the few compliance behaviors that is genuinely easy to unit test, so there's no excuse. Export a gauge for rows where delete_after < now() and purged_at is null, and page someone if it stays above zero for an hour, because a silently wedged reconciler looks exactly like a working one. And keep the sweep idempotent and batched, since the day someone backfills ten thousand expired inspection photos is the day an unbatched job discovers your storage provider's rate limits.

The option I rejected: lifecycle rules as the retention engine

The tempting shortcut is to skip the reconciler and let the bucket expire objects on a schedule. I don't think it holds up for tenant documents, for three reasons that are all about granularity rather than quality.

Lifecycle expiration is expressed in days and applies to a prefix, so it can express "delete inspection photos 90 days after upload" but not "delete this lease 24 months after this particular tenancy ended." It also runs asynchronously — objects are queued for removal after the date, not at a moment you control, which is awkward when the promise you made was a date. Most implementations of the S3 API, MinIO included, share those day-level semantics. And the strongest retention primitive on offer, object lock in compliance mode, deliberately makes deletion impossible until the retention period expires, which is exactly the wrong property when an erasure request arrives for a document that is also under a records-retention obligation. That conflict is a legal question with an engineering consequence, and it's better to discover it while drawing boxes than during a response deadline.

Lifecycle rules are still worth having. Point one at the pending prefix to expire abandoned uploads, and point another at noncurrent versions so the reconciler has a backstop. They are a good broom and a bad clock.

Where would I not build this at all? If the files are meant to be public — listing photos on the marketing site, floor plans in a brochure — the whole private-bucket-plus-signed-URL apparatus is overhead, and a public prefix behind a CDN is the honest answer. If a regulator demands write-once immutability for a fixed number of years, object lock and a documented legal-hold process beat a delete-on-time reconciler, and your erasure workflow turns into paperwork rather than a cron job. And if you're storing three hundred documents total, stream them through the application, skip the signed URLs, and spend the saved complexity somewhere it earns more.

The design that's left is deliberately boring: private bucket, database row that owns the deadline, direct browser transfer, fresh signed link per authorized read, and a sweep that proves its own work. Boring is the goal. Deletion deadlines are one of the few requirements where the failure is invisible until it's expensive.

References

Top comments (0)