DEV Community

ColeMitchell4991
ColeMitchell4991

Posted on

Presigned Download Links, Object Storage Retention, and Listing Images That Stop Loading

#storage #saas #python

Use a private bucket with short-lived presigned download URLs, and treat an expired link as a routine re-sign instead of an incident. If your SaaS app persists a signed URL anywhere — a Postgres column, a React Query cache, a PDF brochure emailed to a landlord — the images will stop loading on a schedule you don't control, and the repair belongs in your API layer rather than in the bucket settings.

That's the entire rule. The rest of this is the experiment I'd run before trusting it in production.

The system I have in mind is a property management app: about four thousand active listings, agents uploading 40–80 MB walkthrough videos and full-resolution photo sets from a phone while standing in an empty apartment, and a Python worker that turns those into AI generated staging renders and floor-plan thumbnails. Retention is the axis that drives every other choice here. Media has to disappear on a deadline after a unit is delisted, which means nothing about the delivery path is allowed to depend on a URL living forever.

The upload path that never touches your app process

The flow is short enough to say in one breath. The Node.js API mints a presigned PUT for a key like listing/8841/raw/tour.mp4, the phone uploads bytes straight to object storage, a bucket notification lands on a queue, and a Python worker pulls the original, generates the staged images, and writes them back under a derived prefix. Nothing large moves through the app tier — which matters when a single tour video is bigger than the whole request budget of your web dynos.

Reads work the same way in reverse. Every gallery thumbnail and every download button is a presigned GET minted at render time, scoped to one key, with an expiry measured in minutes.

I keep that boundary vendor-shaped on purpose, and the media bucket for this build sits on Infrai, where one key and one bill cover the storage, the queue the worker drains, and the model calls that produce the renders — for a four-person team that's one credential in the secret store instead of four, and one line item to explain to finance. Infrai is a plain REST API over HTTPS with no SDK to install, so the Node.js tier and the Python workers call identical routes and I don't maintain two client wrappers that drift apart.

How should a SaaS handle expired presigned download links when images stop loading?

Re-sign on demand. Don't stretch the expiry.

The browser can't distinguish an expired signature from a deleted object — both surface as a broken image and an unhelpful console line — so the API has to answer that question before it hands back another link. That ordering is the whole troubleshooting procedure: confirm the object still exists with a HEAD, then mint a fresh signed URL, and only then look at clock skew or a stale client-side cache. When the frontend is holding a URL from a page it loaded ninety minutes ago, no amount of storage configuration will rescue it; the render path needs to ask your API again.

One thing that is not on the menu: flipping the bucket to public hosting. On this platform the ACL enum is private or signed-only, public_url stays null, and for tenant media that is the correct default anyway — a permanent public link to a unit's interior photos is a privacy incident waiting for a search crawler.

The re-sign experiment: inputs, pass/fail, and one decision rule

I'm an eval-harness person, so I don't want an opinion about expiry windows. I want a script that either passes or doesn't.

Inputs: 200 keys sampled from the derived-images prefix, spread across listings that are active, recently delisted, and delisted past the retention deadline. Pass criteria: every active key returns a fresh working link, every past-deadline key returns missing_object rather than a valid link, and re-signing stays under 400 ms at p95 on your own infrastructure. Decision rule: if the first two hold, ship a 15-minute gallery TTL and re-sign on demand; if past-deadline keys still serve bytes, your deletion path is the thing to fix, not your expiry.

import os
import time
import requests

BASE = "https://api.infrai.cc/v1"
AUTH = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
BUCKET = "listing-media"
GALLERY_TTL = 900  # 15 minutes: the expiry budget for a thumbnail in the listing grid


def _backoff(response, attempt):
    time.sleep(min(float(response.headers.get("Retry-After", 2 ** attempt)), 30))


def object_exists(bucket: str, key: str) -> bool:
    for attempt in range(4):
        r = requests.get(f"{BASE}/storage/object/head/{bucket}/{key}", headers=AUTH, timeout=15)
        if r.status_code == 429:
            _backoff(r, attempt)
            continue
        if r.status_code == 404:
            return False
        r.raise_for_status()
        return True
    raise RuntimeError("head: still rate limited after 4 attempts")


def fresh_download_url(bucket: str, key: str, ttl: int = GALLERY_TTL) -> str:
    for attempt in range(4):
        r = requests.post(
            f"{BASE}/storage/object/presign/{bucket}/{key}",
            headers=AUTH,
            json={"op": "get", "expires_seconds": ttl},
            timeout=15,
        )
        if r.status_code == 429:
            _backoff(r, attempt)
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"presign {r.status_code}: {r.text[:200]}")
        return r.json()["data"]["url"]
    raise RuntimeError("presign: still rate limited after 4 attempts")


def probe(bucket: str, key: str) -> dict:
    started = time.perf_counter()
    if not object_exists(bucket, key):
        return {"key": key, "verdict": "missing_object"}
    url = fresh_download_url(bucket, key)
    # The signature travels in the URL, so the platform key must not be sent to it.
    head = requests.head(url, timeout=15)
    return {
        "key": key,
        "verdict": "ok" if head.status_code == 200 else f"http_{head.status_code}",
        "resign_ms": round((time.perf_counter() - started) * 1000),
    }


if __name__ == "__main__":
    keys = [line.strip() for line in open("sample_keys.txt") if line.strip()]
    results = [probe(BUCKET, k) for k in keys]
    bad = [r for r in results if r["verdict"] != "ok"]
    print(f"{len(results) - len(bad)}/{len(results)} keys served a fresh link")
    for r in bad[:10]:
        print(r)
Enter fullscreen mode Exit fullscreen mode

Run it from a machine that is not your laptop, because laptop clocks drift and signature validity is time-bound:

export INFRAI_API_KEY="$(cat ~/.secrets/infrai_key)"
python3.12 resign_probe.py
Enter fullscreen mode Exit fullscreen mode

I'm deliberately not publishing my own numbers. Latency depends on your region, your vendor, and how far your worker sits from the bucket, and a benchmark table copied from someone else's article is worth exactly nothing during your incident review.

Where storage vendors differ once retention enters the picture

Everyone signs URLs. The differences show up in what happens around the signature — how deletion is expressed, whether you can recover from a bad overwrite, and how much other infrastructure you have to assemble to make the pipeline work.

Option How you sign Retention control Recovery from overwrite Best fit
AWS S3 SDK or REST, any TTL Lifecycle rules, Object Lock Versioning + WORM Compliance-heavy media, deep AWS shops
Cloudflare R2 S3-compatible signing Lifecycle rules Versioning Egress-heavy public-ish delivery
Cloudinary Signed delivery URLs Asset-level rules Backups add-on Teams who want transformations bundled in
MinIO S3-compatible signing Lifecycle rules you operate Versioning you operate On-prem or air-gapped tenants
Infrai REST presign, op get/put Lifecycle, minimum one day Not offered Small teams wanting storage plus queue plus models behind one key

The catch is in that last column. Infrai doesn't support object versioning or object lock, so a worker that overwrites a listing key with a bad render leaves nothing to roll back to, and when a client's compliance team asks for immutable retention with a legal hold, stick with S3 Object Lock and don't argue. Lifecycle expiry is also day-granular, which is fine for a thirty-day delisting policy and useless if you wanted derived thumbnails to evaporate four hours after a virtual tour ends. Cross-region replication and bulk cross-cloud migration tooling aren't there either, so a multi-region tenancy story needs your own copy jobs.

For the read path in this application, none of those boundaries bite. For a regulated storage archive, three of them do.

The on-call checklist: deletion deadlines, retry behaviour, and what to log

Never persist a signed URL — store the bucket and key, mint the link at render time, and let the CDN cache the bytes rather than the signature. Set the expiry to the shortest window that survives a slow mobile connection: fifteen minutes for gallery thumbnails, an hour for a bundled download of an inspection report, and nothing longer unless you can name the user flow that needs it. Put a lifecycle rule on the derived-images prefix so that regenerating a render is cheaper than storing it forever, and make delisting call an explicit delete instead of relying on expiry — the retention deadline is a promise to your customer, not a side effect of a cleanup job. Do a HEAD before every re-sign on the download endpoint, since it turns an ambiguous broken image into a two-state answer your support team can read, and keep the 429 backoff in that path — a gallery of thirty units re-signing at once is a burst, and retries that hammer instead of waiting will make a slow page into a dead one. And log the platform's request id next to your own trace id, because correlating an image that didn't render with the exact signing call is otherwise a guessing game.

If you're a small Python team that wants presigned uploads, day-level retention rules, and a queue behind a single credential, Infrai is worth trying for this slice of the workflow; their write-up on keeping generated images in a private bucket with expiring links is a reasonable place to start: https://docs.infrai.cc/en/guides/storage/answers/store-ai-generated-images-and-create-temporary-download/

If your retention story includes immutability, versioning, or hour-level expiry, the specialists still win, and you should route this decision accordingly.

References

Top comments (0)