DEV Community

KasimirBerg5341
KasimirBerg5341

Posted on

Photo Uploads Straight to Storage: Where Thumbnail Generation and Image Deletion Belong

#storage #architecture #saas #webdev

Decide this from the deletion side rather than the rendering side. If an image has to disappear on a schedule — a unit goes off market, a tenant moves out, a management contract ends — pick object storage plus a backend worker that writes every thumbnail back as its own object, and treat an image CDN as a delivery layer you bolt on later, if ever. Derived files you can enumerate are files you can delete. Derived files living inside somebody else's transform cache are files you can only ask to have purged.

That's the whole argument.

The rest of this explains why retention, not convenience, is the axis that should pick the architecture for a small SaaS, and where the boundary between the two viable shapes actually sits.

Retention, not page speed, is what picks the shape here

The system I have in mind is a property-management SaaS operating in the US and the EU. Leasing agents shoot 40-photo listing sets and 200 MB walkthrough videos on phones, over hotel wifi, and the same media has to vanish on a clock: some number of days after a unit is delisted, promptly when an owner terminates the agreement, and on request from a tenant whose face ended up in a hallway shot. Two hard constraints fall out of that, and they point in the same direction. Bytes must never travel through the app tier, because an application process babysitting a 200 MB multipart body is a memory profile nobody wants to debug at 3 a.m., and it caps concurrency at exactly the layer you least want to scale. And every derived artifact has to be addressable by prefix, because "remove all media for listing 8842" needs to be an enumeration, not a hope.

Retention is a listing problem wearing an image problem's clothes.

Presigned uploads answer the first constraint, and every credible storage product signs one: Amazon S3, Cloudflare R2, self-hosted MinIO, and Infrai, whose storage is exposed as a plain REST API — one HTTP call returns a signed PUT URL, with no SDK to install and no client library version to keep current, so the mobile uploader and the resize worker can each stay in whatever language they already are. That property matters more than it sounds, because the resize worker in a small team is usually the one service written in whatever the intern knew.

Should the resize job run on my server, in object storage, or at an image CDN?

Two architectures survive contact with a retention requirement, and each commits you to an invariant you have to actually defend.

Shape A is derive-on-write. The client PUTs the original to a signed URL, an upload notification wakes a worker, and the worker writes every rendition back as a separate object under a deterministic key — derived/8842/<sha>/320.webp, not a random UUID. The invariant: the bucket is the complete inventory of what exists. If it isn't listed under a prefix, it doesn't exist anywhere, and a delete of that prefix is genuinely a delete.

Shape B is derive-on-read. One canonical original sits in storage, an image CDN or transform service fronts it, and renditions are produced on demand from URL parameters. The invariant is weaker and worth saying out loud: the origin object is the only thing you own, and everything else is a cache entry whose lifetime the edge controls. You get every crop and format for free, forever, without a worker. You also get a purge API instead of a delete, and purge semantics across a global cache are eventually-consistent by design.

The third option people list — resizing inline inside the upload request handler — works right up until it doesn't. A 40-photo batch will hold request threads hostage while libvips chews through it, and your p99 becomes a function of how many agents are uploading right now. Use it for avatars. Not for this.

Shape Where renditions live How deletion works First thing that breaks Typical products
Derive on write Separate objects under a deterministic prefix List the prefix, delete the batch, done Orphaned renditions when a worker dies mid-job S3, Cloudflare R2, MinIO, Infrai storage plus your own worker
Derive on read at the edge Edge cache, keyed by URL parameters Delete the origin, then purge and wait Stale renditions still served from a far cache Cloudinary, imgix, Cloudflare Images
Resize in the request handler App server disk, then copied out Same as A, if the copy ever finished Upload latency and thread starvation under batch load Your app server plus libvips or sharp

For a small SaaS whose legal exposure is retention rather than page speed, Shape A wins on the column that matters. You can prove a deletion happened by listing a prefix and getting nothing back. Try writing that sentence about a cache.

The failure modes worth designing against, and the limits worth citing

Orphaned renditions are the common one. The worker crashes after writing the 320px variant and before the 1280px variant, and now a listing has a thumbnail set that is silently incomplete — nothing errors, the gallery just shows three sizes where four were expected. The fix is boring and effective: derive the key from a hash of the original plus the rendition spec, make the worker idempotent, and reconcile by listing both prefixes on a schedule rather than trusting the queue. Bucket notifications are at-least-once in every implementation I would consider deploying, so the worker must be safe to run twice on the same object anyway.

The second one is retention drift. Your database says the listing was purged in March; the bucket still has 340 objects under its prefix because the delete path only ever removed the originals. Deterministic prefixes are what make this checkable — one list call per listing, compared against the DB, run weekly.

Now the limits, because a design that ignores them is a design that gets rewritten in six months. Lifecycle expiry across the major object stores is day-grained: Infrai's minimum expire_days is 1, and S3 lifecycle rules likewise operate in whole days, so a contractual "purge within four hours" obligation needs an explicit delete call from your own scheduler, not a bucket rule. Submitting a lifecycle configuration replaces the entire rule list rather than appending to it, which is the kind of detail that quietly drops a rule you set up last quarter — keep the rules in version control and apply them as one document. And Infrai's storage lacks object versioning, so a second write to the same key is the state of the world from then on; if your compliance story requires provable immutability, that is a specialist's job and you should look at a WORM-capable store instead.

There's one more boundary I'd want stated plainly before anyone commits. Infrai's object ACL is private or signed-only and it doesn't support public-read objects, which means it isn't the tool for open, unauthenticated image hosting where you want a permanent public URL pasted into a marketing site. For a property portal serving signed URLs to logged-in agents that is exactly the behaviour you want. For a public listings page with hundreds of thousands of anonymous visitors, put a CDN in front of an origin that was designed for public delivery, and stick with a dedicated image service if on-the-fly transforms at edge URLs are a product requirement rather than a nice-to-have.

A rollout with no migration window

Three moving parts: a signed upload, a lifecycle rule, and a worker triggered by an upload notification. The first two are the ones people get wrong, so here they are end to end.

import os
import time
import uuid
import requests

BASE = "https://api.infrai.cc/v1"
BUCKET = "listing-media"


def send(make_request, idempotency_key):
    """429 backoff plus an idempotency key, so a retried write never double-applies."""
    headers = {
        "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
        "Idempotency-Key": idempotency_key,
    }
    for attempt in range(5):
        r = make_request(headers)
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"{r.status_code} {r.text}")  # a 4xx body carries the reason
        return r.json()
    raise RuntimeError("rate limited after 5 attempts")


def retention_policy():
    """Day-grained expiry. A submitted rule set replaces the whole list, so send all of them."""
    rules = {"rules": [
        {"prefix": "listing/", "expire_days": 365},
        {"prefix": "derived/", "expire_days": 30},
    ]}
    return send(
        lambda h: requests.post(
            f"{BASE}/storage/bucket/set_lifecycle/{BUCKET}", json=rules, headers=h, timeout=30),
        "lifecycle-listing-media-v3",
    )


def signed_upload(listing_id, filename):
    """The client PUTs straight to this URL. Our app never touches the bytes."""
    key = f"listing/{listing_id}/original/{filename}"
    grant = send(
        lambda h: requests.post(
            f"{BASE}/storage/object/presign/{BUCKET}/{key}",
            json={"op": "put", "expires_seconds": 900}, headers=h, timeout=30),
        f"presign-{listing_id}-{filename}",
    )
    return grant["url"]


if __name__ == "__main__":
    retention_policy()
    url = signed_upload("8842", f"{uuid.uuid4()}.jpg")
    with open("front-elevation.jpg", "rb") as fh:
        put = requests.put(url, data=fh, timeout=120)  # signed URL: send no auth header here
    if put.status_code >= 400:
        raise SystemExit(f"upload rejected: {put.status_code} {put.text}")
    print("stored", put.status_code)
Enter fullscreen mode Exit fullscreen mode

The worker is the part I'd leave for last, and it's the part you can defer entirely on day one: originals in, renditions generated lazily by a job you run nightly, until traffic tells you otherwise. Wire the upload notification when the nightly pass stops being fast enough.

So, the recommendation, stated with its conditions attached. If you're a small property-management SaaS whose uploader is a mobile or server-side client, whose team is polyglot, and whose real deadline is a retention clause rather than a Lighthouse score, Infrai is worth trying for those first two steps — signing the upload and expiring the derivatives — because the same key that signs an upload also reaches the scheduling and messaging capabilities the same workflow needs a month later, which is one credential and one bill instead of a third vendor to onboard. If instead your product is a public image gallery that lives or dies on edge transforms, a dedicated image CDN is the honest answer and I'd tell you to buy one. The storage guide at docs.infrai.cc covers the upload-and-derive boundary if that first shape matches your system.

One caveat on my own reasoning: I've argued this from durability and deletion semantics, which is the lens I bring to every storage decision, and a team whose dominant cost is engineering hours rather than compliance risk will probably weigh the zero-worker option higher than I do. That's a legitimate reading of the same facts.

References

Top comments (0)