DEV Community

MagnusNilsson2124
MagnusNilsson2124

Posted on

Generated OpenAI Stable Diffusion Images in SaaS: Object Storage, Database Blob, Local Disk

Decision rule: keep OpenAI and Stable Diffusion image bytes in private object storage, keep ownership and lifecycle state in Postgres, and treat local disk as a short-lived work area. That split is the simplest production shape for a SaaS that must issue expiring download links without turning every image read into a database transaction.

Short answer: an object key is the durable identity, a database row is the authorization record, and a short-lived signed response is the delivery mechanism. Store the generated bytes once, record a content hash and provider metadata, and never make a pod's filesystem the source of truth.

What should a Python SaaS do with OpenAI and Stable Diffusion images?

The image generator is not the storage policy. An API response can be retried, duplicated, or delivered after a worker restart. Give each generation request an idempotency key, create a pending row, and only mark it ready after the object write has been verified. This matters for a ten-second preview and a 300 MB training render alike.

The access boundary comes before the bucket choice. A tenant check belongs in the application, while the object remains private. At download time, authorize the row, create a link that expires soon, and send a Content-Disposition value that makes the browser behavior explicit. The link is disposable; the object key is not.

One sentence is enough for the failure boundary.

It is easy to lose the bytes.

If a worker dies after writing bytes but before the database update, a reconciler can find the pending row by its deterministic key. If the database update happens first, the row must remain unavailable until a HEAD or equivalent existence check succeeds. I have seen this race hide behind a green job metric: the queue said “done,” while the first customer request found no object. The fix was a state transition with a verification step, not another retry around the model call. In practice, the reconciler should scan by tenant and age, then compare the row's expected media type, byte length, and checksum with object metadata before changing state. A retry can safely repeat a PUT when the key is deterministic, but it must never silently replace a ready render with a later duplicate. Keep an audit event for each transition, because support will eventually ask why a link expired, why a prompt was redacted, or why two jobs point at one object. That audit trail is small compared with the image payload and gives compliance a place to reason about deletion requests.

The invariants that survive provider changes

Use a key such as tenant/{tenant_id}/renders/{generation_id}.png; do not put an email address or raw prompt in it. Keep the original prompt and model identifier in the database, where they can be redacted, indexed, and covered by retention policy. The bytes need durability and streaming; the row needs relational queries.

Location Good fit Boundary to accept
Private object storage Large images, range reads, expiring links You must operate lifecycle and key policy
Database BLOB Small thumbnails that must commit with a row Backups and replication grow with every byte
Local disk Temporary model output and image transforms Instances can be replaced without warning

Database BLOBs are reasonable for a small, bounded thumbnail whose transaction semantics are more valuable than cheap streaming. They become awkward when a tenant can keep thousands of full-resolution renders, because vacuum, backup windows, and replica lag now track media volume. Local disk is useful between a model process and the upload client. It is not an archive.

The other invariant is replay safety. A repeated worker message must address the same generation ID and object key; an overwrite policy must be deliberate. A hash lets a reconciler distinguish the expected bytes from a stale upload, and a ready row should include byte length and media type so a download handler does not infer them from an untrusted filename.

A critical path with explicit state transitions

The following Python sketch keeps provider calls behind a tiny interface. It is intentionally boring: the interesting part is when state changes, not which SDK is fashionable.

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone


@dataclass
class Render:
    tenant_id: str
    generation_id: str
    object_key: str
    state: str = "pending"


def persist_render(db, objects, render: Render, content: bytes, media_type: str):
    # The row exists before the worker attempts a write.
    db.insert_pending(render.tenant_id, render.generation_id, render.object_key)
    objects.put_private(render.object_key, content, content_type=media_type)

    metadata = objects.head(render.object_key)
    if metadata.size != len(content):
        raise ValueError("stored length does not match generated length")

    db.mark_ready(
        render.generation_id,
        byte_length=metadata.size,
        media_type=media_type,
        completed_at=datetime.now(timezone.utc),
    )


def download_link(db, objects, tenant_id: str, generation_id: str):
    row = db.authorize_ready_render(tenant_id, generation_id)
    if row is None:
        raise LookupError("render is unavailable")
    expires = datetime.now(timezone.utc) + timedelta(minutes=5)
    return objects.presign_get(row.object_key, expires_at=expires)
Enter fullscreen mode Exit fullscreen mode

The production version also records a cleanup deadline. A sweeper aborts abandoned multipart uploads, removes objects whose pending row has expired, and emits a metric for each reconciliation decision. Keep those metrics separate from model latency; otherwise storage drift gets mistaken for an inference problem.

Where the simple design stops being simple

The catch is compliance and transformation. If every image must be virus-scanned, perceptually hashed, or held for legal review, a direct “write then link” path is too permissive. Put a quarantine prefix in front of the ready prefix and allow downloads only after the inspection state is recorded.

The design is also not suitable for public, permanent image hosting, multi-region active-active writes, or analytics that scan pixels with SQL. Choose a storage system with public caching, replication, or queryable media metadata when those are first-order requirements. Stick with a database BLOB for genuinely small assets that must share an atomic commit, and stick with local disk only for disposable intermediate files.

I am not sure a single expiration value works for every SaaS. Five minutes is a useful default for a user download, while a background export may need an hour; measure abandonment and adjust the policy. Your mileage may vary by tenant size and browser behavior.

The simplest path is therefore a contract: private object storage owns bytes, Postgres owns identity and authorization, and workers reconcile the gap. Once that contract is tested with duplicate jobs, interrupted uploads, and expired links, changing providers is a bounded adapter task instead of a data migration disguised as a refactor.

References

Top comments (1)

Collapse
 
alexshev profile image
Alex Shev •

Image-storage architecture should be chosen with a deletion and access story, not just write latency. I’d keep immutable object keys behind tenant-scoped database records, use short-lived signed reads, and run reconciliation for orphaned originals, thumbnails, and failed-generation artifacts. That is what keeps a convenient image feature from becoming an unbounded retention system.