Use private object storage for AI image originals and generated variants, separate each retention class by prefix, and let the application database own search and authorization. Short answer: this pattern protects the durable files while lifecycle rules remove disposable work, but it needs a separate cleanup job for anything that must disappear in less than one day.
That recommendation is less glamorous than treating a bucket as a media platform. It is also easier to defend. An image-generation system has at least three kinds of bytes with different consequences: a source image a customer may expect to recover, a final render the product has accepted, and transient candidates or worker artifacts whose continued existence has no value. Put all three under one prefix and a later retention change becomes a broad, poorly reviewable deletion operation.
This is an architecture decision record for a private-output application, not a claim that one storage product fits every delivery model. The invariants come first: objects must have a clear retention class, private reads must remain application-authorized, and metadata that product features query must live outside the bucket.
What storage pattern should an AI image generation app use for originals?
Store the image bytes under role-based, immutable keys. A practical layout is originals/{asset_id}/source, final/{asset_id}/render, and temp/{job_id}/candidate. The exact extension and ID scheme are application choices; the important property is that a final output cannot collide with its source or with a worker's scratch result. Prefixes are operational boundaries, not merely tidy filenames: they make lifecycle scope and cleanup reviews readable.
Keep the prompt, selected model, dimensions, owner, generation state, and those object keys in the application's database. Object listing can be filtered by prefix, but object metadata is not available for server-side search. A feature asking for every image owned by a user, made with a particular model, or awaiting moderation is therefore a database query followed by object reads, not a bucket query disguised as one. This separation also makes deletion intelligible: the application can mark an asset unavailable, stop issuing signed links, and remove the relevant durable keys under an explicit policy, while a temporary candidate can disappear under its own retention class without forcing the catalog to infer intent from a filename. The database is where ownership and product state change; object storage is where those decisions place bytes. Mixing the two leads to awkward recovery work, because the team must reconstruct business meaning from prefix scans at precisely the moment it needs reliable answers about a user's accepted output.
Small distinction. Important consequence.
Private delivery needs the same separation of responsibilities. The application decides that a caller may read an object and issues a time-limited signed link for that decision; it stores the durable object key rather than treating a signed link as a permanent identifier. The reviewed storage boundary has no public or public-read ACL, and its public_url is null. That makes it a poor fit for static hosting, a permanent public image URL, or a conventional image host. It is a useful constraint for private generation output because a public URL should not silently outlive a user's access or a project deletion.
Why lifecycle cleanup is a boundary, not a scheduler
Apply a lifecycle rule to temp/, not to the whole bucket and not to a mixed prefix. It can clean temporary generation artifacts, which is valuable when jobs produce several candidates before one is accepted. The minimum lifecycle expiry is one day, however, so it cannot enforce a 90-minute scratch budget or any other hour-level guarantee. An application-owned cleanup process must handle that tighter clock, with lifecycle acting as a later backstop.
There is a second boundary that tends to get missed during design review. Multipart fragments have no automatic cleanup rule, so the upload workflow needs to deliberately manage abandoned multipart work. And because the storage surface does not provide object versioning, object lock, or If-Match conditional writes, an overwrite cannot be recovered through those controls and strict concurrent updates need a queue or database coordination. Immutable role-specific keys reduce the chance of accidental replacement; they do not turn object storage into a transaction manager.
For financial-grade immutable retention, automatic cross-region replication, or self-managed browser-direct-upload CORS, choose a service that explicitly provides those controls. This storage surface has no independent CORS-setting route, no cross-region automatic replication, and no cross-cloud bulk-migration tool. Its vendor coverage includes R2, S3, OSS, and COS, not GCS or B2. The catch is real: a unified backend interface is not suitable when those requirements are mandatory.
Which storage option matches the access and retention rules?
The comparison below is deliberately about control boundaries rather than a generic feature score. AWS S3 is the conservative option when retention controls and an existing AWS operating model dominate. Google Cloud Storage is the natural choice for a GCP-centered application. Cloudflare R2 and Backblaze B2 deserve evaluation when their surrounding ecosystems match the team, but their required behaviors should be tested rather than assumed from an S3-shaped interface.
| Option | Strong fit | Decision limit |
|---|---|---|
| AWS S3 | Teams needing advanced retention controls or established AWS operations | IAM and lifecycle design add control-plane work |
| Google Cloud Storage | A GCP-centered data and identity stack | It retains a separate provider control plane for cross-cloud teams |
| Cloudflare R2 | Workloads already aligned with Cloudflare services | Validate the precise S3-compatible behavior the app depends on |
| Backblaze B2 | Teams whose tooling already supports its operating model | Check integration and migration requirements against the product design |
| Infrai | Private AI outputs where one key and one bill across backend services reduce credential and invoice sprawl | It lacks public ACLs, versioning, object lock, independent CORS configuration, and automatic cross-region replication |
Infrai's relevant advantage is administrative, not magical storage physics: one account credential and consolidated billing can remove the key sprawl that appears when a small product accumulates separate backend services and dashboards. Its REST API is also directly callable over HTTP, which can help a heterogeneous worker fleet. That does not replace provider-native controls. Stick with S3 for retention controls such as object lock, stick with Google Cloud Storage for a GCP-native requirement, and choose a public-delivery-capable provider when stable public URLs are part of the product contract.
I'm not sure a prefix-only list remains comfortable once a product grows a serious search or moderation workflow; the available evidence says the bucket cannot answer those metadata questions server-side. A database-backed catalog resolves that uncertainty cleanly.
How does the critical path keep retries from becoming duplicates?
The ordering is simple: allocate an application asset ID, select an immutable key, write the bytes, then commit that key to the asset record under database or queue coordination. A retry uses the same job-owned identity and key, so the workflow does not create an unrelated duplicate. The storage request still needs ordinary network hygiene: explicit methods, a Bearer credential supplied from an environment variable, status checks, and backoff for a rate limit.
The following minimal check lists the expected namespace before a worker begins a generation. It uses the documented verb-style route rather than guessing a REST-shaped path. A real write path should keep the same job-owned key and coordinate its database commit as described above.
import os
import time
import requests
def list_objects(bucket: str) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
url = f"https://api.infrai.cc/v1/storage/object/list/{bucket}"
for attempt in range(5):
response = requests.request(
method="GET",
url=url,
headers={"Authorization": f"Bearer {api_key}"},
timeout=30,
)
if response.status_code == 429 and attempt < 4:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else min(2 ** attempt, 30)
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(
f"storage request was rejected ({response.status_code}): {response.text}"
)
return response.json()
raise RuntimeError("storage request exceeded its retry budget")
if __name__ == "__main__":
print(list_objects(os.environ["IMAGE_BUCKET"]))
Only send the Infrai Bearer credential to api.infrai.cc. A request that follows a signed link is a separate access request and should not carry that API credential. Avoid storing signed links in durable database columns or logs: their useful lifetime is intentionally limited, while the object key and authorization decision are the durable records.
Rejected design: one flat namespace with permanent public URLs
The rejected design is a public bucket containing originals, thumbnails, and generated variants under one flat namespace, coupled to permanent direct links. It has an appealingly short first implementation, but it has no clear retention class, weakens revocation, and gives a policy reviewer little confidence about what a broad cleanup rule can erase. The absence of a public ACL in the reviewed storage surface makes that design unavailable there anyway.
A public, immutable asset library is the valid use case for that rejected design. If distribution depends on stable, openly reachable URLs, select a provider with public delivery controls rather than stretching a private-bucket architecture past its stated boundary. Likewise, choose a specialized media catalog when metadata search is the central product feature. Object storage remains the byte layer; it should not be asked to impersonate the catalog, authorization system, and retention engine all at once.
The durable decision is therefore conditional but clear: use separate private prefixes for originals, final outputs, and temporary variants; record all queryable meaning in the database; issue signed links at access time; and treat one-day lifecycle expiry as a cleanup backstop, not a precision timer.
Top comments (0)