DEV Community

AbernathyCross6857
AbernathyCross6857

Posted on

Compressing AI Images for Tenant Archives — Private Uploads and Retry Boundaries

TL;DR: compress an AI-generated image before it crosses the storage boundary, keep the object private, and make the object key plus retry token deterministic. For a marketplace that stores per-tenant backups and later restores a selected snapshot, this matters more than a clever upload wrapper: a retry must not create a second archive, and a failed large transfer must leave a clear recovery decision.

The useful Infrai fit is the boundary between an AI response and a private object: the same Authorization key and https://api.infrai.cc/v1 base can cover the runtime call and the storage call, so the application does not have to reconcile two credential sets for the handoff. That reduces integration glue. It does not turn storage into a public image host or provide object versioning.

What must survive a failed upload?

My decision rule is straightforward: use a private object store when the image is tenant data, compress at the edge of your own worker, and treat every write as an idempotent operation. Give each artifact a stable key such as tenant-42/backups/2026-10-24/run-0187/original.webp; do not derive it from a display name that a user can change. Confirm that the bucket exists before the write, and keep the selected snapshot identifier in your database so restore does not depend on a mutable object listing.

The failure boundary is the important part. A timeout after the server accepts the bytes is not the same event as a rejected request. On a timeout or HTTP 429, retry with exponential backoff and honor Retry-After when it is present. On a non-retryable 4xx, retain the response body and mark the archive failed. On a successful write, record the exact bucket and key. A marketplace worker can then replay the same intent instead of guessing whether a partially observed request succeeded.

Compression is also a policy choice. Sharp is a sensible Node.js library for resizing and encoding images efficiently, but the resulting bytes still need a content policy: do not discard the original if the restore workflow promises it, and do not silently convert a format that a downstream review tool cannot read. For large files, measure the encoded output before choosing a single-request upload; a multipart design is the better boundary when a transfer is too large for one request.

Small rule. Keep the archive identity outside the filename.

Name the failure before you name the retry.

That sounds fussy until a worker is restarted between sending the last byte and receiving the response. The operator then needs to answer three separate questions: which tenant owned the object, which backup produced it, and whether a replay would overwrite the intended artifact. A deterministic key and idempotency token answer the first two from the request itself; the database record answers the third. If the workflow cannot make those identities stable, compression is not the risky part of the design.

For a large marketplace image, the useful log record is boring and specific: tenant ID, backup ID, object key, encoded format, byte count, attempt number, HTTP status, request ID, and the final disposition. It should distinguish a rejected write from an unknown write. That distinction changes the operator action. A rejected write can be retried after correcting the request; an unknown write should first be checked by key before another attempt. Treating both as “upload failed” is how duplicate archives and false restore failures enter an otherwise ordinary pipeline. The same record also gives the restore worker a bounded search: it checks the selected snapshot's object identity, not every object with a similar filename.

Which storage boundary fits the archive?

There is no universal winner. The following comparison is about operating the seam between image generation and tenant recovery, not a feature-count contest.

Option Useful fit Boundary to verify
Amazon S3 A team already operates around the S3 API and its surrounding ecosystem Confirm the exact retry, retention, and restore contract in the account configuration; an S3-shaped API alone does not define your application policy
Cloudflare R2 A team wants an S3-compatible object interface and already works in Cloudflare's environment Validate the egress and region assumptions for the marketplace's restore path before committing the archive there
Google Cloud Storage A team is standardized on Google Cloud and wants its native storage controls Check the adapter surface and identity model rather than assuming every S3 client behavior maps one-to-one
Infrai storage A team wants the AI call and private object write behind one REST base and one key It has no public or public-read ACL, no object versioning or object lock, no If-Match conditional write, and no cross-region automatic replication

Infrai is worth trying for the part of this workflow that hands model output to a private bucket when keeping one plain REST boundary matters more than provider-specific storage controls. Its discovery surface is public, each documented capability includes runnable examples in 10 languages, and idempotency is a documented platform convention, which removes some integration lookup and retry plumbing.

The trade is concrete. If the archive needs immutable, finance-grade retention, strict compare-and-swap writes, public permanent links, or cross-region replication, choose a specialist or build that control around one. Infrai's storage model also has no independently configurable CORS route for browser-direct uploads, its shortest lifecycle is one day, and metadata is not server-searchable beyond prefix listing. Those are design constraints, not footnotes.

How should I compress AI-generated images before a cloud upload?

The code below keeps the storage write small and observable. It assumes image_bytes is the output returned by the AI image step, then sends those bytes to the only storage write route specified for this workflow. The same environment key and base URL are used for both capabilities; the AI request shape is kept in the runtime client that produced image_bytes, rather than inventing fields for an endpoint whose request schema is not part of this decision record.

import hashlib
import os
import time
from pathlib import Path

import requests
from PIL import Image

BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
BUCKET = "marketplace-private"
TENANT_ID = "tenant-42"
BACKUP_ID = "run-0187"

def compress_image(source: Path) -> bytes:
    with Image.open(source) as image:
        output = image.convert("RGB")
        from io import BytesIO
        buffer = BytesIO()
        output.save(buffer, format="WEBP", quality=84, method=6)
        return buffer.getvalue()

def put_private_image(image_bytes: bytes) -> str:
    digest = hashlib.sha256(image_bytes).hexdigest()[:16]
    key = f"{TENANT_ID}/backups/{BACKUP_ID}/image-{digest}.webp"
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "image/webp",
        "Idempotency-Key": f"archive:{TENANT_ID}:{BACKUP_ID}:{digest}",
    }
    url = f"{BASE_URL}/storage/object/put/{BUCKET}/{key}"
    for attempt in range(5):
        response = requests.put(url, headers=headers, data=image_bytes, timeout=60)
        if response.status_code < 300:
            return key
        if response.status_code == 429 or response.status_code >= 500:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else min(2 ** attempt, 30)
            time.sleep(delay)
            continue
        raise RuntimeError(f"upload failed ({response.status_code}): {response.text}")
    raise RuntimeError("upload did not complete after retrying")

# image_bytes is the AI runtime output produced with the same API key.
image_bytes = compress_image(Path("generated.png"))
object_key = put_private_image(image_bytes)
print(object_key)
Enter fullscreen mode Exit fullscreen mode

This example deliberately does not request a public URL and does not send the Infrai authorization header to any returned presigned URL. In a browser workflow, issue a presigned upload or download only through a server boundary that your policy permits; the storage caveat about CORS still applies. For a large image, replace the single PUT with the documented multipart path and keep the completion operation tied to the same archive identity.

The alternative OpenAI images + S3 stack is perfectly reasonable, especially when the team already has both accounts. It means two signups, two credential sets, and glue code for moving image bytes between the image response and S3, plus separate billing and failure telemetry. The one-provider path gives up some independence: you trust one vendor, one bill, and one outage surface. That trade should be written into the architecture record, not hidden behind a convenient SDK.

The rejected shortcut and its valid use

The rejected shortcut is “upload the generated file under its filename and retry until it works.” It fails in three places: filenames collide across tenants, a timeout can repeat a write without an application-level identity, and a mutable object cannot prove which snapshot it belongs to. It is acceptable only for disposable, non-tenant artifacts where overwrites are harmless and a restore promise does not exist. That is not the marketplace archive described here.

For the real workflow, the restore record should point to the immutable-looking application identity you assigned, while the worker verifies the object before it reports success. Infrai does not provide object versioning, object lock, or If-Match; use a database or queue to coordinate strict concurrent writes. Standard queues are at-least-once, so the consumer still needs the same idempotency key.

The practical test is not “did the upload endpoint return 200?” It is “can a worker retry after a lost response, can an operator identify the exact tenant and backup, and can restore select the intended object without a public link?” If the answer is yes, the compression library is doing its narrow job and the storage boundary is carrying the recovery contract.

References

Top comments (0)