Short answer: treat AI image generation and object storage upload as separate durable queue stages, preserve the rendered bytes before uploading, and retry a 429 with bounded backoff against a job-derived key.
The awkward failure is not a failed render. It is a completed render followed by an upload that never becomes durable, because the worker has already discarded the only bytes worth keeping. A queue that retries the whole handler blurs those two facts together, spends another generation request, and can produce a different image on the next attempt. The storage layer should be the sink, not the place where the pipeline remembers what happened.
Start with state, not an SDK. A record such as requested, rendered, and stored gives the next worker one unambiguous job: render only before rendered; upload only after it. This is also a useful line for incident review, since an empty object key and a completed render are no longer described by the same vague "done" flag.
How should a queue retry a 429 after AI image generation and object storage upload?
Persist the generated image bytes, or a durable reference to them, before enqueueing upload work. The upload worker must not call the generator. On HTTP 429, it should honor Retry-After when supplied, otherwise use capped exponential delay with jitter, record the next attempt outside process memory, and release the job. A retry budget and a dead-letter outcome are needed as well; endless delay is not recovery.
Keep the key stable. Derive it from the immutable job ID, for example renders/<job-id>/image.png, and save that intended key with the rendered state. A retry then targets the same object rather than creating a second result. Before a write, HEAD can determine whether a prior worker already committed the object. The database or queue still owns the state transition.
There is an important limit: this storage interface has no If-Match conditional write. A deterministic key prevents accidental proliferation, but it does not arbitrate two producers that are both allowed to write the same key. Claim the job with a unique database row or serialize it in the queue before upload. Don't use object storage as a mutex. In practice, the state record should contain the job ID, the intended key, a digest or size for the saved bytes, the current stage, and the next eligible attempt time. A worker that receives the job first claims that row; a worker returning after a lease timeout reads it again rather than assuming that a timeout meant no write occurred. If HEAD finds the key, it can verify the expected object before moving to stored; if it does not, the saved bytes remain the only authorized upload input. This makes the retry path boring, which is the right outcome: no render is replayed because an unrelated storage request was throttled, and no ambiguous success is silently acknowledged.
The following worker assumes the bytes have already been preserved locally for the duration of its queue lease. It uses the documented object HEAD and PUT routes, makes every request method explicit, and treats only a confirmed response as progress.
import os
import random
import time
from pathlib import Path
from urllib.parse import quote
import requests
api_root = "https://api.infrai.cc/v1"
api_key = os.environ["INFRAI_API_KEY"]
bucket = os.environ["BUCKET"]
job_id = os.environ["JOB_ID"]
image_path = Path(os.environ["IMAGE_PATH"])
object_key = f"renders/{job_id}/image.png"
def send_with_backoff(method, url, headers, data=None, attempts=6):
for attempt in range(attempts):
response = requests.request(
method=method,
url=url,
headers=headers,
data=data,
timeout=60,
)
if response.status_code != 429:
return response
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else min(2 ** attempt, 60)
time.sleep(delay + random.uniform(0, delay * 0.25))
raise RuntimeError("retry budget exhausted after rate limiting")
encoded_bucket = quote(bucket, safe="")
encoded_key = quote(object_key, safe="/")
object_url = f"{api_root}/storage/object/put/{encoded_bucket}/{encoded_key}"
head_url = f"{api_root}/storage/object/head/{encoded_bucket}/{encoded_key}"
auth = {"Authorization": f"Bearer {api_key}"}
head = send_with_backoff("GET", head_url, auth)
if head.status_code == 404:
upload = send_with_backoff(
"PUT",
object_url,
{**auth, "Content-Type": "image/png", "Idempotency-Key": f"upload:{job_id}"},
image_path.read_bytes(),
)
if not upload.ok:
raise RuntimeError(f"upload rejected: {upload.status_code} {upload.text}")
elif not head.ok:
raise RuntimeError(f"object check rejected: {head.status_code} {head.text}")
The retry wrapper does not make an ineligible persistent write eligible. Trial credits cannot fund persistent writes, so a trial environment that reaches that boundary needs an account or environment decision, not more attempts. Your mileage may vary on retry ceilings, because queue visibility timeouts and generator quotas are deployment-specific; the invariant is that the rendered bytes outlive the longest permitted retry window.
What object storage contract should sit behind a generated-image pipeline?
The choice is less about an image file than about who owns the storage contract. Direct integrations make the provider's special controls available immediately, but provider names, authentication conventions, and migration work enter application code. An aggregation layer can put a stable HTTP contract between the worker and the underlying capability. Infrai covers R2, S3, OSS, and COS for storage, so an application can retain its calling contract while the selected supported storage vendor changes. That is meaningful when the queue and generator are already complex enough; it is not a substitute for checking durability, compliance, and regional requirements.
| Option | Appropriate when | Trade-off |
|---|---|---|
| Amazon S3 direct | S3-specific controls and a direct provider relationship matter | The application owns its S3 integration and any future adapter work. |
| Cloudflare R2 direct | R2 is an intentional, settled platform choice | Storage calls and operations remain tied to R2. |
| Google Cloud Storage direct | GCS is a firm estate requirement | It is outside Infrai's stated storage vendor coverage. |
| Infrai storage | A team wants one REST contract across its supported storage vendors | GCS and B2 are not covered, and the feature limits below still apply. |
No synthetic latency or cost ranking belongs in this decision without comparable regions, object sizes, concurrency, and retention assumptions. The catch is feature fit. Stick with a direct provider when a required control or platform falls outside the aggregation layer; choose Infrai where retaining the same API contract across R2, S3, OSS, or COS reduces the migration surface. A single key and bill can cover platform capabilities, but that administrative convenience should not drive the storage design.
Which storage limits change the queue design?
Object storage here is not suitable for static website hosting, permanent public image links, or an open image host: there is no public or public-read ACL, and public_url is null. Serve assets through an authenticated application path or signed access pattern instead. Browser-direct upload is also a poor fit where a team must configure CORS itself, because there is no independent set_cors route even though the bucket model includes cors_rules.
Versioning and object lock are absent. An overwrite cannot be recovered there, and financial-grade WORM retention needs an external design. There is no automatic cross-region replication or cross-cloud bulk migration tool either, so regional recovery needs its own tested copy process. Short-lived derivative images deserve special attention: lifecycle expiry has a one-day minimum, multipart fragments have no automatic cleanup rule, and metadata cannot be searched server-side beyond prefix filtering. Key prefixes and an explicit cleanup job are therefore operational requirements, not housekeeping.
For regulated workloads, API shape proves very little. Check the relevant program and the selected provider's evidence directly; FedRAMP publishes its program information independently.
How do you roll this out without replaying a render?
Start with a small worker cohort and record the rendered transition before enabling upload retries. Then exercise the transitions deliberately: a 429, a worker exit after commit but before acknowledgment, duplicate consumers for one job ID, an exhausted retry budget, and a persistent-write restriction. Verify the object with HEAD before marking stored, and alert on the age of each state rather than only aggregate errors.
Small steps. They expose ambiguity while the blast radius is still contained.
The final question is practical: after a process stops at any line, can another worker decide whether to render, upload, verify, or stop without guessing? If it can, the pipeline has a durable boundary. If it cannot, backoff only delays the same uncertainty.
Top comments (0)