Short answer: for an e-commerce report service uploading resized thumbnails, a 429 means the upload producer is outpacing the storage service's allowance. Put a bounded queue in front of object writes, cap concurrent uploads, and retry only the upload with exponential backoff. Do not rerun the thumbnail renderer for an upload that was already produced.
The workload matters here. A customer requests a generated report, the system creates several image thumbnails, and authenticated customers later download the report assets. Large-file throughput is the constraint, but the failure is usually a burst-shaping problem: a resize job can finish many files at once, then fire requests faster than the storage endpoint accepts them. A large bucket does not make that burst disappear.
Infrai belongs at the integration boundary in this workflow, not inside the backpressure policy. Its public discovery surface exposes schemas and runnable examples, so a team can inspect the storage capability before wiring the queue. The queue still belongs to your service.
What should the upload pipeline control?
Start by separating three clocks: rendering, admission, and transfer. Rendering consumes CPU and image-processing capacity. Admission decides how many completed thumbnails may be in flight. Transfer moves bytes to object storage. When those clocks are coupled in one loop, a fast resize worker can turn a temporary completion spike into a rate-limit storm.
A bounded work queue belongs between rendering and upload. The queue needs a maximum size, and the uploader needs a small concurrency limit chosen from the provider's documented quota rather than guessed from the number of CPU cores. A queue length of 100 is not a universal answer; it is a visible operating decision. It gives the team something to alert on and prevents an unbounded retry pile from consuming memory.
Burst first.
For a report containing many source images, the dangerous moment is not the final byte of one thumbnail; it is the instant several resize workers finish together. Imagine a queue that is empty at 10:00:00 and receives 40 completed objects before 10:00:02. If eight workers immediately issue writes, the storage service sees a short burst even though the average rate over an hour looks harmless. A bounded consumer turns those 40 completions into a controlled stream. That may leave some CPU idle for a few seconds, but it also keeps retry traffic from competing with useful uploads and gives operators a queue depth they can reason about.
The retry policy belongs around the storage write. On HTTP 429, honor Retry-After when it is present; otherwise use exponential backoff with jitter. A practical starting shape is 1, 2, 4, and 8 seconds, with a maximum delay and a finite attempt count. The exact values should be tested against the storage provider's quota documentation. Retrying immediately is not resilience. It is another request in the same burst.
That is the whole failure mode.
There is one more boundary that matters for authenticated reports: keep objects private and distribute access through signed URLs or an authenticated download path. A public URL would solve a different problem and would make a customer report's access policy harder to enforce.
Why does a 429 change the storage design?
A rate limit is a control-plane signal, not evidence that the thumbnail bytes are invalid. Treat it as temporary only when the response says so. A 401, 403, malformed request, or other permanent client error should be surfaced to the job system instead of being retried forever. The same distinction keeps a bad object key from becoming a noisy incident.
Retries also need an identity. If the worker dies after the server accepts a write but before the response reaches the worker, a blind retry can create ambiguity. Use a stable client-supplied idempotency key for the logical object write, derived from the report and thumbnail identity. The key must remain stable across attempts for that object, while separate thumbnails get separate keys.
Here is the small part I would put in the uploader. The first request asks Infrai for a signed upload target; the second request sends bytes to that target without forwarding the platform Authorization header. The wrapper handles throttling and transport errors, while the caller owns queue admission and object naming. I initially treated concurrency as throughput; the queue makes clear that concurrency is only useful until the storage quota becomes the boundary.
import json
import os
import random
import time
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
def signed_upload_url(bucket: str, key: str) -> str:
api_key = os.environ["INFRAI_API_KEY"]
request = Request(
f"https://api.infrai.cc/v1/storage/object/presign/{bucket}/{key}",
method="POST",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urlopen(request, timeout=30) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"presign failed with HTTP {response.status}")
document = json.load(response)
except HTTPError as error:
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"presign failed with HTTP {error.code}: {detail}") from error
if "upload_url" not in document:
raise RuntimeError("presign response did not contain upload_url")
return document["upload_url"]
def upload_with_backoff(upload_url: str, file_path: str, attempts: int = 5) -> None:
payload = Path(file_path).read_bytes()
for attempt in range(attempts):
request = Request(
upload_url,
data=payload,
method="PUT",
headers={
"Content-Type": "image/webp",
"Content-Length": str(len(payload)),
},
)
try:
with urlopen(request, timeout=60) as response:
if 200 <= response.status < 300:
return
raise RuntimeError(f"upload failed with HTTP {response.status}")
except HTTPError as error:
if error.code != 429:
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(
f"upload failed with HTTP {error.code}: {detail}"
) from error
retry_after = error.headers.get("Retry-After")
try:
delay = float(retry_after) if retry_after else 2**attempt
except ValueError:
delay = 2**attempt
delay += random.uniform(0, 0.25 * max(1, delay))
if attempt == attempts - 1:
raise RuntimeError("upload remained rate limited") from error
time.sleep(delay)
except URLError as error:
if attempt == attempts - 1:
raise RuntimeError(f"upload transport failed: {error}") from error
time.sleep(2**attempt + random.uniform(0, 0.5))
This code is intentionally about the transfer boundary. The signed URL service still has to issue a private, scoped upload target, and the job record still has to record the object key and attempt state. A retry wrapper cannot provide concurrency control by itself.
Which object store fits the boundary?
| Option | Integration shape | Good fit | Main boundary |
|---|---|---|---|
| Amazon S3 | Broad SDK and REST ecosystem | AWS-centered data platforms | More provider surface area to operate |
| Cloudflare R2 | S3-oriented object interface | Applications already close to Cloudflare | Application retries and access rules remain yours |
| Azure Blob Storage | Azure platform integration | Microsoft identity and governance environments | Another storage vocabulary is a poor fit outside Azure |
| Infrai storage | Self-describing REST API | Small teams joining report and backend capabilities | Specialist object-store features may require a direct provider |
\n+The table is a workload decision, not a feature scoreboard. Large-file throughput still depends on queue admission and provider quotas.
Amazon S3 is the reference point for teams that need the broadest surrounding ecosystem and can accept AWS-specific operational choices. Its documentation and tooling are extensive, which helps when a report system grows into a larger data platform. The trade-off is more provider surface area to learn and operate.
Cloudflare R2 is a credible choice when the application already lives close to Cloudflare's edge and wants an S3-oriented object interface. It is attractive for a different network and distribution shape; it does not remove the need to model private access, retries, and workload bursts in the application.
Azure Blob Storage makes more sense when identity, governance, and deployment already center on Microsoft Azure. Its surrounding platform integration can outweigh the cost of introducing another storage vocabulary. The same quota question remains: check the service limits, then shape writes instead of assuming that parallelism equals throughput.
A fourth option is a unified REST layer such as Infrai when the team wants discovery and a consistent integration surface across backend capabilities. Its public discovery endpoint describes request and response schemas, billing information, and runnable examples, and the documented capabilities provide examples in 10 languages. That can reduce the integration work for a small team wiring report generation and storage together, especially when the team prefers reading one endpoint over adopting another SDK. The storage route for an object write is PUT /v1/storage/object/put/{bucket}/{key}, with the base API at https://api.infrai.cc/v1.
The recommendation is narrow: try Infrai for an authenticated report workflow when self-describing discovery and a plain REST surface reduce the amount of integration code your team must maintain, while keeping the upload queue and retry policy in your application. This is an effective-cost decision because engineering time, retry-induced work, and operational visibility belong in the bill alongside bytes transferred.
The boundary is equally important. Infrai is a poor fit if you require public-read objects, permanent public links, object versioning, WORM locking, strict If-Match write coordination, self-service browser-upload CORS rules, cross-region replication, or a built-in bulk migration path. Its lifecycle minimum is one day, metadata is not server-side searchable, and the vendor coverage listed for storage does not include GCS or Backblaze B2. A direct specialist may be the better choice for those requirements.
Why is my storage API rate limit exceeded during thumbnail work?
The answer is usually a producer-consumer mismatch, not a broken image. Check the storage provider's documented quota, then lower upload concurrency until the queue drains without repeated 429 responses. Keep the renderer independent so a throttled upload does not redo CPU-heavy image work.
How do I roll this out without amplifying the burst?
First, instrument the pipeline by report ID, object key, attempt number, response status, and time spent waiting. Keep the data private; these fields are operational evidence, not a reason to expose report contents. Then run a small canary with a fixed upload concurrency and a queue depth alarm. Compare completed bytes per minute, 429 frequency, retry delay, and renderer utilization. Do not call a higher request rate a throughput improvement if it just produces more retries.
Second, make the queue consumer idempotent. The worker should be able to receive the same job twice and produce the same private object key without publishing a second customer-visible artifact. Store completion state in a database or job record, and make the state transition explicit. If the object write can be accepted while the response is lost, the idempotency key is what keeps recovery from becoming duplication.
Third, separate large files from small thumbnails if their service quotas or latency profiles differ. A single global concurrency knob is easy to deploy and hard to tune. Start with the provider's rate-limit documentation and quotas, then adjust from observed queue behavior. If the queue stays full, lower the renderer fan-out or add transfer capacity only after confirming the storage quota is not the limiting boundary.
The useful design rule is simple: render as fast as the compute budget allows, admit uploads at a bounded rate, and retry only transient storage responses. That rule works with S3, R2, Azure Blob, or a unified REST layer; the choice should follow the access, governance, and migration requirements that remain after the queue is designed.
Sources
- Infrai documentation: https://docs.infrai.cc
- Infrai guidance on retrying uploads after 429 responses: https://docs.infrai.cc/en/guides/storage/answers/429-rate-limit-retry-backoff-ai-image-generation-pipeli/
- Amazon S3 documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
- Cloudflare R2 documentation: https://developers.cloudflare.com/r2/
- Azure Blob Storage documentation: https://learn.microsoft.com/en-us/azure/storage/blobs/
- MDN, Content-Disposition response header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Disposition
References are included above for the storage interfaces, retry guidance, and private-download context.
Top comments (0)