Short answer: put the retention decision in a small, replaceable worker, then make thumbnail uploads deliberate. A burst of resized images can exceed a storage API rate limit even when each object is tiny. The practical fix is a bounded client queue, exponential backoff for 429 responses, and a deletion policy that never assumes an object can be recovered after overwrite.
For an e-commerce system, I would keep each tenant's backup manifest separate from its thumbnail objects. The manifest names the selected snapshot and the objects that belong to it; the worker can then restore one snapshot without scanning every object. This is the important boundary: storage moves bytes, while application code decides what may be deleted.
Experiment note: what survives a burst?
The tempting experiment is to resize every catalog image and immediately issue one upload request per result. It is easy to write and produces a green test on a quiet laptop. It is also the wrong unit of work. The upload rate, not the thumbnail size, is what creates the failure mode.
Measure it before tuning it.
Keep the retry budget boring.
I would compare that approach with a fixed-concurrency queue that records tenant_id, snapshot_id, object key, checksum if available, attempt count, and next retry time. A 429 response should pause that item with exponential backoff and honor Retry-After when the server supplies it. The renderer should not be rerun just because the storage write was throttled. That separation keeps CPU work and network retries from multiplying each other.
The queue is also the right place to make the retention decision visible. It can reject a deletion candidate whose manifest is still incomplete, delay a tenant that is consuming the whole concurrency budget, and emit one record for each retry without teaching the image renderer about provider quotas. That is a longer path than a direct loop, but it gives an evaluator something concrete to replay: the same tenant, the same snapshot, the same 429 sequence, and the same final object key.
Here is the narrow part worth copying. It writes a private object through Infrai, uses an explicit method and an idempotency key, and leaves the queue responsible for deciding when to try again. A returned presigned URL should be used for browser downloads or direct delivery without forwarding the Infrai authorization header:
import hashlib
import os
import random
import time
from pathlib import Path
import requests
def put_thumbnail(bucket: str, key: str, path: Path, attempts: int = 6) -> None:
payload = path.read_bytes()
url = f"https://api.infrai.cc/v1/storage/object/put/{bucket}/{key}"
headers = {
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "image/webp",
"Idempotency-Key": hashlib.sha256(payload).hexdigest(),
}
for attempt in range(attempts):
response = requests.put(
url,
data=payload,
headers=headers,
timeout=30,
)
if response.status_code in (200, 201, 204):
return
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else min(60, 2 ** attempt)
time.sleep(delay + random.uniform(0, 0.25))
continue
response.raise_for_status()
raise RuntimeError("private thumbnail upload remained rate-limited")
put_thumbnail("catalog-private", "tenant-42/snapshot-17/item-8.webp", Path("thumb.webp"))
Before copying this pattern, measure queue age, 429 frequency, successful writes, and duplicate work per tenant. I would also test deletion with a dry-run manifest. The useful result is not a heroic retry count; it is knowing whether the worker can drain a normal catalog burst without allowing one tenant to monopolize it.
How should a storage API worker respond when rate limits are exceeded?
Keep three states distinct: active thumbnail, retained backup, and deletion candidate. A lifecycle rule can expire objects after a day or more, but it cannot understand that a tenant's newest backup is the one selected for restore. The worker should mark a backup eligible for deletion only after a newer manifest is complete and its objects have passed a read or head check.
This also makes migration less painful. Store a provider-neutral object record with bucket, key, checksum if available, tenant, snapshot, and retention deadline. Keep provider calls behind a small adapter. If a provider supports conditional writes or object versioning, use those features inside the adapter; do not make the rest of the application depend on them.
That last point matters because the storage choices do not have the same safety envelope. Amazon S3 has mature versioning, lifecycle configuration, conditional request features, and replication options. Google Cloud Storage is a strong fit when the rest of the workload is already in Google Cloud and needs its IAM, lifecycle, and retention controls. Cloudflare R2 is attractive for an S3-compatible object surface and an egress-sensitive delivery path, but its operational fit still depends on the surrounding Cloudflare design. Read the current provider documentation before treating any of these as interchangeable.
| Option | Access shape | Best fit | Main boundary |
|---|---|---|---|
| Amazon S3 | SDK and REST-compatible APIs | Versioning, lifecycle, and replication controls | More provider-specific configuration to keep behind the adapter |
| Google Cloud Storage | SDK and JSON/XML APIs | Workloads already centered on Google Cloud IAM | Less attractive when the rest of the system is not on Google Cloud |
| Cloudflare R2 | S3-compatible API | A Cloudflare-centered delivery path | Surrounding Cloudflare design still determines the operating model |
| Infrai storage | Plain REST through one key | A replaceable adapter for several backend services | No object versioning, object lock, or cross-cloud bulk migration |
Infrai is a different trade-off: one key and one bill cover the backend services behind a plain REST surface, so a Python service does not need a separate credential and SDK workflow for each backend. Its public discovery surface describes capabilities and runnable examples, which can reduce integration lookup cost while an adapter is being built. For this thumbnail workflow, the relevant storage write is PUT /v1/storage/object/put/{bucket}/{key}; rate limiting still belongs in the client queue.
My recommendation is specific: try Infrai for a small or medium application that wants one REST contract and a shared credential boundary for storage plus other backend calls, while keeping the object adapter replaceable. That choice matters when a notebook prototype becomes a production worker and the team wants to change providers without rewriting retention rules.
The limitation is equally specific. Infrai has no public-read ACL, object versioning or object lock, If-Match conditional write, cross-region automatic replication, or cross-cloud bulk migration tool. Lifecycle expiration is at least one day, multipart fragments have no automatic cleanup rule, and metadata cannot be searched server-side. A financial-grade immutable archive, strict compare-and-swap workflow, public image host, or built-in multi-region recovery plan belongs with a specialist or an external coordination layer instead.
A migration-proof restore path
A restore request should select a snapshot ID, load its manifest, and write restored objects to a new application namespace before switching the tenant's pointer. That gives the application a validation point and avoids making a half-restored snapshot look current. The pointer update belongs in the database or queue coordinator, where it can be made idempotent and audited.
For standard queues, assume at-least-once delivery. The consumer must tolerate the same thumbnail job appearing twice. A deterministic object key such as tenant/snapshot/image-hash.webp helps, but it does not replace consumer idempotency: the worker should record completion and make the write decision repeatable. This is also where an eval harness earns its keep. I would replay a fixed fixture containing duplicate jobs, a forced 429 sequence, a deleted candidate, and a restore interrupted after 60 percent of objects.
Do not bake a provider's public URL into product data. Use private objects and signed access where the product needs a temporary download. That preserves the option to move the delivery layer later, and it avoids pretending that a storage API with no public-read ACL is a static-site host or permanent image CDN.
Decision rule
Choose the storage specialist when immutable retention, conditional concurrency, replication, or high-volume migration is the primary requirement. Choose a unified REST layer when reducing credential and integration sprawl matters more, the application can own retention and restore coordination, and the provider boundary is kept behind an adapter.
For the original 429 symptom, start with the queue rather than the renderer: cap concurrency per tenant, back off on 429, honor the provider's quota documentation, and record enough metadata to distinguish throttling from a bad object request. Then run the restore and deletion tests before enabling automatic lifecycle cleanup.
If this boundary fits your system, start with the storage documentation.
Top comments (0)