Use bounded exponential backoff for thumbnail uploads, then make retention and deletion explicit. The retry loop should treat 429 and transient 5xx responses as temporary, respect Retry-After, and stop after a small fixed number of attempts. It should also use a stable object key so a retry does not create a second thumbnail.
That is the decision rule I would put in an architecture record for a property-management system. Uploading a 240 KB image thumbnail is easy; deciding what survives a failed retry, an accidental overwrite, or a tenant move-out is the part that deserves scrutiny.
Why are thumbnail uploads failing when transient errors trigger retries?
Three invariants keep this from turning into a second outage:
- A thumbnail has one deterministic key, such as
properties/1842/units/7/thumb.jpg. - A retry is rate-limited and bounded. It must not turn one storage error into a request storm.
- Deletion is a separate policy decision. A failed upload must not cause the cleanup job to delete the last known-good thumbnail.
The last point is easy to miss. A thumbnail pipeline often has two workers: the image processor writes the new object, and a retention worker removes old objects. If both infer state from a job's success flag, a delayed retry can race with deletion. Keep upload state in the job database, and make deletion conditional on the database record saying that a replacement is durable enough to use.
Here is the trade-off record I would use before choosing a storage surface:
| Option | Good fit for property thumbnails | Important boundary |
|---|---|---|
| Amazon S3 | Mature object storage with broad lifecycle and replication choices | More configuration and surrounding AWS decisions to own |
| Cloudflare R2 | Applications already close to Cloudflare Workers and its storage workflow | The application still owns retry, retention, and deletion semantics |
| Azure Blob Storage | Teams already operating inside Azure identity and data tooling | Azure-specific integration is a poor fit for a provider-neutral service |
| Infrai storage | A plain REST call is useful when a backend wants one key and no SDK dependency | No public ACL, object versioning, object lock, cross-region replication, or server-side metadata search; lifecycle expiry starts at one day |
This is not a durability ranking. It is a boundary map. S3, R2, and Blob Storage can each be the sensible choice when their surrounding platform is already the operational home. A single REST surface is attractive for a small Python worker, but it does not erase the need for an application-level retention record.
How should the Python retry path work?
The critical path below calls the verified object-write route with an explicit PUT. It retries only conditions that are plausibly transient, honors a usable Retry-After value, and adds jitter so several thumbnail workers do not wake at the same instant. The caller supplies the object key, rather than generating a fresh one on every attempt.
import os
import random
import time
from typing import Optional
import requests
def retry_after_seconds(response: requests.Response) -> Optional[float]:
value = response.headers.get("Retry-After")
if value is None:
return None
try:
return max(0.0, float(value))
except ValueError:
return None
def upload_thumbnail(bucket: str, object_key: str, content: bytes) -> requests.Response:
api_key = os.environ["INFRAI_API_KEY"]
api_base = os.environ["INFRAI_BASE_URL"].rstrip("/")
url = f"{api_base}/v1/storage/object/put/{bucket}/{object_key}"
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "image/jpeg",
"Idempotency-Key": f"thumbnail:{bucket}:{object_key}",
}
max_attempts = 5
for attempt in range(max_attempts):
try:
response = requests.put(url, headers=headers, data=content, timeout=15)
except requests.RequestException:
if attempt == max_attempts - 1:
raise
delay = min(16.0, 0.5 * (2**attempt)) + random.uniform(0, 0.25)
time.sleep(delay)
continue
if 200 <= response.status_code < 300:
return response
retryable = response.status_code == 429 or 500 <= response.status_code <= 599
if not retryable or attempt == max_attempts - 1:
response.raise_for_status()
server_delay = retry_after_seconds(response)
delay = server_delay if server_delay is not None else min(16.0, 0.5 * (2**attempt))
time.sleep(delay + random.uniform(0, 0.25))
raise RuntimeError("unreachable")
The Idempotency-Key is stable across attempts. That matters more than the exact backoff constants. Five attempts over roughly 31 seconds at the maximum delay is a deliberate latency trade-off for a thumbnail job; if the user-facing request cannot tolerate it, enqueue the work and report processing status instead of holding an HTTP request open.
One practical trap: a retry metric that counts only successful calls hides the failure pattern. Log the object key, attempt number, status code, elapsed time, and whether the delay came from Retry-After. Do not log the bearer token or the image bytes. Alert on repeated 429s separately from 5xx responses, because increasing concurrency is a reasonable response to neither.
Which retention model matches the workflow?
For thumbnails, I would choose deterministic keys and a database-owned state transition. Write the new object first. Mark the database row as current only after a successful response. Retain the previous key until the replacement has passed whatever read or validation check the product requires, then enqueue deletion. A deletion worker should be idempotent too: deleting an already absent object is not a reason to resurrect a thumbnail or retry the upload.
This makes the failure boundaries visible:
- Upload fails before success: keep the old thumbnail and retry the new key.
- Upload succeeds but the worker crashes before the database update: reconcile by checking the known key, then update state deliberately.
- Database update succeeds but deletion fails: retain the extra object and retry deletion later.
- A human uploads a replacement: use a new content-addressed or job-specific key if overwrites cannot be coordinated safely.
The storage service described here has no If-Match conditional write, so strict mutual exclusion belongs in a queue or database. It also has no object versioning or object lock. That rules it out for records requiring WORM-style immutability unless an external system supplies that guarantee. Lifecycle has a one-day minimum, and multipart fragments have no automatic cleanup rule, so lifecycle is not a substitute for timely application cleanup.
For browser-direct uploads, check the CORS requirement before committing to the design. A bucket model may expose CORS-related data, but there is no independently available CORS-setting route in the stated storage surface. A server-to-server thumbnail worker avoids that browser constraint and keeps credentials out of the client.
When is a different option the better answer?
Choose S3 when the team needs its mature AWS ecosystem, established lifecycle controls, or a larger set of storage-side controls and is willing to operate them. Choose R2 when Cloudflare Workers is already the execution environment and the edge-to-storage path is the real optimization. Choose Azure Blob Storage when Azure identity, monitoring, and governance are already non-negotiable.
Choose the single REST approach when the problem is integration overhead: a Python worker can send HTTPS directly, with no storage SDK version to coordinate. That is a real advantage for a narrow service. It is not a replacement for versioning, object lock, cross-region replication, or searchable metadata, because those capabilities are explicitly outside the stated storage boundary.
The rejected option is “retry until it works.” It confuses availability with correctness. A bounded retry policy, a stable key, and a database-owned retention transition give the system a way to recover without multiplying writes or deleting the only good image.
Top comments (0)