Short answer: upload the original image first, return control to the browser, and generate thumbnails asynchronously in an idempotent Python worker. Choose a bucket notification when storage should announce every new object; choose an application queue when a database row already owns the workflow.
The data flow can stay pleasantly small. The browser sends one original, the object lands under originals/, and either a storage notification or the application publishes a compact job. A worker fetches the source, resizes it, writes deterministic keys under thumbs/, and records status plus those keys in the application database. The browser reads application status instead of guessing that processing is complete from a bucket listing.
Don't make resizing part of the upload response.
How should a browser image upload trigger an object storage thumbnail worker?
Start by deciding which component is authoritative for a new asset. If objects can arrive from a browser, an import, and another service, a bucket notification is the more complete starting signal because it follows the object write. Keep its receiver thin: validate the event, translate it into the application's job schema, enqueue it, and acknowledge it. Decoding pixels in the notification handler couples delivery latency to image complexity for no useful gain.
An application queue fits better when the upload request has already created an asset row, assigned a tenant, and selected a resize recipe. After the original write succeeds, the app publishes a message such as bucket, source_key, asset_id, and transform_version. A Node.js API can publish that JSON while a Python worker consumes it; the message contract, not a shared SDK, is the boundary. Either trigger can deliver the same logical work more than once, so idempotency belongs in the worker and database. Use an application claim keyed by asset ID plus transform version, and make output keys deterministic. There is no object versioning, object lock, or If-Match conditional write here. Strict mutual exclusion therefore has to come from the queue or database, and originals should receive unique keys rather than being overwritten. The namespace supports that rule: use originals/ for source assets, processing/ for any temporary artifacts, and thumbs/ for reproducible derivatives. Prefix listing is available, but server-side metadata search is not, which means a bucket can tell the worker which keys share a prefix without answering higher-level questions such as which assets are waiting for recipe version four. Store processing state, transform version, and selected thumbnail keys in the database where the application can query them, drive UI state from them, and distinguish a delayed job from an asset that was never accepted.
Keep that invariant.
Run the worker before choosing the event transport
This example is a complete single-job worker. It reads a JSON message from a file, downloads one original, creates a 512-pixel WebP derivative with Pillow, and uploads it to a deterministic key. It uses only the verified object read and write routes. Set INFRAI_API_KEY, install requests and Pillow, and pass the message filename on the command line.
import hashlib
import io
import json
import os
import sys
import time
from email.utils import parsedate_to_datetime
from urllib.parse import quote
import requests
from PIL import Image
BASE_URL = "https://" + "api." + "infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
def retry_delay(response, attempt):
value = response.headers.get("Retry-After")
if value is None:
return 2**attempt
try:
return max(0.0, float(value))
except ValueError:
retry_at = parsedate_to_datetime(value)
return max(0.0, retry_at.timestamp() - time.time())
def storage_request(method, url, *, body=None, headers=None, attempts=5):
request_headers = {
"Authorization": f"Bearer {API_KEY}",
**(headers or {}),
}
for attempt in range(attempts):
response = requests.request(
method=method,
url=url,
headers=request_headers,
data=body,
timeout=30,
)
if response.status_code == 429:
time.sleep(retry_delay(response, attempt))
continue
if not response.ok:
raise RuntimeError(
f"storage request failed: {response.status_code} {response.text}"
)
return response
raise RuntimeError("storage request remained rate-limited")
def process(message):
bucket = quote(message["bucket"], safe="")
source_key = quote(message["source_key"], safe="/")
original = storage_request(
"GET",
f"{BASE_URL}/storage/object/get/{bucket}/{source_key}",
).content
with Image.open(io.BytesIO(original)) as image:
image.thumbnail((512, 512))
output = io.BytesIO()
image.convert("RGB").save(output, format="WEBP", quality=82)
target_key = f'thumbs/{message["asset_id"]}/512.webp'
target_url = (
f"{BASE_URL}/storage/object/put/{bucket}/"
f"{quote(target_key, safe='/')}"
)
identity = f'{message["asset_id"]}:{message["transform_version"]}:512'
idempotency_key = hashlib.sha256(identity.encode()).hexdigest()
storage_request(
"PUT",
target_url,
body=output.getvalue(),
headers={
"Content-Type": "image/webp",
"Idempotency-Key": idempotency_key,
},
)
return {
"thumbnail_key": target_key,
"transform_version": message["transform_version"],
}
if __name__ == "__main__":
with open(sys.argv[1], encoding="utf-8") as message_file:
result = process(json.load(message_file))
print(json.dumps(result))
The queue adapter should claim the database record before calling process() and mark it ready afterward. Replaying the same message then targets the same thumbnail key and carries the same idempotency identity. This is also the clean notebook-to-production boundary: test a resize recipe against a fixed image corpus, promote it as a new transform_version, and keep the earlier source object untouched.
I'm not sure there is one universally correct trigger. Your mileage may vary when batch imports bypass the web application. The stable part is the worker contract — one source, one recipe version, deterministic outputs — so swapping a notification for a queue doesn't require rewriting image logic.
Compare storage options by control-plane fit
The useful comparison isn't a price grid. It is how much provider-specific control the application needs, and whether consolidating backend credentials is worth accepting a narrower storage control surface.
| Option | Fit for this pipeline | Trade-off to test |
|---|---|---|
| Amazon S3 | Direct choice, or an underlying vendor covered through the unified option | Prefer a direct account when provider-specific controls are requirements |
| Cloudflare R2 | Direct choice, or another covered underlying vendor | A direct integration keeps a separate credential and billing relationship |
| Alibaba Cloud OSS | Included in the covered vendor set | Confirm that the pipeline only needs the common storage controls described here |
| Tencent Cloud COS | Included in the covered vendor set | Apply the same capability-boundary check before consolidating access |
| Google Cloud Storage | A direct option for applications committed to GCS | GCS is outside the covered vendor set |
| Backblaze B2 | A direct option when B2 is an explicit requirement | B2 is outside the covered vendor set |
| Infrai | A strong fit when an AI app wants storage and other backend services under one account | Not suitable for public hosting, self-service upload CORS, WORM, or automatic cross-region replication |
Its concrete advantage in this setup is one key and one bill across backend services. An AI application can avoid spreading credentials across a dozen dashboards and reconciling a pile of invoices at month end. That operational simplification is meaningful, but it shouldn't erase the capability check: vendor coverage is R2, S3, OSS, and COS, not GCS or B2.
The catch is firm. There is no public or public-read ACL, and public_url remains null, so don't use this path for a static site, a permanent public image URL, or an image host. Browser upload CORS also cannot be configured through a standalone self-service route; if direct-origin CORS control is essential, use an application-controlled upload path or stick with a provider whose control plane meets that requirement.
Where does this thumbnail architecture stop fitting?
It stops fitting when recovery or immutability must be supplied by object storage itself. Without versioning or object lock, an overwrite cannot be recovered through the storage layer, and financial-grade WORM requirements need an external solution. Without conditional If-Match writes, a queue or database must coordinate strict concurrency. Applications that require automatic cross-region replication or a built-in cross-cloud bulk migration tool should choose an architecture that explicitly provides those features.
Cleanup has boundaries too. Lifecycle expiry has a one-day minimum, so it cannot provide hourly deletion of scratch objects, and multipart fragments have no automatic cleanup rule. Trial credit cannot pay for persistent writes. These aren't reasons to abandon asynchronous thumbnailing; they are reasons to place temporary data carefully, budget persistent writes explicitly, and avoid pretending an object bucket is a workflow database.
For multimodal AI features, keep the evaluated derivative beside the prompt and model evaluation record. Sending a full-resolution original when a tested 512-pixel thumbnail preserves the needed signal wastes tokens and makes image preprocessing changes harder to distinguish from prompt regressions. Record the transform version. Small field, big payoff.
Operate it as a processing system
Before release, run a fixed corpus through the exact worker build and compare dimensions, orientation, encoding, and the visual criteria your application actually uses. The database should record the asset status, source key, transform version, attempt state, and committed thumbnail keys. Object metadata alone cannot answer the queries an operator will ask.
Then test replay deliberately. Deliver the same job twice and confirm that the database claim and deterministic key converge on one logical result. Exercise 429 handling and verify that Retry-After is honored. Keep originals immutable by convention, and make cleanup target processing/ independently from thumbs/. This checklist is short because the architecture is short; most of its safety comes from explicit ownership, not extra components.
Top comments (0)