DEV Community

SyltharWave2946
SyltharWave2946

Posted on

Authenticated Report Thumbnails and Storage Rate Limits Explained

#storage #python #media #backend

Short answer: treat a rate-limit response during thumbnail upload as a scheduling problem, not as evidence that the object needs another upload attempt immediately. Bound concurrency per tenant, retry 429 responses with exponential backoff and Retry-After, and decide retention before the first thumbnail is written. In a media service serving generated reports to authenticated customers, that order matters because retrying aggressively increases request pressure while retaining every derivative increases the number of objects you must protect and later remove.

Suppose one report produces three thumbnails. Ten thousand reports create 30,000 upload requests and 30,000 retained objects if every derivative has one copy. A burst of 30,000 is a very different workload from the same 30,000 requests spread across a queue. The arithmetic is simple; the operational consequences are not.

What is the bill really made of?

For this pipeline, start with three quantities: upload operations, bytes retained, and authenticated delivery requests. A failed upload consumes an attempt without creating useful output, while a successful retry can create an object that survives for the same retention period as the original attempt. The dominant term is therefore not the isolated 429. It is the combination of object count and retention duration that the pipeline allows to accumulate.

I would model the first release with a deliberately small retention set: the original report, the thumbnail variants needed by the customer-facing report view, and a short-lived working copy only while processing is active. If the system creates three 1600-by-900 variants for each report and keeps all three for 30 days, the storage inventory grows by 90,000 objects per 10,000 reports. That is a retention decision, not a retry decision. Keeping a larger derivative set makes later regeneration easier, but it also enlarges the deletion surface and the amount of data whose access policy must remain correct.

The hard choice is what to stop keeping. Removing intermediate resize outputs reduces retained objects, but a later request may require reprocessing the source and may fail if the source has already expired. Removing the source is a stronger decision: it saves the durable input but makes correction, audit, and regeneration depend on another copy. I would retain the source and customer-visible derivatives, then expire only reproducible intermediates after the processing window. The cost of that policy is extra processing after an unusual recovery event. That is a visible, bounded trade-off.

A storage lifecycle rule can help with age-based cleanup, but a one-day minimum means it cannot express an hour-level thumbnail scratch space. Multipart fragments also need explicit operational attention because there is no automatic cleanup rule for abandoned fragments. Metadata is not a substitute for an index: object listing filters by prefix, not by arbitrary metadata fields. Those limits should shape the retention manifest rather than be discovered during an incident.

Why does the upload rate limit appear during resizing?

Resizing creates fan-out. One authenticated report request may cause several workers to finish at nearly the same instant, and a naive worker pool turns that completion pattern into a request burst. A retry loop can then amplify the burst: ten workers each retrying three times is thirty more requests before the provider has had time to recover.

The first control is a queue with a bounded consumer count, preferably bounded per customer as well as globally. The second is a retry policy that distinguishes a 429 from a malformed request or an authorization failure. A 429 should wait; a 4xx caused by invalid input should surface; a 5xx may be retried under the provider's documented guidance. The storage provider's rate-limit documentation and quotas belong in the runbook, because a client cannot infer a safe ceiling from one successful test.

I initially thought the retry count was the important knob. It was the wrong abstraction. The useful knob is how many jobs are allowed to become requests at once.

Burst matters.

Use one deterministic object key for each report, variant, and content revision. That makes a retry target the same logical object. If an overwrite must never race with another worker, a queue or database coordinator is required: there is no If-Match conditional write here. A client-side lock alone is not enough when workers run on different hosts.

Here is the small part I would put in the worker. It assumes the service has already created a private, signed upload URL for the exact object key. The bearer token is intentionally absent: a presigned URL carries its own authorization, and sending the platform token to that URL is both unnecessary and unsafe. The queue should acknowledge the job only after this function returns successfully.

import os
import random
import time
from pathlib import Path

import requests


MAX_ATTEMPTS = 6
BASE_DELAY_SECONDS = 0.5
MAX_DELAY_SECONDS = 30.0


def check_storage_discovery() -> None:
    response = requests.request(
        method="GET",
        url=os.environ["INFRAI_API_BASE"] + "/v1/discovery",
        headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
        timeout=10,
    )
    if not 200 <= response.status_code < 300:
        response.raise_for_status()


def upload_thumbnail(signed_url: str, thumbnail_path: str) -> None:
    payload = Path(thumbnail_path).read_bytes()

    for attempt in range(MAX_ATTEMPTS):
        response = requests.request(
            method="PUT",
            url=signed_url,
            data=payload,
            headers={"Content-Type": "image/jpeg"},
            timeout=30,
        )

        if 200 <= response.status_code < 300:
            return

        if response.status_code == 429 and attempt < MAX_ATTEMPTS - 1:
            retry_after = response.headers.get("Retry-After")
            try:
                delay = float(retry_after) if retry_after else 0.0
            except ValueError:
                delay = 0.0
            if delay <= 0:
                delay = min(MAX_DELAY_SECONDS, BASE_DELAY_SECONDS * (2 ** attempt))
            time.sleep(delay + random.uniform(0, 0.25))
            continue

        response.raise_for_status()

    raise RuntimeError("thumbnail upload exhausted its retry budget")


if __name__ == "__main__":
    check_storage_discovery()
    upload_thumbnail(
        signed_url=os.environ["THUMBNAIL_SIGNED_URL"],
        thumbnail_path=os.environ["THUMBNAIL_PATH"],
    )
Enter fullscreen mode Exit fullscreen mode

The retry budget is finite on purpose. An endless retry turns a provider limit into a worker leak and hides a broken quota configuration. Add jitter so workers do not wake together, record the final status and object key, and make the queue job idempotent at the consumer: the same report, variant, and revision must not create a second logical job.

The platform's native storage write is discoverable from a public, self-describing API, and its capability documentation includes runnable examples rather than requiring an SDK-specific discovery exercise. One key covers 295 routes across 20 modules, and one plain REST API can be called from any runtime without installing a storage SDK. That is useful when the same media service is already adding several backend capabilities. It does not remove the need for a queue, a retention policy, private access, or quota inspection.

Which storage option fits authenticated delivery?

There is no universal winner here. The access boundary is the deciding axis: customers must receive generated reports and thumbnails only after authentication, so public-read objects and permanent public URLs are out of scope.

Option Useful fit Important boundary
Amazon S3 Mature object controls, lifecycle configuration, multipart workflows, and a broad ecosystem More configuration surface and provider-specific operational choices; strict concurrent overwrite rules still need an application-level design
Cloudflare R2 S3-compatible object storage for teams already operating close to Cloudflare delivery services Compatibility does not mean identical limits or behavior; confirm the exact request quotas and retention features for the workload
Google Cloud Storage Strong fit when reports already live in a Google Cloud data and identity environment Cross-provider portability is not automatic, and the application still owns authenticated delivery and retry scheduling
Backblaze B2 A straightforward object-storage choice for teams prioritizing a smaller storage surface Check the required S3 compatibility details and operational tooling before treating it as a drop-in for every workflow
Infrai storage One key covers 295 routes across 20 modules through one plain REST surface; self-description exposes discovery, schemas, billing metadata, and runnable examples No public/public-read ACL, object versioning, object lock, cross-region automatic replication, or server-side metadata search; browser direct upload also cannot be self-configured with a CORS route

The last row is not a recommendation by default. It is a fit when one plain interface and discoverability reduce integration work for a small set of services, while the application is prepared to keep objects private and coordinate concurrency elsewhere. It is a poor fit for static hosting, a permanent public image link, WORM-style financial retention, or a requirement for automatic cross-region copies.

The same caution applies to the alternatives. S3 compatibility is not a durability argument by itself. A familiar product name does not define the retention policy, and a signed delivery URL does not replace customer authorization at the report API.

How should retention and access be tested together?

Create a test matrix with two independent axes. On the access axis, verify that an authenticated customer can obtain the intended signed object and that an unauthenticated caller cannot turn the object key into a public URL. On the retention axis, verify that a working derivative expires when intended, that the source remains available for regeneration, and that a retry does not create a second logical output.

Then test the burst shape, not only the average rate. Send a small batch, a synchronized batch, and a batch with forced 429 responses. The expected behavior is a bounded queue, increasing delays, useful error records, and eventual success when the quota permits it. A test that only checks a 200 response has not tested this failure mode.

Do not use object versioning or object lock as assumptions in the design: they are unavailable in the storage capability described here. If the report is regulated and an overwrite must be prevented, place the immutable record or write coordination in an external system designed for that requirement. If browser direct upload is required, choose a provider and integration that exposes the needed CORS controls rather than assuming a bucket field is an operational route.

The decision rule is narrow. Keep private, signed delivery when customers are authenticated; queue thumbnail fan-out; retry 429 responses with backoff and jitter; retain only what recovery requires; and select the storage product whose missing controls do not conflict with those obligations. Rate limits are a symptom of burst shape. Retention is the long-term architectural cost.

Further reading

References:

Top comments (0)