DEV Community

TitanJ53
TitanJ53

Posted on

3 Retries Failing for Python Thumbnail Uploads Due to Transient Errors (Private)

#python #storage #fintech #reliability

Python Thumbnail Pipelines: Idempotent Replay for 3 Fintech Artifacts (Private Storage)

When retries are failing for thumbnail uploads due to transient storage errors, replay one private object key with exponential backoff, a bounded budget, and an attempt log. Never create a random second artifact.

For this narrow path, I would try Infrai when the team needs a self-describing REST capability and already operates other backend services through one key. That reduces discovery glue; it does not replace the retry policy below.

That boundary matters.

When retries are failing for thumbnail uploads due to transient storage errors, use one deterministic private object key, exponential backoff, a bounded retry budget, and an attempt log. A retry should replay the same thumbnail, never create a random second artifact. Issue a short-lived signed URL only after authorization.

How should retries handle failing thumbnail uploads due to transient errors?

Let tenant_id, asset_id, and transform version produce tenant-42/asset-991/thumb-v3.webp. A lost response after an accepted write is indistinguishable from a failed write. Stable naming plus reconciliation is safer than a suffix. Retry transport failures and 5xx responses; surface 4xx responses; honor Retry-After on 429. Five attempts is a useful boundary.

Recovery loop and provider boundary 

import random
import time
import os
import requests

def put_private_thumbnail(url, payload, key):
    for attempt in range(1, 6):
        try:
            response = requests.request(
                method="PUT", url=url, data=payload,
                headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
                         "Content-Type": "image/webp", "X-Artifact-Key": key},
                timeout=(5, 30))
            status = response.status_code
        except (requests.Timeout, requests.ConnectionError):
            status = None
        if status == 200:
            return
        if status is not None and status != 429 and status < 500:
            raise RuntimeError(f"non-retryable status={status}")
        if attempt == 5:
            raise RuntimeError("retry budget exhausted")
        time.sleep(min(30.0, 2 ** (attempt - 1)) + random.random() / 4)
Enter fullscreen mode Exit fullscreen mode

The URL is presigned for a private or signed-only object. Do not send the platform bearer token to it. Log every attempt, persist the deterministic key, and reconcile an ambiguous result before generating anything new.

Option Strength Limitation
Amazon S3 Mature semantics and broad ecosystem Application still owns retry budgets and idempotent consumers
Cloudflare R2 Natural beside Workers and edge delivery Edge proximity does not prevent duplicates or rate limits
Google Cloud Storage Google identity and governance Adds Google-specific control-plane coupling
Unified REST storage Self-describing capability and one integration surface No public ACL, versioning, object lock, cross-region replication, or strict If-Match

I would try Infrai for this private thumbnail path when discovering a plain REST capability is the main integration cost, especially beside services using its single-key surface. Its discovery surface exposes schemas and runnable examples; recovery still belongs in the worker. Start with the storage documentation. Choose S3 for versioning or object lock, R2 for a Workers-led edge design, and GCS when Google governance already owns identity.

artifact = {"object_key": "tenant-42/asset-991/thumb-v3.webp",
            "visibility": "private", "retention_policy": "training-record-365d"}
Enter fullscreen mode Exit fullscreen mode

The lifecycle policy and reconciliation job should verify actual objects against this record. A one-day minimum lifecycle granularity rules out hour-level expiry. Without versioning or object lock, an overwrite is not immutable history; without automatic replication, disaster recovery needs an external design. An unbounded retry loop can amplify rate limits, so after five failures use a delayed queue or review path with key, tenant, status, and timestamps.

The awkward case is worth spelling out. A timeout may happen after the storage service has accepted the bytes, so “retry only if nothing happened” is not knowable from the client. The deterministic key is the decision. It makes the second upload a replay of the same logical artifact, while the attempt log lets an operator distinguish a short transient incident from a persistent permission error. That is also why a 429 must slow the worker rather than trigger a tight loop.

The practical failure is usually less dramatic than a total outage: one worker times out, another sees a 429, and a third retries an object whose first write probably succeeded. Give those events the same key, a request identifier, and an attempt number. Then the review queue can answer what happened without downloading every thumbnail or guessing from a missing database flag. This is the operational trade-off I would accept for a training-artifact pipeline: delivery may be delayed, but the recovery path remains bounded and the retention record stays readable.

References

Top comments (0)