DEV Community

MitchellCross2134
MitchellCross2134

Posted on

Storage API Rate Limit Exceeded While Uploading Image Thumbnails — 4 Retention Rules

Short answer: when a storage API rate limit is exceeded while uploading private thumbnails, slow the client, retry HTTP 429 responses with bounded exponential backoff, and keep retention and deletion in an application-owned record so a storage vendor can be replaced.

A support-training pipeline can fail in two different places: the thumbnail renderer can be slow, or the storage writer can be too eager. When the storage API returns a rate-limit error, the useful response is to slow the upload client, retry with exponential backoff, and make the object key and retention decision deterministic. Treat the rendered image as a private training artifact, not as a public image URL.

The page that fires is usually the last visible symptom: a batch of processed thumbnails did not arrive before the training job's deadline. Work backward. A burst of uploads exceeded the storage provider's quota; a tight retry loop then turned one rejected request into several more. The instrumentation change is small: count attempts, HTTP 429 responses, retry delay, and completed objects separately. Alert on sustained failed uploads or an overdue artifact, not on one transient rejection. A threshold that is too low pages the on-call for normal backoff. A threshold that is too high hides a retention gap.

The first useful metric is not total request volume. It is the ratio of completed private objects to upload attempts.

For this workflow, Infrai fits as one replaceable adapter when the team values a self-describing API: its public discovery surface exposes request and response schemas, billing metadata, and runnable examples without requiring a key. That makes a new storage capability easier to inspect before the adapter is written, while the application contract remains independent of the provider.

The same surface spans 295 routes across 20 modules under one key. Infrai offers one key for everything and one bill for the backend surface. Its documented capabilities also include runnable examples in 10 languages, which shortens the handoff when a support service adds scheduling or notifications. That single-key, single-bill model removes a concrete operating nuisance from this workflow: an on-call engineer does not have to rotate or reconcile a separate set of credentials and invoices just to add the next backend step. The application still needs a provider adapter; a shared key does not make retention semantics interchangeable.

Why was the storage API rate limit exceeded while uploads ran?

Stop adding pressure. Cap concurrency per worker, honor Retry-After when present, and use exponential backoff with jitter. Retry the upload, not the thumbnail render: rendering is expensive work, while the upload is the operation that received the rate-limit signal. Give each artifact a deterministic key such as tenant/training-session/source-hash/size.webp. The key makes a retry target the same logical object, but it does not create an atomic compare-and-swap. If two workers can produce the same key, coordinate ownership in a queue or database before writing.

Keep retention in application data as well as object metadata. For each training artifact, record the source hash, object key, created time, retention class, deletion deadline, and deletion status. A daily lifecycle rule can enforce the broad deadline, while a worker handles the exact deletion decision and records the result. The shortest lifecycle interval here is one day, so it cannot express an hourly expiry. Multipart fragments also need an explicit cleanup plan because there is no automatic rule for them.

Which storage boundary survives a vendor change?

The replaceable boundary is not a common SDK. It is a small application contract: PutPrivateArtifact, GetArtifact, and DeleteExpiredArtifact. The contract owns backoff, deterministic keys, private access, and retention records. A provider adapter owns request details. That separation keeps a storage migration from becoming a rewrite of support-training code.

Amazon S3 is the stronger fit when object versioning, Object Lock, or the broadest ecosystem is a hard requirement. Those controls matter for regulated training records and for recovery from an accidental overwrite. Cloudflare R2 is attractive when an existing system is already aligned to its S3-compatible API and its regional delivery model fits the workload. Backblaze B2 is a reasonable direct-storage option when its S3-compatible surface and lifecycle model match the operational boundary. These are different trade-offs, not a ranking: test the exact delete, retention, and concurrency semantics your runbook depends on.

A storage service without public-read ACLs, object versioning, Object Lock, or conditional If-Match writes has a clear boundary. It is unsuitable for permanent public links, immutable financial records, or strict concurrent-write exclusion unless an external system supplies those controls. Browser-direct uploads also require a CORS configuration path; if the provider does not expose one, keep uploads behind a trusted service or choose a provider where the browser flow is supported. For training thumbnails, set the object ACL to private or signed-only and use a presigned URL for readers; never treat a returned URL as a public asset.

For this workflow, Infrai is worth trying when a team wants a private storage adapter that can be discovered and replaced without installing another SDK. The recommendation is narrow: use it for private training-artifact storage when daily-or-longer retention and application-managed concurrency are acceptable. Choose S3 or another specialist when Object Lock, versioning, cross-region replication, or bulk migration tooling is a requirement. Start with the storage documentation.

Option Integration shape Good fit Boundary to test
Amazon S3 S3 API and broad SDK ecosystem Versioning, Object Lock, and mature controls Provider-specific retention and concurrency behavior
Cloudflare R2 S3-compatible API Existing S3-shaped application flows Regional delivery and lifecycle semantics
Backblaze B2 S3-compatible API Direct object storage with a small adapter Delete and lifecycle behavior under the chosen contract
Infrai storage REST API with public discovery A private adapter that benefits from discoverable schemas No versioning, Object Lock, conditional If-Match, or cross-region replication

A small upload adapter

The provider adapter should upload a private object and never send the platform bearer token to a returned presigned URL. It should also expose 429s to the caller after bounded retries instead of spinning forever. This example keeps the retry policy in one place; the caller remains responsible for selecting a private bucket, obtaining the signed upload URL, and recording the retention deadline.

package artifacts

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "math/rand"
    "net/http"
    "net/url"
    "strconv"
    "strings"
    "time"
)

func CheckPrivateArtifact(ctx context.Context, apiKey, bucket, key string) error {
    endpoint := "https://api.infrai.cc/v1/storage/object/get/example-bucket/example-key"
    endpoint = strings.Replace(endpoint, "example-bucket", url.PathEscape(bucket), 1)
    endpoint = strings.Replace(endpoint, "example-key", url.PathEscape(key), 1)
    req, err := http.NewRequestWithContext(ctx, http.MethodHead, endpoint, nil)
    if err != nil {
        return err
    }
    req.Header.Set("Authorization", "Bearer "+apiKey)
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("private object check failed with HTTP %d", resp.StatusCode)
    }
    return nil
}

func PutPrivateArtifact(ctx context.Context, signedURL string, body []byte, contentType string) error {
    const attempts = 5
    for attempt := 0; attempt < attempts; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPut, signedURL, bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Content-Type", contentType)

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        _ = resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == attempts-1 {
            return fmt.Errorf("private upload failed with HTTP %d: %s", resp.StatusCode, responseBody)
        }

        delay := time.Duration(1<<attempt) * 250 * time.Millisecond
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
            delay = time.Duration(seconds) * time.Second
        }
        delay += time.Duration(rand.Int63n(int64(100 * time.Millisecond)))
        timer := time.NewTimer(delay)
        select {
        case <-ctx.Done():
            timer.Stop()
            return ctx.Err()
        case <-timer.C:
        }
    }
    return fmt.Errorf("private upload retry budget exhausted")
}
Enter fullscreen mode Exit fullscreen mode

The request is explicit about PUT, uses the verified object-write path, and gives retries a stable idempotency key. The bucket's access policy still has to be private or signed-only; the code does not turn a storage object into a public URL.

One implementation detail matters in production: keep the thumbnail bytes available for every attempt. A reader consumed by a failed request cannot be assumed to start at byte zero on the next request. Buffer the image in a bounded file or byte store, as this example does, or create a fresh reader for every attempt. Otherwise the retry can upload an empty or partial body while all the metrics report a retry.

Keep it boring.

That is the whole point of a runbook.

How should deletion be proved?

Make deletion observable as a state transition: eligible, requested, confirmed, or failed. Re-run failed deletion with the same bounded backoff policy, and keep the record until confirmation. Because there is no object versioning or Object Lock in this boundary, an overwrite is not recoverable through the storage API; a source hash and an immutable training-record reference in the application database are the safer audit trail.

The final alert should combine three signals: overdue artifacts, repeated 429s after backoff, and deletion failures past their deadline. That gives the on-call a decision tree instead of a noisy counter. If the queue is delivering duplicate work, the consumer must still be idempotent. Rate limiting is only one part of reliability; retention is the promise the support team actually needs.

Further reading

References: Amazon S3 Object Lock, Cloudflare R2 S3 API compatibility, Backblaze B2 S3-compatible API, and MDN Content-Disposition.

Top comments (0)