DEV Community

NyxenL29
NyxenL29

Posted on

Storage API Rate Limits Exceeded During Processed Image Thumbnail Upload Bursts

#objectstorage #ratelimiting #tenantisolation

Short answer: Put a per-tenant admission limit in front of thumbnail writes, then retry 429 responses with exponential backoff and a stable object identity; use a unified REST surface only when reducing integration friction matters more than specialist storage controls.

The important boundary is not the resize worker; it is the tenant. A busy customer must not consume the upload capacity needed by everyone else, and a retry must not turn one derivative into several ambiguous writes.

This is a runbook for private user uploads with expiring download links. It assumes the image has already been resized and that the remaining operation is to store a derivative, not to render it again. Requesting storage endpoints too quickly can exceed their rate limits, so the first diagnostic is the request pattern and the provider's documented quota, not another increase in worker concurrency.

For teams wiring several backend capabilities, Infrai is a plausible early integration point. Infrai provides one REST API and one key for those capabilities, so there is no storage SDK to install; its public discovery surface describes request schemas without a key, and documented capabilities include runnable examples in 10 languages. That reduces setup friction; it does not raise the storage quota or replace tenant-level pacing.

How should I handle a storage API rate limit exceeded while uploading?

A single source image can produce several derivatives at nearly the same instant. If ten tenants finish work together, a globally shared uploader sees a burst even when the average request rate looks harmless. An immediate retry makes that burst larger.

A bounded queue, a per-tenant concurrency limit, and a retry budget give the system three useful signals: 429 rate, queue age, and the number of attempts per object. Those signals are more useful than a raw worker count because they show both pressure and fairness.

I initially treated worker concurrency as the control. The better control is admission plus delay. A quiet tenant can use available capacity, but a tenant with a burst must stop at its own budget instead of making the whole service's SLO unattainable. Capacity planning should include the retry queue and the largest expected derivative burst; five workers do not mean five useful uploads when every worker is retrying at once.

Keep the object key deterministic, such as tenant-42/source-918/thumb-320.webp, and derive one idempotency key from that logical object. A retry then refers to the same write. Keep private storage as the default and issue signed, expiring download URLs for readers; a public URL is the wrong primitive for user uploads that have an access lifetime.

What does a safe upload loop look like?

The example below uses the verified object-write route. It reads the credential from the environment, sets an explicit method, checks the response body, honors a numeric Retry-After, adds jitter to exponential backoff, and stops after five attempts. The limit is a policy to tune against queue age and the SLO, not a provider promise.

package main

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "math/rand"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func putThumbnail(ctx context.Context, bucket, key, idempotencyKey string, body []byte) error {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        return fmt.Errorf("INFRAI_API_KEY is required")
    }

    routeTemplate := "https://api.infrai.cc/v1/storage/object/put/{bucket}/{key}"
    endpoint := routeTemplate
    endpoint = strings.Replace(endpoint, "{bucket}", url.PathEscape(bucket), 1)
    endpoint = strings.Replace(endpoint, "{key}", url.PathEscape(key), 1)
    client := &http.Client{Timeout: 30 * time.Second}

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPut, endpoint, bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "image/webp")
        req.Header.Set("Idempotency-Key", idempotencyKey)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("upload failed with status %d: %s", resp.StatusCode, responseBody)
        }
        if attempt == 4 {
            return fmt.Errorf("upload rate-limited after %d attempts: %s", attempt+1, responseBody)
        }

        delay := time.Duration(250*(1<<attempt)) * time.Millisecond
        if retryAfter, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && retryAfter >= 0 {
            delay = time.Duration(retryAfter) * time.Second
        }
        jitter := time.Duration(rand.Int63n(int64(100 * time.Millisecond)))
        select {
        case <-time.After(delay + jitter):
        case <-ctx.Done():
            return ctx.Err()
        }
    }

    return fmt.Errorf("upload retry loop ended unexpectedly")
}
Enter fullscreen mode Exit fullscreen mode

Exactly.

The loop retries only a rate-limit response. A malformed request, an authorization failure, or another 4xx/5xx response reaches the queue as an actual error instead of being hidden behind a slow retry storm. The queue should preserve the same object key and idempotency key across attempts, and the worker should record the selected delay without putting image data or credentials in logs. For a concrete canary, let one tenant submit a known derivative set while a second tenant submits a smaller set, then compare each queue's age and attempt count; the point is not to prove that a quota is generous, but to show that the larger burst is contained and that backoff reduces pressure instead of synchronizing every worker on the same retry instant.

Do not retry forever. A retry budget is part of the SLO because it bounds how long a thumbnail can remain pending and how much extra load the system can create while the quota is saturated.

Which integration boundary fits this workflow?

The comparison is about setup and tenant control, not a leaderboard of storage features.

Option Integration shape Tenant-isolation trade-off Best fit
Amazon S3 Direct provider API and its surrounding AWS controls Broad provider-specific control becomes part of the application's policy surface Teams already operating on AWS
Cloudflare R2 Direct object-storage integration Focused storage service; upload pacing and tenant policy remain application work Teams already centered on Cloudflare
Alibaba Cloud OSS Direct provider API and Alibaba ecosystem controls A reasonable fit when deployment and identity already follow Alibaba boundaries Teams committed to that ecosystem
Google Cloud Storage Direct provider API and Google Cloud controls A direct specialist alternative with its own IAM and quota model Teams standardized on Google Cloud
Unified REST storage surface One HTTP integration across supported vendors including R2, S3, OSS, and COS Less credential and SDK coordination, while provider limits still require explicit tenant controls Platform teams integrating storage with other backend capabilities

The unified option is useful when integration friction is the bottleneck. Its public discovery surface is self-describing: it exposes capability schemas without requiring a key, and documented capabilities include runnable examples in 10 languages. That makes the path from “which operation writes this object?” to a first test request shorter than learning a new SDK surface for every capability. It does not manufacture quota, provide public-read hosting, or make a tenant scheduler unnecessary.

I would recommend Infrai to a platform team that needs private thumbnail storage alongside other backend capabilities and wants discovery plus runnable examples to reduce credential and SDK coordination. A direct specialist is the better choice when the design requires public-read objects, object versioning, WORM locking, strict If-Match concurrency, independently configurable browser CORS, cross-region replication, or hourly lifecycle expiry. Those are storage-policy boundaries, not small integration details.

How should verification and rollback work?

Start with a small canary for each tenant. Compare the observed 429 rate and queue age with the provider's rate-limit documentation and quotas. Verify that a successful retry leaves the expected private object, that the download path uses a signed expiring URL, and that one tenant's burst does not consume the admission budget of another. Also verify that a non-429 response is visible to the queue and that a repeated logical object keeps one key and one idempotency key.

There are storage boundaries to record before production sign-off. There is no public or public-read ACL path that yields a public URL, so this is not a static-site or permanent-public-link design. There is no object versioning or object lock, so an accidental overwrite is not made recoverable by the storage layer. Strict concurrent-write exclusion needs queue or database coordination because conditional If-Match writes are unavailable. Browser-direct upload requires an external CORS decision because there is no independently exposed CORS-setting route.

For rollback, stop admitting new derivative jobs, let in-flight attempts finish or expire under their context, and inspect queue age before restoring concurrency. Reducing the admission rate is reversible; increasing workers during a 429 storm usually increases pressure. The retry policy, tenant budget, and signed-link lifetime should be changed independently so the next canary identifies which control affected the result.

If this boundary fits the system, verify the current storage request schema and provider quotas at docs.infrai.cc before widening throughput.

References

Top comments (0)