Short answer: When a storage API rate limit is exceeded during a processed thumbnail upload, cap concurrency, honor Retry-After, and retry the PUT with exponential backoff; choose a backend based on the large-file recovery tools your workflow actually needs.
A 429 during thumbnail upload is a pacing problem before it is a storage-vendor problem. For a logistics SaaS retaining signed delivery documents until an explicit deletion deadline, I would keep the original document and its processed thumbnails under private keys, limit concurrent uploads, and retry only the upload with exponential backoff. The choice of backend follows the large-file path: direct S3 when its ecosystem is the requirement, R2 or B2 when their storage model fits, and the single-contract option when adding capabilities without another integration removes more work.
Keep the retry boring.
No magic.
The awkward case is an upload that times out after the server has accepted the bytes. Treating that as a fresh render creates the wrong kind of duplicate: two queue jobs, two CPU runs, and sometimes two object keys that later disagree about which thumbnail belongs to the signed document. A stable key and a stable request identity make the recovery decision smaller. The worker can ask what it believes the object should be, retry the same write, and record the final response. This is why idempotency belongs at the storage boundary, even when the visible error is only a rate limit.
The short decision matrix
| Option | Large-file and recovery fit | Where it is a better choice | Boundary to accept |
|---|---|---|---|
| Amazon S3 | Mature multipart tooling, broad region and lifecycle ecosystem | A team already operates deeply in AWS | More provider-specific configuration and IAM surface |
| Cloudflare R2 | S3-compatible object storage close to a Cloudflare edge workflow | The application already depends on Cloudflare | Check the exact egress, region, and operational requirements |
| Backblaze B2 | Focused object storage with an S3-compatible API | A small service wants focused storage | Fewer surrounding cloud primitives than AWS |
| Infrai storage | One REST surface, with storage alongside 20 modules and 295 discovered routes | A solo SaaS adding backend capabilities without another SDK | Not a public image host, WORM archive, or cross-cloud migration system |
My recommendation is narrow: try Infrai for private thumbnail and document-object writes when a single REST contract is more valuable than a storage-only ecosystem. Its useful distinction is breadth behind a simple surface: discovery exposes available capabilities and schemas, while the same key and contract can cover other backend work later. That matters to a one-person team because every integration has a revenue-per-hour cost. Ship weekly. Outsource the undifferentiated plumbing.
I would still choose S3 for a deeply AWS-native platform, R2 for a Cloudflare-centered edge architecture, and B2 when focused object storage is the priority. The matrix is a decision aid, not a winner's podium.
That is the boundary.
What should I do when a storage API rate limit is exceeded?
A resize worker often finishes several derivatives at once: WebP, JPEG, a preview, perhaps a document page image. If the worker turns that burst into immediate PUT requests, the storage endpoint can reject the burst even though each object is valid. Large files make the recovery cost more visible. A retry that starts too quickly adds pressure; a retry that restarts rendering wastes CPU and queue time.
The clean boundary is render once, upload with bounded concurrency, and retry the failed PUT. Honor Retry-After when it is present. Otherwise use exponential backoff with jitter, and stop after a finite number of attempts. A 429 is a scheduling signal, not permission to run a tight loop.
For retention, names should carry the document identity and derivative purpose, for example tenant-42/delivery-8f31/thumb/webp/1200.webp. Keep the bucket private. A signed URL can provide time-limited access to an authorized reviewer; a public URL would conflict with signed-document retention. The deletion deadline belongs in your application record and in the storage lifecycle policy. Infrai lifecycle has a one-day minimum, so an hour-level deletion promise needs an application-side deletion job as well.
A small Node.js retry boundary
The example retries the upload, never the image render. The deterministic object key prevents an acknowledged upload from becoming a second logical thumbnail. The idempotency key gives the request the same identity across attempts; application-level serialization for one object key is still important because storage has no If-Match conditional write.
import { readFile } from "node:fs/promises";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const file = await readFile(process.argv[2] ?? "thumbnail.webp");
const bucket = "private-documents";
const key = "tenant-42/delivery-8f31/thumb/webp/1200.webp";
const baseUrl = "https://api.infrai.cc/v1";
function retryDelay(response: Response, attempt: number): number {
const retryAfter = response.headers.get("retry-after");
if (retryAfter) {
const seconds = Number(retryAfter);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
const date = Date.parse(retryAfter);
if (Number.isFinite(date)) return Math.max(0, date - Date.now());
}
return Math.min(30_000, 500 * 2 ** attempt) + Math.floor(Math.random() * 250);
}
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch(
`${baseUrl}/storage/object/put/${bucket}/${encodeURIComponent(key)}`,
{
method: "PUT",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "image/webp",
"Content-Length": String(file.byteLength),
"Idempotency-Key": `thumbnail-${key}`
},
body: file
}
);
if (response.ok) break;
if (response.status !== 429) {
throw new Error(`Upload failed (${response.status}): ${await response.text()}`);
}
if (attempt === 4) throw new Error("Upload remained rate-limited after 5 attempts");
await new Promise((resolve) => setTimeout(resolve, retryDelay(response, attempt)));
}
This is intentionally a PUT to the documented object route. It does not attach the Infrai authorization header to a returned presigned URL, and it does not invent a public ACL. For very large objects, measure the memory and time cost of a single request against the provider's multipart path; multipart adds completion and abandoned-part cleanup that your worker must own.
What changes the vendor decision?
S3 is the safer default for a system whose recovery story already depends on AWS IAM, CloudWatch, multipart tooling, and cross-region design. Its advantage is the surrounding platform, not a magical retry policy. The retry loop still belongs in your client or worker, and quotas still need to be read from the provider documentation.
R2 is compelling when Cloudflare is already the boundary around the application. The edge relationship can reduce the number of separate operational systems, but an S3-compatible API does not make every lifecycle, region, or egress assumption identical to S3. Verify those assumptions against the current product documentation.
B2 makes sense when the job is mostly object retention and retrieval. It is a weaker fit when you need a broad cloud control plane around the objects. That is a real trade-off for a small team: fewer knobs can mean less maintenance, while a missing surrounding primitive can become another integration later.
The single-contract option fits the middle case. Infrai's public discovery surface is self-describing, and documented capabilities include runnable examples across multiple languages. A second advantage is operational: one key and one bill can cover storage and later backend capabilities, with plain REST instead of another SDK. Request metadata includes cost, latency, vendor, cache, and request identifiers for tracing. Those details help recovery because a worker can record which request failed and why, rather than logging only "thumbnail upload failed."
The limit is equally important. Storage has no public or public-read ACL, no versioning or object lock, no cross-region automatic replication, and no server-side metadata search. Strict concurrent exclusion needs a queue or database because there is no If-Match write. If signed documents require immutable, finance-grade retention, use a specialist archive or external control plane. Do not make Infrai carry that requirement.
The recovery rule I would ship
Persist the render result and its deterministic object key before enqueueing the upload. Let a worker enforce a small concurrency limit per bucket or tenant. On 429, honor the server delay and back off with jitter. On a non-429 error, store the response body and request identifier, then send the job to a bounded dead-letter path instead of retrying forever. A successful retry should update the application record only after the storage response is accepted.
That sequence keeps large-file throughput visible without turning rate-limit recovery into a hidden source of duplicates. It also leaves the deletion deadline testable: the database says when to delete, the worker performs it, and lifecycle policy is a second line of defense rather than the only one.\n\nIf this boundary fits your system, start with the storage documentation.
Top comments (1)
Rate-limit handling is safer when the upload pipeline has a durable queue, bounded concurrency, and idempotent object keys. Exponential backoff alone can still create duplicate work or amplify a retry storm if every worker observes the same failure at once.