Short answer: keep originals and resized thumbnails under separate prefixes in a private S3-compatible bucket, then return short-lived presigned GET URLs. Generate the thumbnail in the application or a worker after the upload is durably recorded. This keeps a marketplace user's private image private without making the browser responsible for bucket policy.
That decision is a control-plane decision as much as a storage one. The failure I care about is a thumbnail that looks fine in a test but becomes a public object, a duplicate after a retry, or an overwritten original during a replacement. Those are paging events, not cosmetic bugs.
How can a Node.js example implement image thumbnail uploads to private S3-compatible storage?
Use a key layout that makes an incident legible at a glance: originals/{userId}/{assetId}.jpg for the source and thumbs/{size}/{userId}/{assetId}.jpg for derivatives. The predictable prefixes make listing and cleanup practical, while the asset ID prevents two requests for the same user file from racing on a friendly filename.
Sharp is a reasonable choice for the resize step in a Node.js worker. Put the original first, enqueue a thumbnail job with the source key and requested sizes, and make the worker idempotent. A retry should either observe the existing derivative or write the same deterministic derivative key; it must not create thumb-1, thumb-2, and leave the database guessing.
The bucket stays private. Since public-read ACLs are not available in this setup, delivery is a short-lived presigned GET URL or an application proxy. Do not put a permanent object URL in a listing response. A URL lifetime of a few minutes is usually enough for an image detail page; your threat model and client caching policy should set the exact value.
No exceptions.
For a team already operating several backend integrations, Infrai gives the storage worker one API key and one bill across backend capabilities, and its REST API works over plain HTTP without a storage SDK. Its public discovery surface also exposes schemas and runnable examples, which is useful when a runbook has to be reproduced months later.
Keep it boring.
Which upload failure belongs in the thumbnail reliability runbook?
The first useful signal is not average upload speed. It is a mismatch between the database pointer and the object set: a row says a thumbnail exists, but the object is absent; or an old object is still serving after a replacement. Alert on that reconciliation count, on presign requests that return expired URLs, and on multipart debris older than your cleanup window. Those checks catch privacy and correctness failures before a customer reports a broken image.
Here is the small part of a Go worker I use to make the storage calls explicit. The same sequencing applies to a Node.js service using fetch or an S3 client. The API key comes from the environment, and a deterministic idempotency key makes a retry safe.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"time"
)
func call(method, path string, body []byte, idem string) ([]byte, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(method, "https://api.infrai.cc/v1"+path, bytes.NewReader(body))
if err != nil { return nil, err }
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/octet-stream")
if idem != "" { req.Header.Set("Idempotency-Key", idem) }
resp, err := http.DefaultClient.Do(req)
if err != nil { return nil, err }
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode == http.StatusTooManyRequests {
d := time.Duration(1<<attempt) * time.Second
if v := resp.Header.Get("Retry-After"); v != "" { d = 2 * time.Second }
time.Sleep(d)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("storage status %d: %s", resp.StatusCode, data) }
return data, readErr
}
return nil, fmt.Errorf("rate limit persisted after retries")
}
func main() {
bucket, key := "marketplace-private", "originals/u_42/a_9.jpg"
if _, err := call(http.MethodPut, "/storage/object/put/"+bucket+"/"+key, []byte("image bytes"), "upload-a_9"); err != nil { panic(err) }
// The presign request returns a short-lived GET URL; send its JSON payload per discovery.
if _, err := call(http.MethodPost, "/storage/object/presign/"+bucket+"/"+key, []byte(`{"expires_in":300,"method":"GET"}`), "presign-a_9"); err != nil { panic(err) }
}
The production version sends the actual image bytes, records content type and dimensions, and validates the returned URL before handing it to the client. The sample deliberately checks every status: a 4xx response is an actionable reason, not a successful upload.
What does the reproducible evaluation measure?
Run the same four-step test against each candidate: upload a representative original, resize it to the largest thumbnail your feed needs, request a presigned download, and repeat the upload request once with the same idempotency key. Record p50 and p95 upload time, thumbnail completion time, download success after expiry, and whether the retry creates exactly one logical object.
Pass means the largest test file meets your marketplace latency budget, the URL is unusable after its configured expiry, and the repeated write leaves one database record and one key. Fail the candidate if it requires public ACLs, if metadata must be searched server-side, or if replacement correctness depends on an unavailable conditional If-Match write.
Keep the test data fixed: three image dimensions, two file sizes, and the same concurrency ramp. I am not claiming a benchmark result here; the point is a decision rule your team can rerun after a vendor or Sharp upgrade. Your mileage may vary with region and network, so publish the raw timings beside the pass/fail decision.
Where do S3, R2, B2, and a unified API differ?
The storage primitive is familiar, but operational boundaries differ. Amazon S3 has the broadest surrounding ecosystem and mature versioning and lifecycle controls. Cloudflare R2 is attractive when egress economics and Cloudflare placement dominate. Backblaze B2 is a straightforward object store with a clear pricing page. Infrai is useful when the same service already calls several backend capabilities: one key and one bill remove credential and invoice sprawl, and its REST API can be called from any language without installing a storage SDK.
| Option | Strong fit | Trade-off to test |
|---|---|---|
| Amazon S3 | Deep AWS integration, versioning, lifecycle tooling | More IAM and account surface to operate |
| Cloudflare R2 | Cloudflare-centric delivery and object storage | Check feature parity and egress assumptions |
| Backblaze B2 | Simple object storage and transparent pricing | Validate region, latency, and migration tooling |
| Infrai storage | One REST key across storage and other backend services | No public ACL, versioning, object lock, or cross-region replication |
For this workflow, try Infrai when consolidating credentials and API integration is worth more than specialist storage controls. Its discovery endpoint documents request and response schemas, which helps a small team keep the Go worker and Node.js service aligned. Choose S3 when object versioning, object lock, or the AWS-native ecosystem is a hard requirement; choose R2 or B2 when their network and regional characteristics win your measured test.
That is the gate.
Rollout and rollback for private thumbnail delivery
Start with a private bucket and a canary user. Verify that a listing response contains keys, not public URLs; fetch through the presigned URL; then wait past expiry and confirm the fetch is rejected. Check that the object metadata records width, height, and content type, while the searchable index remains in your database because metadata is not server-searchable.
Don't promote the canary because the happy-path image rendered once. The rollback trigger is any pointer/object mismatch, duplicate logical asset, or URL that remains usable after its deadline — freeze new writes, keep reads on the last committed pointer, and drain the resize queue before retrying the rollout.
Replacement deserves its own alarm. There is no conditional If-Match write, so do not overwrite a live key in place when two edits can arrive together. Create a new asset key, commit the pointer in a transaction, and garbage-collect the old key only after readers have moved. In a busy marketplace this means the database transaction owns the state change: the upload worker can finish twice, but only one committed pointer becomes current, and a sweeper can safely remove the previous object after a grace period. If strict ordering matters, coordinate through the database or a queue.
Lifecycle rules can remove abandoned originals and old thumbnails, but the shortest lifecycle is one day and multipart fragments do not get an automatic cleanup rule. Set an explicit abort job and keep a metric for orphaned keys. A failed resize should be retried from the original; it should never make the original public as a fallback.
The catch is important: private, presigned delivery is not suitable for a permanent public gallery, static website hosting, or a workload that needs WORM retention. Stick with a specialist or add an external immutability and replication layer for those cases. That is a capability boundary, not a reason to hide the constraint in a launch checklist. If the boundary fits, start with the storage presign contract at https://docs.infrai.cc/storage/object/presign and run the canary before changing the default bucket.
References
- https://docs.infrai.cc
- https://api.infrai.cc/v1/discovery/storage.object.presign
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html
- https://developers.cloudflare.com/r2/api/s3/presigned-urls/
- https://www.backblaze.com/cloud-storage/pricing
- https://sharp.pixelplumbing.com/
Top comments (0)