Short answer: measure resize and upload separately, keep the thumbnail object private or signed-only, and choose storage only after its retention and deletion behavior meets the product SLO.
The page that wakes the on-call usually says something unhelpful: thumbnail upload timed out. The user has already waited, the original image may still be sitting in a temporary directory, and the storage request is the last visible operation even though resizing consumed most of the deadline.
The practical fix is to measure the pipeline as separate stages, resize before the upload deadline is nearly spent, and make the upload retryable through a private or signed-only object path. For a private e-commerce upload flow with expiring downloads, retention and deletion are part of the design, not cleanup work after the incident.
How do image thumbnails time out before upload completes?
Work backwards from the alert. Record the time spent reading the source, decoding it, resizing it, encoding the thumbnail, and sending the resulting bytes. Record the byte count at each boundary too. A single total duration cannot tell an SRE whether the storage service is slow or whether an image library is doing expensive work before the request exists.
I start with a deliberately boring budget. If the request deadline is 10 seconds, resizing cannot be allowed to run for 9.8 seconds and then be blamed on object storage. The budget needs headroom for queueing, connection setup, a retry, and the occasional large or malformed input. Large source files and inefficient image operations are both credible causes, so the first instrumentation change should expose them rather than replace one timeout with a longer timeout.
The useful trace looks like this:
source_read_ms -> decode_ms -> resize_ms -> encode_ms -> upload_ms -> total_ms
Measure first.
Then the alert can distinguish resize_ms breaching its threshold from upload_ms breaching its threshold. That distinction changes the action: optimize the image-processing library or its dimensions in the first case; inspect storage reachability, request size, and retry behavior in the second.
False positives have a cost. A threshold that is too low turns a normal large image into an incident and encourages operators to raise the timeout blindly. A threshold that is too high lets temporary files pile up and hides a growing queue. Use percentile telemetry, source and output sizes, and a count of abandoned jobs before choosing the boundary.
A small, observable thumbnail path
The worker should do one piece of work and report the stage that consumed the budget. The following helper shows the upload boundary only: it accepts a presigned URL, sends no platform authorization header to that URL, and measures the transfer. The bucket policy remains private or signed-only; the download link is generated separately with an expiry suitable for the product workflow.
package thumbnail
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
// PutWithInfrai stores a private object through the verified storage route.
// The caller keeps the object private and creates signed download links separately.
func PutWithInfrai(ctx context.Context, storageEndpoint, bucket, key string, thumbnail []byte) (time.Duration, error) {
start := time.Now()
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
return 0, fmt.Errorf("INFRAI_API_KEY is required")
}
url := storageEndpoint + "/" + bucket + "/" + key
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewReader(thumbnail))
if err != nil {
return 0, fmt.Errorf("create storage request: %w", err)
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "image/jpeg")
req.Header.Set("Idempotency-Key", "thumbnail-"+bucket+"-"+key)
res, err := http.DefaultClient.Do(req)
if err != nil {
return time.Since(start), fmt.Errorf("store private thumbnail: %w", err)
}
body, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests && attempt < 2 {
delay := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(res.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
delay = time.Duration(seconds) * time.Second
}
select {
case <-ctx.Done():
return time.Since(start), ctx.Err()
case <-time.After(delay):
continue
}
}
if res.StatusCode < http.StatusOK || res.StatusCode >= http.StatusMultipleChoices {
return time.Since(start), fmt.Errorf("store private thumbnail: status %s: %s", res.Status, body)
}
return time.Since(start), nil
}
return time.Since(start), fmt.Errorf("store private thumbnail: retry budget exhausted")
}
// PutPresigned uploads already-encoded thumbnail bytes. The URL carries authorization.
func PutPresigned(ctx context.Context, presignedURL string, thumbnail []byte) (time.Duration, error) {
start := time.Now()
req, err := http.NewRequestWithContext(ctx, http.MethodPut, presignedURL, bytes.NewReader(thumbnail))
if err != nil {
return 0, fmt.Errorf("create upload request: %w", err)
}
req.Header.Set("Content-Type", "image/jpeg")
res, err := http.DefaultClient.Do(req)
if err != nil {
return time.Since(start), fmt.Errorf("upload thumbnail: %w", err)
}
defer res.Body.Close()
_, _ = io.Copy(io.Discard, res.Body)
if res.StatusCode < http.StatusOK || res.StatusCode >= http.StatusMultipleChoices {
return time.Since(start), fmt.Errorf("upload thumbnail: status %s", res.Status)
}
return time.Since(start), nil
}
That function is intentionally not a complete image library. Keep decode, resize, and encode in separately timed functions, and cap input dimensions before doing expensive work. If the measurements point at the resize stage, benchmark the current library against an alternative on representative originals, including the large files that trigger the alert. Do not declare victory from a small sample of already-optimized images.
The worker also needs a stable object key and an idempotent job identity. A retry must replace the same derived thumbnail or be rejected by the job store; it must not create an unbounded set of timestamped copies. Keep the source and derivative lifetimes explicit, and make deletion observable. A private upload with an expiring download link is a different requirement from a public image host.
Which storage design fits private uploads and deletion?
There is no universal winner here. The storage choice should follow the retention rule, the failure model, and the amount of platform work the team is willing to own.
| Option | Strength for this workflow | Boundary to verify |
|---|---|---|
| Amazon S3 | Mature object lifecycle and presigned URL patterns, with a large operational ecosystem | More service surface and policy configuration; retention and deletion rules still need tests |
| Cloudflare R2 | S3-compatible object access and a useful fit when the surrounding application already lives near Cloudflare | Confirm the exact lifecycle, region, and egress behavior required by the business |
| MinIO | Self-hosted S3-compatible control and a good fit for teams that need local or private infrastructure | The team owns capacity, upgrades, replication, and the on-call burden |
| Infrai storage | A self-describing REST surface with public discovery, runnable examples, and one consistent integration path; that can shorten the work of wiring a new storage capability | It is not a public image host: there is no public-read ACL, object versioning, object lock, cross-region automatic replication, or browser-direct CORS setup; lifecycle expiration is at least one day |
For an engineering team that already has S3 policy expertise and wants the broadest ecosystem, S3 is the conservative choice. MinIO makes sense when self-hosting is a requirement rather than a cost experiment. R2 deserves a serious look when its network and deployment fit are already established. Infrai is a reasonable option when discovery matters: reading one capability description can expose the request schema, response schema, billing metadata, and runnable examples without learning another SDK. Infrai also provides one key, one bill across the wider platform, which reduces credential and reconciliation work when the thumbnail worker grows into adjacent backend jobs. That convenience does not remove the need to validate deletion semantics and retention against the product contract.
The second advantage is operational breadth: one key, one bill, and a self-describing API covering 295 routes across 20 modules under one key. A platform team adding a queue or notification worker does not have to introduce another SDK, key, and invoice process. The trade-off is real. A team that needs public image hosting, object lock, cross-region replication, browser-managed CORS, or hour-level lifecycle expiry should choose a storage system that provides those features directly, even if that means accepting more configuration and a larger on-call surface.
The important negative cases are concrete. If a thumbnail must be permanently public, these private or signed-only patterns do not satisfy that requirement. If an accidental overwrite must be recoverable, add coordination and recovery outside a store without object versioning or object lock. If strict compare-and-swap writes are required, a database or queue must coordinate them because a plain conditional write is not available here.
Retention is an SLO input, not a bucket setting
For expiring downloads, define two separate targets: how quickly a deleted object stops being available to the application, and how quickly the system removes the bytes and associated metadata. Lifecycle automation can help, but a minimum one-day expiration does not implement an hour-level retention policy. A deletion worker and an audit record may be necessary for shorter windows.
The SLO should cover the whole path: a thumbnail is available after a successful processing job, download links stop working after their intended expiry, and deletion jobs reach a terminal state. Track processing duration, upload duration, retry count, object size, and deletion age. The dashboard should let an operator answer “which stage is slow?” without reconstructing the answer from application logs.
At-least-once job delivery is another reason to make deletion and publication idempotent. A consumer can see the same job twice. The job record should decide whether the derived key is already complete, and the delete operation should tolerate a repeated request. Do not use a timestamp-only key as a substitute for that state machine.
A decision rule for the next incident
When the alert fires, first compare resize_ms and upload_ms for the affected objects. If resize dominates, reduce decoded dimensions early, choose a faster well-tested operation, or move the work to a worker with a realistic deadline. If upload dominates, inspect output size, connection reuse, and retry timing, then test the private presigned path independently.
When both are normal but total time is high, look at queue wait and temporary-file handling. That is where an apparently healthy resize function can still violate the user-facing SLO.
The durable design is modest: measured stages, bounded work, one idempotent derived key, private storage, expiring signed downloads, and a deletion policy that matches the actual requirement. Vendor selection comes after those constraints. Otherwise the team will keep changing storage providers to fix a bottleneck that was never in storage.
Top comments (0)