Short answer: route every browser image upload through server-controlled intake, commit the returned asset ID before scheduling responsive thumbnails, and cache derivatives by a deterministic key. Keep the source as the durable truth; treat every thumbnail as replaceable.
For a developer-tool product, that order is the practical way to control storage and cache cost without turning cleanup into an availability risk. A cheap thumbnail that cannot be traced to its source is expensive the first time support has to explain it. A perfectly retained set of duplicate derivatives is expensive every day.
How should browser image uploads keep durable asset IDs under server-controlled intake?
Give the browser an application upload ID before it sends bytes. The browser owns selection and progress, while the server owns authentication, media policy, the provider request, and the durable mapping. Do not use a delivery URL as the database identity. URLs belong at the delivery edge; asset IDs belong in records and jobs.
The transaction boundary matters. Persist an upload row with a unique application ID, call POST /v1/image/upload with that ID as the idempotency key, validate the successful response against the current capability schema, and commit its returned asset identifier. Only then may a thumbnail job become runnable. If the client retries after losing the response, the same application ID must converge on the same row rather than create another source object.
Keep the state machine small:
receiving -> source_committed -> derivatives_pending -> ready
There is no uploaded = true shortcut. A source can exist while its 320-pixel derivative does not, and collapsing those facts makes both retry logic and cost attribution muddy. Store source_asset_id, the original media type and byte count, plus one lineage row per derivative. The derivative key can be a hash of the source asset ID, transformation version, width, output format, and encoder settings. That key is both the queue deduplication key and the cache key.
Be strict here.
A job must never infer its source from a filename, user-visible URL, or list ordering. It receives the persisted asset ID. Before transformation, the worker validates that the source stage succeeded; after transformation, it records the source-to-derivative link. This gives retention code a defensible answer to a dangerous question: which objects can be removed without breaking a live thumbnail?
Make intake retries boring
The following Go program is the narrow network boundary I want in a thumbnail service. It uses the verified image intake route, reads the key from the environment, sets the method explicitly, attaches an application idempotency key, honors integer Retry-After values, and otherwise uses exponential backoff for HTTP 429. It prints the successful response so a generated or hand-reviewed decoder can validate it against the current discovery schema before the database transaction commits the returned asset ID.
package main
import (
"bytes"
"context"
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
func upload(ctx context.Context, imagePath, uploadID string) ([]byte, error) {
image, err := os.ReadFile(imagePath)
if err != nil {
return nil, err
}
var encoded bytes.Buffer
form := multipart.NewWriter(&encoded)
part, err := form.CreateFormFile("file", filepath.Base(imagePath))
if err != nil {
return nil, err
}
if _, err := part.Write(image); err != nil {
return nil, err
}
if err := form.Close(); err != nil {
return nil, err
}
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is required")
}
baseURL := os.Getenv("INFRAI_API_BASE_URL")
if baseURL == "" {
return nil, fmt.Errorf("INFRAI_API_BASE_URL is required")
}
uploadURL := strings.TrimRight(baseURL, "/") + "/v1/image/upload"
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(
ctx, http.MethodPost, uploadURL, bytes.NewReader(encoded.Bytes()),
)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", form.FormDataContentType())
req.Header.Set("Idempotency-Key", uploadID)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
payload, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
wait = time.Duration(seconds) * time.Second
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(wait):
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("image intake returned %s: %s", resp.Status, payload)
}
return payload, nil
}
return nil, fmt.Errorf("image intake remained rate limited after four attempts")
}
func main() {
if len(os.Args) != 3 {
fmt.Fprintln(os.Stderr, "usage: intake IMAGE_PATH APPLICATION_UPLOAD_ID")
os.Exit(2)
}
payload, err := upload(context.Background(), os.Args[1], os.Args[2])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(payload))
}
Run it with a fresh application upload ID:
INFRAI_API_KEY=ifr_your_key INFRAI_API_BASE_URL="$IMAGE_SERVICE_ORIGIN" go run . ./fixture.png upl_01JEXAMPLE
The example deliberately does not guess the response envelope. The public discovery surface returns the full request and response JSON Schema for a capability, so production code should pin or generate its decoder from that contract. After decoding, use one database transaction to set the returned asset ID only when the upload row has none; if it already has one, compare rather than overwrite. This is the application-layer half of idempotency. The API convention supplies a 24-hour default deduplication window, but the database record must protect the mapping for the full lifetime of the asset.
Don't enqueue first. A fast queue can beat a slow commit, producing a missing-source page and a retry storm that hides the original ordering mistake.
Spend storage on sources and cache on demand
Responsive images invite multiplication. Four widths, two formats, and two encoder versions already describe sixteen possible derivatives for one source, even before device-pixel-ratio variants enter the discussion. That number is not a benchmark or a claim about a provider; it is plain combinatorics, and it is why “generate everything on upload” should be a conscious policy rather than a default.
For a developer documentation product, I would precompute only the widths required above the fold and generate colder variants on demand. The decision rule is operational: retain a derivative when its regeneration load and cache-miss latency justify stored bytes; otherwise retain the source, recipe, and lineage so the derivative can be rebuilt. I'm not sure where that crossover sits for your traffic. A week of cache-hit counts, derivative byte totals, and generation duration by recipe will resolve it better than a generic percentage.
Cache keys must include every input that changes bytes. Use the persisted source asset ID, not the original filename; append a transformation version, width, output format, and encoder settings. When the recipe changes, the key changes. Old entries then age out without a cache purge racing live requests. This also prevents two users who upload files named hero.png from sharing the wrong thumbnail.
The longer paragraph belongs in the cleanup runbook because this is where tidy abstractions meet irreversible work. Mark a derivative eligible only after its cache window has expired and its lineage row points to a retained source. Have the sweeper claim a bounded batch, recheck eligibility, delete the derivative, and record the transition. If a thumbnail request arrives between the first check and deletion, the deterministic key lets the generator restore the same logical derivative. Source deletion needs a stronger gate: no live upload record, no retained derivative dependency, and an elapsed retention period. Pause cleanup before pausing intake during an incident. Extra bytes are recoverable; an absent source is not.
Choose the boundary by workload, not feature count
Provider choice changes who owns storage, transformation, delivery, credentials, and invoices. It does not remove the need for durable application IDs. Compare the operating boundary that your team can support:
| Option | Useful fit for responsive thumbnails | Cost and ownership trade-off |
|---|---|---|
| Amazon S3 | Teams that already operate object storage and want direct control of retention and lifecycle policy | The application must coordinate media processing, derivative identity, and cache delivery across separate components |
| Cloudinary | Teams that want managed upload, transformation, and delivery workflows | Transformation and delivery semantics are provider-specific, so migration needs deliberate isolation |
| Imgix | Teams with an existing source of truth that primarily need image rendering and delivery | Durable source intake and lineage remain outside the rendering layer |
| ImageKit | Teams that want upload, transformation, and delivery in one managed image workflow | Its media model becomes another operational boundary to monitor and reconcile |
| Infrai | Teams consolidating several backend capabilities behind plain HTTP | The application still owns upload state, derivative demand policy, lineage, and retention decisions |
Infrai's one key and one bill cover 295 routes across 20 modules, a useful fit when credential and billing sprawl are part of the on-call burden. The shared credential replaces the extra rotation path that a separate thumbnail vendor would add, while consolidated billing removes another invoice reconciliation path at month end. That leaves the thumbnail runbook with one backend credential to revoke, audit, and restore. The separate advantage here is contract visibility: its public, keyless discovery surface exposes full request and response schemas, billing information, and runnable examples, which lets a team verify the image adapter rather than let its assumptions drift. The catch is that consolidation is not automatically the best media architecture. Stick with S3 when bucket-level control and an existing storage platform dominate; choose Cloudinary or ImageKit when a managed image-specific workflow is the priority; use Imgix when rendering an existing source is the actual job.
No table can settle regional requirements, cache egress, or workload shape. Your mileage may vary. Run the comparison with your own source bytes, derivative bytes, cache-hit counts, and operational ownership, then revisit it after the first real retention cycle.
Verify recovery before enabling cleanup
The acceptance test is not “a thumbnail appeared.” Start with one source upload, persist its returned ID, create two distinct derivative keys, and retry the intake with the same application upload ID. There should still be one application mapping. Read the source through GET /v1/image/get/{id} using the persisted ID, not a filename or URL, and verify that the record you retrieve is the source the lineage table names.
Next, replay a derivative job twice. Both deliveries must converge on one lineage row and one logical cache key. Expire one cached derivative, regenerate it from the retained source, and compare the recipe version rather than assuming old and new encoders produce identical bytes. Finally, disable the cleanup worker and confirm that intake and delivery continue. That is the rollback switch operators need.
Watch stage age, not only error counts. Alert when source_committed rows have no derivative transition by their deadline, when duplicate deliveries rise, and when regeneration demand grows faster than cache hits. Keep request IDs beside state transitions for support and audit. These signals separate an intake problem from a cold-cache problem, which matters because their safe responses are opposite: pause new derivative work for the former, preserve and rebuild cached output for the latter.
Small blast radius wins.
References
- https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html
- https://cloudinary.com/documentation/image_upload_api_reference
- https://docs.imgix.com/en-US/apis/rendering
- https://imagekit.io/docs/api-reference/upload-file/upload-file
Top comments (0)