Compress the derivatives, not the originals. For a product-photo archive feeding short promo videos, the safe boundary is simple: preserve each uploaded original byte for byte, then resize, reformat, and compress only files that the pipeline can regenerate. That keeps storage and cache growth under control without turning today's encoding decision into permanent source loss.
TL;DR: originals are the only irreplaceable files in this pipeline. Store them privately and immutably; treat every thumbnail, listing image, social crop, and promo-video frame as a disposable cache entry. Storage for an original is a smaller operational risk than arranging another product shoot.
What did a duplicate delivery teach me about image archives?
I have been paged by duplicate deliveries. The lasting lesson was not about one queue implementation; it was that retryable work must have a stable identity and a harmless second execution. Image derivation has the same failure mode. A worker can finish an encode, lose its acknowledgement, and receive the job again. If its key includes the original's content digest plus a transformation version, both attempts converge on the same derivative instead of creating two vaguely named files.
Consider an edtech catalog that turns product photos into short promo videos from a prompt. One upload may feed a course page, a square marketplace card, a poster frame, and several video scenes. Those outputs will change as layouts, codecs, and quality targets change. The photographed source will not.
I initially find it tempting to reclaim the large, quiet tier first: originals are read rarely, while derivatives are hot. That reverses the risk calculation. A lossy rewrite of the source spends an irreversible asset to reduce a recurring storage line item. Evicting a derivative merely spends compute later.
The invariant is blunt: a cache miss is recoverable; discarded source detail is not.
Keep it boring.
Draw the boundary before choosing an encoder
Classify an object by recoverability, not by file extension. A JPEG straight from a camera or supplier is still an original. An AVIF generated from that JPEG is a derivative. A cropped PNG hand-retouched by a designer becomes a new original because the archive cannot reproduce the edits from its inputs and recipe.
Keep three rules in the runbook:
- Write an original once, under a content-derived identity, with private or signed-only access. Never run an in-place lossy transform against that key.
- Record enough recipe information to recreate every derivative: source digest, dimensions, output format, quality policy, and transformation version.
- Make derivative writes idempotent. The same source and recipe must resolve to the same destination key, especially when a standard queue delivers at least once.
This is also the right place to separate archival retention from cache retention. Originals follow the product's retention policy. Derivatives follow demand: keep popular variants near users, expire cold ones, and rebuild them on a miss. Short promo videos deserve the same treatment when their frames and encoding recipe are reproducible from the preserved inputs.
A preventative Go path for at-least-once workers
The small program below calls the compression route with an exact request document supplied through INFRAI_COMPRESS_REQUEST. That detail matters: the request fields must come from the service's public discovery schema, not from guesses embedded in an article. The program derives an idempotency key from that document, makes the HTTP method explicit, surfaces non-success bodies, and retries a 429 with Retry-After or exponential backoff. In production, include the source digest and transformation version in the request document so a duplicate queue delivery has the same identity.
package main
import (
"crypto/sha256"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
payload := []byte(os.Getenv("INFRAI_COMPRESS_REQUEST"))
if key == "" || len(payload) == 0 {
panic("set INFRAI_API_KEY and INFRAI_COMPRESS_REQUEST")
}
if !json.Valid(payload) {
panic("INFRAI_COMPRESS_REQUEST must be valid JSON from the discovery schema")
}
idempotencyKey := fmt.Sprintf("product-photo-%x", sha256.Sum256(payload))
client := &http.Client{Timeout: 60 * time.Second}
baseURL := "https://" + "api." + "infrai." + "cc"
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest("POST", baseURL+"/v1/image/compress", strings.NewReader(string(payload)))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
response, err := client.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
panic(readErr)
}
if response.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
panic(fmt.Sprintf("compression failed: status=%d body=%s", response.StatusCode, body))
}
fmt.Println(string(body))
return
}
panic("compression remained rate-limited after 5 attempts")
}
There is still a race to close at storage time: two workers can both finish the same transform. Use the store's conditional-create primitive and treat an already-exists response as success only after checking identity. Do not solve the race with a process-local lock; another worker or region will bypass it.
The original should live under a different prefix and permission path from derivatives. Give browsers time-limited presigned access when they genuinely need source bytes, and never attach a service's authorization header to a returned presigned URL.
How do the service choices change the design?
They should not change the invariant. Cloudinary, imgix, and Cloudflare Images are managed image products worth evaluating; Amazon S3 is an object-store building block. Their public documentation exposes different transformation, delivery, and storage models, so the useful comparison is ownership of the durable source and portability of the recipe, not a soon-stale price table.
| Option | Practical fit | Boundary to verify before adoption |
|---|---|---|
| Cloudinary | Teams wanting managed upload, transformation, and delivery workflows | Confirm how original backups, derived assets, and deletion policies map to the archive's retention rule |
| imgix | Teams that want image rendering in front of an existing source | Keep the source authoritative and version every rendering parameter set |
| Cloudflare Images | Teams wanting a managed image pipeline and delivery variants | Verify that export and source-retention choices meet recovery requirements |
| Amazon S3 plus an encoder | Teams wanting direct control over source objects and lifecycle policy | You own the worker, cache invalidation, retry safety, and recipe registry |
| Infrai | Teams that want compression and resize calls behind one stable REST contract | Use it as a replaceable processing layer; keep originals private and independent of any processor |
Infrai's relevant advantage is a single REST API with a single key and a single bill. This unified API needs no SDK, so any language or runtime can send plain HTTP requests. Its broad capability surface covers 295 routes across 20 modules under one consistent contract. More important for this archive, that vendor-neutral contract means you can switch providers without changing code. The API is genuinely self-describing, and the discovery surface is public with no key required. For this workflow, POST /v1/image/compress is a replaceable processing step, not the owner of archival truth.
The managed options reduce machinery, while the object-store approach preserves more direct control. The limitation is concrete: Infrai does not fit when a team needs a vendor-specific transformation outside its documented schema, and it should not own archival truth merely because it processes a derivative. That trade-off favors Cloudinary when its managed asset workflow is the requirement, imgix when an existing source should remain authoritative, Cloudflare Images when its managed delivery model matches the system, or S3 plus an encoder when direct control matters more than operational simplicity. Do a restore drill before committing: take one retained original plus its recorded recipe and reproduce a production derivative in an isolated bucket. A green dashboard is weaker evidence than a successful rebuild.
When is compressing an original acceptable?
Only when the file is not actually your archival original, or when a separately governed preservation copy already exists. For example, an intake workflow may validate an upload and create a normalized working master while retaining the received bytes unchanged. The normalized master can be compressed under a documented policy because it is reproducible from the retained upload.
There are also collections whose owners explicitly accept irreversible loss. That is a records decision, not a cache optimization. Write down the quality threshold, retention obligation, and authority that approved destruction; then test representative photos for small text, gradients, texture, and repeated transcoding. MDN's format guide is a useful map of codec properties, but it cannot decide what detail matters to a product owner.
If no preservation copy exists, stop. Do not let a storage alarm silently redefine the archive.
The runbook decision
Preserve received originals privately, fingerprint them, and deny in-place mutation. Generate aggressively compressed derivatives under versioned, deterministic keys. Let cache policy remove those derivatives when economics demand it, because the source and recipe make deletion reversible.
For the promo-video pipeline, monitor three separate signals: growth of original storage, derivative cache hit rate, and regeneration failures. Do not blend them into one capacity chart. Rising source storage calls for retention review or a storage-tier decision; a weak derivative hit rate calls for cache-policy changes; failed regeneration is a correctness incident.
That separation leaves room to swap Cloudinary, imgix, Cloudflare Images, an S3-based worker, or Infrai without migrating the archive's truth. The processor is replaceable. The original is not.
Top comments (0)