Storefront Media Lifecycles: Keep Original Image Inputs for Reprocessing

An e-commerce catalog has one constraint that changes this decision: a product photo will probably outlive the thumbnail specification that happens to be current when the seller uploads it. TL;DR: retain the original as an immutable source, generate the derivatives required by today's storefront at upload time, and keep a repeatable reprocessing path for tomorrow's design. On-demand transformation can cover long-tail or experimental sizes, but it should not be the only way the primary product grid gets its images.

That choice separates two promises that are often confused. The original is the durable input. A thumbnail is a disposable, versioned output produced for a particular crop policy, pixel geometry, codec, quality setting, and interface. Keeping only the current thumbnails quietly turns a reversible operation into a future content-acquisition problem.

Why should we keep the original image for derivative reprocessing?

Consider a bounded tabletop incident rather than a claimed production anecdote. A marketplace accepts one product upload, emits square listing images, and discards the upload after processing. Months later, a storefront refresh calls for a wider card with a different crop. The team can resize the square file, but it cannot recover pixels that were cropped away, undo a lossy encode, or reliably infer the seller's intended framing. The refresh is now blocked on another upload or a visibly compromised asset.

The invariant is blunt: a derivative cannot be treated as a lossless archive of its source. Raster resizing discards samples. Cropping removes content. Lossy formats may discard additional information. A later encoder may be better suited to browsers or devices that did not matter when the image first arrived, yet it can work only with the pixels still available.

Pixels do not grow back.

I use a stricter wording in design reviews: deletion of the source is an irreversible schema migration for the media itself. That framing forces the same questions we would ask before dropping a database column. Can the data be reconstructed exactly? Who has verified that every future consumer is represented? What is the rollback plan? Usually, the honest answers are no, nobody, and none.

This does not mean serving the original to shoppers. It means putting the source behind a private object boundary, recording its digest and media type, and exposing only controlled derivatives. MDN's image-format guide is a useful reminder that formats differ in compression, animation, transparency, and browser support; one derivative policy cannot be assumed to remain optimal forever.

The source-and-derivative contract

The smallest workable model has an immutable source object and a derivative key that describes the transformation. A database row can point to the source by a stable asset ID, while generated objects use a policy version rather than a mutable filename such as thumbnail-new-final.

For example, a derivative identity might be computed from the source digest plus a normalized specification:

package media

import (
    "crypto/sha256"
    "encoding/hex"
    "fmt"
)

type Spec struct {
    Width         int
    Height        int
    Fit           string
    Format        string
    Quality       int
    PolicyVersion string
}

func DerivativeKey(sourceDigest string, s Spec) string {
    canonical := fmt.Sprintf(
        "%s|%dx%d|%s|%s|%d|%s",
        sourceDigest, s.Width, s.Height, s.Fit,
        s.Format, s.Quality, s.PolicyVersion,
    )
    sum := sha256.Sum256([]byte(canonical))
    return "derivatives/" + hex.EncodeToString(sum[:])
}

This key makes retries idempotent. Two workers processing the same source and specification converge on the same location, and a policy change creates a new object instead of mutating the old one underneath caches. The code deliberately does not implement decoding or resizing: production image processing should use a maintained codec implementation, enforce resource limits before decoding, normalize orientation according to an explicit policy, and reject unsupported inputs. The contract matters here, not an image library disguised as ten lines of sample code.

Metadata should preserve what is needed to reproduce and audit the result: source digest, detected media type, byte size, pixel dimensions, derivative specification, processor version, and completion state. Treat client-provided filenames and MIME declarations as untrusted labels. Access control belongs at the source-object boundary because retaining originals should not accidentally make full-resolution catalog uploads public.

Backups deserve a separate sentence. Versioned object storage is useful for operator recovery, but object versioning is not the same thing as a declared source-retention policy. The catalog record, source object, lifecycle rules, and backup restoration procedure need a consistent deletion contract.

Should thumbnails be generated on upload or on demand?

For a known storefront set, upload-time processing makes latency and failure visible before the product becomes ready for display. It also gives operators a bounded queue to observe. On-demand processing moves work into a shopper request or a cache miss, which is attractive for rarely used dimensions but couples decoding cost and transformation failure to read traffic.

Neither choice removes capacity planning. Suppose an import contains N sources and each source requires D eager derivatives. The initial work count is N x D; a design refresh that adds one policy version creates another N x D_new jobs. Measure decode time, peak memory, output bytes, and failure rate with representative source dimensions and formats, then set worker concurrency from the limiting resource. Average execution time alone is a poor admission-control signal because a few very large images can dominate memory.

No mode is free.

My default split is operational, not ideological:

Decision	Upload-time generation	On-demand generation
Primary listing and product views	Predictable reads; readiness waits for required outputs	A cold request may pay processing latency
Experimental or rare dimensions	Creates outputs that may never be read	Creates only observed variants
Refresh rollout	Queue is explicit and can be rate-limited	Demand and rollout work compete at cache misses
Failure ownership	Ingestion pipeline owns retries and readiness	Serving path must define fallback and retry behavior
Capacity signal	Upload and backfill arrival rates	Request distribution and cache-miss rate

The limitation of the upload-time default is deliberate overproduction: every required derivative consumes processing and storage even when a product receives no views, and ingestion cannot declare the image ready until its required set succeeds. It is a poor fit for a catalog whose consumers invent many sparse dimensions or whose source is fetched from an authoritative archive only after a request. In those cases, constrained on-demand generation is the better alternative, provided the serving path has a latency budget, request coalescing, a fallback asset, and a finite specification allowlist. The opposite trade-off is just as concrete: on-demand generation avoids unused outputs, but a cold miss now depends on source availability and transformation capacity. A hybrid exists because neither failure domain is universally preferable.

Set an SLO for asset readiness, not merely worker uptime. A worker can be healthy while a queue is old enough that new products miss their publication window. Useful signals include queue age, jobs completed by policy version, permanent rejection count, retry count, source-decode duration, and derivative verification failures. Alert on customer-visible risk, such as the oldest required job approaching the readiness objective; a raw queue-depth threshold without arrival-rate context tends to page during harmless bursts.

The same reasoning applies to on-demand paths. Track cache-miss transformation latency and the proportion of product-image requests that fall back to a known-good derivative. Do not let an arbitrary width query create unlimited cache keys or unbounded compute. Map requests to an allowlist of specifications.

Reprocessing without turning refresh day into an incident

A design refresh should be a controlled backfill. Register the new policy version, generate a small validation set from retained sources, inspect crop behavior and output compatibility, then enqueue the catalog in bounded batches. New uploads can dual-write the current and next policy while the historical queue catches up. The storefront switches only after the required coverage and error budget are acceptable.

Keep the old derivative set during the rollout. Rollback then changes a policy pointer rather than initiating another full encode. Once the new version has met its observation window and old cached references have expired, the obsolete derivatives can enter a lifecycle deletion process. The originals remain because they are the input for the next change, not because every generated object must be kept forever.

A preventative worker path needs explicit state transitions and idempotency. This sketch shows the control flow without pretending that storage or image decoding is trivial:

package media

import (
    "context"
    "errors"
)

var ErrRejectedInput = errors.New("rejected image input")

type Store interface {
    ReadSource(context.Context, string) ([]byte, error)
    DerivativeExists(context.Context, string) (bool, error)
    WriteDerivative(context.Context, string, []byte) error
}

type Transformer interface {
    Transform([]byte, Spec) ([]byte, error)
}

func Process(ctx context.Context, store Store, tx Transformer, sourceID, digest string, spec Spec) error {
    key := DerivativeKey(digest, spec)
    exists, err := store.DerivativeExists(ctx, key)
    if err != nil || exists {
        return err
    }

    source, err := store.ReadSource(ctx, sourceID)
    if err != nil {
        return err
    }
    output, err := tx.Transform(source, spec)
    if err != nil {
        return err
    }
    return store.WriteDerivative(ctx, key, output)
}

Real workers also need bounded input bytes and dimensions, timeouts, cancellation, atomic publication, output validation, and a distinction between permanent rejection and transient failure. Retry a temporary storage error. Do not repeatedly retry a source that violates the accepted-format contract. Keep that classification observable so an operator can tell backlog from poison input.

Test reprocessing as a release path. Golden-image tests can catch unexpected crop and orientation changes, while property tests can enforce maximum dimensions and stable key generation. A staging backfill should use a representative sample that includes unusually tall, wide, transparent, and large sources. After deployment, compare counts by policy version and verify that every active catalog item has its required derivative set before changing reads.

Buy, build, or combine the boundaries

The buy-versus-build decision is wider than an encoder benchmark. A managed transformation service may reduce codec patching and operational labor, while a self-hosted worker may give tighter control over data location, queue behavior, and migration. Lock-in becomes material when asset identity, transformation syntax, or source storage exists only inside one provider's model.

Boundary	Build or self-host favors	Managed service favors	Exit question
Source archive	Direct retention and access-policy control	Lower storage operations burden	Can all originals and metadata be exported intact?
Transformation workers	Custom crop policy and explicit resource controls	Less codec maintenance and fleet ownership	Are specifications portable and reproducible?
Delivery cache	Control over keys and invalidation	Broad operational coverage	Can URLs change without editing catalog data?

I would keep the source ID and derivative specification in an application-owned schema in either case. That is the cheapest place to preserve optionality. The SLO comparison should include on-call load, restore testing, security patching, backfill throughput, dependency failure behavior, and the engineering time required to leave; a monthly invoice by itself answers very little.

There are conditions where retaining every original is the wrong policy. A legal deletion request, contractual retention limit, or data-minimization rule can require source removal. Some workflows also receive an authoritative master from a separate digital-asset system, in which case the pipeline may keep a verifiable reference rather than another indefinite copy. Document that authority and test retrieval before deleting local sources. “We can ask the seller again” is not a recovery design.

The decision rule

Keep an immutable original whenever future presentation requirements can change and the upload is the highest-fidelity input under your control. Generate the small, known set of revenue-critical thumbnails during ingestion; reserve constrained on-demand generation for uncertain or sparse demand. Version every transformation, make reprocessing idempotent, and operate refreshes as rate-limited migrations with rollback.

The capacity question then becomes manageable: how quickly must required derivatives become ready, how much backfill can run without consuming the serving error budget, and how long can old outputs coexist during a rollout? Those are measurable platform decisions. Trying to reconstruct missing pixels is not.

Sources

References:

• MDN Web Docs, “Image file type and format guide”: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types
• OWASP, “File Upload Cheat Sheet”: https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
• RFC 9111, “HTTP Caching”: https://www.rfc-editor.org/rfc/rfc9111