When a digital asset management library promises a thumbnail, a preview stream, or a downloadable master, its transformation presets become an operational contract. The hard choice is where to perform the work: during upload, when the original is still warm, or on demand, when a user actually asks for a rendition.
Short answer: use upload-time processing for derivatives that every search or browse request needs, and on-demand processing for expensive or rarely used variants; keep both paths governed by the same transformation preset and status model.
How should digital asset management teams treat transformation presets as contracts?
A preset is more than a width, codec, or bitrate. It is the named agreement between ingestion, storage, search, delivery, and the people who operate them. poster-small-v3 should mean the same output requirements no matter which worker runs it. That name belongs in metadata, job events, tests, and retention policy.
I write the contract before choosing a queue. For each preset, record the source assumptions, output container, codec, dimensions, audio policy, color handling, maximum processing time, and what happens when a source cannot satisfy the request. A missing audio track is not the same state as a failed transcode; callers need to distinguish unavailable, queued, ready, and rejected.
Three fields carry surprising weight: a stable preset identifier, a content hash of the source, and a transformer version. Together they make an idempotency key. Re-uploading the same object should not create a second billing event or two conflicting thumbnails, while changing the preset should produce a new derivative rather than silently replacing the old one.
The contract also defines observability. Emit an event with the asset ID, preset ID, source hash, attempt number, duration, and terminal state. Do not put user-supplied filenames in metric names. They create an accidental high-cardinality bill and make delivery incidents hard to see.
How do upload-time and on-demand choices change failure surfaces?
Upload-time work gives the indexer predictable data. A search result can point to a ready poster without waking a worker on the first click. The trade-off is ingest latency and storage: a library that accepts ten-minute 4K videos may spend substantial compute creating derivatives that nobody opens.
Cold starts hurt.
On-demand work keeps ingestion responsive and avoids speculative output. It also moves latency into the reader journey. A cold cache, a burst of first views, or a failed retry can turn a harmless browse action into a blank player. This is where rate limits and back-pressure matter more than nominal transcoding speed.
The useful compromise is a small eager set plus lazy expansion. Generate the search thumbnail and a low-bandwidth preview at ingest; defer unusual aspect ratios, editorial mezzanines, and alternate language tracks until requested. Put a per-asset lock around creation so twenty simultaneous requests collapse into one job and nineteen wait on the same result.
Here is the decision record I keep beside the preset registry:
| Derivative class | Default timing | Why | When to change it |
|---|---|---|---|
| Search thumbnail | Upload | Search pages need a bounded, cacheable response | Skip for private assets never indexed |
| Browser preview | Upload for small files; on demand for large video | Balances first-view latency with ingest cost | Move lazy when ingest queues exceed the SLO |
| Rare crop or format | On demand | Avoids unused storage and compute | Precompute after an observed demand threshold |
A preset registry keeps workers honest
The registry should be data, not a scattered set of command-line flags. A worker receives an immutable preset snapshot with the job. That prevents a deploy halfway through a batch from producing two meanings for the same name.
from dataclasses import dataclass
@dataclass(frozen=True)
class Preset:
name: str
container: str
video_codec: str
max_width: int
max_height: int
audio_required: bool
version: int
PRESETS = {
"poster-small-v3": Preset(
name="poster-small-v3",
container="webp",
video_codec="none",
max_width=640,
max_height=360,
audio_required=False,
version=3,
),
"preview-h264-v2": Preset(
name="preview-h264-v2",
container="mp4",
video_codec="h264",
max_width=1280,
max_height=720,
audio_required=False,
version=2,
),
}
def idempotency_key(asset_id: str, source_sha256: str, preset: Preset) -> str:
return f"{asset_id}:{source_sha256}:{preset.name}:{preset.version}"
The code does not decide upload versus on demand. That policy belongs to the asset class and its access pattern. The code does make the output addressable, testable, and safe to retry.
What should a DAM measure before switching processing timing?
Start with user-visible measures: time from upload acceptance to searchable status, time from rendition request to first byte, and the percentage of requests served from a ready derivative. Pair them with queue age, worker saturation, retry counts, and bytes retained per source hour. Averages hide the painful tail, so inspect p95 and p99 for first-view latency.
I once treated a rising queue as a capacity problem and added workers. The real issue was a preset that preserved a source frame rate no browser needed. CPU usage fell after the contract capped that output, but the more important fix was making the policy explicit so it could not return in the next migration. The investigation took an afternoon because the dashboard showed only aggregate queue depth: it did not split jobs by preset version, source duration, or whether the request came from ingest or playback. Now those dimensions are attached to each event, and an alert points to the contract field that changed. That extra context costs a few columns and saves a long incident call.
Test the contract with fixtures that resemble the library, including variable frame rate video, rotated phone footage, silent clips, transparent images, and objects with misleading extensions. Assert metadata as well as pixels: duration, orientation, color profile, and content type are part of what downstream systems consume.
There is uncertainty in any cost forecast because demand changes with the catalog. Your mileage may vary. Run a shadow period that records which lazy derivatives would have been requested, then use that trace to set an evidence-based eager threshold.
Roll out the contract in small, reversible steps
Version presets instead of editing them in place. During migration, write both the old and new derivative references, compare dimensions and playback metadata, and keep the old object until consumers have switched. A canary collection containing difficult media catches regressions before the whole library is reprocessed.
The catch is operational complexity: two timing paths mean two retry policies, two alert shapes, and a cache invalidation story. This design is not suitable when the team cannot operate asynchronous jobs or retain enough storage for a short overlap. Stick with a single eager path for a small catalog, or choose a managed media pipeline when owning workers, codecs, and patching would distract from the product.
Make deletion part of the contract too. When an asset is removed, enqueue derivative cleanup and verify that search indexes no longer expose stale URLs. A transformation preset is successful only when the complete lifecycle is predictable.
Top comments (0)