Short answer: process a small, immutable preview at upload, then keep full promo-video transformations on demand behind a versioned preset contract. That split limits retained bytes and queue pressure while preserving a reproducible audit trail for a payment-facing system.
The bill is made of more than encoder CPU. For a digital asset management (DAM) system, the dominant term is usually retained media multiplied by replicas, derivative count, and delivery egress. A 30-second promo can produce several resolution and codec variants; keeping every experiment forever turns a convenient preview feature into a retention liability. The operational question is therefore not “can the worker render this file?” It is “which bytes deserve to exist before a user asks for them, and for how long?”
What should digital asset management transformation presets promise?
A preset is an operational contract, not a bag of encoder flags. Its identity should include an immutable name, an input constraint, an output MIME type, a bounded resolution or duration, and the policy that governs retention. The contract also states what happens when a request is repeated: the same asset revision, preset version, and source digest must map to one idempotency key and one ledger-visible result.
That is the contract.
I write that identity into a manifest before scheduling work. The manifest is append-only; a corrected preset gets a new version. This matters during reconciliation because an operator can answer which definition produced a file without trusting a mutable configuration table.
package transform
import "crypto/sha256"
type Preset struct {
Name string
Version int
InputMime string
OutputMime string
MaxWidth int
MaxSeconds int
RetainDays int
}
type Request struct {
AssetID string
Revision int
SourceHash [32]byte
Preset Preset
}
func (r Request) IdempotencyKey() [32]byte {
return sha256.Sum256([]byte(r.AssetID + ":" +
string(rune(r.Revision)) + ":" +
r.Preset.Name + ":" + string(rune(r.Preset.Version))))
}
This key illustrates the boundary, not a substitute for canonical serialization: production code should encode each field in a stable binary or JSON form, then hash it. The important property is that a retry cannot silently create a second billable derivative or a second audit event.
Upload processing or on-demand rendering: how do you choose?
Start with the user journey and the evidence obligations. Upload-time work is appropriate for a low-cost poster frame, a short H.264 preview, and metadata extraction that makes search useful immediately. It gives reviewers fast feedback and lets policy checks run before an asset enters a campaign. On-demand work is better for rarely requested 4K or alternate-audio variants, because the system pays the compute and storage cost only when a consumer asks.
| Decision point | Upload-time derivative | On-demand derivative |
|---|---|---|
| First preview latency | Predictable | Depends on queue and cold workers |
| Retained bytes | Always present | Kept only under a retention policy |
| Retry behavior | Must absorb upload bursts | Must absorb request spikes |
| Audit shape | One event per asset revision | One event per request and cache hit |
| Best fit | Moderation and catalog preview | Long-tail campaign formats |
The split is a policy choice, not a permanent architecture. Measure request frequency by preset version, then promote a hot on-demand variant to an upload-time derivative only after the retention cost is accepted. Your mileage may vary when campaign traffic is seasonal; a monthly average hides a launch-day queue that can violate a review deadline.
Making cost and retention visible
I keep source objects, derived objects, and transient worker files in separate accounting domains. A source digest links all three, while a retention record says when a derivative may be deleted. Deletion is an auditable state transition, never an implicit object-store lifecycle surprise. The ledger records bytes reserved, bytes released, and the policy version that authorized release.
This is where teams often stop keeping the wrong thing: intermediate image sequences and failed encodes. They are useful during a debugging window, but retaining them as customer assets inflates storage and creates a second privacy surface. The catch is that deleting them removes forensic detail; when a disputed render arrives after the window, you may have only the source hash, preset version, and error classification. That trade-off should be explicit in the contract. In a payment campaign, a reviewer may report a subtitle mismatch weeks after publication; without the intermediate timeline, the team can prove which source and preset were used, yet cannot inspect the exact frame ordering that led to the disputed result, so the retention decision directly changes the evidence available during reconciliation.
Use bounded queues and a per-tenant concurrency budget. A retryable decode error can be retried with backoff; an input that violates the preset's declared MIME or duration limit should be rejected deterministically and recorded once. A worker acknowledgement happens after the derivative and audit event are durably committed, so a process crash cannot mark a job complete while the file is missing.
A small Go worker with an exactly-once mindset
The worker below separates claim, render, and commit. The repository implementation would wrap the commit in a database transaction and enforce a unique constraint on the idempotency key. No API-specific SDK is required; the renderer can be a local process or an HTTP service behind the same interface.
type Renderer interface {
Render(input []byte, preset Preset) ([]byte, error)
}
type Store interface {
Claim(key [32]byte) (alreadyDone bool, err error)
PutDerivative(key [32]byte, data []byte, mime string) error
AppendAudit(key [32]byte, event string) error
Commit(key [32]byte) error
}
func Process(r Request, src []byte, renderer Renderer, store Store) error {
done, err := store.Claim(r.IdempotencyKey())
if err != nil {
return err
}
if done {
return nil
}
data, err := renderer.Render(src, r.Preset)
if err != nil {
_ = store.AppendAudit(r.IdempotencyKey(), "render_rejected")
return err
}
if err := store.PutDerivative(r.IdempotencyKey(), data, r.Preset.OutputMime); err != nil {
return err
}
if err := store.AppendAudit(r.IdempotencyKey(), "render_stored"); err != nil {
return err
}
return store.Commit(r.IdempotencyKey())
}
The render_rejected event is intentionally distinct from a service failure. Compliance reviewers need to distinguish an invalid customer input from an unavailable dependency, and the retry policy differs. Keep request IDs, source hashes, preset versions, and timestamps in structured logs; do not put payment credentials or raw prompts in them.
Failure modes that break the contract
Three failures recur in production designs. First, a mutable preset changes while a job is queued, so a retry produces a different video under the same identifier. Versioning the preset and copying its manifest into the job removes that ambiguity. Second, a cache hit is treated as a new transformation and charged twice internally. Make cache hits observable audit events with zero additional reservation. Third, deletion runs before downstream consumers finish reading; use a lease or reference count and record the release decision.
Small details decide the outcome.
Standards still matter at the edge. Validate declared MIME types against detected content, publish accurate Content-Type and duration metadata, and provide fallbacks that browsers can decode. The MDN media formats guide is a useful compatibility checklist, but it cannot define your retention or reconciliation policy; those are contractual decisions owned by the DAM service.
The approach is not suitable when every upload must be available in many high-resolution formats within a strict, fixed deadline. In that case, precompute the required matrix and budget the storage before accepting the asset. Stick with a simpler upload-only pipeline when the catalog is small, derivatives are never regenerated, and there is no audit or cost-allocation requirement; adding a versioned contract would be needless machinery.
References
- https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats
- https://datatracker.ietf.org/doc/html/rfc9110
- https://12factor.net/logs
Further reading
The MDN formats guide covers container and codec support. RFC 9110 documents HTTP semantics relevant to idempotent requests and cache behavior. The Twelve-Factor logs essay gives a concise baseline for treating event records as streams that operations can route and retain deliberately.
Top comments (0)