Use an asynchronous, persisted state machine for marketing video jobs: moderate the logistics team's uploaded images first, generate one approved campaign rendition, poll your own job record, and expose a download only after the record reaches a terminal ready state. That is the least complex design that preserves idempotency and an audit trail without making a browser request own a long-running operation.
The cost question comes before the workflow diagram. For each campaign, write the bill as source storage + generation work + status traffic + derivative storage + delivery traffic. Don't start by optimizing status requests because they are easy to count. Measure every term using the same campaign cohort and retention window; if generated video bytes retained over time dominate, shortening derivative retention changes the bill, while shaving a status check does not. The exact dominant term depends on duration, dimensions, encoding, request cadence, and viewing behavior, so I'm not sure which term wins in an unmeasured workload.
Keep the source image set and the approved output, then expire rejected drafts, superseded outputs, and stale download grants under an explicit policy. The catch is forensic depth: deleting intermediate artifacts reduces retained bytes, but after deletion an operator can reconstruct the decision path only from immutable metadata, hashes, transition records, and provider request identifiers, not from the discarded media itself.
What actually creates the retention bill?
A useful model separates objects from events. Source uploads, generated derivatives, and delivery copies consume byte-time; generation consumes compute or an external operation; polling consumes requests; downloads consume delivery traffic. Put those units in separate ledger columns. A single blended "video cost" number can't tell an architect which policy to change, and it makes reconciliation needlessly ambiguous when a campaign is retried or replaced.
The unit of accounting should be the logical job, not an individual attempt. Give the job an immutable ID, attach each generation attempt beneath it, and assign an idempotency key to the user's intent, such as the combination of tenant, campaign revision, and approved asset-set hash. Repeated submissions with that key return the same logical job. A fresh creative revision gets a fresh key. This is an exactly-once mindset implemented over operations that may be delivered more than once: the durable database decides whether an intent already exists, and workers remain safe under replay.
Consider a logistics campaign built from a user-uploaded depot photograph, a vehicle photograph, and approved copy. Moderation belongs before generation because a rejected input should never become part of a public derivative. The source objects remain distinct from the generated video, and the manifest records their stable identifiers rather than treating a mutable folder as the job input. If an editor replaces the vehicle photograph, the asset-set hash changes, the old approved output stays attributable to its old inputs, and the new revision can proceed without rewriting history.
Retention then becomes a policy table rather than an ad hoc cleanup task:
| Record or object | Keep until | Reason for keeping | Cost of early deletion |
|---|---|---|---|
| Source asset | Contract and privacy policy permit deletion | Re-generation and evidence of input | The exact prior output may not be reproducible |
| Transition log | Audit policy expires | Reconciliation and authorization review | State changes become difficult to prove |
| Approved derivative | Campaign expiry plus the defined recovery window | Playback and controlled download | A legitimate late retrieval requires regeneration |
| Rejected or superseded derivative | Short, documented investigation window | Diagnose policy and quality decisions | Less evidence during a dispute |
| Download grant | Its brief authorization lifetime | Controlled delivery | The client must request another grant |
No universal duration belongs in that table. Legal, privacy, security, and commercial owners must approve the limits for the data actually present; engineering's job is to make those limits executable, observable, and testable. A cleanup worker should record what policy selected an object and when deletion completed, while avoiding sensitive content in logs.
How should marketing video jobs handle generation, polling, and download?
Treat generation, polling, and download as three different permissions over one lifecycle. Generation is a command accepted once for an approved manifest. Polling is a read of your persisted projection, not permission for the browser to interrogate a rendering system indefinitely. Download is a short-lived capability issued only after readiness, tenant authorization, and campaign policy have all been checked. The server returns quickly after creating or finding the logical job. A worker claims the queued record, starts an attempt, and records the external operation identifier if one exists. Another worker advances nonterminal attempts according to a backoff policy. The UI asks your application for the current projection; it doesn't infer truth from elapsed time, and it doesn't turn a network timeout into a second generation command. Once the output is ready, the application verifies that the stored result belongs to the expected job and revision before publishing a download grant. Seven transitions are enough for this scenario: uploaded -> moderating, moderating -> rejected, moderating -> queued, queued -> generating, generating -> ready, generating -> failed, and ready -> expired. Rejection and generation failure are separate terminal outcomes because they answer different operational questions. An operator may permit a new attempt after a generation failure, but changing a moderation rejection requires a new input revision or an authorized review event; silently moving the same record backward would damage the audit trail.
State is evidence.
Poll on the server with bounded exponential backoff and jitter chosen for the workload, and stop at any terminal state. Clients can poll the application more slowly, resume after reconnecting, and use an entity version or update timestamp to avoid repainting unchanged data. There is no honest "exactly once" network call here. There can be exactly one committed transition for a particular state version, enforced by a conditional database update, while duplicate deliveries become no-ops that are still visible in metrics.
Short-lived delivery authorization matters because ready means the artifact exists; it does not mean every holder of a job ID may retrieve it forever. Check tenant membership and campaign access on each grant request, issue the grant for a deliberately narrow window, and audit issuance separately from the actual media object. Never place permanent object credentials or unrestricted storage locations in job status responses.
Encode legal transitions in Go
The state machine should reject impossible movement in the domain layer before persistence. This compact Go example models the seven permitted transitions and emits an audit event as part of the same proposed change. It deliberately omits transport and vendor calls; those are adapters around this invariant, not the invariant itself.
package lifecycle
import (
"errors"
"time"
)
type State string
const (
Uploaded State = "uploaded"
Moderating State = "moderating"
Rejected State = "rejected"
Queued State = "queued"
Generating State = "generating"
Ready State = "ready"
Failed State = "failed"
Expired State = "expired"
)
var allowed = map[State]map[State]bool{
Uploaded: {Moderating: true},
Moderating: {Rejected: true, Queued: true},
Queued: {Generating: true},
Generating: {Ready: true, Failed: true},
Ready: {Expired: true},
}
type Job struct {
ID string
TenantID string
AssetSetHash string
IdempotencyKey string
State State
Version uint64
}
type AuditEvent struct {
JobID string
From State
To State
Actor string
ReasonCode string
FromVersion uint64
OccurredAt time.Time
}
func (j Job) Transition(to State, actor, reason string, now time.Time) (Job, AuditEvent, error) {
if !allowed[j.State][to] {
return Job{}, AuditEvent{}, errors.New("illegal lifecycle transition")
}
if actor == "" || reason == "" {
return Job{}, AuditEvent{}, errors.New("audit identity and reason are required")
}
event := AuditEvent{
JobID: j.ID,
From: j.State,
To: to,
Actor: actor,
ReasonCode: reason,
FromVersion: j.Version,
OccurredAt: now.UTC(),
}
j.State = to
j.Version++
return j, event, nil
}
Persist the updated job and event atomically with a condition equivalent to WHERE id = ? AND version = ?. Zero updated rows means another worker won the race; reload, classify the delivery as duplicate or stale, and do not invoke generation again. This is a routine conflict, not a reason to erase history. Use an outbox in that same transaction when a committed transition must schedule downstream work, then mark each outbox message delivered with its own stable identifier.
Tests should enumerate every allowed edge and sample forbidden edges, but the more revealing tests inject repetition and reordering: deliver the queue message twice, process an old status observation after a newer one, request a grant after expiry, and submit the same idempotency key concurrently. Assert both the final state and the event sequence. A green response alone proves too little.
Should moderation run at upload time or on demand?
For this logistics workflow, run moderation after upload and before queuing generation. It blocks disallowed source images before expensive derivative work and establishes a stable approved manifest. Generate the campaign video only when the editor submits that manifest, rather than immediately after every image upload; uploads are often partial and mutable, while submission expresses a coherent user intent.
On-demand processing is still the better choice in some systems. Stick with on-demand moderation or rendering when most uploaded assets are never viewed or published, latency on first access is acceptable, and retaining a derived object would violate the chosen data-minimization policy. It is not suitable when publication has a fixed deadline, the first viewer must not pay processing latency, or a policy decision must be recorded before any public exposure. Your mileage may vary — especially when upload volume is high but publication volume is low — so compare ratios from representative cohorts instead of adopting a rule from a different product.
The media output itself also needs an explicit compatibility contract. Container, codec, audio, and browser support are separate choices; a familiar filename alone doesn't prove that a target client can decode the streams inside it. The MDN media formats guide documents those distinctions and the compatibility considerations. Define accepted source types, the exact target profile, maximum dimensions and duration, and what counts as an unacceptable output, then test representative campaign assets on the clients the product actually supports.
Operate the lifecycle as a reconciled ledger
Dashboards should expose state age, attempts per logical job, duplicate command count, transition conflicts, moderation outcomes, generation latency, expired outputs, and grant issuance. Keep identifiers consistent across the job, attempt, audit event, and structured log. Don't log source URLs, temporary grants, prompts containing sensitive customer material, or raw uploaded media; identifiers and reason codes should carry the operational story without copying the payload into every system.
Reconciliation closes the gap between a worker's last action and durable truth. Periodically scan nonterminal jobs older than their state-specific expectation, compare their current attempt record with the latest authoritative observation, and propose only a legal forward transition. Scan ready records for a matching derivative manifest and expired records for completed deletion. The reconciler must be idempotent because it will revisit healthy records, and each correction needs an actor such as reconciler, a reason code, and the observed version.
Deployment deserves the same caution. Add new states in code before any worker can write them, keep older readers tolerant of unknown reason codes, and deploy schema changes before behavior changes. During a rollback, don't force new records into an older state merely to satisfy old code; pause claims, retain the evidence, and restore a compatible reader.
Correctness beats motion.
The deliberate limitation of this design is operational weight. A persisted state machine, conditional writes, an outbox, reconciliation, and retention evidence require more work than one synchronous handler. For an internal tool where generation reliably finishes inside the request budget, no user can retry concurrently, and no regulated or contractual audit trail is needed, a synchronous operation may be sufficient. Once jobs outlive requests or control publishable customer media, the smaller implementation externalizes complexity into duplicate renders, ambiguous ownership, and missing evidence.
Stop keeping superseded video bytes after the approved investigation window; keep their hashes, provenance, policy decision, and transition events until the applicable audit policy expires. If a customer later disputes the visual result, you can prove which inputs and decisions produced it, but you may be unable to replay the exact discarded artifact.
That loss is real.
It should be approved as a retention trade-off, not discovered during an incident.
Top comments (0)