For AI marketing video jobs, the operational constraint is what generation polling can actually prove when search tags are part of the publish contract. A generated video is not useful to a marketing team merely because a renderer has finished; the file must be durably stored, identifiable, searchable, and safe to retrieve while retries are happening.
Short answer: model generation, polling, download, and tagging as six durable states, tag at upload for the fields required by search, and reserve on-demand tagging for enrichment that can be late without hiding the asset.
That answer puts the storage record ahead of the worker process. It also rejects a tempting shortcut: treating a successful generation response as proof that a downloadable, indexed object exists. Those are separate claims, with separate evidence.
Start with the storage invariant
The governing invariant is narrow: once the application tells a user that a marketing video is available, the database row, object key, content metadata, and minimum search tags must agree on the same immutable asset version. A polling loop can disappear. A worker can run twice. The user can click download during enrichment. None of those events should create a second logical video or make an earlier version silently replace a later one.
Six states are enough for this design: submitted, generating, retrieving, stored, indexed, and failed. They describe evidence the application owns, not vague percentages reported by a remote job. retrieving matters because a completed generation job and a verified object are not the same thing. stored means the application has committed an immutable object reference and the metadata needed to serve it. indexed means the required tags are visible to search.
Keep failure terminal for a particular attempt, not for the logical asset. A retry creates a new attempt under the same asset identifier, while the successful attempt supplies the immutable version. This distinction prevents an old poll response from moving a newer attempt backward. It also leaves an audit trail without making a collection of half-written objects look like published media.
The state machine should reject backward transitions.
How should marketing video generation, polling, and download jobs handle search tags?
Split tags by consequence, not by which model can produce them. Required tags are the small set the search contract depends on: for example, campaign identifier, locale, asset type, and approved visibility. Enrichment tags can include inferred subjects, scenes, or creative themes. The first group belongs in the upload transaction boundary; the second can run on demand or asynchronously after storage.
This is the practical reason to prefer upload-time tagging for a media library. A user who uploads or accepts a generated asset expects it to appear under deterministic filters immediately. If the object becomes visible before its required tags do, search has a false negative: the asset exists but a correct query cannot find it. If tags become visible before the object does, search has a false positive: the result points at something that cannot yet be downloaded. Both are consistency failures, even if every individual request returned success.
On-demand enrichment is different. It avoids spending work on assets nobody inspects, and it lets the application add new classifiers without rewriting the publish path. The catch is latency and cache invalidation: the first search that needs an enrichment field either waits, returns an explicitly partial result, or queues work for a later query. There is no honest fourth option. Don't let an optional classifier hold the core asset hostage.
The boundary should look like this:
| Concern | At upload | On demand |
|---|---|---|
| Identity and campaign scope | Required before publish | Not suitable |
| Access and visibility tags | Required before publish | Not suitable |
| Codec and container metadata | Record from the verified object | Re-read only for repair |
| Semantic scene labels | Useful when search requires them immediately | Better when demand is sparse |
| Newly introduced classifiers | Backfill deliberately | Good for gradual adoption |
| Search behavior during work | Asset stays hidden until the minimum index is ready | Return a named partial state, never pretend completeness |
Upload-time processing is not universally correct. Stick with on-demand tagging when enrichment is expensive, searches rarely use it, or the taxonomy changes faster than assets arrive. Conversely, on-demand processing is not suitable for authorization, tenant boundaries, retention class, or any field whose absence changes whether a user may discover the object. Those aren't enrichment.
Make polling advance evidence, not time
A poller should answer one question: has the generation attempt produced new evidence that permits a forward transition? Fixed sleep loops answer a different question, and they tend to create synchronized traffic, ambiguous timeouts, and duplicate retrievals after restarts. Persist the provider job identifier, current attempt, next eligible poll time, and last accepted state. Then use bounded backoff with jitter as a scheduling policy, while keeping the state transition conditional in storage.
The following Python sketch deliberately leaves transport behind a generic interface. Its important behavior is the compare-and-set around the durable state and the stable key derived from the logical asset plus attempt. A repeated completion observation writes the same destination; it does not mint another published video.
from dataclasses import dataclass
from enum import Enum
from typing import Protocol
class State(str, Enum):
SUBMITTED = "submitted"
GENERATING = "generating"
RETRIEVING = "retrieving"
STORED = "stored"
INDEXED = "indexed"
FAILED = "failed"
@dataclass(frozen=True)
class Attempt:
asset_id: str
number: int
state: State
remote_job_id: str
class Repository(Protocol):
def compare_and_set(self, attempt: Attempt, expected: State, new: State) -> bool: ...
def record_object(self, attempt: Attempt, object_key: str, checksum: str) -> None: ...
def publish_required_tags(self, attempt: Attempt) -> None: ...
def accept_completed_job(repo: Repository, attempt: Attempt, checksum: str) -> None:
if not repo.compare_and_set(attempt, State.GENERATING, State.RETRIEVING):
return
object_key = f"assets/{attempt.asset_id}/attempt-{attempt.number}/video"
repo.record_object(attempt, object_key, checksum)
if not repo.compare_and_set(attempt, State.RETRIEVING, State.STORED):
return
repo.publish_required_tags(attempt)
repo.compare_and_set(attempt, State.STORED, State.INDEXED)
There is a subtle race here that deserves more attention than the polling interval. Suppose attempt 1 finishes after a timeout caused the scheduler to create attempt 2. Attempt 1's late completion may be a valid object, but it must not become the current version merely because its callback arrived last. Promotion therefore needs a condition on both the logical asset's selected attempt and the attempt state. The object may remain available for audit or later cleanup, while only the selected attempt can move the asset's published pointer. This is why “last response wins” is a storage policy disguised as a convenience, and usually the wrong one.
Poll less cleverly. Commit more carefully.
Treat download as a verified handoff
Download begins after generation completes, but publication begins only after verification. Stream into a temporary, non-public object; calculate a checksum while writing; inspect the media metadata needed by the application; then commit the durable record and promote visibility. The MDN media formats guide is a useful reminder that a file extension alone does not settle browser compatibility: containers can carry different codecs, and support varies. Store the observed container and codec information rather than inferring playback behavior from a name.
Name the failure modes in tests. A truncated body must never be promoted. Two completion notifications must converge on one object version. A checksum mismatch must leave search unchanged. A late result from an obsolete attempt must not replace the selected attempt. A download request during retrieving should return a stable “not ready” application state, while a request for an already published version should keep working even if optional enrichment is running.
I'm not sure which polling interval is right for a given workload without the renderer's completion distribution and rate-limit behavior. Measure both, then set a ceiling on poll age and an explicit attempt deadline. The architecture does not depend on guessing that interval correctly; after a crash, the scheduler can reconstruct eligible work from durable timestamps.
Observability should follow the same state model. Count transitions, not worker log lines. Track age in each nonterminal state, conditional-update conflicts, bytes retrieved, checksum failures, and the delay between stored and indexed. A growing retrieving age points toward transfer or verification pressure; a growing indexing delay points toward tag publication. One generic “job latency” chart blurs those diagnoses.
Roll out the lifecycle without reprocessing everything
Begin by shadow-writing the six-state record while the existing path remains authoritative. Compare final object identity, required tags, and selected attempt for a bounded sample; then move new uploads to the state machine while old assets retain their existing references. Backfill only the minimum tags needed for consistent search, and let optional classifiers populate when demand justifies them.
The rollback boundary is the published pointer. Keep it independent from worker deployments and tag-enrichment releases, so reverting a consumer does not rewrite objects or erase evidence. Once transition-conflict rates, state age, and search visibility agree with the service objective, remove the shadow path. Migration is complete when recovery no longer depends on a particular worker still remembering what it was doing.
Top comments (0)