Designing Model-Agnostic Prompts for AI Video Workflows

jiaoshuo1997 — Thu, 23 Jul 2026 14:05:45 +0000

AI video tools change quickly. A prompt that works well in one model may behave differently in another, and model-specific syntax can become obsolete after an update. A more durable approach is to represent the creative intent first, then translate that representation into the vocabulary of a chosen generator.

This article describes a model-agnostic prompt architecture for reference-based video workflows.

Start with an intermediate representation

Instead of sending a long natural-language paragraph directly to a model, store the shot as structured fields:

{
  "duration": 5,
  "aspect_ratio": "9:16",
  "subject": "a cyclist wearing a yellow rain jacket",
  "environment": "wet city street at dusk",
  "composition": "medium tracking shot, subject in right third",
  "camera": "truck left at cycling speed, stable horizon",
  "action": "cyclist passes two parked cars and looks over shoulder",
  "lighting": "cool ambient light with warm shop reflections",
  "end_state": "cyclist exits frame left"
}

This is not a prompt yet. It is an intermediate representation: a stable description of what the shot must accomplish.

The advantages are practical:

missing information is visible
fields can be validated independently
one shot can be rendered for multiple models
revisions can target a single layer
duration and aspect-ratio constraints stay explicit

Separate invariants from preferences

Not every detail has equal importance. Divide the representation into two groups.

Invariants are requirements that should survive every model translation. Examples include the subject, action order, camera direction, duration, and final state.

Preferences are desirable but negotiable details: a subtle film grain, a specific color mood, shallow depth of field, or a minor background prop.

When generation fails, preserve invariants first. A visually beautiful result is still wrong if the subject moves in the opposite direction or never completes the required action.

P0: action order, camera direction, end state
P1: framing, environment, lighting
P2: texture, secondary props, stylistic accents

Use a two-stage renderer

A model adapter can transform the intermediate representation in two stages.

Stage 1: build the visual state

The first stage describes the opening frame: subject, wardrobe, environment, framing, lens cues, lighting, and spatial relationships. It should be possible to sketch the first frame from this section alone.

Stage 2: build the temporal change

The second stage describes motion: what the subject does, how the camera moves, when events occur, and what the final frame should contain.

Keeping these stages separate makes debugging easier. If the first frame is wrong, revise the visual state. If the action stalls or the camera drifts, revise the temporal instructions.

Translate, do not copy

Each model adapter should map the same fields into the model’s preferred style. One generator may respond well to concise cinematic prose. Another may need explicit timing beats. A third may benefit from negative constraints.

The adapter should be responsible for ordering fields by importance, converting durations into supported values, translating camera vocabulary, omitting unsupported controls, and adding only constraints the target model understands.

The source representation remains unchanged. This prevents model quirks from leaking into the analysis layer.

Validate before generation

A lightweight validator catches many expensive errors:

- duration is supported by the target model
- aspect ratio is valid
- subject and action are present
- camera instruction has direction or lock behavior
- action has an observable end state
- timing beats do not exceed total duration

For reference-video workflows, validation should also compare every field against the source. Do not invent wardrobe, dialogue, or camera movement just to make the description sound richer.

Measure structural fidelity

Pixel similarity is a poor measure for generative video. Evaluate structure instead:

Was the shot type preserved?
Did the action occur in the correct order?
Did the camera move in the intended direction?
Was the final state reached?
Did subject identity and wardrobe remain consistent?
Was the pacing close to the reference?

These checks create useful revision signals. If camera direction is wrong, change the camera layer. If the action is incomplete, simplify the action or allocate more time. If identity drifts, strengthen the visual-state constraints.

Keep the workflow auditable

Store the intermediate representation, rendered prompt, target model, generation settings, and revision notes together. This produces an audit trail and makes successful prompts reusable.

source segment -> structured shot -> model adapter -> rendered prompt
-> generation result -> evaluation -> revision

We use this separation between analysis and model-specific rendering in promptvv, where reference videos can be decomposed into shot-level tasks before prompts are adapted and refined.

The main idea is simple: treat prompts as compiled output, not as the source of truth. A stable intermediate representation makes an AI video workflow easier to test, debug, and migrate when models change.

DEV Community: jiaoshuo1997