Adapted for DEV Community from an article originally published on the Oimi blog. The engineering framing, prompt schemas and failure analysis below are written for people building generation pipelines rather than for end users.
Originally published at oimi.ai/en/blog/gpt-image-2-seedance-2-workflow
Everyone building on image-to-video models hits the same wall. You have one great still. You feed it to the video model with a motion prompt. You get something that looks plausible for two seconds and then falls apart.
We watched a creator burn five generations on the same 15-second concept — a continuous shot through a medieval market ending on a lone knight in a tavern. Every attempt failed differently:
- Crowd geometry scrambled into noise
- Transitions arrived as hard cuts with no spatial logic
- A wagon that was supposed to mask a cut dropped out of existence mid-move
- The tavern door opened at the wrong beat, breaking the one-shot illusion
The sixth attempt succeeded on the first try. The only thing that changed was what got passed between the two calls: instead of a single image, a storyboard with a timeline and camera motivations.
That's the whole article. The interesting part is why it works, and it has nothing to do with prompt wording.
A single image is an underspecified contract
Here's the framing that made it click for me: you are not sending an image to a model. You are sending an interface — and a single frame is a lossy one.
A still image encodes: appearance, composition, lighting, style. That's it.
It does not encode: camera path, shot boundaries, pacing, what triggers each transition, or where the viewer's attention should be at second 9. You are implicitly asking the model to invent all of that, and its priors for "what should happen next" are generic. Hence shuffle.
The failure isn't randomness. It's missing constraints being filled in with the average case. Same class of bug as an API that accepts options: any and guesses.
So the fix isn't a better motion prompt. It's choosing a richer intermediate representation.
What the intermediate representation has to encode
A storyboard panel is only half of it. The panel gives you appearance; the annotations give you the schedule. What actually needs to survive the handoff:
| Must encode | Why it can't be deferred |
|---|---|
| Subject invariants | Identity has to be fixed before any frame is generated, or every shot reinterprets the face |
| Shot list with boundaries | If the model picks its own shot count, you get whatever it feels like |
| Camera motivation per transition | This is the one everyone skips, and it's the one that matters most (below) |
| Pacing per segment | Without explicit durations, the model front-loads the action and starves the ending |
| Lens / camera height | Otherwise shot 4 has a different focal character than shot 3 and reads as a different film |
Notice that four of these five are temporal, not visual. A single image can only carry the first one. That asymmetry is the entire argument for the storyboard step.
Encoding the IR: prompts as schemas
The practical version of this: stop writing prompts as prose and write them as a structured spec that gets rendered to prose. Not literally JSON in the API — but thinking in fields keeps you from silently dropping a constraint.
For the image step, the spec looks roughly like:
task: "storyboard infographic, 16:9, 12 panels"
style:
base: "rough graphite storyboard sketch, monochrome pencil shading"
no: ["cartoon", "modern elements"]
character:
identity: "locks once, reused verbatim across all panels"
scene:
location: "medieval market street, stone city, dusk"
props: ["wooden stalls", "banners", "livestock", "carts"]
shots:
- { n: 1, lens: 50mm, framing: "street-level close", action: "woman buys apples, pays coins" }
- { n: 2, lens: 50mm, framing: "medium close-up", action: "hands exchanging coin and fruit" }
- { n: 3, lens: 35mm, framing: "foreground interruption", trigger: "horse crosses frame" }
- { n: 4, lens: 35mm, framing: "medium tracking", follows: "wooden cart" }
# ...
- { n: 12, lens: 35mm, framing: "interior reveal", action: "knight alone, candlelight, lifts gaze" }
annotations:
- "motion arrows"
- "lens focal lengths written on each panel"
- "terms like tracking / push-in / redirect / focus handoff / foreground interruption"
Then the video step is a different, much smaller spec — because everything visual is already locked:
format: "continuous shot, motivated camera movement, 15s"
timeline:
- { t: "0:00-0:03", shot: "street-level, woman selecting fruit, hands coins" }
- { t: "0:03-0:05", shot: "cart crosses foreground, camera catches and tracks it" }
- { t: "0:05-0:07", shot: "cart brushes banner, banner swings, reveals chickens scattering" }
- { t: "0:07-0:09", shot: "boy chases chickens, camera follows boy" }
- { t: "0:09-0:12", shot: "boy runs past tavern, door swings open" }
- { t: "0:12-0:15", shot: "camera glides through doorway, knight at corner table lifts gaze" }
The division of labour that makes this work: the image model owns appearance, the video model owns motion. Every constraint you push into the image step is a constraint the video model doesn't have to guess. Every constraint you leave in the video step costs you a retry.
The part that actually fixes the cuts: transition motivation
This is the highest-value idea in the whole pipeline, and it comes from film rather than ML. Spielberg's staging rule: every camera movement must have a motivation — something on screen justifies why the camera moves now.
Applied to generation, it turns the shot list into a chain where each transition is triggered by an on-screen event:
| Transition | Trigger in frame | Camera response |
|---|---|---|
| 1 → 2 | coins exchanged | push in on hands |
| 2 → 3 | horse enters frame | hold, let it occlude (interruption) |
| 3 → 4 | cart passes | pick up and track the cart |
| 4 → 5 | cart brushes banner | banner sweeps across lens |
| 5 → 6 | chickens scatter | follow the movement outward |
| 6 → 7 | boy runs past tavern door | chase into the doorway |
Read that as a system and it's not a prompt anymore — it's a state machine where every transition is event-driven. Nothing moves the camera without a cause in frame. The model doesn't need to invent continuity logic, because the continuity logic is already in the input.
Compare the two attempts on measurable outcomes:
| Single image + prompt | Storyboard + timeline | |
|---|---|---|
| Attempts needed | 5+ | 1 |
| Transitions | Random jump cuts | Every cut motivated by scene action |
| Shot coverage | Elements lost | All 12 shots reproduced |
| Camera | Moves arbitrarily | Every move has a motivation |
The 5× retry reduction is the number to care about if you're paying per generation. The storyboard call is not overhead; it's the cheapest call in the pipeline, and it's the one that eliminates the expensive retries.
Where this still breaks
Being honest about the limits, because a pipeline diagram that only shows the happy path is useless:
- Occlusion has to be physically plausible. The interruption trick (something crossing the lens to mask a cut) works only if the object's size, speed and path make sense at that focal length. Get it wrong and you get a smear, not a wipe.
- Detail decay. The storyboard locks identity, but hands, jewellery and small text still degrade. Shoot the subject large in frame and don't ask for detail the model can't hold.
- Duration ceilings. Every model has a length past which coherence collapses. The storyboard doesn't raise that ceiling; it just makes you hit it in a predictable place.
- The IR itself is a prompt. It's a text description of a storyboard image, then a text description of a video. It's more structured, not lossless. Debug it like any other text interface: change one field, regenerate, compare.
Engineering rules worth stealing
- Storyboard, not a single keyframe. Three panels minimum. One image is an underspecified contract.
- Specify motivation, not movement. "Wagon crosses frame, camera follows the wagon" beats "pan camera". Action-driven motion reads as natural; arbitrary motion reads as random.
- Static before motion. Visual quality sets the ceiling for video quality. Perfect the storyboard first — it's cheaper to iterate there.
- Iterate deliberately. Both cases above needed revision. Change one variable per retry or you can't attribute the improvement.
- Write prompts in English first. Both models parse English more accurately; localize afterwards if you need to.
The framing I'd leave you with: treat the generation stack like any other multi-stage pipeline. Define the interface between stages, make the handoff lossy in only one direction, and you stop debugging outputs and start debugging inputs.
I work on Oimi AI, which is where this storyboard-to-video pipeline runs end to end — the image step, the storyboard annotations, and the video call on one canvas instead of exporting between apps. If you want the specs rather than my paraphrase, the fashion case we ran is published as a runnable template.
If you've got a better answer for the retry-economics problem — fewer calls, or a cheaper way to validate a shot before paying for the video model — I'd like to hear it in the comments.
Top comments (0)