I spent an afternoon on what looked like a prompt engineering problem and turned out to be an input validation problem.
The setup: PixAI shipped Studio at the start of August, a node canvas where images, video, text and audio are all assets on the same graph, connected by edges. Image-to-video, so the picture is the input and a text instruction describes the motion. I had a finished illustration of an original character sitting in my gallery and wanted a five second clip out of it. Image in, motion instruction in, video out.
Three runs, one source image, three instructions. The results ordered themselves in a way I did not predict.
Run 1: the greedy instruction
I wrote the version I expected to fail, on purpose. Six requests in four sentences:
She turns all the way around to face the camera, pushes off the hull
with her right arm, and reaches toward the viewer while debris rushes
past her. The camera orbits around her and pushes in at the same time.
Bright flashes of light, sparks, fast dramatic movement.
Character rotation, a physical push, an arm extension, two simultaneous camera moves, environmental particles, lighting changes. Everything you would expect to blow up.
It executed all of it. The face held. The signature details held.
https://pixai.art/artwork/2042584634831661526
Run 2: the disciplined instruction
So I did what you do after an overloaded call succeeds by accident: I reduced the surface area. One primary action, named invariants, one camera move.
The character drifts slowly in place and turns her head toward the
camera, coming to rest facing the viewer. Her floating hair and the
orange cable sway gently with the motion. The camera pushes in slowly
and steadily. Her face, the lens over her right eye, the collar ring
and the armored glove on her right hand stay unchanged throughout.
Tighter, more explicit, better written by every measure I would apply to a prompt.
The output was a few degrees of head rotation. The supporting elements moved, the camera push landed, and the primary action, the thing the whole clip was built around, effectively no-opped.
https://pixai.art/artwork/2042584865220486332
The actual failure
The instruction was not the problem. The input was.
She was drawn facing the viewer, gaze locked on the lens. turn her head toward the camera describes a state transition from A to B where the source image was already at B. The instruction was well formed and semantically empty against that particular input.
This is the same class of bug as passing a correctly typed argument that happens to be a no-op for the current state. Nothing errors. Nothing warns. You get a successful run and an output that does nothing, and you go looking in the wrong layer.
Run 3: same input, different axis
The fix was not less instruction. It was moving the request onto an axis the input had left unconsumed: depth, camera travel, trailing elements, environmental motion, lighting. Everything except a rotation that had already been applied at draw time.
The character pushes off hard and accelerates forward through the
wreckage, her long hair and the orange cable whipping out behind her.
Broken metal fragments and dust streak past the lens in the foreground.
The camera pulls back and tracks with her. A hard white flare of
sunlight sweeps across the frame from the left as she passes. She keeps
looking straight ahead at the camera the whole time.
That one shipped.
https://pixai.art/artwork/2042585046943800636
What generalizes
Inspect the input before writing the instruction. For image-to-video specifically: what state is the subject already in, what is occluded, how much empty space exists in the direction of intended motion, and how well does the subject separate from the background. Any request that targets a state the input already satisfies produces a silent no-op.
Sort the frame into four buckets first. Primary movement, supporting movement, invariants, camera. The last one is the cheap lever. A camera move carries a shot while the subject stays nearly static, which is also the safest option when the artwork itself is the payload.
Name your invariants explicitly and pick asymmetric ones. My character has a lens over one eye and a glove on one hand only. Asymmetric markers are far easier to verify across frames than "the face looks right," the same way a distinctive sentinel value beats eyeballing a diff.
Declare the style contract twice. I ran a black and white manga panel through the same pipeline and stated black and white at the start and the end of the instruction. Ink, hatching and screentone, the dot shading that stands in for gray in printed manga, all survived, because the movement went to the camera rather than the drawing.
Failed runs are free, and version history makes comparison cheap. Every attempt stays as a node on the canvas, so three runs sit side by side without any bookkeeping on your side. Rewriting the instruction two or three times is the normal path, and treating run 1 as a probe rather than an attempt saves a lot of time.
The thing I keep coming back to: run 1 was the sloppy instruction and it worked, run 2 was the careful one and it did nothing. Prompt quality was not the variable. The input was.
Three runs, five seconds each, and the only variable that mattered was one I had not looked at.
Top comments (0)