DEV Community

Yan Gao
Yan Gao

Posted on Fully Autonomous

Two-Step Photo-to-Video: Edit the First Frame, Then Animate

Turning one portrait into a short AI video sounds like a single-model problem: upload an image, write a prompt, and ask a video model to move it. In practice, I get more control by splitting the job into two steps.

First, an image-edit model redraws the uploaded photo as the opening frame of the intended scene. Then a video model animates that prepared frame.

This is the pipeline I use for many preset effects in a small photo-to-video product. The resulting videos are AI generated. The input photos must show consenting adults; a technically successful generation does not remove the need for permission.

Here is why I chose this design, what it costs, and what I learned while tuning it.

Why not prompt the video model directly?

A video prompt has several jobs competing for attention. It must describe the person, the new setting, the action, the camera, the timing, and sometimes speech and sound. If the input portrait does not already resemble the desired opening shot, the video model has to redesign the scene while also inventing motion.

That is a lot of change at once.

The two-step approach gives each model a narrower task:

  1. Create a coherent opening frame while keeping the recognizable face.
  2. Add motion, speech, sound effects, and music from that frame.

This does not guarantee identity preservation. Faces can still drift, especially during large turns, fast movement, or occlusion. But preparing the scene before animation gives the video model a cleaner starting point. It no longer has to transform an ordinary portrait into a birthday set, costume, and camera composition during the same frames in which the person starts moving.

For a birthday effect, for example, the image step can place the adult subject in a decorated room, establish the clothing and lighting, and leave space for a cake. The video step can then focus on a few beats: look at the cake, react, speak, and make one clear gesture.

The current pipeline and cost

As of November 2026, I use OpenAI's gpt-image for the image-edit step in many effects. It costs about $0.05 per edited image in this workflow. MiniMax H3, called through fal, turns that image into a 10-second, 9:16 video with two or three short spoken lines, sound effects, and music. That step costs about $0.25 per clip.

The direct model cost is therefore about $0.30 for a successful run, before storage, application infrastructure, payment fees, failed experiments, or support time.

That last part matters. The price of one successful API call is not the cost of operating the feature. Prompt testing can become the larger expense if every small edit triggers another paid video.

My rule is to iterate the prompt on a cheap or free model first, then use the same prompt once on the expensive target model. I cap test runs at two per effect and ask before paying for additional retries.

Even then, a prompt that performs well on one model is not automatically good on another. I had a “muscle” effect score well on one model and poorly on H3. Models interpret timing, body movement, framing, and prompt emphasis differently. Cheap-model testing is useful for catching obvious problems, but the target model still needs its own acceptance test.

A generic implementation shape

The application code can stay simple if the two stages have explicit inputs and outputs. This is intentionally generic pseudocode, not production code or a copy of a proprietary prompt.

type EffectRequest = {
  sourcePhoto: Blob;
  adultConsentConfirmed: boolean;
  effect: "birthday";
};

async function generateClip(request: EffectRequest) {
  if (!request.adultConsentConfirmed) {
    throw new Error("Consent is required");
  }

  const openingFrame = await editImage({
    image: request.sourcePhoto,
    prompt: openingFramePrompt(request.effect),
    aspectRatio: "9:16",
  });

  return animateImage({
    image: openingFrame,
    durationSeconds: 10,
    aspectRatio: "9:16",
    prompt: videoStoryboard(request.effect),
    audio: true,
  });
}
Enter fullscreen mode Exit fullscreen mode

The prompt functions might return a structure like this:

OPENING FRAME
- Keep the adult subject recognizable and front-facing.
- Place the subject chest-up in a simple birthday setting.
- Leave clear space around the face and hands.
- Use clothing without words, logos, or numbers.

VIDEO STORYBOARD
- 0–3s: subject notices the cake; eyes move toward it.
- 3–6s: subject smiles and says one short line.
- 6–10s: subject makes one small hand gesture; camera moves in slowly.
- Keep the camera stable and the action readable.
Enter fullscreen mode Exit fullscreen mode

The important part is not the exact wording. It is the separation of responsibilities and the timeline.

Write the video prompt as a storyboard

My prompts improved when I stopped writing a pile of visual wishes and started writing beats.

For each beat, I specify the expression, eye-line, hand action, spoken line, and camera behavior. One clear action every few seconds is easier for the model to follow than several simultaneous actions.

Positive instructions also work better for me than a long list of prohibitions. “The subject keeps both hands visible near the cake” is more useful than ten variations of “do not hide the hands.” Negative constraints still have a place in validation and safety rules, but they should not become the whole creative prompt.

I also avoid visible text in the generated scene. Text on shirts, logos, and numbers on sleeves often drift between frames. A clean garment is less interesting on paper but more stable in motion.

The source photo still determines the ceiling

A strong input is one clear, front-facing adult, framed roughly chest-up, with some room around the face. Covered faces and group photos fail more often. A tightly cropped face also gives the image editor less room to build a believable vertical scene.

The upload checks should explain this before the user pays or waits. They should also be honest that results vary. AI video generation is probabilistic; a prepared opening frame reduces one class of uncertainty but does not turn the process into deterministic rendering.

If you want to see the kind of preset this architecture supports, here is a birthday video effect. The useful idea is broader than that example: when one model is being asked to redesign an image and animate it at the same time, consider separating composition from motion.

That extra step adds a small image-generation cost. In return, it gives the pipeline a visible checkpoint, a reusable opening frame, and a more focused job for the expensive video model.

I'm the maker of inreel.

Top comments (0)