How to Benchmark Image-to-Video Models Without Fooling Yourself

AI video demos are easy to compare badly.

One model gets a cinematic portrait, another gets a product shot, and a third gets a complicated action scene. The winner is usually the model that received the easiest prompt, not the model that would perform best in production.

A useful image-to-video benchmark needs to control the input, define what success means, and separate prompt failures from model failures. This guide describes a small evaluation workflow that a creator or engineering team can run without building a full research lab.

Start with a shot contract

Before choosing a model, describe the shot as a compact contract. The contract should make the non-negotiable requirements explicit.

{
  "subject": "presenter holding a black product box",
  "identity_lock": ["face", "hair", "outfit", "logo", "box geometry"],
  "primary_motion": "presenter raises the box and turns it toward camera",
  "secondary_motion": "subtle breathing and fabric movement",
  "camera": "locked medium shot with a slow five-percent push-in",
  "duration_seconds": 8,
  "finish": "hold the front-facing product for the final second"
}

This is not necessarily the prompt sent to the model. It is the specification used to write prompts and score outputs.

The distinction matters. If the final logo must be readable, that requirement belongs in the test contract even if you later decide that the safest workflow is to add the exact logo in post-production.

Build a small but representative test set

Do not test only the shot your current project happens to need. Use three or four scenes that expose different failure modes.

1. Identity test: a close or medium portrait with a visible face, hairstyle, and wardrobe.
2. Product test: a rigid object with text, straight edges, and recognizable proportions.
3. Motion test: a full-body action with clear start and end states.
4. Camera test: a scene with foreground and background depth where a controlled camera move is visible.

Keep the source images at comparable quality. A heavily compressed input can make a strong model look weak, while a studio-quality reference can hide a model's limitations.

For each scene, store the source image, shot contract, prompt version, model settings, job ID, credit quote, output, and score. This turns a folder of attractive clips into an auditable test.

Hold the variables still

The first pass should change only the model.

Use the same source image, semantic prompt, duration target, aspect ratio, and resolution class wherever the models support them. If a model has a fixed duration or resolution, record the mismatch instead of silently changing the comparison.

You can inspect several supported image-to-video models and their model-specific controls in the CreateForge AI image-to-video workspace. The important idea is not that every model exposes identical settings. It is that the differences should be visible before the job is submitted.

Run at least three outputs per model and test case. Generative systems have variance, so a single excellent or terrible result is not a reliable benchmark.

Score the output in layers

A single “looks good” score hides too much. Use separate dimensions, each scored from 0 to 5.

1. Identity preservation

Did the face, product, wardrobe, and environment remain recognizable? Penalize gradual drift, not only catastrophic changes.

2. Motion correctness

Did the requested action happen? Check the start state, transition, and end state separately. A smooth but incorrect movement is still a failure.

3. Temporal stability

Look for flicker, duplicate limbs, melting geometry, texture crawling, and objects appearing or disappearing between frames.

4. Camera compliance

Did the camera stay locked, push in, orbit, or pan as requested? Also check whether the model added an unrequested zoom or reframed the subject.

5. Editability

Can the clip enter a real timeline? A useful output should have a clean opening, a deliberate final frame, and enough visual continuity for trimming.

6. Cost and latency

Record the accepted credit cost and wall-clock completion time. The visually strongest model may not be the best choice for high-volume variations.

A simple weighted score might look like this:

final_score =
  identity * 0.25 +
  motion * 0.20 +
  stability * 0.20 +
  camera * 0.10 +
  editability * 0.15 +
  efficiency * 0.10

Change the weights for the deliverable. Identity deserves more weight in spokesperson content; motion and camera may matter more in a cinematic sequence.

Test multimodal references as a separate mode

Do not mix a basic first-frame test with a multimodal reference test. They answer different questions.

A multimodal workflow may use an image for appearance, a video for motion, and audio for timing. Each reference needs one declared job. Otherwise the model must reconcile an unstructured pile of signals, and the benchmark becomes difficult to interpret.

For a concrete example, the CreateForge AI Seedance 2.5 workflow exposes image, video, and audio references alongside duration, aspect ratio, resolution, synchronized-audio, last-frame, and output-format controls. Evaluate this mode against its own shot contract rather than comparing it directly with a first-frame-only request.

Diagnose failures before rewriting everything

When an output fails, map the symptom to one likely variable.

• Face drift: strengthen the identity lock and reduce competing motion.
• Dead motion: make the main action more concrete and give it a clear end state.
• Scene warping: simplify the camera move or reduce the number of simultaneous actions.
• Abrupt ending: reserve the final second for a stable pose or product hold.
• Reference conflict: assign one role to each input and remove redundant references.

Change one variable, render again, and record the new version. If you rewrite the prompt, switch the model, change duration, and replace the source image at the same time, you learn almost nothing from the result.

Choose a model by failure tolerance

The “best” model is the one whose common failures your production can tolerate.

An experimental social clip may accept small identity drift in exchange for dramatic motion. A product advertisement may prefer restrained movement if it preserves packaging geometry. A talking presenter may need stable facial identity and synchronized audio more than an elaborate camera move.

CreateForge AI keeps multiple image and video models, parameter-aware quotes, job state, credit use, and finished media in one workflow. That makes it easier to preserve the evidence behind a model decision instead of selecting from isolated demos.

The goal of benchmarking is not to crown a permanent winner. It is to build a repeatable map from shot requirements to the model, prompt structure, and settings most likely to produce an editable result.