A minimal experiment log for image-to-video workflows

An image-to-video workflow has at least two quality gates: whether the source image is usable, and whether the generated motion follows the brief. If you change the image prompt, video prompt and model together, a better result tells you very little about which change helped.

Here is a small experiment record you can use without building an evaluation service. It is a proposed workflow, not a report of measured model performance. I work on FreyaVideo; the linked product pages and guides below are ours.

Freeze the input before comparing video models

Give each accepted source image a stable filename. Keep a copy of the original and record the exact edit instruction. An image that looks better at thumbnail size can still introduce a changed face, bent lettering or an object that will make motion harder to judge.

A minimal record might look like this:

{
  "case_id": "window-shot-001",
  "source_file": "window-shot-001-approved.png",
  "source_revision": 2,
  "must_preserve": ["subject identity", "window position", "shirt color"],
  "requested_edit": "reduce background clutter",
  "video_brief": "subject turns toward the window; static camera",
  "duration_seconds": 5,
  "aspect_ratio": "16:9"
}

These values describe an example brief, not an actual generation. Keep the source revision fixed while comparing video results. If you need another image edit, create a new revision and label the subsequent runs accordingly.

For the image stage, our GPT Image 2.5 workspace exposes generation and reference-image editing. The accompanying noise and editing guide describes how to inspect edits. Regardless of the tool, compare against the last acceptable source rather than continuing indefinitely from a damaged result.

Separate the three clocks

The time a user experiences can include upload, queueing, generation and delivery. A provider completion timestamp is not necessarily the moment the user can play the output.

Record timestamps only when the application exposes the relevant events:

request_started_at
provider_accepted_at
provider_completed_at
playable_output_ready_at

Then calculate the intervals whose endpoints you actually have. Do not label the time between acceptance and completion as pure compute time if it includes queueing. Do not substitute polling frequency for an exact provider completion timestamp.

For example, if a poll discovers completion at 12:00:30, all you know may be that the job completed between that poll and the previous one. Preserve that uncertainty in the result rather than reporting a precise processing duration.

Our video generation speed guide provides context for assessing turnaround. For your own comparisons, total time to a playable result is often the most useful user-facing measure, while queue and processing intervals help diagnose delays.

Keep a failure row

A failed or rejected run is still an observation. Dropping it makes both turnaround and cost look better than the experience a user actually had.

Useful columns are:

case_id,source_revision,model,settings,status,total_seconds,credits,identity_ok,motion_ok,artifact_notes

Leave unknown values blank. Keep cancellation, validation rejection and provider failure separate: a request rejected before it reaches the provider is not evidence of model reliability. Likewise, an output that renders successfully can fail your visual acceptance criteria.

Decide the acceptance criteria before seeing the output. For the example brief, ask whether the subject remained recognizable, whether the turn happened, whether the camera stayed static and whether the result has distracting artifacts. A short note is more informative than an unexplained overall score.

Change one variable per comparison

Start by holding the source image, brief, duration and aspect ratio constant. Record any settings that cannot be matched between the models. If one output uses a different resolution or audio setting, make that difference visible instead of presenting it as a controlled comparison.

The H3 Max versus Kling guide is a starting point for choosing cases to investigate. It does not replace recording your own settings and observations. A single clip should not be treated as a universal ranking.

If you are setting up an H3 run, the H3 Max tutorial and model workspace cover the practical next steps. Check current settings and credit estimates before generating.

A stopping rule prevents endless tweaking

Before running a case, define how many attempts you are willing to spend and what would make the output acceptable. If the source image keeps causing ambiguity, go back to the image gate. If the action is too complex, simplify the brief and record that as a new case.

The resulting table will be imperfect, but it will answer a concrete question: which combination met this brief, at what observed cost and turnaround, under which settings? That is much more actionable than collecting unrelated good-looking clips.