DEV Community: FreyaVideo

A minimal experiment log for image-to-video workflows

howard hua — Thu, 10 Sep 2026 06:16:30 +0000

An image-to-video workflow has at least two quality gates: whether the source image is usable, and whether the generated motion follows the brief. If you change the image prompt, video prompt and model together, a better result tells you very little about which change helped.

Here is a small experiment record you can use without building an evaluation service. It is a proposed workflow, not a report of measured model performance. I work on FreyaVideo; the linked product pages and guides below are ours.

Freeze the input before comparing video models

Give each accepted source image a stable filename. Keep a copy of the original and record the exact edit instruction. An image that looks better at thumbnail size can still introduce a changed face, bent lettering or an object that will make motion harder to judge.

A minimal record might look like this:

{
  "case_id": "window-shot-001",
  "source_file": "window-shot-001-approved.png",
  "source_revision": 2,
  "must_preserve": ["subject identity", "window position", "shirt color"],
  "requested_edit": "reduce background clutter",
  "video_brief": "subject turns toward the window; static camera",
  "duration_seconds": 5,
  "aspect_ratio": "16:9"
}

These values describe an example brief, not an actual generation. Keep the source revision fixed while comparing video results. If you need another image edit, create a new revision and label the subsequent runs accordingly.

For the image stage, our GPT Image 2.5 workspace exposes generation and reference-image editing. The accompanying noise and editing guide describes how to inspect edits. Regardless of the tool, compare against the last acceptable source rather than continuing indefinitely from a damaged result.

Separate the three clocks

The time a user experiences can include upload, queueing, generation and delivery. A provider completion timestamp is not necessarily the moment the user can play the output.

Record timestamps only when the application exposes the relevant events:

request_started_at
provider_accepted_at
provider_completed_at
playable_output_ready_at

Then calculate the intervals whose endpoints you actually have. Do not label the time between acceptance and completion as pure compute time if it includes queueing. Do not substitute polling frequency for an exact provider completion timestamp.

For example, if a poll discovers completion at 12:00:30, all you know may be that the job completed between that poll and the previous one. Preserve that uncertainty in the result rather than reporting a precise processing duration.

Our video generation speed guide provides context for assessing turnaround. For your own comparisons, total time to a playable result is often the most useful user-facing measure, while queue and processing intervals help diagnose delays.

Keep a failure row

A failed or rejected run is still an observation. Dropping it makes both turnaround and cost look better than the experience a user actually had.

Useful columns are:

case_id,source_revision,model,settings,status,total_seconds,credits,identity_ok,motion_ok,artifact_notes

Leave unknown values blank. Keep cancellation, validation rejection and provider failure separate: a request rejected before it reaches the provider is not evidence of model reliability. Likewise, an output that renders successfully can fail your visual acceptance criteria.

Decide the acceptance criteria before seeing the output. For the example brief, ask whether the subject remained recognizable, whether the turn happened, whether the camera stayed static and whether the result has distracting artifacts. A short note is more informative than an unexplained overall score.

Change one variable per comparison

Start by holding the source image, brief, duration and aspect ratio constant. Record any settings that cannot be matched between the models. If one output uses a different resolution or audio setting, make that difference visible instead of presenting it as a controlled comparison.

The H3 Max versus Kling guide is a starting point for choosing cases to investigate. It does not replace recording your own settings and observations. A single clip should not be treated as a universal ranking.

If you are setting up an H3 run, the H3 Max tutorial and model workspace cover the practical next steps. Check current settings and credit estimates before generating.

A stopping rule prevents endless tweaking

Before running a case, define how many attempts you are willing to spend and what would make the output acceptable. If the source image keeps causing ambiguity, go back to the image gate. If the action is too complex, simplify the brief and record that as a new case.

The resulting table will be imperfect, but it will answer a concrete question: which combination met this brief, at what observed cost and turnaround, under which settings? That is much more actionable than collecting unrelated good-looking clips.

Do not stop your AI video timer at provider success

howard hua — Fri, 04 Sep 2026 06:30:24 +0000

A successful video-generation response is not the same event as a video becoming playable in a browser. If a dashboard stops its timer at provider success, the number can exclude output transfer, browser loading, and the creator's review.

Here is a small, provider-independent pattern for instrumenting that gap. It is an example you can adapt, not a benchmark or a claim that one model is faster than another.

Disclosure: this engineering note was written with AI assistance and is published by FreyaVideo, an AI video tool. The related company-blog guides are linked where they provide background.

Give each milestone a precise meaning

Start by naming what the browser actually observes:

Event	Meaning in this example	What it does not prove
`submitted`	The user starts the generation request	The provider has accepted or begun work
`acknowledged`	The app receives a job acknowledgement	Rendering is finished
`provider_done_observed`	A poll or notification tells the app that output is ready	The browser can play that output
`playing`	The browser emits its first `playing` event for the output	The result meets the creator's brief
`accepted`	A person marks the output usable after review	Every previous attempt was successful

The word observed matters. A polling client can learn about completion several seconds after it happened. That delay belongs in a user-facing waiting-time measurement, but it is not evidence about GPU inference time.

Similarly, playing is a practical playback milestone, not an exact “first pixel appeared on the screen” measurement. It can also occur after playback resumes. Record the first event for a particular output, and document the definition.

This distinction is the central question in our guide to comparing AI video speed claims: which clock is the claim actually measuring?

Use one local clock for a single page visit

For elapsed time within one document, performance.now() provides a monotonic clock. It avoids the wall-clock adjustments that can affect a subtraction of two Date.now() values. The MDN reference also documents precision limits and differences in how browsers handle operating-system sleep.

Do not subtract a server timestamp from performance.now(). Do not take a value from an old page load and subtract it from a new page's reading. Both cases mix clock contexts.

The following ledger deliberately exists only in memory. Call start() directly before initiating the generation request, rather than when the page loads:

function createRunLedger(runId, now = () => performance.now()) {
  const startedAt = now();
  const marks = new Map();

  return {
    runId,
    mark(name) {
      // Keep the first observation of each milestone.
      if (!marks.has(name)) marks.set(name, now() - startedAt);
    },
    snapshot(outcome) {
      return {
        runId,
        outcome,
        clock: "browser-document-monotonic",
        elapsedMs: Object.fromEntries(marks),
      };
    },
  };
}

function start(runId) {
  const ledger = createRunLedger(runId);
  ledger.mark("submitted");
  return ledger;
}

The now argument makes the ledger easy to check with a deterministic clock. Calling mark("acknowledged") twice must keep the first value. An event that never happened stays absent; it is not recorded as zero milliseconds.

This code is only the ledger. Your application still needs request handling, authentication, polling, cancellation, persistence, and an explicit job state machine. Keeping those responsibilities separate makes the timing definitions easier to inspect.

Attach playback observation to the correct output

After the application observes a completed job, mark provider_done_observed. Before assigning the new output URL to its video element, register a listener for the media element's playing event:

function observeFirstPlayback(video, ledger) {
  const onPlaying = () => ledger.mark("playing");
  video.addEventListener("playing", onPlaying, { once: true });
  return () => video.removeEventListener("playing", onPlaying);
}

Use this with an element dedicated to that output. Call the returned cleanup function when abandoning the run or replacing the element. Otherwise, a listener from an old run could record playback of a different result.

A play button and autoplay have different implications. If the browser waits for the user to click Play, submission-to-playing includes that human delay. If autoplay is blocked, the job may still be perfectly valid even though no playing mark exists. Store the playback mode alongside the measurements and avoid mixing the two populations.

If your question is instead “when was media data available?”, choose and document a different milestone. Do not silently rename a media-loading event “visible to the user.”

Keep run outcomes and review decisions explicit

One local run should have its own identifier. A retry should receive a new run identifier and retain a reference to the overall user request. This avoids overwriting the time and outcome of the first attempt with the successful second attempt.

The application, not the sample ledger, must decide when a run is failed, cancelled, timed out locally, or complete. A local timeout means the observer stopped waiting; it does not prove that the provider stopped working. Keep that distinction in reconciliation and avoid treating it as permission to dispatch another chargeable job automatically.

Record acceptance through a separate, deliberate review action. This gives submission-to-acceptance for one run; measuring brief-to-accepted-result across retries needs a separate workflow-level starting point. An output can be playable but unusable because it changed the product, missed a requested action, or broke continuity. Conversely, “not reviewed yet” should remain unknown rather than being converted into rejection.

For a first experiment, an empty worksheet is enough:

request_id	run_id	model_version	settings	outcome	elapsed_ms	accepted
Fill from a real run	Fill from a real run	Record exact version	Record actual settings	Explicit state	Milestone map	yes / no / unknown

This table contains no measurements. Populate it from real observations before publishing performance conclusions. Avoid storing prompts, signed media URLs, or personal data unless they are necessary for the analysis and have appropriate handling.

Compare the same brief before comparing the clocks

Timing data is difficult to interpret when one run uses a shorter duration, lower resolution, or a different input image. Record those settings and keep the creative brief stable. For model-specific context, our H3 Max and Kling comparison separates documented controls and specifications from measured results.

For an initial H3 Max workflow, the first-shot tutorial provides a scoped starting point. The instrumentation above applies regardless of the selected provider; a tutorial's example settings are not a substitute for recording the settings your application actually sent.

Only after collecting enough comparable runs should you summarize distributions. Report the number of observations, the outcome breakdown, and the measurement definition next to a median or percentile. If you report successful runs separately, label that selection clearly and show how many attempts failed or remain unresolved.

A useful dashboard lets its reader answer three different questions: when did the application learn that rendering was finished, when did playback begin, and did a person accept the result? Keeping those events separate makes the numbers much easier to trust.