DEV Community: guowu zong

How to Evaluate an AI Video Generator Without Wasting Credits: A Repeatable Test Harness

guowu zong — Sat, 01 Aug 2026 10:13:57 +0000

Most AI video demos show the same thing:

A polished final clip
A prompt
A claim that the result was generated in seconds

What they usually do not show is everything that happened between the first prompt and the final result:

rejected generations
identity drift
broken product geometry
unwanted camera motion
deformed text
inconsistent lighting
multiple retries
credits spent on ideas that were never usable

That missing process is the part developers, product teams, and creators actually need.

After testing different text-to-video, image-to-video, product-animation, and talking-avatar workflows, I stopped treating AI video generation as a prompt-writing contest.

I now treat it as a controlled experiment.

In this article, I will share a repeatable evaluation workflow that helps answer three practical questions:

Is the model following the requested motion?
Is the result stable enough to edit or publish?
How many generations will the workflow realistically require?

Disclosure: I work on Haiper AI, an independent creator-facing AI video site. The framework below is tool-agnostic, although I use our public workflow for several examples.

The Unit of Work Is a Shot, Not a Finished Video

A common mistake is asking an AI video generator to create an entire commercial, product demo, or social video in a single prompt.

That makes the result difficult to control and almost impossible to debug.

A better unit of work is one short shot.

A well-defined shot normally contains:

one primary subject
one primary action
one camera behavior
one visual objective
one set of details that must remain unchanged

For example, this is not a useful shot definition:

Create an exciting luxury advertisement for a perfume brand.

It leaves too many decisions to the model.

A more testable definition is:

A clear perfume bottle remains centered on a reflective black surface. The camera performs a slow close-up push-in while a narrow light sweep moves across the glass. Preserve the bottle shape, cap, label position, liquid color, and background.

The second version gives us something we can evaluate.

Did the camera move forward?

Did the bottle stay stable?

Did the lighting change without changing the product?

Did the label remain in the correct location?

That is the difference between generating randomly and running a test.

1. Turn Creative Intent Into a Structured Motion Brief

Instead of writing prompts directly, I first describe the shot as structured data.

Here is a simple TypeScript representation:

type MotionBrief = {
  subject: string;
  action: string;
  setting: string;
  camera: string;
  tone: string;
  preserve?: string[];
  avoid?: string[];
};

A product shot might look like this:

const brief: MotionBrief = {
  subject: "A matte-black running shoe",
  action: "rotates approximately fifteen degrees",
  setting: "on a dark studio pedestal",
  camera: "slow close-up orbit from left to right",
  tone: "controlled commercial lighting, premium ecommerce style",
  preserve: [
    "shoe silhouette",
    "sole geometry",
    "material texture",
    "logo placement",
    "original color"
  ],
  avoid: [
    "new text",
    "extra shoes",
    "hands",
    "invented accessories",
    "large object deformation"
  ]
};

We can then compile that object into a prompt:

export function compileMotionPrompt(brief: MotionBrief): string {
  const sections = [
    `${brief.subject} ${brief.action} ${brief.setting}`,
    brief.camera,
    brief.tone,
    brief.preserve?.length
      ? `Preserve ${brief.preserve.join(", ")}`
      : "",
    brief.avoid?.length
      ? `Avoid ${brief.avoid.join(", ")}`
      : ""
  ];

  return sections
    .filter((section): section is string => Boolean(section))
    .map(section => section.trim())
    .join(". ");
}

The resulting prompt is:

A matte-black running shoe rotates approximately fifteen degrees on a dark studio pedestal. Slow close-up orbit from left to right. Controlled commercial lighting, premium ecommerce style. Preserve shoe silhouette, sole geometry, material texture, logo placement, and original color. Avoid new text, extra shoes, hands, invented accessories, and large object deformation.

This structure has several advantages.

First, it separates creative intent from model-specific prompt syntax.

Second, it becomes easier to generate prompt variants programmatically.

Third, when a result fails, we can identify which field should change instead of rewriting the entire prompt.

2. Choose the Generation Mode Before Choosing the Model

The most expensive mistake is often not choosing the wrong model.

It is choosing the wrong generation mode.

I use the following routing rule:

Situation	Recommended starting mode
The visual direction does not exist yet	Text-to-video
The first frame is already approved	Image-to-video
A strong still image is needed before animation	Text-to-image, then image-to-video
A product must stay close to an existing photo	Image-to-video
A portrait must speak using prepared audio	Talking-avatar workflow
The team is still testing ideas and visual tone	Text-to-video
Brand, character, or product appearance matters	Image-to-video

Text-to-video is useful for discovery.

Image-to-video is useful for control.

When a text-to-video result keeps drifting away from the desired composition, repeatedly expanding the text prompt is usually not the answer. It is often better to create or select the first frame, then animate it.

That is why the workflow on the Haiper AI video generator separates text-to-video and image-to-video rather than treating them as interchangeable inputs.

For some projects, I also create the visual direction first with an AI image generator, approve the frame, and only then move into video.

This introduces an additional step, but it reduces uncertainty.

3. Change Only One Variable Per Generation

Suppose the baseline prompt produces a weak result.

A common response is to change everything:

rewrite the subject
add more action
change the camera
change the lighting
add a new style
switch models
switch resolution

The next result may be better, but we do not know why.

A controlled test changes one variable at a time.

For example:

Run A: Baseline

A black running shoe on a dark studio pedestal.
Slow close-up orbit.
Premium commercial lighting.

Run B: Action Test

A black running shoe rotates approximately fifteen degrees
on a dark studio pedestal.
Slow close-up orbit.
Premium commercial lighting.

Run C: Camera Test

A black running shoe on a dark studio pedestal.
Slow forward camera push.
Premium commercial lighting.

Run D: Lighting Test

A black running shoe on a dark studio pedestal.
Slow close-up orbit.
A narrow light sweep moves from left to right.

Keep the following settings fixed while comparing these runs:

source image
aspect ratio
duration
resolution
model
output objective

Now each generation answers a specific question.

Does object rotation increase deformation?

Does a forward push preserve shape better than an orbit?

Does moving the light produce a cleaner result than moving the object?

That information becomes reusable in future projects.

4. Score the Result Instead of Asking Whether It “Looks Good”

“Looks good” is not a useful evaluation criterion.

A clip can look impressive while being unusable for the project.

I score each result across five dimensions:

Dimension	What to inspect	Weight
Subject fidelity	Identity, product shape, materials, layout	30%
Motion obedience	Whether the requested action happened	20%
Temporal stability	Flicker, morphing, disappearing details	20%
Camera control	Direction, speed, framing, composition	15%
Editability	Whether the clip can be used in a real timeline	15%

A small TypeScript scoring function makes the process consistent:

type Evaluation = {
  fidelity: number;
  motion: number;
  stability: number;
  camera: number;
  editability: number;
};

const weights: Record<keyof Evaluation, number> = {
  fidelity: 0.3,
  motion: 0.2,
  stability: 0.2,
  camera: 0.15,
  editability: 0.15
};

export function calculateScore(evaluation: Evaluation): number {
  const keys = Object.keys(weights) as Array<keyof Evaluation>;

  const score = keys.reduce(
    (total, key) => total + evaluation[key] * weights[key],
    0
  );

  return Number(score.toFixed(2));
}

Each dimension can be scored from 1 to 5.

const result = calculateScore({
  fidelity: 5,
  motion: 3,
  stability: 4,
  camera: 4,
  editability: 5
});

console.log(result);
// 4.25

The weighted score is not intended to prove that one model is universally better.

It helps a team compare results against its own requirements.

A product team may give fidelity the highest weight.

A storyboard team may care more about camera direction and composition.

A social creator may accept small detail changes if the first second creates a strong hook.

5. Build a Failure Diagnosis Table

AI video failures become less frustrating when they are classified.

Here is the diagnosis table I use:

Symptom	Likely cause	Next test
Camera motion feels random	No clear camera instruction	Add one explicit camera direction
Product shape changes	Object motion is too aggressive	Move the camera or light instead
Character identity drifts	Weak or ambiguous reference	Use a clearer image and smaller motion
Background invents objects	Scene contains too much empty ambiguity	Describe the environment more precisely
Scene becomes unstable	Too many subjects or actions	Reduce the shot to one action
Logo or label changes	Model is regenerating fine typography	Add exact text during editing
Motion is technically correct but boring	No visual objective	Define reveal, tension, scale, or product focus
Avatar mouth motion looks weak	Portrait or audio is unclear	Use a front-facing face and cleaner speech
Output cannot be edited cleanly	Important action begins too early or ends too late	Add visual handles before and after the action

The important pattern is that most fixes involve removing uncertainty, not adding more adjectives.

6. Use Camera Motion Before Object Motion

For product, furniture, vehicle, jewelry, architecture, and real-estate visuals, camera motion is often safer than large object movement.

Compare these two instructions:

Make the entire chair spin around rapidly.

Keep the chair stationary while the camera performs a slow
fifteen-degree arc from left to right.

The first instruction asks the model to imagine unseen parts of the object.

The second instruction asks it to move the viewpoint while preserving more of the source evidence.

The same principle works for lighting:

Keep the bottle stationary. Move a soft highlight across the glass
from left to right.

A light sweep can create visible motion without requiring the product to change shape.

For this reason, the product photo to video workflow emphasizes controlled movements such as:

small camera orbits
slow push-ins
side slides
restrained light sweeps
subtle background movement

The objective is not maximum movement.

The objective is useful movement with minimum unwanted change.

7. Add Exact Text After Generation

Product labels, prices, ingredient lists, interface text, captions, and calls to action are all high-risk elements.

Even when a first frame contains correct typography, generated motion may distort individual letters between frames.

My default workflow is:

Generate the clean visual without new text.
Preserve enough negative space for copy.
Export the selected clip.
Add exact typography in an editor.
Add captions, legal text, pricing, and calls to action after generation.

This is not a limitation unique to one tool.

It is a production decision that separates generative visuals from deterministic brand assets.

The AI generator creates the motion layer.

The editor creates the accuracy layer.

8. Estimate the Cost of the Experiment Before Running It

The cost of an AI video project is not simply the cost of one successful generation.

A more realistic formula is:

Estimated budget =
number of concepts
× variants per concept
× credits per run
× retry multiplier

For example, assume:

3 concepts
2 variants per concept
20 credits per run
25% retry allowance

The estimate is:

3 × 2 × 20 × 1.25 = 150 credits

This estimate is still simple, but it is much more realistic than planning for a single generation.

I use a three-pass funnel:

Pass 1: Direction

Test multiple concepts using the simplest reasonable configuration.

The question is:

Is this idea worth developing?

Pass 2: Control

Take only the strongest concepts and improve motion, camera behavior, and preservation constraints.

The question is:

Can this direction become stable and repeatable?

Pass 3: Quality

Use higher-quality or more expensive generation settings only for the finalists.

The question is:

Is the improvement large enough to justify the extra cost?

A useful generator interface should show the cost before submission. That allows the budget calculation to happen before the credits are consumed instead of after the experiment is already expensive.

9. Treat Talking Avatars as a Separate Workflow

A talking portrait is not simply an image-to-video task with a longer prompt.

It has a different input contract:

one portrait
one voice clip
one speaking subject
one short message
small facial and upper-body movement
identity and permission checks

For a talking avatar, I evaluate:

Does the face remain recognizable?
Does the mouth movement roughly follow the speech?
Do the eyes, teeth, jaw, and hair remain stable?
Does the motion support the message instead of distracting from it?
Is the result clearly disclosed when viewers may mistake it for a real recording?

A clean front-facing portrait usually provides a stronger starting point than a dramatic side profile.

A clean single-speaker audio file is easier to drive than speech mixed with music, reverb, or several voices.

The current AI avatar video generator uses an uploaded portrait and uploaded audio rather than pretending that avatar generation, script writing, and voice cloning are the same task.

Keeping those stages separate makes the workflow easier to review:

Script
  ↓
Approved or licensed voice recording
  ↓
Authorized portrait
  ↓
Avatar generation
  ↓
Identity and lip-motion review
  ↓
Captions and final edit

Permission also matters.

Do not animate a real person's portrait or voice without authorization. Avoid deceptive endorsements, impersonation, fabricated statements, and misleading political, medical, or financial content.

10. What a Production-Oriented AI Video Interface Should Expose

A useful AI video interface needs more than a prompt box.

At minimum, I want to see:

Mode

Is this text-to-video, image-to-video, image generation, or avatar generation?

Input requirements

Which files, dimensions, durations, and formats are accepted?

Model or quality choice

Can I choose between speed, quality, resolution, or a different generation behavior?

Cost before submission

How many credits will this specific configuration use?

Progress state

Is the task uploading, queued, generating, saving, completed, or failed?

Failure handling

Are reserved credits released when an upstream generation task fails?

Generation history

Can I return to earlier results instead of losing them after the session?

Download and comparison

Can I inspect multiple variants and export the one that scored best?

These product details are not as exciting as a viral demo clip, but they determine whether a generator can support a repeatable workflow.

What I Learned From Building the Workflow

The most useful lessons were not about finding one magical prompt.

They were about reducing variables.

Simple prompts are easier to debug

A short prompt with one subject, one action, one camera move, and one tone is easier to improve than a paragraph containing ten competing ideas.

The first frame is a control mechanism

When appearance matters, an approved source image often provides more control than adding more descriptive text.

Camera motion is often safer than object motion

Moving the camera, light, mist, reflection, or background can create useful motion while reducing product deformation.

Exact typography belongs in post-production

Generate the visual first. Add accurate words, prices, captions, and calls to action afterward.

The best result is not always the most spectacular result

A restrained clip that preserves the subject and fits into an edit can be more valuable than a visually dramatic clip that cannot be used.

AI video generation should produce information

Even a failed generation can be useful when the test changes one variable and reveals what the model does not handle well.

A random failure wastes credits.

A controlled failure improves the next experiment.

Final Take

The practical question is not:

Can this AI video generator make one impressive clip?

The better questions are:

Can it follow a repeatable motion brief?
Can it preserve the details that matter?
Can I diagnose failures without rewriting everything?
Can I estimate the cost before generating?
Can the result enter a real editing workflow?

Treat the process as:

Define → Generate → Score → Diagnose → Change one variable → Repeat

That turns AI video generation from a prompt lottery into an evaluation process.

You can use the framework with any video model or platform. To test it in a public text-to-video and image-to-video workflow, start with one real shot inside Haiper AI, keep the settings fixed, and score the result before changing the prompt.

The goal is not to generate more clips.

The goal is to learn more from every generation.

How to Turn a Research Paper Method Section Into a Diagram With AI

guowu zong — Sun, 12 Jul 2026 08:46:16 +0000

Creating a research figure often takes much longer than expected.

The scientific idea may already be clear, the experiments may be complete, and the paper may be nearly finished. But then comes the frustrating part: moving boxes in PowerPoint, aligning arrows, rewriting labels, adjusting spacing, and redrawing the same methodology diagram after every revision.

Generic AI image generators can create attractive images, but research figures are a different problem. A useful academic diagram must preserve the correct stages, relationships, labels, and data flow.

That is the problem I wanted to explore with PaperBanana, an AI-assisted workflow for creating methodology diagrams and research figures from paper text, notes, sketches, and reference images.

Why Research Figures Are Difficult to Generate

A normal image-generation prompt might say:

Create a beautiful AI system architecture diagram.

That instruction is too vague for a scientific figure.

The model does not know:

Which components must appear
What order the stages follow
Which arrows represent data flow
Which labels must remain unchanged
Whether the figure is for a paper, poster, or presentation
Which visual details are scientifically important

The output may look polished while still being structurally incorrect.

For research figures, visual quality is only one part of the task. The figure must also remain faithful to the underlying method.

Start With Source Context, Not Just a Prompt

A better workflow is to provide the model with the actual source material.

This can include:

A methodology section from the paper
A short system description
Notes explaining the experimental pipeline
A rough diagram drawn on a whiteboard
An existing figure that needs to be cleaned up
A caption describing the intended figure

For example, instead of writing:

Make a RAG diagram.

You could provide:

Our system contains four stages.

First, the user question is converted into an embedding. The embedding
is used to retrieve the top five relevant documents from a vector
database.

The retrieved documents are compressed by a context-selection module.
The compressed context and the original question are then passed to
the language model.

Finally, the model produces an answer with citations to the retrieved
sources.

Then add a figure instruction:

Create a left-to-right methodology diagram showing the four stages:
query embedding, vector retrieval, context compression, and cited
answer generation.

Use clear academic labels, a white background, readable arrows, and a
layout suitable for a machine-learning paper.

This gives the generator both the scientific structure and the desired presentation.

A Practical AI Figure Workflow

Here is the workflow I recommend.

1. Extract the Essential Method

Do not paste the entire paper unless every section is relevant.

Start with the smallest amount of text that completely explains the figure. Include important component names, stage order, inputs, outputs, and relationships.

The goal is not to make the input shorter at any cost. The goal is to remove information that does not belong in the figure.

2. Write the Figure Caption First

A good caption acts as a specification.

For example:

Overview of the proposed multi-agent research assistant. A planner
decomposes the user request, a retriever collects external evidence,
an executor performs the task, and a critic evaluates the result
before the final response is returned.

Even when the final caption changes later, writing it first helps define what the diagram must communicate.

3. Choose the Correct Figure Type

Different research content requires different visual structures.

Common options include:

Methodology diagrams
Model architecture diagrams
Data-processing pipelines
Multi-agent workflows
Benchmark comparison panels
Ablation-study figures
Graphical abstracts
Biological or chemical mechanism diagrams
Conference poster panels

Choosing the figure type early helps the model decide whether to use a pipeline, layered architecture, multi-panel composition, or another layout.

4. Generate a Draft Before Chasing Perfection

The first output should be treated as a structured draft.

Check the scientific logic before spending time on appearance:

Are all required stages present?
Are the arrows pointing in the correct direction?
Are any components duplicated?
Did the model change technical terminology?
Is the most important contribution visually clear?
Can the figure be understood without reading the full paper?

A beautiful but incorrect diagram is not useful.

5. Revise With Specific Instructions

Avoid vague feedback such as:

Make it better.

Use targeted revision instructions:

Keep the current four-stage structure.

Move the vector database below the retrieval module.
Add a feedback arrow from the critic to the planner.
Shorten the labels inside each block.
Increase spacing between the retrieval and generation stages.
Do not change any technical component names.

Specific instructions make iterative editing much more reliable.

6. Verify Everything Manually

AI-generated academic figures must still be reviewed by a human.

Always verify:

Technical terminology
Mathematical notation
Dataset names
Numerical values
Chart axes
Legends
Arrow direction
Component relationships
Citations and attributions

The generator should reduce repetitive design work, not replace scientific review.

Starting From a Sketch

Sometimes text is not the best starting point.

You may already have a rough diagram in PowerPoint, a whiteboard photo, or a tablet sketch. In that case, the sketch can define the approximate composition while written instructions explain what should be preserved or changed.

For example:

Preserve the five-block layout and the current left-to-right flow.

Replace the handwritten labels with clean academic typography.
Align all blocks to the same height.
Use consistent arrow thickness.
Keep the feedback loop shown at the bottom.
Remove the gray background.

This approach is useful when the structure is correct but the figure is not visually ready for a paper or presentation.

What I Added to PaperBanana

I built PaperBanana around several common research-figure workflows:

Generating methodology diagrams from source text
Creating model architecture and pipeline figures
Using sketches or reference images as structural guidance
Editing existing figures with text instructions
Improving the resolution of low-quality drafts
Exploring reusable prompts for common research diagrams

There is also a research figure prompt library with examples for RAG pipelines, transformers, diffusion models, multi-agent systems, benchmark charts, biological mechanisms, and other figure types.

The intention is not to create generic AI artwork. It is to provide a more focused starting point for researchers who need structured technical visuals.

Where This Workflow Is Most Useful

AI-assisted figure generation is especially helpful during:

Early paper drafting
Internal lab reviews
Conference submission preparation
Reviewer-response revisions
Camera-ready updates
Poster creation
Technical presentations
Documentation for research prototypes

It is also useful for developers writing about complex systems. Architecture diagrams, agent workflows, retrieval pipelines, and infrastructure overviews often have the same structural requirements as academic methodology figures.

Current Limitations

There are still important limitations.

Long labels may be rendered incorrectly. Complex equations may need to be added manually. Dense figures can become crowded. Statistical plots containing exact values should be generated from real data whenever possible rather than reconstructed from a text description.

For final publication, vector editing tools may still be necessary.

The most practical workflow is therefore:

Research context
      ↓
AI-generated structured draft
      ↓
Scientific verification
      ↓
Targeted AI revision
      ↓
Manual final cleanup

AI is strongest at accelerating the first draft and repeated visual revisions. Human researchers remain responsible for correctness.

Try It With One Section of Your Paper

A simple experiment is to take one paragraph from your methodology section and turn it into a figure specification.

Identify the stages, write a one-sentence caption, define the desired layout, and generate the first draft.

You can try the workflow at PaperBanana.

I would be interested to hear which research figures are still the hardest to create: model architectures, experimental pipelines, result panels, graphical abstracts, or something else?

Disclosure: PaperBanana.com is an independent third-party web service built to provide online access to workflows inspired by the open-source PaperBanana research direction. It is not affiliated with, authorized by, or officially connected to Google, Peking University, or the original project authors.