AI Image and Video Agents Need an Evaluation Loop, Not Just a Better Prompt

An image generator can produce a beautiful result and still fail the job.

The subject may be cropped. The product color may be wrong. A word in the poster may be misspelled. A video may look convincing frame by frame but lose the character's identity after three seconds. The soundtrack may have nothing to do with the scene.

A one-shot generation script cannot respond intelligently to any of this. It accepts a prompt, calls a model, saves a file, and stops. Even if you wrap that script in a chat interface, it is still a generator—not an agent.

A useful generative media agent must be able to answer a harder set of questions:

1. What does success mean for this specific task?
2. Which model or tool should handle each step?
3. How should the result be evaluated?
4. If it fails, which part should be regenerated?
5. When should the system stop spending money and ask for human review?

That changes the problem from generation to control.

The core system is not a larger prompt. It is a closed loop:

Goal -> Plan -> Generate -> Evaluate -> Decide
                    ^          |
                    |          +-> Accept
                    |          +-> Retry the smallest failed unit
                    +----------+-> Escalate to a human

Every step records inputs, outputs, model versions, scores, latency, and cost.

This article turns the main ideas from DeepLearning.AI and Google's AI Agents for Image and Video Generation course into an engineering pattern for systems that need to run more than once.

The four roles in a minimal generative media agent

A reliable image or video agent can be divided into four roles. They do not need to be four separate LLMs or services. They do need clear contracts.

1. The planner converts intent into a testable contract

The planner should not merely make the user's prompt longer. It should turn a vague brief into structured requirements.

For an image, that contract might include:

• subject and setting;
• composition and camera angle;
• visual style and lighting;
• elements that must appear;
• elements that must not appear;
• exact text, if text rendering matters;
• invariants that must survive later edits.

For video, add:

• action and temporal progression;
• camera movement;
• dialogue, sound effects, and music;
• character and object continuity;
• constraints across scenes.

The same contract drives both generation and evaluation. If the plan never says the person must appear full-body, the evaluator has no defensible basis for rejecting a waist-up crop.

2. The generator has a narrow job

The generator receives the plan and reference assets, then returns a candidate artifact with operational metadata:

generate(plan, references) -> {
  artifact_uri,
  model,
  parameters,
  latency,
  estimated_cost
}

It should not decide whether its own output passed. Combining creation and approval makes failures difficult to classify and gives one model's biases control over the whole loop.

3. The evaluator turns taste into a decision

An evaluator that returns only 7.8/10 is not very useful. The controller still does not know what failed or what to do next.

A better result is structured:

{
  "passed": false,
  "failure_type": "visual",
  "score": 0.68,
  "confidence": 0.91,
  "failed_checks": ["subject_crop", "brand_color"],
  "feedback": "Restore the subject's full body and use the approved navy background."
}

For video, the result should also identify the scene and time range. “The video is inconsistent” forces a full regeneration. “The character's jacket changes from 00:04 to 00:06 in scene 2” creates a repairable task.

4. The controller owns state, money, and exit conditions

The controller reads the evaluation, selects the next action, tracks attempts, and guarantees that the loop ends.

An LLM can help classify a failure, but deterministic code should enforce:

• maximum attempts;
• maximum elapsed time;
• per-task cost limits;
• allowed failure types and actions;
• non-retryable safety or policy failures;
• artifact storage and versioning;
• mandatory human approval points.

Prompts are probabilistic controls. Budgets and stop conditions should not be.

Evaluation should be a funnel, not a single score

Generated media rarely has one correct answer. Several images can satisfy the same prompt, and a technically valid video can still be unusable for a brand campaign.

The practical answer is a layered evaluation funnel, ordered from cheap checks to expensive judgment.

Layer	Method	Best at	Blind spot
Mechanical validation	File integrity, format, dimensions, duration, policy rules	Rejecting corrupt or obviously invalid artifacts	Cannot judge meaning or aesthetics
Semantic alignment	Image-text similarity models such as SigLIP	Detecting a major mismatch between prompt and image	Weak on composition, typography, and brand character
Multimodal judgment	LLM-as-a-Judge with explicit criteria	Explaining composition, readability, consistency, and instruction following	Sensitive to rubric design and model bias
High-stakes approval	Structured rubric plus human review	Brand sign-off, safety, cultural context, final selection	Expensive and slow

SigLIP is useful as a filter, not as an art director. It can help detect that an output is about the wrong subject. It cannot reliably decide whether a layout feels professional or whether a visual belongs to a particular brand.

Multimodal LLMs can make richer judgments, but vague questions produce unstable answers. “Is this image good?” is a weak evaluation prompt. Ask separate questions about subject completeness, instruction following, composition, visual defects, text accuracy, and brand consistency. Require structured output.

A rubric goes one step further by decomposing the source prompt into answerable checks:

• Is there exactly one cat?
• Are the flowers to the cat's right?
• Is all required copy present and spelled correctly?
• Is any unrequested text visible?

Google Research's Gecko work uses this kind of question decomposition to make text-to-image evaluation more diagnostic. The point is not merely to rank outputs. It is to reveal which capability failed.

Machines should remove obvious failures at scale. Humans should spend their attention on expensive judgment.

Why PASS is a dangerous production API

The course notebook includes a useful demonstration: fetch a blog post, generate an infographic, evaluate factual accuracy, spelling, and aesthetics, then retry up to three times if the evaluator does not return PASS.

That is enough to teach the loop. It is not enough to operate one.

Several engineering gaps appear immediately:

1. Taking the first 5,000 characters of raw HTML may capture navigation and scripts instead of the article body.
2. An exact string comparison fails on PASS., whitespace, or a helpful explanation after the word.
3. Free-form feedback does not distinguish content, visual, audio, safety, and infrastructure failures.
4. Every failure regenerates the complete asset, even when a smaller retry would work.
5. Attempt count is limited, but token use, media generation cost, latency, and artifact lineage are not recorded.

The first production improvement is not necessarily a better model. It is a stable decision schema.

from dataclasses import dataclass
from enum import Enum


class FailureType(str, Enum):
    NONE = "none"
    CONTENT = "content"
    VISUAL = "visual"
    AUDIO = "audio"
    SAFETY = "safety"
    INFRA = "infra"


class Action(str, Enum):
    ACCEPT = "accept"
    RETRY_PLAN = "retry_plan"
    RETRY_IMAGE = "retry_image"
    RETRY_VIDEO = "retry_video"
    HUMAN_REVIEW = "human_review"


@dataclass(frozen=True)
class Evaluation:
    passed: bool
    failure_type: FailureType
    score: float
    feedback: str
    confidence: float


def choose_action(result: Evaluation) -> Action:
    if result.passed:
        return Action.ACCEPT
    if result.failure_type is FailureType.CONTENT:
        return Action.RETRY_PLAN
    if result.failure_type is FailureType.VISUAL:
        return Action.RETRY_IMAGE
    if result.failure_type is FailureType.AUDIO:
        return Action.RETRY_VIDEO
    return Action.HUMAN_REVIEW

This is deliberately plain. The controller should be easy to test. Model sophistication belongs behind typed boundaries, not inside an unbounded while not satisfied loop.

Retry the smallest failed unit

“The agent can retry” sounds impressive until every failure triggers another full video render.

The useful capability is failure routing: preserve what is valid and regenerate only the earliest broken dependency.

Failure	Preserve	Regenerate	Why
Audio mismatch	Scene plan and reference frame	Video clip or audio layer	The visual anchor is still valid
Broken motion or temporal continuity	Scene plan and reference frame	Video clip with revised motion prompt	The error appears over time
Wrong subject, composition, or style	Scene plan	Reference frame and downstream clip	The visual anchor is already wrong
Factual script error	Original brief	Scene plan and downstream assets	Rendering cannot repair an upstream semantic error
Safety or rights risk	All artifacts for evidence	Nothing automatically	Rewording should not be used to evade review

This principle controls both cost and regression risk. A full regeneration may fix one problem while changing five things that were already correct.

Image agents need explicit invariants

Multi-turn image editing creates the appearance of continuity, but continuity is not automatic. Each edit should state what must remain unchanged.

Examples:

• Change only the background color; preserve subject, pose, and composition.
• Correct the spelling; do not introduce new text.
• Preserve the logo's proportions and clear space.
• Use the reference image for style only; do not treat it as the image to edit.

These constraints should live in a stable task-level BrandProfile or design contract. A retry-level image artifact should not be allowed to rewrite the brand rules that judge it.

A clean tool boundary looks like this:

analyze_brand(reference) -> BrandProfile
plan_concepts(brief, profile) -> list[Concept]
generate_image(concept, reference) -> ImageArtifact
evaluate_image(artifact, concept, profile) -> Evaluation

Video agents add time as a failure dimension

A single video frame may look correct while the complete clip fails. Motion can jump. A face can drift. The camera can violate the scene plan. Dialogue can lose sync. Three individually attractive scenes can look like three unrelated films.

A practical video pipeline separates appearance from motion:

1. Convert the brief into structured scenes.
2. Generate a reference frame for each scene under a shared style contract.
3. Animate each frame with an explicit motion, camera, and audio plan.
4. Evaluate visual quality, temporal continuity, prompt adherence, and audio.
5. Route each failure to the smallest affected stage.

Reference frames act as visual anchors. Image-to-video, first-and-last-frame generation, and reference-image controls make it possible to separate “what should this scene look like?” from “how should it move?”

The evaluator must also become temporal. A video-level score hides too much. Evaluate by scene, time span, character, motion, camera, and audio track.

The production features that demos usually omit

Once the loop works, six less glamorous systems determine whether it remains trustworthy.

Observability

Record inputs, outputs, model and prompt versions, parameters, latency, cost, scores, failure types, and next actions. For asynchronous video jobs, also save the operation ID and polling state. A production trace must answer: Why did the system retry? What changed? Why was the final result accepted?

Idempotency and artifact lineage

Never silently overwrite an earlier attempt. Use paths such as task_id/attempt_id, hash artifacts, and point to the accepted version explicitly. “The newest file in the folder” is not a reliable state model.

Budgets and exit conditions

Limit attempts, elapsed time, and estimated spend. Three text-planning retries and three video generations have radically different costs.

Evaluator calibration

Keep a small human-labeled set. Compare automatic decisions with human decisions, especially after changing a judge model or evaluation prompt. A higher judge score does not automatically mean greater user satisfaction.

Safety and rights review

Track permissions for input assets, likenesses, trademarks, and licensed material. A safety failure should route to a policy decision, not an automated prompt-variation loop.

Human approval

Low-risk bulk assets may pass automatically. Brand campaigns, public advertisements, and expensive final videos should retain an explicit human sign-off.

When an ordinary pipeline is the better design

Not every media workflow needs an agent.

If the steps are fixed, inputs are structured, and deterministic rules can judge each output, an ordinary pipeline will be cheaper and easier to test.

Agentic control becomes useful when:

• goals arrive underspecified and require dynamic decomposition;
• different failures need different recovery paths;
• outputs have no single correct answer but can be judged by multiple criteria;
• tool order changes in response to intermediate results;
• quality, latency, and cost must be balanced at runtime.

The strongest design is usually hybrid: deterministic code owns state, limits, and policy; multimodal models handle interpretation, planning, generation, and open-ended judgment; humans make high-risk decisions.

A build order that prevents expensive confusion

Start with the smallest loop you can measure:

1. Build one-shot generation and save the complete input and raw artifact.
2. Define three to five task-specific quality checks.
3. Add an evaluator in report-only mode. Do not retry automatically yet.
4. Compare automated judgments with human review and calibrate thresholds.
5. Add a targeted retry for one frequent, well-understood failure.
6. Expand failure types, budgets, concurrency, and approval workflows only after the first path is stable.

This sequence may look conservative. It prevents the most common failure mode: an agent that “keeps improving” an artifact while cost rises and nobody can explain whether quality actually changed.

The real product is the control loop

The generator is only one component.

The planner creates a testable contract. The generator produces a candidate. The evaluator returns a structured diagnosis. The controller selects the smallest valid repair and enforces budgets. Humans retain authority over outcomes where context, rights, or brand risk matter.

When a system can explain why an artifact failed, what it regenerated, how much it spent, and why the final version passed, it stops being a slot machine with an API.

It becomes a production system.

References

• AI Agents for Image and Video Generation — DeepLearning.AI
• Evaluation — Google Agent Development Kit
• Veo video generation prompt guide — Google Cloud
• Revisiting Text-to-Image Evaluation with Gecko — Google Research
• Sigmoid Loss for Language Image Pre-Training — SigLIP paper