Your AI Video Passed QA—and Still Wasn't Publishable

I defined this acceptance model after rejecting technically valid AI-generated videos in my own production pipeline. AI helped reorganize the Japanese framework for DEV readers and draft the examples. I am not claiming a measured quality increase or a completed three-video stability proof. #ABotWroteThis

The MP4 rendered.

It had an audio track. It had subtitles. ffprobe returned the expected codec, resolution, streams, and duration. The timeline checks passed.

I still rejected the video.

The background did not support the explanation. Sentence boundaries and visual changes disagreed. Speaker turns had no believable pause. The characters moved, but the result felt like two avatars standing still and opening their mouths.

The pipeline had succeeded technically. The artifact had failed as media.

That distinction changed how I define "done" for AI video generation.

I no longer use one completion state. I use four:

1. moves
2. usable
3. accepted
4. stable

Automated QA can prove the first. It cannot prove the other three.

Automated QA proves validity, not publishability

Automated checks still matter. They should catch deterministic failures before a person spends time watching the result.

For example:

• the video file opens;
• video and audio streams exist;
• duration, codec, and resolution are within contract;
• subtitles are present and their timestamps are ordered;
• the timeline has no empty or overlapping segments;
• every required artifact and source trace exists.

Those checks should not be weakened. They should be made stricter.

The mistake is using their PASS result as a proxy for audience acceptance.

A valid audio stream does not prove that pronunciation and pauses feel natural. A subtitle file does not prove that captions and visuals are not fighting for attention. One image per segment does not prove that the background, framing, or character movement carries meaning.

Technical validity is necessary. It is not the release decision.

Separate four release states

State	What it proves	Exit condition
moves	The pipeline produced a technically valid artifact	MP4, audio, subtitles, provenance, and automated QA pass
usable	A first-time viewer can follow it without production-side explanation	No major first-watch disruption; any required fixes are localizable
accepted	The release owner and target viewer accept the actual artifact	publishable_as_is or narrowly defined publishable_with_minor_edits
stable	The same production method works beyond one lucky example	Three different topics are consecutively Accepted under one series-template hash

This is a state machine, not a score.

moves does not gradually become usable because the codec score is high. accepted is not usable + 10 points. Each transition needs different evidence.

To move from moves to usable, somebody has to experience the artifact at normal speed.

To move from usable to accepted, the production team cannot be the only judge.

To move from accepted to stable, one successful video is insufficient.

Review the same artifact three different ways

A normal first watch mixes script, pronunciation, pacing, framing, captions, and visual continuity. When the result feels wrong, that integrated impression is real—but it is often too entangled to tell the pipeline where to return.

I use three passes in a fixed order.

Pass 1: audio only

Close the video or turn the display away. Listen without captions or visuals.

Check:

• pronunciation of names and technical terms;
• phrase boundaries, emphasis, and speed;
• pauses between statements;
• pauses when the speaker changes;
• the gap between a question and its answer;
• whether the argument and causal chain are understandable from audio alone;
• whether the delivery fits the intended speaker or character.

If the explanation cannot be followed from audio, do not add more captions and images to conceal it. Return to the script, narration treatment, or affected audio segments.

Pass 2: silent visual

Mute the audio and hide the captions.

Check:

• whether the background communicates topic, location, or state change;
• whether characters do more than stand and lip-sync;
• whether framing, expression, gaze, objects, and cuts carry information;
• whether repeated background, placement, and motion create visual stagnation;
• whether a viewer can recover at least the current topic from the screen.

If meaning disappears when the captions disappear, adding more text is not a visual fix. Return to the storyboard, visual treatment, or shot plan.

Pass 3: one integrated first watch

Now watch the complete artifact once, at normal speed.

Do not pause to explain what the production system intended.

Check:

• whether audio, visuals, and captions compete for attention;
• whether visual changes align with the same semantic units as the narration;
• whether the background or character performance lowers the credibility of the explanation;
• where the first urge to stop watching occurs;
• which release verdict applies.

Use a small verdict set:

publishable_as_is
publishable_with_minor_edits
revise
redesign

The most important output is not an average score. It is the first material blocker and the layer that owns it.

Record the first blocker, not an average

Suppose pronunciation scores 8/10, visuals 7/10, and captions 9/10. An average of 8/10 looks healthy.

It can still be unpublishable.

If a product name is mispronounced in the first few seconds, later quality does not cancel the error. If the background implies the opposite of the narration, a polished art style does not make the claim trustworthy.

I store a review result like this:

review:
  pass: integrated_first_watch
  verdict: revise
  first_blocker:
    category: pacing
    segment_id: s07
    evidence: "The question ends and the answer starts without a perceptible turn-taking gap."
  return_to: audio_treatment
  allowed_fix_scope:
    - pause_before_s08
    - regenerate_audio_s07_s08
  material_revision: false

This record answers the operational questions that a score cannot:

• What failed first?
• Where did it fail?
• What evidence supports the rejection?
• Which production layer owns the repair?
• What may change in the next run?
• What must remain fixed?

Make the release boundary machine-readable

A human may supply the observations, but the decision boundary can still be explicit.

from dataclasses import dataclass
from enum import Enum


class Verdict(str, Enum):
    PUBLISHABLE = "publishable_as_is"
    MINOR = "publishable_with_minor_edits"
    REVISE = "revise"
    REDESIGN = "redesign"


@dataclass
class PassResult:
    name: str
    verdict: Verdict
    first_blocker: str | None = None
    return_to: str | None = None


def release_decision(results: list[PassResult]) -> dict:
    for result in results:
        if result.verdict in {Verdict.REVISE, Verdict.REDESIGN}:
            return {
                "accepted": False,
                "failed_pass": result.name,
                "first_blocker": result.first_blocker,
                "return_to": result.return_to,
            }

    return {
        "accepted": True,
        "minor_edits_only": any(
            result.verdict == Verdict.MINOR for result in results
        ),
    }

This is not a claim that computer vision or an LLM can reliably infer every perceptual defect.

The purpose of the code is narrower: once observations exist, do not let a scoring average erase an observable release blocker.

Define "minor edits" narrowly

publishable_with_minor_edits easily becomes a polite name for "not actually accepted."

I limit minor edits to changes such as:

• a typo;
• a local volume adjustment;
• one segment's pronunciation;
• one caption timing correction;
• a short replacement that does not change the argument or structure.

I treat these as material revisions:

• restructuring the script;
• regenerating a substantial part of the narration;
• changing the background or overall screen design;
• changing the character's role or personality;
• redesigning pacing across several segments.

If a material revision is required, that run does not count as Accepted. The repaired artifact becomes a new run and goes through all three passes again.

This rule prevents acceptance metrics from improving only because every rejected artifact was renamed "accepted with edits."

After the same rejection twice, stop micro-tuning

Small parameter changes are useful for a local defect:

• add a little more pause;
• reduce speech speed;
• adjust one transition;
• change one crop.

But if the same rejection category appears twice, the problem may no longer be a parameter.

I use this return rule:

same_rejection_category >= 2
    -> stop micro-tuning
    -> return to audio treatment or visual treatment
    -> create a new run

Repeating the same ambiguous prompt with slightly different numbers produces more artifacts without improving diagnosis.

Moving up one design layer makes the next hypothesis testable.

One Accepted video is not a stable pipeline

A template may work for one topic because the script happens to fit its timing and visual rhythm.

Change the topic and the pipeline can fail immediately.

I separate a single success from operational stability by keeping a consecutive-acceptance streak tied to one series-template hash.

def update_stability_streak(
    streak: int,
    accepted: bool,
    same_template: bool,
    material_revision: bool,
) -> int:
    if not accepted or not same_template or material_revision:
        return 0
    return streak + 1

My current release rule calls the template Stable only after three different topics are consecutively Accepted.

Three is not a universal statistical threshold. It is an operating rule for separating "one successful sample" from "a template I am willing to reuse."

The important part is the reset behavior:

• revise resets the streak;
• redesign resets the streak;
• a material revision resets the streak;
• changing the series-template hash starts a new streak.

Do not publish three successes selected after the fact and ignore the failures between them. The sequence is the evidence.

Keep machine QA and human QA in different jobs

The answer is not to replace automation with subjective review.

It is to assign each type of evidence to the right owner.

Check	Machine responsibility	Human responsibility
Codec, resolution, required streams	Primary	None
Duration, timeline gaps, subtitle ordering	Primary	Exception review
Required artifacts and source trace	Primary	Meaning and rights judgment
Natural pronunciation and conversational pauses	Assistance	Primary
Whether backgrounds and shots carry meaning	Assistance	Primary
Attention conflict across audio, visuals, and captions	Limited	Primary
Whether the first-watch artifact should be published	None	Primary
Repeatability across different topics	Record and calculate	Interpret and approve

With this split:

• automated QA proves moves;
• first-watch review proves usable;
• owner and target-viewer decisions prove accepted;
• run history proves stable.

Each claim has evidence appropriate to it.

A checklist for the next AI video

Before generation:

• [ ] Decide which state this run is trying to prove.
• [ ] Fix the series-template hash.
• [ ] Define the boundary between minor and material revision.
• [ ] Name the target viewer.
• [ ] Prevent reviewers from receiving production-side explanations.

After generation:

• [ ] Prove moves with automated QA.
• [ ] Review audio only.
• [ ] Review silent visual.
• [ ] Perform one integrated first watch at normal speed.
• [ ] Record one first blocker.
• [ ] Map the blocker to a return layer and allowed repair scope.
• [ ] After the same rejection twice, return to treatment design.
• [ ] Reset the stability streak after anything other than Accepted under the same template.

What this model does not guarantee

This framework does not automatically create a good video.

It does not guarantee:

• that every viewer will like the result;
• that one review catches every quality problem;
• that three accepted videos will prevent all future regressions;
• that AI can replace the release owner or target viewer;
• that one model or video provider is superior;
• that production time or cost will decrease.

Its purpose is to stop treating technical generation as finished media.

A video that moves is an engineering result.

A video that somebody can publish without explanation is a different result.

A pipeline that can repeat that outcome across topics is different again.