AI Video Needs CI More Than Better Prompts

Here I'll share what I built for Track 2 of the Global AI Hackathon Series with Qwen Cloud, and what it taught me about testing things you can't see.

My expensive little problem

I generated a five-second video clip. The first frame looked perfect.

Then I watched the other four seconds.

The camera drifted left. My brief said hold static.

Here's the part that irritates me: I found out after I paid. On Qwen Cloud's Wan models a clip is a fixed five seconds of premium quota whether it's usable or garbage. The bill doesn't care.

Text models don't work like this. You read the answer, you see it's wrong, you move on. The artifact and the verdict show up together and being wrong costs you a few hundred tokens.

Video fails in motion. The failure lives in frames you haven't looked at yet.

So you pay first, and find out second.

But it gets worse...

The obvious build for this track is a premise-in, episode-out generator. I started there. Within a couple of days I had something that produced a demo reel.

But demo reels hide the interesting failures.

Every bad take that reaches your final render is money you already spent. And I kept asking myself a question that had nothing to do with generation:

Can the system tell whether the video obeyed the brief, before it spends premium budget on it?

That question has a name in software. It's CI.

And almost nobody is running it on generated video.

So why bother? Don't we just prompt better?

That was my first instinct too. Write better prompts, get better clips.

But a prompt is not a contract. It's too easy to rewrite, stretch, and forget as an artifact moves between models. You can't assert against a prompt. You can't fail a build with one.

Before I burned a week on this bet, I wanted to know it wasn't just a slogan I liked the sound of. So I counted. Twice.

Test 1 — the simulated field. I ran 15 independent LLM brainstorms over this track's public brief. Two model families, 14 personas, none of them seeing my repo or each other. Around 150 ideas.

All 15 produced a generator.

Zero produced a way to check whether the generated episode came out right.

Two of them even emitted the phrase "CI for AI-generated video" verbatim — and then bolted it onto a generator anyway.

Test 2 — the observed field. Two days before the deadline I read the eight most substantial shipped entries in this track's public GitHub field. Every one routes its quality checks through a token-billed model call. Six of the eight import no computer-vision library at all. None ships a closed assertion vocabulary. None can gate a video it didn't itself generate.

Then I found this line in the config of a shipping logline-to-video pipeline:

DAILIES_QC = False   # off = faster pipeline

Someone built a review stage. Named it dailies. Shipped it switched off. For speed.

That's the whole problem in one line.

When the gate lives inside the generator's codebase, the gate is the first thing traded away for throughput.

Time to build the tests instead

In film production, dailies are the screening where the crew watches yesterday's footage before more money is spent shooting on top of it. It's the daily quality gate.

So that's the name. Dailies. The review stage that can't be quietly switched off, because it doesn't belong to the pipeline it judges.

It reads frames, not generator internals. Point it at any mp4 from any model.

Here's how it works.

You write what a shot has to do. Every take gets measured against it before it ships. A take that misses gets one bounded repair attempt. A take that still misses never reaches your cut.

A spec is a short program in a closed assertion DSL — ten sentence types across three tiers. The compiler translates each sentence into a call in a shared check library, and rejects anything outside the grammar before a single token is spent.

Send it something it doesn't recognize and parse_assertions raises. Send {"type": "vibe_check"} to the MCP server and it comes back isError, not a guess.

That rejection is the literal compile error in "CI for generated video."

Assertion type	Tier	Compiles to
duration_between	A · deterministic	clip length within bounds
brightness_range	A · deterministic	mean-luma within bounds
flicker_below	A · deterministic	inter-frame luma std under threshold
scene_cuts	A · deterministic	HSV-histogram cut count at or under max
camera_motion	A · deterministic	optical-flow pan direction
palette_deltae	A · deterministic	dominant-color ΔE to a brand palette
subject_present	0 · pre-render still	subject recognizable in the t2i still
identity_consistent	B · advisory	VLM: same identity across frames
action_completed	B · advisory	VLM: the briefed action visibly completes
title_card_present	B · advisory	VLM: a title / text card is visible

Notice there's an organizational idea hiding in that table.

Right now, the person who defines "correct" — brand, legal, marketing — is coupled to the person who operates the generator. They're joined by one overworked reviewer eyeballing every clip.

The stakeholder writes these assertions once, in plain language. Every shot gets tested against them automatically.

Spec-driven development, for video.

The tiers are boring on purpose

Not every check deserves the same budget. So the cascade runs cheapest-and-most-certain first.

Tier-A is deterministic OpenCV. Duration, brightness, flicker, scene cuts, optical-flow camera motion, palette ΔE. Zero tokens, so it runs on every take.

It's deliberately unglamorous. OpenCV has no idea whether your story works. But it cheaply catches a huge amount of broken video, and it never has an opinion it can't defend with a number.

Measured over 25 iterations: all six checks on a 5-second clip cost a median 317 ms of CPU. Nothing else.

Tier-B is Qwen-VL, and it's advisory. A VLM is good at "does this appear to match the brief." But a model judgment is softer evidence than a pixel measurement — so Tier-B flags for the human and never blocks promotion.

Deterministic checks are the foundation. Model-graded judgment sits on top. Never underneath.

Tier-0 is a still-image pre-screen. Before any motion is generated, a wan2.1-t2i-plus still renders at roughly 1/25th of a video's cost, and qwen-vl-plus gets asked one question: is the briefed subject even in the frame?

If a prompt can't render its own subject as a still, paying for motion just buys you the same lesson at 25× the price.

Only after surviving all that does a shot earn promotion:

premise → script + specs (qwen-plus) → compiled assertion checklist
        → Tier-0 still pre-screen (wan2.1-t2i-plus)
        → [human review gate — the one checkpoint, before any video spend]
        → drafts (wan2.1-t2v-turbo)
        → Tier-A CV (deterministic, zero tokens) + Tier-B VLM (qwen-vl-plus, advisory)
        → bounded prompt-repair + retake (qwen-plus)
        → promote passing shots (wan2.2-i2v-flash, anchored on the approved frame)
        → narration, one voice per character (qwen3-tts-flash)
        → ffmpeg assembly → certified episode, with sound

A rendered architecture diagram was a required deliverable anyway, so here's the same thing drawn formally. One convention worth knowing: red is reserved for the two paths that matter most — a spec rejected before any spend, and a blocking Tier-A fail.

Two human roles that today collapse into one exhausted reviewer. The stakeholder owns "correct" and authors the spec once. The operator is the only human in the loop at runtime, at the single gate before any video is paid for.

Level 2 answers the question the hackathon actually asked — how Qwen Cloud connects to the frontend, backend, and store:

That was the design.

Then I ran it for real, and reality started correcting me.

The rule that kept saving me: let the model choose, never let it spell

Before the war stories, one design decision that turned out to matter more than anything else I did.

That closed vocabulary started as an assertion-layer detail. It ended up being the rule the whole system runs on.

There are three places where a Qwen model makes a real decision in Dailies. In all three, it emits a choice from a set the server owns — never the artifact itself.

Where	The model emits	The server produces	The failure it prevents
Assertions	a type from a closed set + typed params	a call into the check library	a check that means something other than what it says
Pipeline topology	run parameters (premise, pack, max_shots, custom_checks)	the canonical node/edge graph	a structurally malformed run
Narration casting	a speaker — a character name	a voice from a probed roster	a silent hole in a certified episode

That middle row became my headline demo.

You type a request in plain language. qwen-plus calls a build_pipeline_graph tool with tool_choice="auto". The server deterministically expands those parameters into the graph the run executes, rendered live in React Flow off the same 2.5-second poll the dashboard already uses.

So "an agent wired the pipeline" is a claim I can make without flinching. The model authored the run. It could not have emitted a broken graph, because it never emitted topology at all.

This is the difference between validating the model's output and making invalid output unrepresentable.

Validation is a check you can forget to run. Or run in only one of two code paths — which I did, and I'll get to that.

A parameter contract is a shape the bad value cannot fit through.

The day my green tests lied to me

Every end-to-end test I'd written used synthetic clips. A fake Wan client returning generated mp4s so I could exercise the pipeline for free.

That validates control flow. And nothing about the API's contract.

A fake client can't reject a request the real service would reject.

So I built a fixtures mode. Real Wan client, real Tier-A, real Tier-B, real assembly — with only the two non-deterministic text stages pinned to fixed prompts. (Prompts are cache keys, so a pinned run replays identically at zero quota.)

Then I ran it cold, against the real API, for the first honest time.

It found four bugs.

Seventy-nine passing tests had missed every single one.

Bug	Why 79 passing tests missed it
wan2.2-t2v-plus rejects 1280*720 — so every premium promotion the project had ever made had silently failed	_promote treats a failed promote as "keep the passing draft," so a rejected final and a healthy skip produced identical state. The premium tier — my entire wedge — had never actually run.
The budget governor asked the cache about the wrong resolution key for finals, calling every cached final "fresh"	Under judge mode that would have refused free replays during judging — the one thing the live URL depends on.
The ledger wrote an empty note for both a no-op and a hard failure	Which is precisely how the first two hid. The audit trail didn't audit.
state.json was write-only — snapshots atomically written, never read back	Every redeploy silently discarded every run a viewer had made, while my architecture diagram promised state persists across restarts.

Sit with that first row for a second.

My entire pitch is that generated video ships unverified because nobody runs the claim against the artifact.

And my own premium tier had been dead for the project's whole life. In a repo whose thesis is a conformance gate.

The gate caught its own maker.

I fixed all four in one commit, each with a regression test I verified by reverting the fix and watching the test fail. But this is the line I wrote in the log afterward, and it's the one I keep coming back to:

Mocks test the code you wrote. Fixtures test the assumption you made. This bug lived in the gap.

A mock returns what you told it to. So a suite of mocks can only ever confirm your own mental model. It re-asserts your assumptions back at you, in green.

The real contract lives at the boundary with the external service. The only test that touches it is one that actually calls the thing.

If you're integrating any generative API — spend the tokens on one real end-to-end run early. It's the cheapest bug-finder you'll ever build.

There was a fifth of the same family, and it's the one I'd hand to a test-design skeptic.

subject_present — my Tier-0 check — was declared, marked blocking-class, and evaluated by nothing. All three runtimes wired it to lambda spec, still: [], while the pipeline dutifully generated and billed a still for every shot, then stored the empty list it got back.

A green suite couldn't catch that. Because of one line:

take.passed = not [r for r in results if not r.advisory and r.status is Status.FAIL]

A check that returns no result and a check that passes are the same empty list.

Wiring it to a real qwen-vl-plus read costs 325 tokens per shot to avoid a 5-second premium clip. I measured it in both directions — present subject and absent — because a check that only ever answers PASS is not a check.

The receipt that made me believe in the thing

Run 3e1f628d4acf, shot 0, contract camera_motion: static.

The turbo draft passed cleanly at optical-flow magnitude |v|=0.30.

Then the premium wan2.2-t2v-plus render came back drifting left at |v|=0.92, and failed the same contract the draft had passed.

The gate did exactly what it exists for. Rejected the premium clip. Certified the cheaper draft.

Working as designed — but now my episode was stuck with the lesser render.

So I built targeted repair.

Tier-A doesn't just say a check failed. It keeps the per-frame series behind each measurement, so a failure carries a time window. Here it placed the drift at 0.4s–3.6s.

That window gives you an anchor: the last good frame, at 0.2s. A frame-anchored Wan model (wan2.2-i2v-flash) re-renders from that frame forward, Tier-A re-verifies, and a passing patch re-cuts the episode.

	before (premium, take 1)	after (patch)
camera_motion	FAIL — drifting left, `\	v\
flicker / brightness / duration / cuts	pass	pass

Cost: 5 video-seconds on the i2v free-tier pool, plus one {% raw %}qwen-plus repair call.

The premium look, recovered under contract, by the same gate that rejected it.

My best idea was only half right

That receipt made me greedy.

If anchoring on a good frame rescues a broken clip, why not anchor everything?

The reasoning felt airtight. Draft, repair, and final were three independent rolls from the same prompt — three separate lotteries — so a certified episode could look like three different films of the same script.

Anchor the retake on the last good frame. Anchor the promotion on the frame the human approved. Every take of a shot inherits one continuous look.

Same primitive, generalized. I shipped it.

Then I ran it on real Wan output. It was wrong in two specific, measurable places.

1. An anchor can't fix a defect that has no "before."

Shot 1 asserts camera_motion: right. Its first draft came back static — |v| = 0.005 against a 0.4 threshold — with Tier-A localizing the failure to fail_window_s [0.0, 5.33].

The entire clip.

There's exactly one frame before that window. Frame 0. And anchoring the retake there pinned the exact staticness the retake existed to remove.

The anchored retake measured |v| = 0.112. Still a FAIL.

A fresh t2v roll on the same repaired prompt reached |v| = 0.745 'right' and passed.

2. An anchor frame carries composition. It does not carry motion.

This one I did not see coming.

I took that approved rightward pan — |v| = 0.745 'right' — and promoted it by anchoring at 0.1s.

The premium final came back panning the other way. |v| = 6.15 'left'. Failed re-verify.

A single frame encodes framing, lighting, wardrobe, palette. It encodes nothing about which direction the camera was travelling when that frame was taken.

So the model just invents one.

The rule those two measurements produced is now code, not prompt advice:

Anchor to preserve. Re-roll to change.

Concretely: _anchor_for_retake returns None when the failure window opens at t = 0, so a whole-clip defect re-rolls from the corrected prompt instead. And a shot whose contract asserts camera motion skips promotion entirely — the clip that satisfied the contract is the clip that ships.

That second consequence is my favourite number in the whole project. Look at the ledger for a certified three-shot batch:

Stage	Calls	Billed
drafting (wan2.1-t2v-turbo)	4	20 video-s
promoting (wan2.2-i2v-flash, frame-anchored)	2	10 video-s

Three shots certified. Two promotions.

The missing one is the shot with the motion contract.

So verification here isn't only rejecting bad output. It's deleting a generation the run would otherwise have paid for, because measurement proved that generation could only make things worse.

A quality gate that reduces spend is a much easier thing to sell than one that adds to it.

Both findings cost one clip each to discover. 10 seconds out of a 600-second budget.

And notice which half of my idea survived. "Anchor for visual continuity" was right. "Anchoring beats re-rolling" was an unexamined extension of it, and it took two real generations to pull them apart.

When a primitive works beautifully in the case you built it for, the temptation is to extend it by resemblance.

Every bug I found was the same bug

Here's the postscript, and it's the most uncomfortable finding in my log.

The promotion path and the manual re-render button computed their anchors in two separate places.

pipeline.py learned the t=0 rule. patch.py had its own anchor_second, which clamped happily to 0.0.

So for another full day, the button in my UI cheerfully offered — and the endpoint faithfully performed — precisely the move I had just measured as useless.

No test caught it. And the reason is worth stating plainly:

Both surfaces agreed with themselves.

Each was internally consistent. Nothing in the suite asserted they agreed with each other. It only surfaced because someone finally clicked the button.

There's a third instance lurking in my deploy runbook. The budget governor wraps gen_video_fn — but promotion now renders through patch_video_fn, deliberately unwrapped so anchored work draws the separate i2v pool instead of competing with drafts.

Correct decision. Unintended consequence: fresh_final_cap no longer bounds a live run.

The safety rail didn't break. The thing it guarded moved out from under it.

Now line up every bug this project has found:

· a fallback that returns the success value on failure
· a missing result indistinguishable from a passing one
· two code paths deriving the same value independently
· a wrapper whose subject moved

Not one of them failed loudly.

Every single one presented as success.

Which is, uncomfortably, the exact thing my product exists to catch. Generated video ships unverified because a bad clip and a good clip both look like output.

Giving it a voice (and the roster the API won't tell you)

Wan's video models return silent clips. So my certified episode was silent.

Listing the models available to my key turned up 149 of them, including the qwen3-tts-flash voice family. A 107-character line synthesizes in 2.4 seconds.

Easy win. Except for two things.

The twelve-hour episode. The returned WAV is streamed, so its header declares an effectively infinite length. A ~7-second clip reports 44,739 seconds to both wave and ffprobe.

Anything that trusts that header to size the output produces a half-day-long file.

My assembler survives because -shortest bounds every segment by its video, and a paired apad stops a short narration line from truncating the video the other way. Verified: a 5.00s clip muxed with that 44,739s-declaring WAV comes out at exactly 5.00s.

Never trust a container's self-reported duration for a stream. Bound it by something you do know.

The roster you can't list. Narration shipped with one hardcoded voice, so every character sounded identical.

Giving a cast distinct voices means knowing which voices this account may actually use. And the API won't say:

400 InvalidParameter: Invalid voice specified, the requested voice does not exist
                      or is not licensed for use—please select a supported voice.

That message enumerates nothing.

And "not licensed for use" means availability is an account property, not a model property. So the roster has to be probed, not read.

I fired 20 candidate names at qwen3-tts-flash. Got 200-with-audio for all 20:

Cherry · Ethan · Nofish · Jennifer · Ryan · Katerina · Elias · Jada · Dylan · Sunny · Li · Marcus · Roy · Peter · Rocky · Kiki · Eric · Serena · Chelsie · Aiden

Which brings back the rule from earlier. My script agent emits a speaker — a character name. The server casts the voice from that probed roster.

The model cannot emit an invalid voice, because it never emits a voice at all.

That closure earns its keep here more than anywhere. Because this failure is silent: an unlicensed voice degrades that shot to silence rather than failing the run. A certified episode would just have a hole in it, and nothing would say so.

One more detail that's load-bearing. Casting is by order of first appearance, fixed at scripting time — not by hashing the character's name.

A hash can collide and hand two characters one voice with nothing to signal it. An ordinal is equally deterministic, so a re-run casts identically and every narration cache key still hits.

That key is sha1(model|voice|text). Unstable casting would silently re-synthesize every line on a replay that was supposed to be free.

Making it survive the judges

A gate is only real if someone can click it. And generated media is slow and quota-sensitive — exactly the wrong properties for a live demo.

A content-addressed cache. Identical (model, prompt, seed, size) requests replay from disk for free:

	billed video-seconds	wall clock	shots certified
Cold (2 uncached finals)	10 s (~$3.00 est.)	207 s	3/3
Warm (everything cached)	0 s ($0.00)	6 s	3/3

Re-certifying real 1080p video costs zero video quota. Across the entire life of this project, my honest end-to-end runs have spent 35 video-seconds. Total. Ever.

A real deployment. The backend runs on an Alibaba Cloud Simple Application Server in us-west-1 — nginx serving the SPA on :80, proxying /api to a uvicorn app, bind-mounted volume so cache and run-state survive restarts. A push to main triggers a GitHub Actions job that SSHes into the box, rebuilds, and gates on the health check.

A catalog deliberately kept off the critical path. Finished runs publish into a Postgres 18 sidecar with media in a private Alibaba OSS bucket, addressed by content hash.

It's additive and flag-gated (CATALOG_ENABLED, default off). Live runs stay on the in-memory store. A run gets mirrored only once it has finished. Publishing tolerates a dead database or unreachable bucket and never raises.

Why? Because a storage outage must not fail a run that already passed its gates. The conformance verdict is the product. Archiving it has a lower right to fail.

One thing bit me immediately, and it's a direct consequence of my own determinism: media paths are many-to-one against content.

Deterministic runs produce byte-identical episodes. So ~40 project paths collapsed onto 13 objects.

That's the cache working as designed. It also means the path→hash mapping needs its own media_paths table — not a column on the object, where it would keep only the last path written and 404 every other one.

The deploy had its own small war stories. A plan upgrade that grew the cloud disk but not the partition inside it — grow the partition first, then the filesystem, because a filesystem can't stretch past its partition. And the discovery that this box is Alibaba Cloud Linux 3, RHEL-family, so it's dnf, not apt.

The kind of thing you only learn by actually putting compute on the cloud, instead of calling its APIs from your laptop.

Why this isn't just a demo

The engine has no coupling to the generator, because its checks take frames, not generator internals.

So it lifts out. It's exposed as an MCP server — the same primitive coding agents already speak — runnable as a package:

uvx --from '.[mcp]' dailies-mcp

ListTools returns two tools, and the pair is the point:

run_shot_tests — zero-token, deterministic Tier-A CV, runs on any mp4. The model-agnostic claim, made executable. It reports.

patch_clip — localizes the first blocking failure in time, anchors a Wan i2v regeneration just before it, re-verifies, and reports patched only if the retake passes. It acts, and spends one generation to do it.

Watching a Qwen agent drive this loop was genuinely satisfying. It calls the free tool. Gets back a real FAIL — camera static, |v|=0.00. Explains it in plain English. Then offers the repair — naming it correctly as keyframe-to-video generation, which it only knows because ListTools handed it patch_clip's description.

It never fires it, because its system message forbids the tool that spends quota.

A gate an agent can read is a gate an agent can act on.

Both ends of that loop are mine. Dailies is an MCP producer, and the client consuming it is a Qwen-Agent Assistant with my own server in its mcpServers block.

That same engine is reachable three ways — a native function-calling tool, a Qwen-Agent custom skill via @register_tool, and the MCP server — over one shared core that imports neither openai nor qwen_agent.

Let me be honest about the edges, because the whole ethos here is not claiming what the engine can't do.

Assertions are per-shot and evaluated whole-clip. There's no audio transcript check, no OCR, no time-windowing (a "logo in the first 3 seconds" check), no episode-level concept yet.

When a rule needs a modality the vocabulary lacks, the compiler omits it rather than approximating it with a type that means something else. Known gap: that omission is currently silent.

Those are roadmap, not shipped features.

What Qwen Cloud actually gave me

The integration splits along a line the platform draws for you, and picking the right side per capability is most of what "sophisticated use" means here.

The OpenAI-compatible endpoint (dashscope-intl.aliyuncs.com/compatible-mode/v1) fronts everything text- and vision-shaped. qwen-plus for scripting, repair, and both function-calling loops. qwen-vl-plus for Tier-0 and Tier-B verdicts.

The native DashScope task API handles the media models, because generation is long-running and the OpenAI-compatible shape has no vocabulary for it. wan2.1-t2v-turbo drafts, wan2.2-i2v-flash frame-anchored finals and repairs, wan2.1-t2i-plus Tier-0 stills, qwen3-tts-flash narration.

Video generation is mandatorily asynchronous — create with X-DashScope-Async: enable, then poll. And it carries a gotcha I'm very glad I found on day one, at zero cost:

HTTP 200 on create does not mean the request was valid.

Validation surfaces asynchronously, on the first poll. So retries must branch on the polled task_status, never the POST's status code. (The image endpoint, confusingly, validates synchronously and returns a 400.)

One more finding I'd reuse anywhere: the video endpoints accept a base64 data: URI in place of an image URL. That's what my entire repair feature depends on, because a frame extracted on a 127.0.0.1 box has no public URL and now needs no upload step.

And notice what's missing from that model list. The deterministic tier.

Tier-A is OpenCV. It calls nothing. The Qwen surface gets spent on generation and judgment — never on measurement.

Which is the whole reason measurement can run on every take.

Key Takeaways:

1. Test the artifact, not the prompt. Orchestrating models is only half the problem, and it's the easy half. The hard half — the one almost nobody demos — is deciding when not to call the expensive model, and being able to prove with a number that the thing you didn't render was one you were right to reject.

2. A mock can only confirm your own mental model. It returns what you told it to, so a suite of mocks re-asserts your assumptions back at you in green. The real contract lives at the boundary with the external service. If you're integrating any generative API, spend the tokens on one real end-to-end run early. Cheapest bug-finder you'll ever build.

3. Let the model choose, never let it spell. Don't validate the model's output — make invalid output unrepresentable. Ask what the model actually needs to decide, which is almost always far less than what you're asking it to emit. A closed vocabulary, a parameter contract, a probed roster: three shapes a bad value can't fit through.

4. Generalize by measurement, not by resemblance. My best idea worked beautifully in the case I built it for, so I extended it everywhere by analogy. Two real generations proved half of it wrong. The failure mode isn't that the primitive is bad — it's that you've silently changed what you're asking it to preserve.

5. Watch for the bugs that present as success. A fallback returning the success value. A missing result that reads as a passing one. Two code paths deriving the same value independently. A wrapper whose subject moved. None of these fail loudly, and green tests love them.

So, here's the recap:

· AI video fails in motion, after you've already paid for it
· The fix isn't better prompts — it's a spec you can assert against
· Deterministic checks first, model judgment on top, never underneath
· Verification doesn't just reject bad output; sometimes it deletes spend
· Every bug worth finding looked like success first

Dailies is open source (MIT), built for Track 2 of the Global AI Hackathon Series with Qwen Cloud, and deployed on Alibaba Cloud.

Go test your generated video like you test your code.

Happy shipping!