A plan-gate check on the main video-generation service exists to catch a bad plan before it becomes a paid render. Over several days, a cluster of bugs kept showing up in the gap between "the plan" and "the thing checking the plan" — not because the gate was weak, but because the planner and the gate were, in several distinct ways, reading two different specifications of the same ad. A grading system is only as good as whether the thing being graded was ever told what it's being graded on, and this cluster is a tour of every way that assumption can quietly fail.
A gate that flags a defect it cannot repair, and a repair that fixes a defect it was never told about, agree on nothing except the plan that slipped through both of them.
Detection and repair, reading two different vocabularies
The first bug in this chain is also the cleanest statement of the whole pattern. scanPlanConsistency flagged "the shot says there is no person in it, but its role puts one there" as a plan error. The repair function meant to fix exactly that defect had its own, narrower pattern for recognizing a personless shot — so the gate reported a contradiction the repair code literally could not see, and the contradiction reached the clip vendor anyway. Measured directly against real scenes: of five ways a script can say "no person here," three were misclassified as person shots, including one where the negation trap was almost poetic — the phrase "no face visible" contains the word "face," and "face" was one of the terms the code used to detect a person's presence. A negated mention counted as a positive one. The concrete cost was recorded on a real job: a scene explicitly staged as hands-free rendered a man holding pens.
The fix wasn't a smarter pattern. It was making detection and repair share one exported constant instead of two independently-maintained ones, with a test asserting the import rather than a redeclaration — because a second copy of the same vocabulary is, on its own, the regression waiting to happen again.
A prompt that told the planner two contradictory things
The same PR surfaced a second, unrelated contradiction sitting a few lines apart in the planner's own system prompt: an explicit "the presenter never holds, touches, lifts, or shows the product — no exceptions" rule, followed five lines later by a role definition that offered "(or holding the product while talking)" as an option. The planner followed the parenthetical often enough to be measurable — five scenes on one reference described handling the product in the line while the staging correctly showed none, because the renderer strips the product from every presenter shot regardless of what the script says. Script and picture were pulling in opposite directions, and nothing surfaced it until the gate started persisting its findings instead of only flashing them in a terminal.
A role assigned after the prose was already written for a different one
The largest single insight in this cluster is an ordering bug, not a vocabulary bug. A scene's role — presenter, narrated, or b-roll — determines whether the clip renders with native audio, mute with a voiceover muxed in later, or as a pure cutaway. The planner picks a role and writes motion and acceptance criteria consistent with it. Then a separate measurement of the reference video's own dominant shot type overwrites that role afterward — correctly, because a real measurement of the reference should win over a model's guess. The prose and the role then disagree through no fault of either step individually: seven scenes on one job shipped as narrated — meaning the clip renders mute — while their own motion description had the presenter talking straight to the lens. A narrated clip's rendering path appends an instruction that the subject does not speak, lips at rest, "observed by the camera rather than performing to it" — turned into a mouth moving under someone else's voiceover, which is precisely the "dead avatar" complaint a human reviewer had reported independently, with no idea it traced back to a role reassignment happening after the prose was already committed.
The tempting fix was a large one: rename the whole role enum to make the distinction unambiguous, a forty-four-site migration touching every persisted job state. The investigation that actually shipped rejected that plan on its own evidence — the roles were not conflated, they were correctly and unambiguously defined; the bug was purely that the role got decided twice, once by the planner and once by the reference measurement, in the wrong order. The real fix derives the role from the reference before the planner writes a single word of motion, and states it to the planner as already decided — naming the exact phrases that would then be wrong ("talks to lens," "facing camera") and where that content belongs instead. Deliberately kept as two separate functions rather than one shared helper: the pre-plan derivation is advice that may be undefined for an ambiguous window, while the reference measurement remains the authority applied afterward. Collapsing them into one function would make the second call look redundant — an invitation to delete the exact override that keeps a real measurement from losing to a language model's guess.
Criteria describing a beat that no longer exists
A related but distinct bug: the same transform that rewrites a personless shot into a person shot left the scene's acceptance criteria — the list a downstream critic grades the render against — completely untouched, still describing the product-focused, person-free beat that had just been overwritten. One omission produced two separate errors on the same scene, and a clip critic checking the render against those stale criteria would have failed a render for not matching a beat that the pipeline itself had already deleted. The fix drops the stale criteria rather than trying to rewrite them into something new — inventing replacement criteria would mean authoring plan content inside what's supposed to be a pure transform step, a different kind of contamination than the one being fixed.
The last three contradictions, on beats no per-scene check could ever see
After the two fixes above, a measured gate run against the same reference went from nine total plan errors to zero in two of the three defect categories — genuine evidence the fixes were complete rather than merely narrower. Three contradictions remained, and they were structurally different from everything before them: every one occurred on a cutaway window where the reference measurement legitimately had no opinion about who was on camera, because a cutaway by definition doesn't answer that question. The role then got clamped downstream to "narrated" for an equally sound, independent reason — the reference video simply never puts a talking head on camera at all, so no presenter role should ever be assigned regardless of scene. Two individually correct decisions produced a contradiction that neither one could see on its own, because the actual missing fact — does this reference ever show a speaker at all — is a property of the whole reference video, not of any individual scene, and no per-scene check was ever going to be capable of answering it. The fix computes that fact once, for the reference as a whole, before the plan exists at all, and lets every beat inherit it.
A judge whose verdict a re-roll simply ignored
A semantic plan-QA step asks a genuinely useful question — does this scene's staging actually illustrate what its own line is saying — and for a while, the answer went nowhere. The check ran, logged its findings, displayed them on the gate, and handed them back to nobody. A human reviewing a flagged plan could re-roll it, and the re-roll would re-author a fresh plan and then read the old plan's cached verdict, because the cache key for that verdict never moved when the plan's content did. A re-roll meant to fix a mismatch would show the exact mismatch it had just fixed, or fail to show a new one the fresh plan had earned. The fix threads the judge's findings back to the author on a dedicated feedback channel — deliberately not the same channel used for human-requested edits, because that channel carries a separate license to author entirely new spoken lines, and reusing it for an automatic pass would have handed an unattended re-plan a privilege the system explicitly reserves for a human asking for it.
That automatic re-author was itself later removed, in an unusually candid piece of self-audit: measured across five actual fires, it improved exactly one plan out of five, and every fire cost roughly thirty cents for a second authoring call. Three of the five failures were a different, already-fixed bug in disguise; the fourth and fifth simply didn't help. The PR that removed it is explicit that a redo mechanism which mostly reports "no better" shouldn't get a pass just because its author wrote it — the same standard applied to a different feature earlier in the same investigation.
A flag that argued itself out of existing
The last bug in the chain is the funniest and, in its way, the most instructive: a judge model reasons about a scene inside its own message text and sometimes talks itself into "actually this is fine" — while still emitting the structured flag that parks a render. One flagged message read, verbatim, that a scene's timing was "acceptable; no issue," attached to a flag that had already stopped a real render pending human review. The model's own prose had refuted its own verdict, and nothing was reading the prose. The fix narrows the parser to drop retractions specifically — a scene report that says "acceptable, no issue" doesn't count as a flag — while carefully preserving softer hedges like "a minor mismatch" that are genuine findings and need to keep surfacing. Getting that distinction wrong in either direction would have either let broken plans through as false negatives or blinded the gate entirely.
The pattern: ask whether the judge and the author share a spec
Every bug in this chain reduces to the same root question, asked about a different pair of components each time: does the thing writing the content and the thing grading it agree on what's being written and graded? A shared vocabulary constant answers it for detection and repair. Ordering the role-assignment step before the prose-writing step answers it for the planner and the reference measurement. A feedback channel that actually reaches the author's prompt answers it for the judge and the planner. None of these fixes involved a smarter model or a better prompt in the sense of more eloquent instructions — every one of them was closing a gap where two systems were each behaving correctly according to a specification the other one didn't have.
Top comments (0)