Episode 09's visual-prompt file passed every automated check I had. Green across the board. And I was looking straight at it, so I knew the file was broken.
That green build is the most dangerous state a pipeline can be in. A failing test asks for attention. A passing test over a known defect manufactures confidence in the broken thing, and nobody goes looking for bugs in a green build.
This is the story of what it took to make green mean something, on an unusual codebase: a git repository that works as a one-person film studio. The bug is specific to my domain. Every fix is transferable.
I was filming while it happened, and not by accident. I had been catching this class of error by hand for eight episodes, one file at a time, and I was tired of it. When I sat down to record the build-along sessions for Episode 09, the plan was to show the problem and fix it properly in the same run. The sessions are public, so you can watch the problem and the rebuild rather than take my summary of them: the green that shouldn't have been and the rebuild that followed.
The setup, in one paragraph
The repo directs a 10-episode film series. The protagonist is an android whose body moves through three phases across the series (pristine, damaged, gold-mended), and each phase has its own reference image so the generated frames stay consistent shot to shot. A visual-prompt file describes every shot of an episode: a block of prose for the image generator, plus metadata fields naming which reference image to attach. The rule that matters here is that a damaged-body episode must attach the damaged reference. Episode 09 attached the pristine one, scene after scene. A human catches that instantly. My validator reported success.
Two blind spots, compounding
The first one searched the prose for the wrong word. The check looked for the string "robotiko" in the prompt text to decide whether a scene was about the protagonist and needed policing. But prompts never say "robotiko". They say "the chrome android", because that is what an image generator needs to hear. The check never engaged. Zero matches, zero findings, green.
The second one never read the field where the bug lived. The wrong reference was not in the prose at all. It sat in a metadata field the validator never parsed. Right tool, pointed at the wrong place: I was checking the description of the painting while the mislabelled canvas hung untouched beside it.
Generalize both and you get the first transferable rule. Check the data where the rule actually lives. Any invariant that sits in structured fields (references, upload lists, config) is invisible to a prose-level or naming-level check, and it fails silently as a category, not as one bug.
The rule nobody had written down
The deeper problem was that the rule the file broke existed only in my head. "Which body belongs in which episode" was tribal knowledge with a tribe of one.
So it became data: a phase_reference_map in the character-profiles JSON, each phase mapped to its reference image, with episode-level exceptions recorded explicitly. One episode keeps the damaged body even after the story turns. Another transitions mid-episode at a named scene. The reasoning went into an Architecture Decision Record, dated.
Without a written source of truth, every check is just one more person's opinion hard-coded in a script. There is nothing to check against.
Red first, or it doesn't count
The rewritten check read the metadata fields and compared them against the map. First run on EP09 came back red: exact file, exact scenes, the forbidden reference, the fix. That red was the proof that the check worked at all.
Then I fixed the generator, not just the file. EP09's references were corrected, and the skill file that authors prompt documents was patched so the next episode could not re-create the bug. If you only fix the artifact, the same unchanged process produces the same defect on the next run.
Grade the graders
A checker with no proof that it works is a more confident way to be wrong. So the test got tested.
Two fixtures were frozen permanently. The real broken file, captured exactly as it shipped, must fail. Its corrected twin must pass. Meta-tests assert both directions, and they also assert that the broken fixture fails only on the reference check, which pins down the explanation of why it ever shipped green. The fixtures never change, and that is the whole point of them.
This bidirectional fixture discipline became mandatory for every checker in the suite. The meta-test count now stands at 245. If I loosen a regex six months from now and the BROKEN fixture goes green, the meta-test turns red and tells me the grader has stopped grading.
Two rules hold the whole thing up:
- Make the check fail first. A green you have never seen turn red is not evidence. It might be checking nothing.
- Never loosen a check without a both-directions proof: one test that it still catches a real bug, one that it now ignores the intended case.
The scarier green
Sweeping the new checks across all episodes lit up reds in seven of them, EP02 through EP08, and it exposed something worse hiding under a green. EP06 used a slightly different scene-header format. The parser matched zero scenes, so the reference check had been passing over nothing for weeks.
A green that means "I checked everything" and a green that means "I checked nothing" look identical. The suite now asserts that every shipped file parses a non-zero number of scenes. If your test loops over query results, parsed blocks, or discovered files, assert the count is non-zero. An empty iteration passes every test you write against it.
A red isn't always a bug
The sweep's reds needed triage, and "fixing" all of them would have been its own disaster. Three buckets:
- FIX. Real defects. EP04 and EP05 carried the same wrong-reference error. Corrected in the text only; the shipped images were kept, because the films were already released and the record is what had to become honest.
- WHITELIST. Intentional and documented. One episode's dream sequence deliberately surrounds the damaged hero with hundreds of pristine copies. Another pits him against a pristine conformist foil. The forbidden word legitimately describes something that is not the hero. Exceptions were recorded scene-pinned and reasoned, never as a blanket "ignore this word here", so a real slip still fires.
- REFINE. The check over-fired. It flagged "pristine shelves", which is scenery, not the character. It learned to judge the subject instead of the set.
Treat every red as a bug and you corrupt correct files. Treat every red as noise and you silence real errors. The triage is where the work is.
The ladder
None of this stays fixed by itself. The repo names the pattern the lesson graduation ladder, where a correction climbs from a one-time fix toward a guarantee that outlives the person who made it:
- Incident. Someone pays for a mistake once.
- Dated rule. One plain-English line in a lessons file, with the date and the case that produced it.
- Validator check. A function that reads the artifact and fails on the pattern.
- Bidirectional fixture. A frozen BAD input it must fail, a GOOD twin it must pass, meta-tests asserting both.
- ADR. The reasoning, recorded so the next person inherits the trade-off instead of quietly undoing it.
- CI gate. One command runs everything, and a red blocks the merge.
Not every lesson climbs all six rungs. Many stop at rung 2 forever because they are matters of taste that no function can judge. The discipline is knowing which rung a lesson can honestly reach, and not claiming a higher one than it earns.
You can start the ladder on any repo without much ceremony: write the dated rule, write the smallest checker that can fail on it, freeze one BAD fixture. The third step is the one I would have skipped, and it is the one that keeps the checker honest.
What green certifies now, and what it doesn't
The suite's coverage ledger currently tracks 41 invariants: 30 machine-enforced, 4 heuristics that can misfire and say so, 12 human taste-gates with no automation claimed, and 1 acknowledged gap. Some rows span two categories. That last category matters most. A gap you have named is on a list somewhere. A gap you have not named is being covered by a green check that does not cover it.
Three checkpoints in this pipeline are human on purpose: approval after dramaturgy, after reference authoring, and after the motion script. Each one is recorded with a sha256 of the approved artifact, so post-approval drift is visible. No function decides whether a scene serves the story. The point of mechanizing everything else is to leave the humans free to argue about the things that actually need arguing about.
When a green run comes back now, it certifies exactly the 30 machine rows and not one inch more. That is what I wanted green to mean.
The repo (validators, frozen fixtures, meta-tests, the coverage matrix, and the film series they protect) is open source, method MIT: github.com/fibuladev/robotiko-v2. The ladder is documented in docs/method-lesson-graduation.md, and the full case study in _management/case_study_validation_backbone.md.
Claude drafted this article. I directed it, checked every claim against the repo, and edited it.
Top comments (0)