The Renderer Dropped What the Script Ordered. Every Check Watched the Script.

Originally published on hexisteme notes.

I run a renderer that turns geographic data into short map-animation episodes — country polygons filled in over a basemap, driven by a script that lists, per scene, which countries to focus on, which are neighbors, and which get highlighted. On a single day, it produced two defects, and both had the exact same shape: the renderer silently dropped part of what the script had ordered it to draw. Both had passed everything I had running at the time.

Two ways to drop what was ordered

The interior ring that got dropped

The code that draws a country's polygon was only using the outer boundary:

exterior = polygon[0]  # simplification: ignore interior holes

South Africa's polygon carries a real interior ring at Lesotho's location — an actual hole, because Lesotho sits entirely inside South African territory. Drop that ring and South Africa paints as one solid shape. Countries draw in alphabetical order, so Lesotho — drawn first — got painted, and then South Africa, drawn afterward with no hole cut into it, covered it over. The episode this happened in is titled "A COUNTRY INSIDE A COUNTRY." The subject of the episode had disappeared from its own frame.

The countries that only glowed

The code that decides which countries get a filled polygon at all was built from two of the scene's three country lists:

target_codes = set([*scene.focus, *scene.neighbors])   # no highlight

The glow effect around a highlighted country is generated from the separate highlight field, so a country that's only in highlight — not in focus or neighbors — never enters target_codes. It gets the glow. It gets no polygon. One episode's label was "Three, side by side"; only one of the three countries actually had a filled shape, the other two were a blur of red light around nothing. In a different episode, the same gap meant all five highlighted countries came out that way.

Three checks were green. None of them looked at the picture.

At the time, this renderer had three separate verification passes, and all three were green on both episodes.

Check	What it actually asserts
Schema validation	The input is well-formed — country codes exist, fields are populated
Compliance gate	Policy was followed — sourcing, AI disclosure, label conventions
501 unit tests	The renderer's internal behavior matches expectations — patch counts, the color at a given coordinate, label-placement rules

Not one of the three asks what the rendered artifact is actually saying. All three either look at the input or at an intermediate structure the renderer builds on the way to a frame. So "the label claims three countries and the picture shows one" got produced, cleanly, underneath three green signals — because none of the three signals was ever pointed at the thing that was wrong.

The structural reason input checks can't see this

This is worth stating plainly, because it isn't a case of the checks being lazy or thin — it's structural. In both bugs, the input was exactly correct. The script ordered the right countries; scene.focus, scene.neighbors, and scene.highlight all had the right values; the country codes existed; the fields validated. What failed was delivery: code that ran precisely as written and, by design, dropped something on the way to the frame. exterior = polygon[0] is not a defect a schema can see, because the schema was never told a polygon might carry more than one ring. A discard bug like this is invisible to an input assertion by construction — there is no wrong input to catch, because the input was never wrong.

I've written before that you should verify the surface the consumer actually sees instead of the payload you built — that's the general principle, and this is a sharp case of exactly why the substitution fails: the input check wasn't weaker evidence, it was evidence about a different fact. A separate note covers a renderer measured against the wrong source geometry entirely, upstream of rendering — a different failure from a mid-render discard, where the source was right and the render dropped part of it anyway. And another one is about an agent's inability to perceive its own render at all; this isn't a perception problem, it's a check aimed at the wrong target from the start.

Looking at the frame doesn't fix it either

I opened the frame and saw a black shape sitting inside South Africa's outline — easy to look at and conclude "there's the hole" and move on.

Back-projecting those pixels through the camera transform put them at longitude 30.8–32.2, latitude −27.4 to −25.6: Eswatini, not Lesotho. Eswatini borders Mozambique, so it was never an enclosed country to begin with — whatever put a dark shape there was unrelated to the ring-dropping bug. Looking at the frame produced a confident answer. It just wasn't the right one.

The first attempt to check this programmatically was also wrong. matplotlib's Path.contains_point returns True for a point inside an interior ring, same as for a point inside the fill — true as of matplotlib 3.11, and make_compound_path doesn't change that. Ask the geometry object whether a point is "inside the polygon," and a hole doesn't register as an exception to that; it's still inside the outer boundary.

What actually answered the question was connectivity, not color and not the eye: flood-fill from the frame's outer border, then count the pixels that fill can never reach.

Before the fix: 0 enclosed pixels
After the fix: 3,900 px, x644-731 / y878-955 — matches the back-projected location of Lesotho

The rule

"Look at the output" is not a verification method. "Ask the output the same question the claim is making" is.

When an artifact is something a person has to interpret — an image, a PDF, a chart, a rendered document, an audio track — a test that inspects intermediate structure cannot, by construction, catch that the artifact stopped saying what it was supposed to say. Those tests are all answering "did the pipeline run the way I coded it." The question that actually matters is "does the artifact say what the claim says," and the two questions do not overlap. A pipeline can run exactly as coded and still discard part of the order, because "ran as coded" was never a promise that nothing gets dropped.

Three rules came out of fixing this:

1. Write one question per claim, and take the question from the claim, not the code. "Is Lesotho a hole?" becomes: flood-fill from the frame border, count the enclosed components. "Are three countries highlighted?" becomes: count the connected components in the highlight color. The question has to come from what the episode is claiming, because the code will happily tell you it's doing exactly what it's doing.
2. Answer the question in the same medium as the artifact. An image's claim gets asked of pixels. A sound's claim gets asked of the waveform. Ask a data structure instead and you get the data structure's answer — contains_point will tell you a hole is "inside," because that's a true fact about the geometry object and a false one about the picture.
3. Don't use what you saw with your eyes as evidence. A human eye is reliable up to "something is there." It is not reliable for "that is what it is" — mine confirmed a dark shape and had nothing to say about whether it was the hole the bug produced or something else entirely. It only becomes evidence once you can back-project it to coordinates or count it into a named, located component.

A side effect: the question is also a blast-radius scanner

Once a claim becomes a precise, mechanical question, it can be run against everything you've already shipped, not just the thing in front of you. The second defect's condition writes exactly as highlight ⊄ focus ∪ neighbors — a highlighted country the focus/neighbor set doesn't already cover. Running that condition against the ten episodes already published took about a minute and confirmed none of them triggered it — the condition happened not to be satisfied anywhere in what was already out the door, so nothing already published needed a fix. Turning a claim into a question about the artifact doesn't just catch the next instance of the bug. It tells you, mechanically, how far the last one reached.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.