I built a gate that checks that the five agent readiness categories are declared the same way on six surfaces. It went green and it stayed green. 79 static checks with no failures, 212 checks against the live site with no failures, and 53 mutation cases across three runners that all reported no holes. Then I had a hostile reader go through the gate itself, and it came back with three holes. All three were one defect in three shapes.
The two posts before this one came at the same thing from other sides. The first was about a check that keeps passing after it has stopped measuring the right thing. The second was about a red reading that turned out to be about my own client rather than the server. This one is neither. The gate measured what it measured correctly, and it was blind by construction to everything it did not measure.
The gate asked the wrong question
On every surface the gate asked whether these five were present. It never asked what else was there. That is the whole finding, and it is worth putting plainly. A check that asks whether these five are present is not a check of the set. It is a check of a list you wrote yourself, and it is blind to every element you did not know to write down.
A sixth category went through three of the four static surfaces untouched. I measured each one by running it rather than by reading the code, because reading my own code is how I ended up here. Add a sixth category to the surface, run the gate, read the output. RESULT: OK, zero failures, sixth category sitting on the page.
Three shapes of one defect
The first was a filter that ran before the count. The table rows in the README are collected, filtered down to the ones that resolve to a known category, and only then counted. A row whose name the fact file does not recognise disappeared in the filter, so the counter never went up. The filter is there to keep unrelated rows out of the count, and it kept the finding out with them.
The second was a span that ends on the last element's own sentence. The gate lifts a paragraph out of the audit guide by anchoring on the first category and the last one, and the closing anchor is commerce's own sentence. That is exactly where a sixth gets appended. Anything added after it falls outside the span the gate is reading.
The third was a substring test standing in for an equality test. One surface carries a twin sentence built entirely from the fact file, and the gate compared the page against it with includes. A sixth appended to the end of the list was caught. A sixth inserted at the front was not, because the string the gate wanted was still in there. It just was not the whole of it any more.
The fourth one was in the tool doing the checking
There was a fourth, and it sat in the thing I was using to prove the other three. One of the mutation runners reported no holes across ten cases. Seven of those ten had come back with applied=0 and HTTP 429, which means the rate limit refused the request and the mutation was never in place at all. The runner counted a case that never ran as a case with nothing wrong. It read a red run as a pass.
That is the same defect one level up, and it is the one I would rather report than the three below it. The runner asked whether the failures it knew to look for had appeared, and it never asked whether the run had happened. A tool that certifies the other tools is where this class of mistake does the most damage, and a green summary sitting on top of seven refused requests is exactly what it looks like from the outside.
The same blindness turned up in a different gate a few hours later. Cases aimed at an agent skills file were matching any agent skills line instead of the one they named, so a digest check stood in for a content check and one real hole went through. The mutation had put a wrong business identifier in the declared field. The check passed because the correct identifier still appeared elsewhere in the same file, inside a registry link, and the check was searching the file rather than reading the field.
What a set check has to do instead
A check that watches a set must not search. Searching answers a question about one string at a time, and a set is not a bag of strings you happened to remember. The check has to enumerate what the surface actually contains, then compare that enumeration against the set. Three conditions, all of them at once.
- The enumeration and the set are the same length.
- Every element found on the surface resolves to a member of the set.
- No member of the set is resolved twice.
The third condition earns its place. Without it a surface can carry the right number of rows with one member listed twice and another missing entirely, and the first two conditions are still satisfied.
Then the part my original code had backwards. Every element that does not resolve is a finding. In the gate I wrote, an unresolvable row was a row to skip, and it was a row to skip because the code had been written to answer whether the known five were present. The question shapes the data structure, and the data structure then makes the other question impossible to ask.
The version that transfers
None of this is new and none of it is rare. Anyone who has written against a schema has met it under the name additionalProperties, and the repair is the same repair. The reportable part is not the class of defect. It is that three instances of it were sitting in my own code, in a gate I had negative tested, on a day it reported green three separate ways.
The version that transfers to anyone with a test suite: your test asserts that the fields you remembered are present in the response. It does not ask what else is in the response. Add a field to the payload and the test stays green, which is correct behaviour for a compatibility check and wrong for anything you believe is watching a set. If a test's name says it validates a schema or a contract, open it and check whether it enumerates or whether it searches.
About the hostile read
It was a tool of mine in a session of mine. It is not an audit and I am not going to call it one. It produced 17 observations, 13 of them carrying evidence and 4 flagged as suspicion, and I verified the four most serious by running them myself before I believed any of them. A second reader's finding is a lead. It becomes a measurement when you run it, and until you run it you have only swapped one reader's confidence for another's.
What these gates still do not read
The gates watch what they watch. The category set is now compared across six surfaces where it used to be compared across two. Two things they do not touch at all: the markdown twins on pages other than the homepage, and the values inside the well-known manifests. Nothing here says the site cannot drift. It says that one specific way of drifting now costs a red run, and that I know the names of the next two.
Top comments (0)