Our release gate told us to ship a build it had never successfully looked at. Every one of the six checks had crashed. The report underneath the verdict said, in bold, Decision: BLOCK. The field the API actually returns said ship, and the one-line summary said "Cleared, all checks passed."
Nothing was broken in the way you would expect. The model reasoned correctly. It wrote out its reasoning, and the reasoning was right. The failure happened in the two inches between the model's prose and the enum, and it failed in the direction that lets bad code through.
We found that bug because we spent a day pointing our own agents at two dozen real marketing sites belonging to companies we wanted to talk to. The idea was to find something genuinely useful on each one and send it to them. What we got instead was a very fast education in the ways an automated QA tool can be confidently, articulately wrong, and all of them turned out to be the same mistake wearing different clothes.
Every false positive in this post is this diagram.
A green check on a build that never ran
Start with the worst one, because it sets up everything else. Two of the sites we pointed the runner at were behind infrastructure that our browser could not get through. All six agents errored: Audit returned 500, Probe returned 500, Page returned 0, Could not capture rendered page snapshot.
The judge that produces the final call saw all of this. Its narrative said the release could not proceed. Its report rendered a table of six failed agents and the heading Decision: BLOCK. Its reasoning field opened with the words "The decision is block based on hard guardrail (b)". There was no error anywhere in the pipeline: judgeError was null.
And the structured field that every integration is told to gate on read ship.
The extraction step that turns the model's answer into an enum did not find the decision it was looking for, and fell back to a default. The default was the permissive value. In pull request mode that is a passing status check on a build nobody checked, sitting above a report that says, if you scroll, that it should have been blocked. Almost nobody scrolls.
Absence is not evidence of absence
The second shape is quieter and, for us, more dangerous, because it produces a finding rather than a non-finding.
On one company's homepage our SEO agent reported that there was no canonical tag. We were about to send that to their engineering lead as a helpful observation. The page has a canonical tag. It is right there in the served HTML: <link href="https://example.com/" rel="canonical">.
What happened is that the page capture had partially failed. Two other agents in the same run reported Page returned 0. The SEO agent got a fragment of a document, looked for a canonical tag inside it, did not find one, and reported absence as a finding. The internal verifier then marked it confirmed, because it was checking that the agent had said the thing, not that the thing was true.
This is the class of bug that ends a sales conversation before it starts. Telling a stranger their site is missing something that is visibly present demonstrates, in one message, exactly the failure mode you are asking them to pay you to prevent.
Every absence-based check has this property. Missing alt text, missing lang attribute, missing meta description, missing heading, missing tag of any kind. All of them are only meaningful if you are certain you saw the whole document, and almost none of them check.
Blocked is not broken
Our link checker reported a company's team page as containing a broken link, because a LinkedIn profile URL returned HTTP 999.
999 is not a real status code. It is what LinkedIn returns to any client it does not believe is a browser. The control takes about four seconds to run: fetch the LinkedIn profile of someone unambiguously famous and still employed, and watch it also return 999. The profile is fine. The link is fine. The checker was blocked, and reported being blocked as evidence of breakage.
We had already fixed this exact bug for HTTP 403 a day earlier, after a different site's trust page got flagged as broken when it was merely behind bot protection. The fix worked and the reasoning was sound. It just enumerated status codes instead of naming the concept, so the moment a different code with the same meaning appeared, the bug came back with a new number.
The general rule turns out to be short: a status code that is not in the IANA registry cannot support a conclusion about whether a page exists. Neither can 403, 429, or anything else that means "I am refusing to talk to you specifically."
Whose failure was that, exactly?
The fourth shape is the most embarrassing, because the tool blames its target for its own problems.
On one site, our console and network agent reported twelve console errors and fourteen failed requests. Most of them were net::ERR_CONNECTION_RESET against the site's CDN, and the specific files were the core framework bundles: the React bundle, the framework runtime, the animation library.
Those files are fine. Fetched from an ordinary machine they return 200, at 170KB and 506KB respectively. The tell was available in the report itself and we nearly missed it: if a page's core React bundle had genuinely failed to load, the page would not have rendered, and the other agents would not have been able to describe its contents. The connection resets happened on our side. We reported them as fourteen defects in someone else's website, and our verifier confirmed every one.
The same shape shows up whenever the measuring environment is unusual, which it always is. Headless browsers get served different content. Datacenter IPs get rate limited. Analytics scripts refuse to run for suspicious user agents, and then log a warning that looks a lot like a bug in the page.
The third state
Four failures, one cause. In each case the system had two labels available, pass and fail, and the truth was neither. It had not measured. Forced to pick, it picked, and the choice was arbitrary in a way the output did not reveal.
The fix is not smarter agents. It is a third outcome that sits alongside pass and fail and means something specific:
- pass means the check ran to completion and found nothing wrong.
- fail means the check ran to completion and found something wrong.
- undetermined means the check did not run to completion, and the tool is declining to guess.
Our link checker already had this concept, which is what makes the 999 bug so instructive. In the very same run that misreported the LinkedIn profile, it correctly emitted was not checked because the origin rate-limited the check (429). This is not a broken link, it is a link we could not adjudicate. The vocabulary existed. One status code simply was not routed into it.
That is the usual state of affairs. Teams that build this kind of tooling almost always discover the third state eventually, in one subsystem, and almost never propagate it to the others.
Why the default matters more than the detection
Here is the part worth arguing about, because it is where we got it wrong in the most costly way.
You will not catch every unmeasurable case. New ones appear constantly, because the web is adversarial to automated clients and getting more so. So the question is not only how you detect a failure to measure. It is what your code does when detection fails.
In our verdict bug, the parser hit an input it could not classify and chose a default. That is reasonable. The problem is that the default was ship. Somebody, at some point, wrote a fallback and reached for the value that would not annoy anyone, and that decision sat quietly in the codebase until the day it approved a build that had never been examined.
The performance agent did the same thing in a different costume. On a page that returned 403 to our runner, and where three sibling agents correctly reported the 403, it produced Performance 100/100. It had no measurements at all. Every metric was zero, and a composite built from zeros came out perfect. 100/100 is the most reassuring number the product can print, and it printed it for a page it had never seen.
The rule we now hold to: when you cannot classify, degrade toward the answer that makes a human look. Undetermined, not pass. An inconclusive build is annoying. A green build that was never tested is how a defect reaches production with documentation saying it was checked.
What we fixed, and what is still wrong
Being specific about this is the only reason a post like this is worth reading, so: our own defect backlog stands at 37 filed, most of them from pointing the tool at real sites over the past few weeks rather than at our own fixtures. 34 are fixed. Three are open as of today, all high severity, all customer-facing, and all three are the blocked-is-not-broken and whose-failure-is-it shapes described above.
The verdict bug is fixed, and the fix has two halves, because one would not have been enough. The extraction is stricter, and the fallback now degrades to inconclusive rather than to ship. The summary line is derived from the agent results rather than from the decision, so "all checks passed" can no longer be printed over a table of six failures.
The underlying capture bug is fixed too, and it was mundane in the way these things usually are. A screenshot call inside the navigation helper had a 30 second Playwright timeout. On a page that animates continuously, the screenshot never settles, the timeout raises, and because the screenshot lived inside navigate() rather than beside it, the exception took out the navigation and therefore all six agents at once. One incidental screenshot, six checks, no results. The fix was to stop letting a picture fail a page load.
What is still open is a fair reflection of how hard the general problem is. We can enumerate the bot-protection status codes we have seen. We cannot enumerate the ones we have not.
A checklist you can steal
If you build, buy, or depend on anything that automatically checks a running application, these are worth asking. They took us a day of real use to learn and about an hour to write down.
- Can your tool say undetermined? If the only outcomes are pass and fail, every unmeasurable case is already being silently mapped to one of them.
- When a check cannot classify its own result, what is the fallback value? Read the code, not the docs. If it is the permissive one, you have a fail-open gate.
- Do absence-based checks verify that the document fully loaded before concluding something is missing? A partial page makes everything look absent.
- Does the tool distinguish "this origin refused me" from "this resource does not exist"? Test it with a live URL behind bot protection and see what it says.
- Does it distinguish its own transport failures from the target's? A CDN that resets your connection is not a bug in the site you are measuring.
- Can a composite score be produced from zero measurements? Point it at a URL that returns 403 and see whether you get a number or an error.
- When your verifier confirms a finding, is it confirming the finding is true, or only that the agent said it? These are very different checks, and the second one is much easier to build by accident.
We build ShipGuarde, which is where all of these came from. Writing this up was uncomfortable for the obvious reason, and worth it for a less obvious one: the day we found five ways our own tool could be wrong was the day it started being useful. A checker that knows the difference between a clean result and a failed measurement is a different product from one that does not, and you cannot tell them apart from the outside until something ships that should not have.
Top comments (0)