DEV Community: Shubham Shrivastav

Pass, Fail, and the Third State Your QA Tool Is Missing

Shubham Shrivastav — Wed, 19 Aug 2026 11:03:16 +0000

Our release gate told us to ship a build it had never successfully looked at. Every one of the six checks had crashed. The report underneath the verdict said, in bold, Decision: BLOCK. The field the API actually returns said ship, and the one-line summary said "Cleared, all checks passed."

Nothing was broken in the way you would expect. The model reasoned correctly. It wrote out its reasoning, and the reasoning was right. The failure happened in the two inches between the model's prose and the enum, and it failed in the direction that lets bad code through.

We found that bug because we spent a day pointing our own agents at two dozen real marketing sites belonging to companies we wanted to talk to. The idea was to find something genuinely useful on each one and send it to them. What we got instead was a very fast education in the ways an automated QA tool can be confidently, articulately wrong, and all of them turned out to be the same mistake wearing different clothes.

Every false positive in this post is this diagram.

A green check on a build that never ran

Start with the worst one, because it sets up everything else. Two of the sites we pointed the runner at were behind infrastructure that our browser could not get through. All six agents errored: Audit returned 500, Probe returned 500, Page returned 0, Could not capture rendered page snapshot.

The judge that produces the final call saw all of this. Its narrative said the release could not proceed. Its report rendered a table of six failed agents and the heading Decision: BLOCK. Its reasoning field opened with the words "The decision is block based on hard guardrail (b)". There was no error anywhere in the pipeline: judgeError was null.

And the structured field that every integration is told to gate on read ship.

The extraction step that turns the model's answer into an enum did not find the decision it was looking for, and fell back to a default. The default was the permissive value. In pull request mode that is a passing status check on a build nobody checked, sitting above a report that says, if you scroll, that it should have been blocked. Almost nobody scrolls.

Absence is not evidence of absence

The second shape is quieter and, for us, more dangerous, because it produces a finding rather than a non-finding.

On one company's homepage our SEO agent reported that there was no canonical tag. We were about to send that to their engineering lead as a helpful observation. The page has a canonical tag. It is right there in the served HTML: <link href="https://example.com/" rel="canonical">.

What happened is that the page capture had partially failed. Two other agents in the same run reported Page returned 0. The SEO agent got a fragment of a document, looked for a canonical tag inside it, did not find one, and reported absence as a finding. The internal verifier then marked it confirmed, because it was checking that the agent had said the thing, not that the thing was true.

This is the class of bug that ends a sales conversation before it starts. Telling a stranger their site is missing something that is visibly present demonstrates, in one message, exactly the failure mode you are asking them to pay you to prevent.

Every absence-based check has this property. Missing alt text, missing lang attribute, missing meta description, missing heading, missing tag of any kind. All of them are only meaningful if you are certain you saw the whole document, and almost none of them check.

Blocked is not broken

Our link checker reported a company's team page as containing a broken link, because a LinkedIn profile URL returned HTTP 999.

999 is not a real status code. It is what LinkedIn returns to any client it does not believe is a browser. The control takes about four seconds to run: fetch the LinkedIn profile of someone unambiguously famous and still employed, and watch it also return 999. The profile is fine. The link is fine. The checker was blocked, and reported being blocked as evidence of breakage.

We had already fixed this exact bug for HTTP 403 a day earlier, after a different site's trust page got flagged as broken when it was merely behind bot protection. The fix worked and the reasoning was sound. It just enumerated status codes instead of naming the concept, so the moment a different code with the same meaning appeared, the bug came back with a new number.

The general rule turns out to be short: a status code that is not in the IANA registry cannot support a conclusion about whether a page exists. Neither can 403, 429, or anything else that means "I am refusing to talk to you specifically."

Whose failure was that, exactly?

The fourth shape is the most embarrassing, because the tool blames its target for its own problems.

On one site, our console and network agent reported twelve console errors and fourteen failed requests. Most of them were net::ERR_CONNECTION_RESET against the site's CDN, and the specific files were the core framework bundles: the React bundle, the framework runtime, the animation library.

Those files are fine. Fetched from an ordinary machine they return 200, at 170KB and 506KB respectively. The tell was available in the report itself and we nearly missed it: if a page's core React bundle had genuinely failed to load, the page would not have rendered, and the other agents would not have been able to describe its contents. The connection resets happened on our side. We reported them as fourteen defects in someone else's website, and our verifier confirmed every one.

The same shape shows up whenever the measuring environment is unusual, which it always is. Headless browsers get served different content. Datacenter IPs get rate limited. Analytics scripts refuse to run for suspicious user agents, and then log a warning that looks a lot like a bug in the page.

The third state

Four failures, one cause. In each case the system had two labels available, pass and fail, and the truth was neither. It had not measured. Forced to pick, it picked, and the choice was arbitrary in a way the output did not reveal.

The fix is not smarter agents. It is a third outcome that sits alongside pass and fail and means something specific:

pass means the check ran to completion and found nothing wrong.
fail means the check ran to completion and found something wrong.
undetermined means the check did not run to completion, and the tool is declining to guess.

Our link checker already had this concept, which is what makes the 999 bug so instructive. In the very same run that misreported the LinkedIn profile, it correctly emitted was not checked because the origin rate-limited the check (429). This is not a broken link, it is a link we could not adjudicate. The vocabulary existed. One status code simply was not routed into it.

That is the usual state of affairs. Teams that build this kind of tooling almost always discover the third state eventually, in one subsystem, and almost never propagate it to the others.

Why the default matters more than the detection

Here is the part worth arguing about, because it is where we got it wrong in the most costly way.

You will not catch every unmeasurable case. New ones appear constantly, because the web is adversarial to automated clients and getting more so. So the question is not only how you detect a failure to measure. It is what your code does when detection fails.

In our verdict bug, the parser hit an input it could not classify and chose a default. That is reasonable. The problem is that the default was ship. Somebody, at some point, wrote a fallback and reached for the value that would not annoy anyone, and that decision sat quietly in the codebase until the day it approved a build that had never been examined.

The performance agent did the same thing in a different costume. On a page that returned 403 to our runner, and where three sibling agents correctly reported the 403, it produced Performance 100/100. It had no measurements at all. Every metric was zero, and a composite built from zeros came out perfect. 100/100 is the most reassuring number the product can print, and it printed it for a page it had never seen.

The rule we now hold to: when you cannot classify, degrade toward the answer that makes a human look. Undetermined, not pass. An inconclusive build is annoying. A green build that was never tested is how a defect reaches production with documentation saying it was checked.

What we fixed, and what is still wrong

Being specific about this is the only reason a post like this is worth reading, so: our own defect backlog stands at 37 filed, most of them from pointing the tool at real sites over the past few weeks rather than at our own fixtures. 34 are fixed. Three are open as of today, all high severity, all customer-facing, and all three are the blocked-is-not-broken and whose-failure-is-it shapes described above.

The verdict bug is fixed, and the fix has two halves, because one would not have been enough. The extraction is stricter, and the fallback now degrades to inconclusive rather than to ship. The summary line is derived from the agent results rather than from the decision, so "all checks passed" can no longer be printed over a table of six failures.

The underlying capture bug is fixed too, and it was mundane in the way these things usually are. A screenshot call inside the navigation helper had a 30 second Playwright timeout. On a page that animates continuously, the screenshot never settles, the timeout raises, and because the screenshot lived inside navigate() rather than beside it, the exception took out the navigation and therefore all six agents at once. One incidental screenshot, six checks, no results. The fix was to stop letting a picture fail a page load.

What is still open is a fair reflection of how hard the general problem is. We can enumerate the bot-protection status codes we have seen. We cannot enumerate the ones we have not.

A checklist you can steal

If you build, buy, or depend on anything that automatically checks a running application, these are worth asking. They took us a day of real use to learn and about an hour to write down.

Can your tool say undetermined? If the only outcomes are pass and fail, every unmeasurable case is already being silently mapped to one of them.
When a check cannot classify its own result, what is the fallback value? Read the code, not the docs. If it is the permissive one, you have a fail-open gate.
Do absence-based checks verify that the document fully loaded before concluding something is missing? A partial page makes everything look absent.
Does the tool distinguish "this origin refused me" from "this resource does not exist"? Test it with a live URL behind bot protection and see what it says.
Does it distinguish its own transport failures from the target's? A CDN that resets your connection is not a bug in the site you are measuring.
Can a composite score be produced from zero measurements? Point it at a URL that returns 403 and see whether you get a number or an error.
When your verifier confirms a finding, is it confirming the finding is true, or only that the agent said it? These are very different checks, and the second one is much easier to build by accident.

We build ShipGuarde, which is where all of these came from. Writing this up was uncomfortable for the obvious reason, and worth it for a less obvious one: the day we found five ways our own tool could be wrong was the day it started being useful. A checker that knows the difference between a clean result and a failed measurement is a different product from one that does not, and you cannot tell them apart from the outside until something ships that should not have.

Running Untrusted Code Safely: A Field Guide for AI and CI Pipelines

Shubham Shrivastav — Fri, 10 Jul 2026 04:42:53 +0000

Every bot that reads your pull requests is executing code it did not write. Here is how to make sure that is not the end of the story.

There is a quiet assumption baked into most developer tooling: that the code we run is code we trust. Your CI runs your tests. Your linter loads your config. Your build installs your dependencies. For years that held, because the person triggering the job and the person who wrote the code were the same person.

That assumption is now false, and a lot of teams have not noticed.

The moment you add an AI reviewer, a CI job that runs on pull requests from forks, a dependency bot, or any automation that touches contributor code, you have invited untrusted code into a trusted environment. A stranger can now open a pull request and, if you are not careful, get your infrastructure to run something on their behalf. The unsettling part is not the code in the diff. It is everything that runs around it.

This is a field guide to doing that safely. It is written for anyone building or operating a tool that reads other people's repositories, but the same principles apply to any CI pipeline that accepts pull requests from people you do not know.

The threat is bigger than the diff

When people picture untrusted code, they imagine a malicious function hidden in a pull request. That is real, but it is the least of your problems, because you are probably not executing the diff line by line. The real surface is everything your pipeline does automatically on the way to looking at that diff.

Walk through what a normal review or CI job touches:

Package installation. Running npm install, pip install, or the equivalent executes lifecycle scripts. A single postinstall hook, or one hostile transitive dependency, runs arbitrary code before a human has looked at anything.
The repository's own commands. Test runners, build steps, and task files are code. If your pipeline runs npm test on a contributor's branch, you are running whatever that contributor decided test should mean today.
Tool configuration. This is the one almost everyone misses. Linters and formatters read configuration from the repository, and that configuration can load plugins, custom rules, and local modules. A crafted linter config is a remote code execution primitive wearing a yaml costume. The tool is yours. The config it obeys is theirs.

Here is the sharp version. The attacker does not need you to run their function. They need you to run your tool against their configuration. That is a far wider door, and it is usually left unlocked.

The failure that turns a bug into a breach

A code execution bug inside an isolated sandbox is a contained incident. The same bug becomes a company-ending event when the environment running untrusted code also holds the keys to everything else.

This is the anatomy of the worst tooling breaches. An automation platform runs contributor code in an environment that, for convenience, also contains its cloud credentials, its database connection string, and, in the most painful cases, the private key of the integration that grants it access to every customer repository. One escape from the sandbox, and the blast radius is not one job. It is every tenant at once.

The lesson is not "sandbox harder." Sandboxes have escapes; assume yours will. The lesson is that the thing reading untrusted code should have nothing worth stealing and nowhere to pivot. If your isolation fails and the intruder finds an empty room with a dead network, you have had an incident worth a postmortem. If they find your master keys, you have had an extinction event. Design so that the worst realistic day is the boring one.

Seven principles that actually hold

Here is the posture I would defend in a design review. None of it is exotic. It is the discipline of assuming your isolation will eventually fail and making that failure uneventful.

Never run the repository's own scripts. No install, no test, no build, no task file from untrusted code. If you need to understand the project, parse its manifests yourself instead of executing its lifecycle. This single rule deletes the most common execution path outright.
Keep secrets out of the room. The process that touches untrusted code should not have your platform credentials in its environment, its filesystem, or its memory. Not the database password, not the signing key, not another tenant's token. What is not there cannot leak.
Scope every credential to the minimum. When the sandbox genuinely needs access, hand it the narrowest possible grant: a token scoped to the single repository, read-only, and short-lived. A token that can read one repo for ten minutes is a completely different liability from an app key that can write to a million.
Deny egress by default. Most analysis never needs the open internet, and it certainly does not need your internal services or the cloud metadata endpoint. Block outbound traffic and allow-list only what is required. This closes both data exfiltration and the classic metadata-credential theft in one move.
Treat tool configuration as untrusted input. If you run linters or analyzers, run them with plugin loading and config discovery disabled, or parse the repository-controlled config yourself rather than letting the tool obey it. Assume any file the attacker can write is an attempt to make your tools work for them.
Make the environment ephemeral, non-root, and constrained. One run, one throwaway environment, destroyed when it finishes. Run as an unprivileged user, with a read-only filesystem except for a scratch directory, and hard limits on CPU, memory, and wall-clock time. Ephemerality alone defeats an entire category of persistence.
Practice least privilege on the integration itself. Before you ever reach the sandbox, ask what your GitHub App or CI integration is even allowed to do. If it does not need write access to file contents, do not request it. The permissions you never hold cannot be abused when something downstream goes wrong.

What this looks like in practice

I care about this because I build a tool squarely in this category. ShipGuarde is an AI reviewer that reads pull requests and QAs the running app, which means it reads untrusted code every hour of every day. The posture above is not aspirational for us; it is the architecture. Each pull request is analyzed in a single-use sandbox that holds none of our platform's secrets. The only credential it ever receives is a short-lived token scoped to that one repository, read-only, and it has no route to internal services or metadata. We do not run a repository's own scripts, and we would rather parse a config than obey it. If that sandbox were fully compromised, the intruder would find one repository's readable source and a network cable that goes nowhere. That is the entire point.

I mention it not to sell you anything, but because the constraint shaped every early decision we made, and I think far more tools should be built this way. If you are evaluating any automation that touches your source, this is the posture to make it prove.

A checklist you can steal

If you take nothing else, take this. Before you let any pipeline run on untrusted code, confirm every line:

It does not execute the repository's install, test, or build.
Its environment holds no platform or tenant secrets.
Any credential it carries is repo-scoped, read-only, and short-lived.
Outbound network access is denied by default and allow-listed narrowly.
Repository tool configs cannot load plugins or arbitrary modules.
The environment is ephemeral, non-root, and resource-limited.
The integration requests the minimum permissions and nothing more.

Run down that list honestly. If you cannot check every box, you do not have a hardening problem. You have an untrusted-code problem in a hardening costume.

The takeaway

The industry spent a decade tuning pipelines for speed and convenience on the assumption that everything flowing through them was trusted. AI tooling quietly broke that assumption, and the tools built next will be judged on whether they noticed. The good news is that the safe design is not the slow one. When you stop running other people's code and refuse to hold secrets you do not need, you end up with a system that is both harder to breach and easier to reason about. For once, the secure path and the fast path are the same path. Take it.

Written from the trenches of building an AI reviewer. If your team is working through any of this, I am always happy to compare notes.