The most common way a verification lies is not by erroring: it is by going green.
These four checks cost seconds and prevent the “tested, all good” that tested nothing.
1. A 200 is not a result
An anti-bot page, a consent wall and a login redirect all return 200. Before parsing,
assert a marker you expect to see (the product name, a word from the title). If the marker is not
there, the body belongs to another page.
2. The count has to add up
The tool says total: 5 and you parsed 3. Do not report the 3 you have: re-fetch and
parse in code. A report with fewer items than reality is worse than no report, because it looks
complete.
3. Same host, another app
A hostname can point at two different applications (the last rule wins, with no error). Validate
by comparing the response body, not just the status: another app's endpoint can answer 200
in place of the one you expected.
4. Unverifiable is not verified
A command that does not exist, a service that is down, a full disk, a stuck machine: the absence
of proof is never proof of success. Write “unverifiable” and the reason, in one line —
instead of letting it look like a pass.
In a gate that becomes three states, not two: passed, failed (the only one that is
yours) and unverifiable (the machine's). Collapsing the third into “passed” makes the gate
approve without proving; collapsing it into “failed” makes the gate stop enforcing.
The test that catches this
Write the check so it fails when the verification does not happen — not one that passes by
omission. A check that only fails when the command runs and errors is measuring the command, not
the result.
See the AI Test Loop skill
See all skills
Want this wired into your project? Guided setup from
R$ 1.500, with the first cycle run together.
Top comments (0)