My scanner's README is tested in CI: 19 claims, PASS/FAIL, build breaks if one stops being true

Every security tool README makes claims. "Detects SQL injection." "Finds
hidden parameters." Some of those claims were true the day they were
written. A few of them are still true today. Nobody checks.

We got tired of that, so Summit (our web security CLI) does something a
bit odd: the README is part of the test suite.

How it works

There is a deliberately vulnerable app in examples/vuln_app.py. It binds
to 127.0.0.1 only, seeds 16 families of vulnerabilities (SQLi, XSS, SSTI,
command injection, CRLF, an IDOR admin panel, a hidden debug parameter,
an exposed .env, ...), and -- importantly -- seeds one perfectly healthy
outbound link to example.com as a negative control.

Then examples/evidence.py runs the real tool against that app and checks
each README claim with measured numbers:

[PASS] full scan: 16 seeded check families detected, zero false positives
       53 findings in 22.9s (344 requests), missing=none,
       control link flagged=no
[PASS] WAF bypass: naive filter blocks, tampered payload lands decoded
[PASS] OS command injection: marker executed and echoed by the server
...
evidence summary: 19/19 claims verified

CI runs this on every push. A healthy link getting flagged, or a seeded
SQLi going undetected, fails the build.

Why not just unit tests?

Unit tests assert the engine. The evidence runner asserts the promise.
When I add a check and forget to mention it, or "improve" a check until
it only works on the unit test's fixture, the README evidence turns red.
Docs rot is a build failure, not a slow leak.

The confirmation rules are the interesting part

The hard part of a scanner is not firing payloads -- it is deciding when
a finding is real:

• Boolean SQLi: AND 1=1 must match the untouched baseline, AND 1=2 must differ. Pages that vary per request stay quiet.
• Command injection: an exec marker compared against a control marker. A server that merely reflects shows both; one that executes shows the exec marker more often. Only that asymmetry reports.
• Sensitive files: confirmed by content signature (APP_KEY=...), never by status code alone. A soft-404 page that returns 200 is not an .env.
• WAF bypass: reported only when the tampered payload lands decoded in the response. A 200 with a still-encoded payload is not a bypass.
• Every finding carries a confidence label: confirmed (observed fact) or potential (heuristic). CVE banner matches stay potential because banners are trivially spoofed.

Try it without pointing anything at a real site

git clone https://github.com/pennacle-security/summit.git && cd summit
pip install -e .
python examples/vuln_app.py          # terminal 1
summit scan http://127.0.0.1:8765    # terminal 2

MIT, Python 3.10+. The demo app is yours to break.

Where this approach breaks down

Honest list: checks that depend on external services are flaky in CI
(we cap and degrade instead of failing); time-based checks are
nondeterministic so we verify calibration instead of raw delay; and the
evidence runner only proves the demo app is detected -- your target is
different. It bounds the lies, it does not eliminate them.

What would you add to a "docs are tests" runner for a security tool?