In my last post I pointed Bite at its own code. This time I wanted the opposite: projects I trust, to see whether Bite says anything useful about good code, or just makes noise.
I had Claude run Bite's free check on three well-known projects, 40 recent changes each:
- Rich, the Python library that makes terminal output pretty,
- Click, the Python library behind a huge number of command-line tools,
- qs, the JavaScript query-string parser underneath a lot of the web.
They passed
| Project | Changes checked | Noticed by a test | Real gaps |
|---|---|---|---|
| Rich | 40 | 15 | 1 small: <kbd> keys in Markdown |
| Click | 40 | 30 | 1 small: help on custom setting types |
| qs | 40 | 40 | none |
In all three projects, every bug fix Bite could check had a test that noticed when the fix was taken out. qs was the cleanest: all 40 changes had one.
The two gaps are small. Rich's Markdown support for <kbd> tags (it draws <kbd>Ctrl</kbd> in bold yellow) has no test. In Click, a change let every kind of parameter take a help text, but its tests only try the two kinds that could already do that. Neither is a bug, just something nothing would warn about if it broke.
Every unnoticed change was read by hand. Apart from those two, they were things a test can't be expected to catch: type hints, speed-ups, Windows-only code, unused code being removed, and rewrites that work the same.
My checker didn't
Rich: too many false alarms. On the first run, 21 of Bite's 22 flags on Rich were noise. That's a different problem from last time. Those flags weren't wrong about the code; they just weren't worth anyone's time. So Bite now also skips commits that only touch type hints, changes for another Python version, and speed-ups, since tests check results, not speed. On Rich that removes 11 of the 21, and the <kbd> gap is still flagged. On Click the new rule fired once, correctly.
Click: my old bug, back in a new shape. In the last post, Bite was fooled by an installed copy of its own code. Click hit a cousin of that. Click keeps its code in a src/ folder and is installed with pip install -e ., like many Python projects. So when Bite checked an old commit, Python quietly loaded today's Click instead of the old one, and all 40 checks came back "tests already fail". I'd fixed the installed-copy trap for one case and missed this one. Bite now makes each copy use its own src/ folder.
What I took from it
- Good projects are the best test for a checker. When the code is solid, every flag is either a real gap or Bite's mistake, and that makes the mistakes easy to see.
- Quiet isn't the goal, honest is. Bite finding almost nothing on Rich, Click and qs is the right answer. The work was making sure that "almost nothing" was real, not Bite looking at the wrong code.
- The same bug can come back wearing different clothes. One fix for "Python loads the wrong copy" wasn't enough. That's exactly the kind of thing Bite is meant to catch in other people's code, so it's a little funny it caught me twice.
The full table, with links to each gap, is in Bite's README.
Disclosure: I'm not a coder. Claude ran these checks, wrote Bite's fixes and helped me draft this post. Every flag was read by hand, and the numbers match the README table.
Top comments (1)
This is exactly the kind of failure that makes a validator valuable: it exposes an assumption in the checker rather than quietly blessing the wrong thing. The next durable step is to add the failing real-world shapes as regression fixtures, then test the checker’s claims at its boundary—not only the implementation paths you expected it to see.