In my last post, Bite checked some well-known projects and found a few small gaps: changes that no test would notice if they broke. Two of them:
- In Rich, Markdown
<kbd>Ctrl</kbd>is drawn in bold yellow, and no test checks it. - In chi, a Go web router, a change made XML responses get compressed by default. The commit did add a test, but that test counts the list against itself, so it still passes with the change taken out.
Finding a gap is the easy half. Someone still has to write the test. So this week Bite got a new option, --write-tests, that tries to write it.
How it works
For each change no test noticed, a fresh AI worker writes one small test on a copy of the project. Then Bite doesn't take its word for it. It runs the test itself:
- With the change in, the test must pass, twice.
- With the change taken out, the test must fail, and fail for a real reason (not just a crash on loading).
If either part goes wrong, Bite throws the test away and says so. It uses your own Anthropic API key, and --budget caps what a run spends.
What happened
| Gap | Bite's test | With the change | Without it | Cost |
|---|---|---|---|---|
Rich <kbd>
|
prints <kbd>Ctrl</kbd>, checks Ctrl is bold yellow |
passes, twice | fails: Ctrl not bold yellow | $0.14 |
| chi XML | sends XML through chi's compressor, checks it comes back gzipped | passes, twice | fails: XML not gzipped | $0.10 |
Total: $0.34. That's more than the two rows add up to, because of the next part.
The first chi try got thrown away, and that was Bite's fault. Go runs tests by folder, not by file, and Bite asked Go to run one file alongside the whole project. Go refused before running anything. Bite's rule caught it: a test that fails before it even runs doesn't count. So it threw the test out instead of showing a fake result. The fix is in PR #67, and the second try worked.
Did it write the right test?
Before Bite could do this, Claude had already written a test for each gap the slow way, to confirm the gaps were real. So there was an answer key.
- Rich: Bite's test checks the same thing as the answer key: Ctrl comes out bold bright yellow.
- chi: the answer key (chi #1246, still open) adds XML to chi's existing list test. Bite's test goes further: it sends a real XML response through the compressor and checks it gets zipped. Different route, same claim.
Here's the Rich one, exactly as Bite wrote it:
# Guards that <kbd>…</kbd> in Markdown renders its text in the markdown.kbd style.
from rich.console import Console
from rich.markdown import Markdown
def test_kbd_inline_html_is_styled():
console = Console(
file=__import__("io").StringIO(),
force_terminal=True,
color_system="truecolor",
width=40,
)
console.print(Markdown("Press <kbd>Ctrl</kbd> now"))
output = console.file.getvalue()
assert "\x1b[1;93mCtrl\x1b[0m" in output
Then we tried the whole loop
On my own copy of Rich, Bite ran as a GitHub Action on a pull request. It found the <kbd> gap, wrote the test, and opened a small pull request with the test in it. I merged that pull request, and Bite's check on the original pull request turned green. Gap found, test written, test merged, without anyone typing a test.
What I took from it
- Don't trust a test you didn't watch fail. The pass-and-fail check is the whole point. An AI can write a test that looks right and checks nothing, just like the chi commit's own test did.
- Throwing things away is a feature. The chi test that never ran would have looked like a catch. Bite saying "no" there is what makes its "yes" worth something.
- Two gaps is a small trial. It worked twice, on two languages. That's a start, not proof it always works.
It's in v0.7.0 (pipx upgrade bite-check). On your own computer: bite --write-tests. In GitHub Actions, the setup is in Bite's README. Since this trial, Bite also asks a second fresh check whether each test is fair before showing it.
Disclosure: I'm not a coder. Claude ran these trials, wrote Bite's code and helped me draft this post. The numbers come from the run logs.
Top comments (0)