I had a small web shop with four end-to-end tests. All four passed, every time. I felt good about them.
Then I broke the shop on purpose, ten different ways, and ran the same four tests against each broken version. They caught zero.
That's the problem I built Saboteur to measure: a green suite tells you nothing about what it misses. I built it for the KaneCLI Online Hackathon by TestMu AI.
Live report: https://softalpha0.github.io/saboteur/
Code: https://github.com/softalpha0/saboteur
The idea in one paragraph
Saboteur plants one small, realistic bug at a time, builds the app, and runs your test suite against it. If every test still passes, the bug "slipped". For each bug that slipped, Claude writes a new test, and the test is only kept if it passes on the correct app and fails on the bugged one. No test gets in on the model's word alone.
This is basically mutation testing, but for end-to-end browser tests instead of unit tests, with an AI closing the gaps and a real browser acting as the judge.
The pieces
The tests are Kane CLI _test.md files: plain English steps that Kane runs in a real Chrome browser. Here is one of the four starter tests:
---
mode: testing
url: http://localhost:4173
tags: [starter, cart]
max_steps: 30
---
# Cart shows the added item
## Add the mug
Open the home page and add the Ceramic Mug to the cart.
Verify the cart badge in the header shows 1.
## Open the cart
Open the cart page.
Verify a line item named "Ceramic Mug" is listed with a unit price of $14.50.
The bugs are git patches. Each one is the kind of mistake that ships for real. This one makes the "+" quantity button do nothing:
- onClick={() => onChange(value + 1)}
+ onClick={() => onChange(value)}
The ten bugs in the demo shop include a cart total that skips the last item, a discount code that takes 1% instead of 10%, a checkout that accepts the email a@, and a cart that empties when you refresh.
The loop for each bug:
- Copy the app to a temp folder,
git applythe patch, build it, serve it. - Run the whole suite with Kane CLI, with self-healing off (
--no-adaptive-heal). With healing on, Kane would try to work around a broken page, which is great for flaky selectors and terrible for catching bugs. - Score it: caught or slipped.
Being strict about what counts as "caught"
This took most of the work, and it's the part I'd tell anyone building something similar to care about most.
A failed test is not the same as a caught bug. While building this I saw tests fail because:
- Chrome crashed (
browser_disconnected) - my disk filled up mid-run and Chrome couldn't write screenshots
- Kane's own page check errored (
analyzer_failed) - a hover simply didn't land the first time
If you count those as catches, your score is a lie. So Saboteur:
- treats crashes, disk errors and Kane-side errors as errors, never catches, and retries them
- re-runs every catch on its own. If the test doesn't fail a second time, the first failure is treated as flaky and the whole suite runs again
That second rule saved me on a real run. On two bugs, a test failed with analyzer_failed, which had nothing to do with the bug. The confirm run passed, the suite re-ran clean, and both bugs were correctly scored as slipped instead of fake catches.
Closing the gaps with Claude
For every bug that slipped, Claude gets the bug's description, the app's routes and facts, and the existing tests, and writes a new _test.md. Then Kane runs it twice:
| Run | Build | Healing | Must |
|---|---|---|---|
| A | correct app | on | pass |
| B | bugged app | off | fail |
If run A fails, the test is wrong about the app. If run B passes, the test doesn't catch the bug. Either way, Kane's failure text goes back to Claude for another try, up to three attempts. Only tests that pass both checks join the suite.
Results (all from real Kane runs)
Demo shop: the 4 starter tests caught 0 of 10 bugs. With Claude's Kane-proven tests: 10 of 10.
Duel mode. I then pointed two Claude agents at each other. Red reads the current tests and invents new bugs they would miss. Blue writes tests to catch them. Kane referees, and a bug only counts if Kane can see it in the browser (the check passes on the bugged app and fails on the correct one).
Final score: Red 3, Blue 3. Red found three real holes:
- the cart badge counted products instead of items
- free shipping was checked before the discount was applied
- signing out left you signed in
Blue closed all three, and the suite grew from 18 to 21 tests.
An app I didn't write. A demo app I built myself is easy to game, so I also ran it on the official React TodoMVC with ten new bugs. Four typical starter tests (add, complete, delete, edit a todo) caught 2 of 10. Claude has written Kane-proven tests for 6 of the 8 misses so far. The final two were cut short when Kane lost its sign-in mid-run.
Things that went wrong
- The first full attack took hours. Every bug is a fresh build plus a full browser run. Running two tests at a time, stopping at the first failure on attacks, and saving progress so an interrupted run can resume brought a caught bug down to a few minutes.
-
A Kane run hung for over two hours. On Windows, kane-cli runs behind a
cmdlauncher, so my timeout killed the launcher and left the real process alive. The fix was killing the whole process tree, plus a watchdog that stops a run when no output arrives for ten minutes. -
A full disk looks like flaky tests. Kane keeps screenshots for every run. When my disk hit zero, tests started failing in odd ways. Saboteur's
doctorcommand now checks free space before it starts. - Claude got every test right on the first attempt in my real runs, so the "send Kane's failure back to Claude" loop has never had to fire for real. It's covered by automated tests, but I want to see it in a live run.
Try it
git clone https://github.com/softalpha0/saboteur
cd saboteur
pnpm i
kane-cli login
pnpm saboteur doctor --replay
pnpm saboteur run --replay
--replay needs no Claude key. It uses the tests Claude already wrote and Kane already proved. To point it at your own app, edit saboteur.config.json (build command, serve command, URL) and add your own bug patches.
The question I keep coming back to: how many of your tests would still pass if the feature they test was broken? I'd love to hear what you find if you try it.
Top comments (6)
The double-confirm rule is the part I would steal first, since a flaky failure counted as a catch would quietly inflate the score. The free-shipping hole is a good example of why: a test that only checks checkout succeeds will pass it, while one that asserts the exact total for a discounted cart catches it.
Thanks, that's exactly the trap. On a real TodoMVC run, two tests failed with a Kane-side analyzer error on bugs they had nothing to do with. Without the confirm step, both would have counted as catches.
And yes on free shipping: the suite already had a checkout test, but it only checked that an order went through. The test Claude wrote for that hole builds a SAVE10 cart that drops under $50 and checks the shipping line. That's the difference between "it works" and "it's right".
That SAVE10 example is exactly the pattern worth stealing — the shipping-line assertion is what turns a happy-path check into a real guard. I will try the same shape on my own checkout tests: build the cart, apply the discount, then assert the exact total.
Glad it helps! One addition that made the biggest difference for me: pick numbers that sit right on a boundary.
For the free-shipping hole, the cart that catches it is one that starts over $50 and drops under after the discount: Canvas Tote + Desk Plant = $54.98, SAVE10 takes $5.50, subtotal $49.48, so $5.00 shipping comes back and the total is $54.48. A cart that stays well above $50 passes either way.
Then break the discount on purpose and make sure the test goes red. That last step is the part Saboteur automates.
把“测试失败”等同于“抓到缺陷”这个坑点得很准,我以前也吃过浏览器崩溃、磁盘满了导致的假失败。最值得借鉴的是二次确认那条:只有在原版通过、注入缺陷版稳定失败时才算真正抓到,否则分数很容易被不稳定的测试掩盖。
完全同意。我在 TodoMVC 的真实运行里就遇到过:两个测试因为 Kane 自身的 analyzer 错误失败了,跟植入的缺陷无关。单独重跑一次就通过了,所以这两个缺陷被正确地记为"漏掉",没有虚高分数。浏览器崩溃和磁盘写满也一样,只算错误,永远不算抓到。