<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: billbin1969-debug</title>
    <description>The latest articles on DEV Community by billbin1969-debug (@billbin1969debug).</description>
    <link>https://dev.to/billbin1969debug</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4216103%2F78a5befb-0a84-4076-b4f6-1cbb0162044c.png</url>
      <title>DEV Community: billbin1969-debug</title>
      <link>https://dev.to/billbin1969debug</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/billbin1969debug"/>
    <language>en</language>
    <item>
      <title>I tested my AI bug hunter on 44 real bugs from Express, ESLint, Hexo and others. It caught 32.</title>
      <dc:creator>billbin1969-debug</dc:creator>
      <pubDate>Mon, 09 Nov 2026 12:43:25 +0000</pubDate>
      <link>https://dev.to/billbin1969debug/i-tested-my-ai-bug-hunter-on-44-real-bugs-from-express-eslint-hexo-and-others-it-caught-32-4fn4</link>
      <guid>https://dev.to/billbin1969debug/i-tested-my-ai-bug-hunter-on-44-real-bugs-from-express-eslint-hexo-and-others-it-caught-32-4fn4</guid>
      <description>&lt;p&gt;AI writes more and more of our code, and code review can't keep up. So I built &lt;strong&gt;BuildProof&lt;/strong&gt;: a tool that looks at a code change and tries to &lt;strong&gt;prove&lt;/strong&gt; a bug, not just guess one.&lt;/p&gt;

&lt;h2&gt;
  
  
  How it works
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;The AI reads the change and suspects a bug.&lt;/li&gt;
&lt;li&gt;It writes a test, and that test runs on the old code AND the new code.&lt;/li&gt;
&lt;li&gt;If the test passes on the old code and fails on the new one, that's a real behaviour change.&lt;/li&gt;
&lt;li&gt;A separate AI "judge" decides whether that change was intended or a bug.&lt;/li&gt;
&lt;li&gt;You only get bugs that come with a failing test as proof.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The test
&lt;/h2&gt;

&lt;p&gt;BugsJS is a public dataset of real bugs from real projects (Express, ESLint, Hexo, Karma, Shields, Bower, hessian.js). I re-introduced each bug and let the hunter look at the change, with no hints.&lt;/p&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Caught &lt;strong&gt;32 of 44&lt;/strong&gt; real bugs.&lt;/li&gt;
&lt;li&gt;In the reverse direction (the actual fix, which is correct code), it raised a false alarm 4 times out of 44.&lt;/li&gt;
&lt;li&gt;Cost: about $0.15 per check.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It also knows when to stay quiet: I ran it on 4 open pull requests in lodash, koa and axios. It found nothing, and it was right not to. It doesn't invent bugs to look busy.&lt;/p&gt;

&lt;p&gt;Your code runs in a locked container with no network and is deleted when the check ends. There's also a free mode that uses your own local model (Ollama).&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;30-second demo, no project needed:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;npx buildproof try
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;On your own git project (3 free cloud checks):&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;npx buildproof hunt --base main --cloud
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;I'd really like feedback, especially on the misses. Which kinds of bugs should it catch that it doesn't?&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>node</category>
      <category>ai</category>
      <category>testing</category>
    </item>
    <item>
      <title>Claude Code said "all tests pass". It had skipped them. So I built a fail-closed gate.</title>
      <dc:creator>billbin1969-debug</dc:creator>
      <pubDate>Fri, 06 Nov 2026 19:20:01 +0000</pubDate>
      <link>https://dev.to/billbin1969debug/claude-code-said-all-tests-pass-it-had-skipped-them-so-i-built-a-fail-closed-gate-3dfm</link>
      <guid>https://dev.to/billbin1969debug/claude-code-said-all-tests-pass-it-had-skipped-them-so-i-built-a-fail-closed-gate-3dfm</guid>
      <description>&lt;p&gt;If you use an AI coding agent long enough, you will see this: the agent says &lt;strong&gt;"Done - all tests pass"&lt;/strong&gt;, and it's true only because a test was skipped, deleted, or quietly weakened in the same change.&lt;/p&gt;

&lt;p&gt;It's a known failure mode (see &lt;a href="https://github.com/anthropics/claude-code/issues/7074" rel="noopener noreferrer"&gt;anthropics/claude-code#7074&lt;/a&gt;). Asking the agent nicely in CLAUDE.md doesn't fix it, because the agent is the one grading its own work.&lt;/p&gt;

&lt;h2&gt;
  
  
  The idea: don't trust "done", verify it
&lt;/h2&gt;

&lt;p&gt;I built &lt;strong&gt;BuildProof&lt;/strong&gt; around one rule: the agent's claim is never accepted on its word. A separate check decides, and it fails closed - if it can't prove the work is good, the change is rejected.&lt;/p&gt;

&lt;p&gt;After Claude Code finishes a turn (as a hook), or on a pull request (as a GitHub Action), it:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Re-runs the real test suite&lt;/strong&gt; - not the agent's summary of it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checks the tests really ran&lt;/strong&gt; - zero tests executed is a failure, not a pass.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Diffs the tests themselves&lt;/strong&gt; - any test that was skipped, deleted, or had assertions weakened in the same change blocks it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checks for regressions&lt;/strong&gt; against the previous state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rolls back&lt;/strong&gt; rejected changes automatically.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Signs&lt;/strong&gt; accepted changes with an ed25519 certificate, so "this passed" is something you can verify later, not just a log line.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What it looks like
&lt;/h2&gt;

&lt;p&gt;The demo page shows real output - a change that "passes" by weakening a test, and how it gets rejected:&lt;/p&gt;

&lt;p&gt;👉 &lt;a href="https://buildproof.mgrprosnetwork.com/demo" rel="noopener noreferrer"&gt;https://buildproof.mgrprosnetwork.com/demo&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-g&lt;/span&gt; buildproof
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Package: &lt;a href="https://www.npmjs.com/package/buildproof" rel="noopener noreferrer"&gt;https://www.npmjs.com/package/buildproof&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Full disclosure: I'm the maker. It's a commercial tool with a 14-day free trial, not open source.&lt;/p&gt;

&lt;p&gt;I'd really like feedback from people who run agents on real codebases: &lt;strong&gt;what other ways have you seen an agent "fake" a green build?&lt;/strong&gt; Every trick you've seen becomes a new check.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>claudecode</category>
      <category>devtools</category>
    </item>
    <item>
      <title>My AI assistant kept saying "all tests pass". So I built a gate that checks.</title>
      <dc:creator>billbin1969-debug</dc:creator>
      <pubDate>Thu, 29 Oct 2026 22:01:09 +0000</pubDate>
      <link>https://dev.to/billbin1969debug/my-ai-assistant-kept-saying-all-tests-pass-so-i-built-a-gate-that-checks-2ae9</link>
      <guid>https://dev.to/billbin1969debug/my-ai-assistant-kept-saying-all-tests-pass-so-i-built-a-gate-that-checks-2ae9</guid>
      <description>&lt;p&gt;If you code with AI assistants, you've probably seen this: the agent says "done, all tests pass", and then you find out the tests never ran, were skipped, or were quietly edited so they would pass.&lt;/p&gt;

&lt;p&gt;I hit this often enough that I stopped trusting the claim and built a tool to check it: &lt;strong&gt;BuildProof&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does
&lt;/h2&gt;

&lt;p&gt;BuildProof is a CLI that sits between the AI and your repo:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fail-closed test gate&lt;/strong&gt; - a change is accepted only if the real test suite actually ran and passed. If anything is unclear, it counts as a failure, not a pass.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Guard&lt;/strong&gt; - detects tampering: weakened assertions, skipped tests, or test files edited to force a pass.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rollback&lt;/strong&gt; - a rejected change is rolled back to the last good state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Signed certificates&lt;/strong&gt; - every accepted run gets an ed25519-signed proof that anyone can verify.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why "fail-closed"
&lt;/h2&gt;

&lt;p&gt;Most CI setups fail open: if a step is missing or misconfigured, the pipeline can still go green. With AI agents writing more of the code, that gap gets exploited, usually by accident. BuildProof flips the default. No real proof, no pass.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;npm i -g buildproof
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;There's a 14-day free trial, and plans start at $15/month.&lt;/p&gt;

&lt;p&gt;Website: &lt;a href="https://buildproof.mgrprosnetwork.com" rel="noopener noreferrer"&gt;https://buildproof.mgrprosnetwork.com&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Your turn
&lt;/h2&gt;

&lt;p&gt;I'd really like feedback, especially if you can think of ways an AI agent could still sneak a fake "pass" past it. Tell me in the comments.&lt;/p&gt;

</description>
      <category>cli</category>
      <category>tooling</category>
      <category>testing</category>
      <category>ai</category>
    </item>
  </channel>
</rss>
