<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Oscar Alejandro Santana Leal</title>
    <description>The latest articles on DEV Community by Oscar Alejandro Santana Leal (@oscar_alejandrosantanal).</description>
    <link>https://dev.to/oscar_alejandrosantanal</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4256879%2Fce39cc9b-e98f-4fca-b673-38407bba34f6.png</url>
      <title>DEV Community: Oscar Alejandro Santana Leal</title>
      <link>https://dev.to/oscar_alejandrosantanal</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/oscar_alejandrosantanal"/>
    <language>en</language>
    <item>
      <title>No proof, no bug: making an AI code reviewer prove each defect with a failing test</title>
      <dc:creator>Oscar Alejandro Santana Leal</dc:creator>
      <pubDate>Sat, 21 Nov 2026 08:11:33 +0000</pubDate>
      <link>https://dev.to/oscar_alejandrosantanal/no-proof-no-bug-making-an-ai-code-reviewer-prove-each-defect-with-a-failing-test-1k64</link>
      <guid>https://dev.to/oscar_alejandrosantanal/no-proof-no-bug-making-an-ai-code-reviewer-prove-each-defect-with-a-failing-test-1k64</guid>
      <description>&lt;p&gt;When a coding agent finishes a change, the usual "review" is the same model reading its own output. It tends to agree with itself.&lt;/p&gt;

&lt;p&gt;During Meta's AI developer hackathon I tried a stricter rule for the reviewer: it may only report a defect if it ships a test that fails against the code in a sandbox. No failing test, no bug report.&lt;/p&gt;

&lt;h2&gt;
  
  
  The benchmark
&lt;/h2&gt;

&lt;p&gt;I built FinCode-30: 30 small finance tasks (rounding, business-day calendars, payment splits). The hard part was not writing code from scratch, a cheap model does that fine. The hard part is &lt;strong&gt;code that already exists, passes its tests and is still wrong&lt;/strong&gt;. So every task has:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;public tests and hidden tests&lt;/li&gt;
&lt;li&gt;three buggy variants: plausible one-line bugs that pass every public test and fail at least one hidden test&lt;/li&gt;
&lt;li&gt;the clean reference, which no reviewer is allowed to break&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On top of that there are 6 packages of three modules each, with one bug hidden in one module.&lt;/p&gt;

&lt;h2&gt;
  
  
  Results with an open-weights model
&lt;/h2&gt;

&lt;p&gt;Held-out split (20 tasks, 60 buggy variants), prompts frozen before the run, the same open-weights model in every role:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Review mode&lt;/th&gt;
&lt;th&gt;Bugs fixed&lt;/th&gt;
&lt;th&gt;Correct code damaged&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Free-text review&lt;/td&gt;
&lt;td&gt;41/60 (68%)&lt;/td&gt;
&lt;td&gt;0/20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Self-check (the model writes its own tests)&lt;/td&gt;
&lt;td&gt;54/60 (90%)&lt;/td&gt;
&lt;td&gt;0/20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Proof-gated review&lt;/td&gt;
&lt;td&gt;57/60 (95%)&lt;/td&gt;
&lt;td&gt;0/20&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the three-module packages: 17/18 for proof-gated review against 12/18 for free-text review. Cost was about 6 to 9 cents per target.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it did not help
&lt;/h2&gt;

&lt;p&gt;With Meta's strongest hosted model as the reviewer, every condition fixed almost everything, and plain self-check was as good as proof-gated review and far cheaper. So the rule mostly lifts weaker models. What it adds for strong models is an audit trail: every reported defect comes with a test that fails, and each run keeps code, claims, tests and results, hashed with SHA-256 so it can be re-checked later.&lt;/p&gt;

&lt;p&gt;One surprise: the reviewer's proven claims sometimes also failed on my own reference solutions. I read each of them against the spec; none contradicted it, and about half were real defects my references shared (for example dates with a trailing newline).&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;One model family, 30 tasks, finance only. I don't know yet how much of this carries over to larger codebases or messier domains.&lt;/p&gt;

&lt;p&gt;Code, benchmark and full result tables: &lt;a href="https://github.com/matrixagentsocial/matrix-globe" rel="noopener noreferrer"&gt;https://github.com/matrixagentsocial/matrix-globe&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you run agent code review in CI, I'd like to hear where this breaks.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This article was written with the help of AI (Claude) and checked against the published result tables before posting.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>codereview</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
