<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: tappers</title>
    <description>The latest articles on DEV Community by tappers (@tappers_).</description>
    <link>https://dev.to/tappers_</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4219060%2Ffa6a2007-fa93-4181-9139-db2a6ba2f787.png</url>
      <title>DEV Community: tappers</title>
      <link>https://dev.to/tappers_</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/tappers_"/>
    <language>en</language>
    <item>
      <title>Stop Returning Bare Linter Error Codes to Coding Agents: Opaque Error Codes vs. Actionable Fix-Hint Linters (-69.6% Input Tokens)</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Sat, 21 Nov 2026 00:15:04 +0000</pubDate>
      <link>https://dev.to/tappers_/stop-returning-bare-linter-error-codes-to-coding-agents-opaque-error-codes-vs-actionable-fix-hint-1h2b</link>
      <guid>https://dev.to/tappers_/stop-returning-bare-linter-error-codes-to-coding-agents-opaque-error-codes-vs-actionable-fix-hint-1h2b</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E17&lt;/code&gt; (Feedback Sensors &amp;amp; Self-Correction)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E17&lt;/code&gt; (Backlog Candidate &lt;code&gt;C013&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e17-vm221-20261028-01-03&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM221&lt;/code&gt; (&lt;code&gt;ai-e17-221&lt;/code&gt;, 0 NICs, &lt;code&gt;PrivateNetwork=yes&lt;/code&gt;, &lt;code&gt;ProtectSystem=strict&lt;/code&gt;, UID split &lt;code&gt;labexp17&lt;/code&gt; / &lt;code&gt;labjudge17&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; &lt;em&gt;"Human-Oriented vs. Agent-Oriented Linter Messages"&lt;/em&gt; (Marmelab / &lt;code&gt;agentlint&lt;/code&gt; / OpenAI &amp;amp; Anthropic Harness Guides) — &lt;strong&gt;Bare Error Codes (&lt;code&gt;OpaqueErrorLinter&lt;/code&gt;) vs. Dynamic &lt;code&gt;💡 Fix Hint&lt;/code&gt; Remediation Recipes (&lt;code&gt;ActionableFixHintLinter&lt;/code&gt;)&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;Traditional compilers and linters were designed for human engineers who already know their team's domain conventions. Returning a terse one-line error like &lt;code&gt;E402 [settlement.py:29]: Ordering invariant or reserve threshold invariant violated&lt;/code&gt; feels "token-efficient" at first glance because the linter stdout is only &lt;code&gt;502&lt;/code&gt; characters long.&lt;/p&gt;

&lt;p&gt;For an AI coding agent, however, &lt;strong&gt;an opaque error code is an invitation to go on an expensive documentation hunt&lt;/strong&gt;. When a linter tells the LLM &lt;em&gt;what&lt;/em&gt; failed (&lt;code&gt;E401&lt;/code&gt;–&lt;code&gt;E404&lt;/code&gt;) without telling it &lt;em&gt;how&lt;/em&gt; to fix it, the agent has to spend an extra turn calling &lt;code&gt;read_rule_docs&lt;/code&gt; or grepping the repo—pulling tens of thousands of characters of unrelated rule manuals into its context window.&lt;/p&gt;

&lt;p&gt;We tested this head-to-head inside isolated VM &lt;code&gt;VM221&lt;/code&gt; on a 3-module &lt;strong&gt;payment transaction validation, chargeback reserve, and settlement statement pipeline&lt;/strong&gt; (&lt;code&gt;validator.py&lt;/code&gt;, &lt;code&gt;settlement.py&lt;/code&gt;, &lt;code&gt;statement.py&lt;/code&gt;) containing 4 custom domain contract violations (&lt;code&gt;E401&lt;/code&gt;–&lt;code&gt;E404&lt;/code&gt;):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;OpaqueErrorLinterRunner&lt;/code&gt;):&lt;/strong&gt; Returns standard terse error codes and line numbers (&lt;code&gt;E401&lt;/code&gt;–&lt;code&gt;E404&lt;/code&gt;, &lt;code&gt;502&lt;/code&gt; chars). To learn the exact domain constants and thresholds, the agent must call &lt;code&gt;read_rule_docs&lt;/code&gt; on Turn 2, loading the 85-rule enterprise rulebook (&lt;code&gt;27,651&lt;/code&gt; chars, including 64 unrelated legacy SWIFT rules) before patching on Turn 3.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;ActionableFixHintLinterRunner&lt;/code&gt;):&lt;/strong&gt; Embeds a concrete, 1-line &lt;code&gt;💡 Fix Hint&lt;/code&gt; remediation recipe directly beneath each triggered error (&lt;code&gt;2,145&lt;/code&gt; chars total JSON), allowing the agent to self-correct all 3 modules immediately on Turn 2 with zero external doc lookups.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Both modes passed &lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt; on the hidden grading suite. Embedding dynamic &lt;code&gt;💡 Fix Hint&lt;/code&gt; recipes directly into the linter output &lt;strong&gt;eliminated documentation lookup turns completely (&lt;code&gt;doc_lookups: 1 ➔ 0&lt;/code&gt;, &lt;code&gt;-100%&lt;/code&gt;)&lt;/strong&gt;, compressed &lt;strong&gt;cumulative tool output by &lt;code&gt;-92.0%&lt;/code&gt; (&lt;code&gt;28,252 ➔ 2,248&lt;/code&gt; chars, &lt;code&gt;12.57x&lt;/code&gt; smaller)&lt;/strong&gt;, cut &lt;strong&gt;LLM calls by &lt;code&gt;-33.3%&lt;/code&gt; (&lt;code&gt;3 ➔ 2&lt;/code&gt;)&lt;/strong&gt;, reduced &lt;strong&gt;cumulative input tokens by &lt;code&gt;-69.6%&lt;/code&gt; (&lt;code&gt;13,367 ➔ 4,064&lt;/code&gt;)&lt;/strong&gt;, lowered &lt;strong&gt;API cost by &lt;code&gt;-43.3%&lt;/code&gt; (&lt;code&gt;$0.0160 ➔ $0.0091&lt;/code&gt;)&lt;/strong&gt;, and ran &lt;strong&gt;&lt;code&gt;1.45x&lt;/code&gt; faster (&lt;code&gt;21.60s ➔ 14.86s&lt;/code&gt;, &lt;code&gt;-31.2%&lt;/code&gt;)&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Bare Error Codes (&lt;code&gt;OpaqueErrorLinter&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Fix-Hint Linter (&lt;code&gt;ActionableFixHintLinter&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (&lt;code&gt;labjudge17&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Identical 100% accuracy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Rulebook Lookup Turns (&lt;code&gt;doc_lookups&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1 lookup&lt;/code&gt; (85-rule manual)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 lookups&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-100% (zero doc exploration)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Tool Output (&lt;code&gt;tool_output_chars&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;28,252 chars&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2,248 chars&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-92.0% (&lt;code&gt;12.57x&lt;/code&gt; compression)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3 calls&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2 calls&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-33.3% (immediate self-heal)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 1 Tokens (In / Out)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1,591 in&lt;/code&gt; / &lt;code&gt;11 out&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1,614 in&lt;/code&gt; / &lt;code&gt;12 out&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Initial linter call identical&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 2 Tokens (In / Out)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1,795 in&lt;/code&gt; / &lt;code&gt;10 out&lt;/code&gt; (&lt;code&gt;read_rule_docs&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2,450 in&lt;/code&gt; / &lt;code&gt;1,592 out&lt;/code&gt; (Done)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Mode B finishes on Turn 2&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 3 Tokens (In / Out)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;9,981 in&lt;/code&gt; / &lt;code&gt;1,569 out&lt;/code&gt; (Done)&lt;/td&gt;
&lt;td&gt;&lt;em&gt;— (completed on Turn 2)&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Avoided &lt;code&gt;9,981&lt;/code&gt;-token manual bloat&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;13,367 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;4,064 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-69.6% (&lt;code&gt;3.29x&lt;/code&gt; input efficiency)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,590 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,604 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Identical code volume&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;21.60 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;14.86 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-31.2% (&lt;code&gt;1.45x&lt;/code&gt; faster)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0160&lt;/code&gt; (&lt;code&gt;15,989 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0091&lt;/code&gt; (&lt;code&gt;9,064 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-43.3% cost reduction&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Experiment Architecture: Opaque Linter vs. Actionable Fix-Hint Linter
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM221 (0 NICs, PrivateNetwork=yes, grader labjudge17)  |
|  Target Task: 3-Module Payment &amp;amp; Chargeback Settlement Pipeline         |
|               validator.py ➔ settlement.py ➔ statement.py               |
|               Initial State: 4 Domain Linter Violations (E401..E404)    |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: OpaqueErrorLinterRunner (Human-Style Bare Error Codes)    |  |
|  |  - Turn 1: Calls `run_opaque_linter` ➔ gets 502 chars of bare     |  |
|  |            error codes (`E401`..`E404`) without target values     |  |
|  |  - Turn 2: Forced to call `read_rule_docs` ➔ dumps 27,651 chars   |  |
|  |            (all 85 rules, including 64 legacy SWIFT rules)        |  |
|  |  - Turn 3: Submits fix with a bloated 9,981-token prompt          |  |
|  |  --&amp;gt; 3 calls | 13,367 in | 28,252 tool chars | 21.60s | 20/20     |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ActionableFixHintLinterRunner (Agent-Style Fix Recipes)   |  |
|  |  - Turn 1: Calls `run_actionable_linter` ➔ gets 2,145 chars with  |  |
|  |            exact `💡 Fix Hint` code recipes for E401..E404 only   |  |
|  |  - Turn 2: Skips `read_rule_docs` (0 lookups) ➔ patches all 3     |  |
|  |            files in one shot (2,450 input tokens)                 |  |
|  |  --&amp;gt; 2 calls | 4,064 in  | 2,248 tool chars  | 14.86s | 20/20     |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Side-by-Side Comparison: Bare Error vs. &lt;code&gt;💡 Fix Hint&lt;/code&gt; Output
&lt;/h2&gt;

&lt;p&gt;Here are the exact linter messages returned for &lt;code&gt;E401&lt;/code&gt;–&lt;code&gt;E404&lt;/code&gt; inside &lt;code&gt;VM221&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Rule ID &amp;amp; Location&lt;/th&gt;
&lt;th&gt;❌ Mode A (&lt;code&gt;OpaqueErrorLinter&lt;/code&gt; — &lt;code&gt;502&lt;/code&gt; chars total)&lt;/th&gt;
&lt;th&gt;✅ Mode B (&lt;code&gt;ActionableFixHintLinter&lt;/code&gt; — &lt;code&gt;2,145&lt;/code&gt; chars total)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;E401&lt;/code&gt; (&lt;code&gt;validator.py:21&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;E401 [validator.py:21]: Contract violation in transaction normalization, currency allowlist, or input mutation guard.&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Same header + &lt;strong&gt;&lt;code&gt;💡 Fix Hint (E401):&lt;/code&gt;&lt;/strong&gt; &lt;code&gt;Set ALLOWED_CURRENCIES = ('USD', 'EUR', 'JPY'), reject empty raw_txns (len(raw_txns) == 0), reject bool on int fields (isinstance(num, bool)), allow unit_cents &amp;gt;= 0 and 0 &amp;lt;= chargeback_bps &amp;lt;= 1000, and append a new dict {...} without mutating item.&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;E402&lt;/code&gt; (&lt;code&gt;settlement.py:29&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;E402 [settlement.py:29]: Ordering invariant or reserve threshold invariant violated.&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Same header + &lt;strong&gt;&lt;code&gt;💡 Fix Hint (E402):&lt;/code&gt;&lt;/strong&gt; &lt;code&gt;Iterate for k in order: (remove sorted(order)) to preserve first-seen key order, and change reserve condition to strictly g['chargeback_bps'] &amp;gt; 300 (reserve = (gross * 10) // 100 if g['chargeback_bps'] &amp;gt; 300 else 0).&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;E403&lt;/code&gt; (&lt;code&gt;settlement.py:37&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;E403 [settlement.py:37]: Tier discount schedule or fee rebate cap invariant violated.&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Same header + &lt;strong&gt;&lt;code&gt;💡 Fix Hint (E403):&lt;/code&gt;&lt;/strong&gt; &lt;code&gt;Use subtotal &amp;gt;= 15000 (inclusive) for GROWTH (10%) and ENTERPRISE (20% if subtotal &amp;gt;= 15000 else 10%: (subtotal * 10) // 100), and cap rebate_applied = min(fee_rebate, after_discount).&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;E404&lt;/code&gt; (&lt;code&gt;statement.py:9&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;E404 [statement.py:9]: Canonical statement delimiter or signed zero footer format violated.&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Same header + &lt;strong&gt;&lt;code&gt;💡 Fix Hint (E404):&lt;/code&gt;&lt;/strong&gt; &lt;code&gt;Format item lines with # (f"{item['merchant_id']}#{item['currency']}: tx=...") and unconditionally format footer discounts with a leading minus sign (f"TIER_DISCOUNT: -{batch['tier_discount']}" and f"REBATE_APPLIED: -{batch['rebate_applied']}").&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  4. Why "Shorter" Linter Output Cost 3.29x More Input Tokens
&lt;/h2&gt;

&lt;p&gt;Look at the paradox in the turn-by-turn token trace (&lt;code&gt;usage.json&lt;/code&gt;):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;On Turn 1, Mode A's linter output was &lt;code&gt;1,643&lt;/code&gt; characters &lt;strong&gt;shorter&lt;/strong&gt; than Mode B's (&lt;code&gt;502&lt;/code&gt; vs. &lt;code&gt;2,145&lt;/code&gt; chars).&lt;/li&gt;
&lt;li&gt;Yet because &lt;code&gt;502&lt;/code&gt; characters of bare error codes didn't tell the agent that &lt;code&gt;"JPY"&lt;/code&gt; was the missing currency or that &lt;code&gt;15000&lt;/code&gt; was the inclusive tier boundary, Mode A had to call &lt;code&gt;read_rule_docs&lt;/code&gt; on Turn 2—dumping &lt;strong&gt;&lt;code&gt;27,651&lt;/code&gt; characters&lt;/strong&gt; (all 85 enterprise rules, including 64 irrelevant SWIFT rules &lt;code&gt;E101&lt;/code&gt;–&lt;code&gt;E164&lt;/code&gt;) into the chat history.&lt;/li&gt;
&lt;li&gt;That single doc lookup inflated Mode A's Turn 3 prompt to &lt;strong&gt;&lt;code&gt;9,981&lt;/code&gt; input tokens&lt;/strong&gt;—more than &lt;strong&gt;4x larger&lt;/strong&gt; than Mode B's Turn 2 prompt (&lt;code&gt;2,450&lt;/code&gt; input tokens).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;By dynamically attaching only the 4 triggered &lt;code&gt;💡 Fix Hint&lt;/code&gt; recipes at the point of failure, Mode B acted as a &lt;strong&gt;Just-in-Time (JIT) Context Injector&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: Turn Every Custom Linter Into a JIT Fix-Hint Teacher
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Never emit an error code without a 1-line remediation recipe:&lt;/strong&gt; When wrapping &lt;code&gt;ruff&lt;/code&gt;, &lt;code&gt;eslint&lt;/code&gt;, &lt;code&gt;mypy&lt;/code&gt;, or custom AST linters (&lt;code&gt;E11&lt;/code&gt;) for an AI agent, map each rule ID to a concrete &lt;code&gt;💡 Fix Hint&lt;/code&gt; showing the exact helper function, import path, or idiom expected by your codebase.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep the full style manual out of the prompt:&lt;/strong&gt; Instead of pre-loading a 28k-character rulebook (or forcing the agent to read it), inject only the remediation snippet for the specific rule that just failed (see also &lt;strong&gt;&lt;code&gt;E39&lt;/code&gt; Guides &amp;amp; Sensors Remediation&lt;/strong&gt;).&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🥉 &lt;strong&gt;Previous Sequential Post (&lt;code&gt;E16&lt;/code&gt; — Rank #3):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e16-cli-test-output-token-killer" rel="noopener noreferrer"&gt;Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🧭 &lt;strong&gt;Next Sequential Post (&lt;code&gt;E18&lt;/code&gt; — Rank #7):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e18-advisor-executor-routing" rel="noopener noreferrer"&gt;Benchmarking the Claude Code &lt;code&gt;/advisor&lt;/code&gt; Pattern: Solo Executor Loops vs. Upfront Advisor Blueprint Routing&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🚦 &lt;strong&gt;Next Newly Published Deep Dive (&lt;code&gt;E22&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e22-reproduction-first-gate" rel="noopener noreferrer"&gt;Fail-Before-Fix in AI Coding Agents: Speculative Patching vs. a Reproduction-First Gate (-95.3% Input Tokens, 100% Accuracy)&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>softwareengineering</category>
      <category>agenticai</category>
    </item>
    <item>
      <title>Why LLMs Fail at Unified Diffs: Standard Diff Hunks vs. Line-Number-Free SEARCH/REPLACE Blocks (2.41x Faster)</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Fri, 20 Nov 2026 12:15:11 +0000</pubDate>
      <link>https://dev.to/tappers_/why-llms-fail-at-unified-diffs-standard-diff-hunks-vs-line-number-free-searchreplace-blocks-2m4i</link>
      <guid>https://dev.to/tappers_/why-llms-fail-at-unified-diffs-standard-diff-hunks-vs-line-number-free-searchreplace-blocks-2m4i</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E15&lt;/code&gt; (Edit Primitives &amp;amp; Tool Ergonomics)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E15&lt;/code&gt; (Backlog Candidate &lt;code&gt;C011&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e15-vm219-20261028-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM219&lt;/code&gt; (&lt;code&gt;ai-e15-219&lt;/code&gt;, 0 NICs, &lt;code&gt;PrivateNetwork=yes&lt;/code&gt;, &lt;code&gt;ProtectSystem=strict&lt;/code&gt;, UID split &lt;code&gt;labexp15&lt;/code&gt; / &lt;code&gt;labjudge15&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; &lt;em&gt;"Unified Diffs vs. SEARCH/REPLACE Blocks"&lt;/em&gt; (Aider / Claude Code / Antigravity Edit Tool Design) — &lt;strong&gt;Standard Unified Diff (&lt;code&gt;@@ -l,s +l,s @@&lt;/code&gt;) + Whole-File Rewrite Fallback vs. Line-Number-Free &lt;code&gt;SEARCH/REPLACE&lt;/code&gt; Anchor Blocks&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;If you have ever built a coding agent from scratch, your first instinct for a file-editing tool was probably &lt;strong&gt;Unified Diff (&lt;code&gt;git apply&lt;/code&gt; format: &lt;code&gt;@@ -old_start,old_count +new_start,new_count @@&lt;/code&gt;)&lt;/strong&gt;—after all, &lt;code&gt;git diff&lt;/code&gt; is ubiquitous in pre-training data and much shorter than rewriting whole files.&lt;/p&gt;

&lt;p&gt;Yet nearly every top-performing production coding agent—including Aider, Claude Code (&lt;code&gt;Edit&lt;/code&gt;), Cursor, and Google Antigravity (&lt;code&gt;edit_file&lt;/code&gt;)—abandoned standard Unified Diffs in favor of &lt;strong&gt;line-number-free &lt;code&gt;SEARCH/REPLACE&lt;/code&gt; (&lt;code&gt;Target&lt;/code&gt; / &lt;code&gt;Replacement&lt;/code&gt;) exact-match blocks&lt;/strong&gt;. Why?&lt;/p&gt;

&lt;p&gt;Because LLMs reason over subword tokens, &lt;strong&gt;not 1-indexed line numbers&lt;/strong&gt;. Asking an LLM to mentally count lines across docstrings and constant tables produces off-by-3-to-11 hunk header errors, triggering &lt;code&gt;patch failed&lt;/code&gt; retries or expensive whole-file rewrites.&lt;/p&gt;

&lt;p&gt;We verified this empirically inside isolated VM &lt;code&gt;VM219&lt;/code&gt; on a &lt;strong&gt;201-line, 3-module enterprise inventory fulfillment pipeline&lt;/strong&gt; (&lt;code&gt;catalog.py&lt;/code&gt; &lt;code&gt;[67L]&lt;/code&gt;, &lt;code&gt;allocator.py&lt;/code&gt; &lt;code&gt;[98L]&lt;/code&gt;, &lt;code&gt;manifest.py&lt;/code&gt; &lt;code&gt;[36L]&lt;/code&gt;) containing 6 coupled bugs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;UnifiedDiffAndRewriteRunner&lt;/code&gt;):&lt;/strong&gt; Generates standard Unified Diffs (&lt;code&gt;apply_unified_diff&lt;/code&gt;) on Turn 1, falling back to &lt;code&gt;rewrite_whole_files&lt;/code&gt; on Turn 2 if strict hunk verification rejects the patch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;SearchReplaceBlockRunner&lt;/code&gt;):&lt;/strong&gt; Generates line-number-free &lt;code&gt;SEARCH/REPLACE&lt;/code&gt; anchor pairs (&lt;code&gt;apply_search_replace&lt;/code&gt;) specifying only the exact existing code snippet and its replacement.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Both modes passed &lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt; on the hidden grading suite. However, on Turn 1 of Mode A, &lt;strong&gt;every single one of the 3 Unified Diff files failed strict hunk verification due to LLM line-counting offsets (&lt;code&gt;3&lt;/code&gt; to &lt;code&gt;11&lt;/code&gt; lines off)&lt;/strong&gt;, forcing a Turn 2 whole-file rewrite (&lt;code&gt;2,693&lt;/code&gt; output tokens). Mode B applied &lt;strong&gt;all 6 &lt;code&gt;SEARCH/REPLACE&lt;/code&gt; blocks cleanly on Turn 1 with zero patch failures (&lt;code&gt;patch_failures = 0&lt;/code&gt;)&lt;/strong&gt;, cutting &lt;strong&gt;LLM calls by &lt;code&gt;-50.0%&lt;/code&gt; (&lt;code&gt;2 ➔ 1&lt;/code&gt;)&lt;/strong&gt;, &lt;strong&gt;input tokens by &lt;code&gt;-58.9%&lt;/code&gt; (&lt;code&gt;8,196 ➔ 3,366&lt;/code&gt;)&lt;/strong&gt;, &lt;strong&gt;output tokens by &lt;code&gt;-63.0%&lt;/code&gt; (&lt;code&gt;3,775 ➔ 1,398&lt;/code&gt;)&lt;/strong&gt;, &lt;strong&gt;API cost by &lt;code&gt;-61.7%&lt;/code&gt; (&lt;code&gt;$0.0203 ➔ $0.0078&lt;/code&gt;)&lt;/strong&gt;, and running &lt;strong&gt;&lt;code&gt;2.41x&lt;/code&gt; faster (&lt;code&gt;29.34s ➔ 12.15s&lt;/code&gt;, &lt;code&gt;-58.6%&lt;/code&gt;)&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Unified Diff + Rewrite (&lt;code&gt;UnifiedDiffAndRewrite&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Exact Anchor Blocks (&lt;code&gt;SearchReplaceBlocks&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (&lt;code&gt;labjudge15&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Identical 100% accuracy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Patch Application Failures (&lt;code&gt;patch_failures&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1&lt;/code&gt; (failed on all 3 files in Turn 1)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-100% (zero hunk offset errors)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2 calls&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1 call&lt;/code&gt; (1-shot)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-50.0% (&lt;code&gt;2 ➔ 1&lt;/code&gt; turn)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 1 Tokens (In / Out)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;3,423 in&lt;/code&gt; / &lt;code&gt;1,082 out&lt;/code&gt; (rejected)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3,366 in&lt;/code&gt; / &lt;code&gt;1,398 out&lt;/code&gt; (applied)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;6 anchor blocks applied cleanly&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 2 Tokens (In / Out)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;4,773 in&lt;/code&gt; / &lt;code&gt;2,693 out&lt;/code&gt; (full rewrite)&lt;/td&gt;
&lt;td&gt;&lt;em&gt;— (completed on Turn 1)&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Eliminated 201-line full rewrite&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8,196 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3,366 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-58.9% (&lt;code&gt;2.43x&lt;/code&gt; input efficiency)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3,775 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,398 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-63.0% (&lt;code&gt;2.70x&lt;/code&gt; output efficiency)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;29.34 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;12.15 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-58.6% (&lt;code&gt;2.41x&lt;/code&gt; faster)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0203&lt;/code&gt; (&lt;code&gt;20,304 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0078&lt;/code&gt; (&lt;code&gt;7,767 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-61.7% cost reduction&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Experiment Architecture: Unified Diff vs. &lt;code&gt;SEARCH/REPLACE&lt;/code&gt; Blocks
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM219 (0 NICs, PrivateNetwork=yes, grader labjudge15)  |
|  Target Task: 201-Line Enterprise Inventory Fulfillment Pipeline        |
|               catalog.py (67L) ➔ allocator.py (98L) ➔ manifest.py (36L) |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: UnifiedDiffAndRewriteRunner                               |  |
|  |  - Turn 1: Emits `apply_unified_diff` (1,082 out)                 |  |
|  |     * catalog.py:   predicted @@ -17,3 &amp;amp; -40,9 (actual: L20, L51) |  |
|  |     * allocator.py: predicted @@ -57,4         (actual: L67)      |  |
|  |     * manifest.py:  predicted @@ -21,7         (actual: L22)      |  |
|  |     --&amp;gt; PATCH FAILED (strict hunk line-offset mismatch)           |  |
|  |  - Turn 2: Falls back to `rewrite_whole_files` (2,693 out)        |  |
|  |  --&amp;gt; 2 calls | 8,196 in | 3,775 out | 29.34s | $0.0203 | 20/20    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: SearchReplaceBlockRunner                                  |  |
|  |  - Turn 1: Emits `apply_search_replace` (6 exact-match blocks:    |  |
|  |     3 in catalog.py, 2 in allocator.py, 1 in manifest.py)         |  |
|  |     --&amp;gt; Zero line math needed; untouched docstrings skipped       |  |
|  |  --&amp;gt; 1 call  | 3,366 in | 1,398 out | 12.15s | $0.0078 | 20/20    |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Anatomy of an LLM Unified Diff Failure (&lt;code&gt;actions.json&lt;/code&gt; Trace)
&lt;/h2&gt;

&lt;p&gt;Why did &lt;code&gt;gemini-3.8-flash&lt;/code&gt; get the Python business logic 100% right on Turn 1 of Mode A, yet fail to apply a single file via &lt;code&gt;apply_unified_diff&lt;/code&gt;?&lt;/p&gt;

&lt;p&gt;Inspecting &lt;code&gt;sequence: 0&lt;/code&gt; in &lt;code&gt;actions.json&lt;/code&gt; shows how multi-line docstrings and helper blocks throw off an LLM's internal line counter across every file:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Target File (Total Lines)&lt;/th&gt;
&lt;th&gt;Model's Predicted Hunk Header&lt;/th&gt;
&lt;th&gt;What Actually Lived at That Line&lt;/th&gt;
&lt;th&gt;True Target Line&lt;/th&gt;
&lt;th&gt;Line-Count Drift&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;catalog.py&lt;/code&gt; (&lt;code&gt;67L&lt;/code&gt;) — Hunk 1&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;@@ -17,3 +17,3 @@&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;End of module docstring (&lt;code&gt;"be preserved without raising..."&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Line &lt;code&gt;20&lt;/code&gt;&lt;/strong&gt; (&lt;code&gt;ALLOWED_WAREHOUSES&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;-3 lines&lt;/code&gt; off&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;catalog.py&lt;/code&gt; (&lt;code&gt;67L&lt;/code&gt;) — Hunk 2&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;@@ -40,9 +40,15 @@&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Middle of &lt;code&gt;_check_order_dict_item&lt;/code&gt; docstring/setup&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Line &lt;code&gt;51&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;-11 lines&lt;/code&gt; off&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;allocator.py&lt;/code&gt; (&lt;code&gt;98L&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;@@ -57,4 +57,4 @@&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;"backorder_bps": item["backorder_bps"],&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Line &lt;code&gt;67&lt;/code&gt;&lt;/strong&gt; (&lt;code&gt;for k in sorted(order):&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;-10 lines&lt;/code&gt; off&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;manifest.py&lt;/code&gt; (&lt;code&gt;36L&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;@@ -21,7 +21,7 @@&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;# BUG_MANIFEST_FORMAT_START&lt;/code&gt; comment&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Line &lt;code&gt;22&lt;/code&gt;&lt;/strong&gt; (&lt;code&gt;for item in allocation["lines"]:&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;-1 line&lt;/code&gt; off&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Notice that the drift grows cumulatively deeper into a file (&lt;code&gt;-3&lt;/code&gt; lines at the top of &lt;code&gt;catalog.py&lt;/code&gt; ➔ &lt;code&gt;-11&lt;/code&gt; lines by line 51; &lt;code&gt;-10&lt;/code&gt; lines by line 67 of &lt;code&gt;allocator.py&lt;/code&gt;). Once Turn 1's diff failed, Mode A had to re-ingest its failed attempt (&lt;code&gt;4,773&lt;/code&gt; input tokens) and re-emit all 201 lines of untouched docstrings, constants, and helpers (&lt;code&gt;2,693&lt;/code&gt; output tokens).&lt;/p&gt;

&lt;h3&gt;
  
  
  Why &lt;code&gt;SEARCH/REPLACE&lt;/code&gt; Anchor Blocks Nailed It in 1,398 Output Tokens
&lt;/h3&gt;

&lt;p&gt;In Mode B (&lt;code&gt;sequence: 2&lt;/code&gt;), the tool schema simply accepted a list of &lt;code&gt;{"search": "...", "replace": "..."}&lt;/code&gt; objects per file:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;catalog.py&lt;/code&gt; (&lt;code&gt;3 blocks&lt;/code&gt;):&lt;/strong&gt; Added &lt;code&gt;"CENTRAL"&lt;/code&gt; to &lt;code&gt;ALLOWED_WAREHOUSES&lt;/code&gt;, fixed &lt;code&gt;bool&lt;/code&gt;/&lt;code&gt;unit_price == 0&lt;/code&gt;/&lt;code&gt;backorder_bps == 1000&lt;/code&gt;/non-mutation in &lt;code&gt;_check_order_dict_item&lt;/code&gt;, and rejected empty lists &lt;code&gt;[]&lt;/code&gt; in &lt;code&gt;validate_orders&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;allocator.py&lt;/code&gt; (&lt;code&gt;2 blocks&lt;/code&gt;):&lt;/strong&gt; Preserved first-seen &lt;code&gt;order&lt;/code&gt; (removing &lt;code&gt;sorted()&lt;/code&gt;) + fixed &lt;code&gt;backorder_bps &amp;gt; 300&lt;/code&gt;, and fixed &lt;code&gt;GROWTH&lt;/code&gt;/&lt;code&gt;ENTERPRISE&lt;/code&gt; &lt;code&gt;&amp;gt;= 15000&lt;/code&gt; discount tiers + &lt;code&gt;min(voucher_credit, after_discount)&lt;/code&gt; cap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;manifest.py&lt;/code&gt; (&lt;code&gt;1 block&lt;/code&gt;):&lt;/strong&gt; Switched line separator from &lt;code&gt;:&lt;/code&gt; to &lt;code&gt;#&lt;/code&gt; and formatted zero discounts/credits as &lt;code&gt;-0&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Because LLMs excel at &lt;strong&gt;copying exact token spans from their context window&lt;/strong&gt;, all 6 &lt;code&gt;search&lt;/code&gt; anchors matched uniquely on the first try—skipping untouched code and completing the entire pipeline fix in &lt;code&gt;12.15 seconds&lt;/code&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Practical Takeaway: Design Edit Tools Around Token Strengths, Not Line Math
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Use exact-match &lt;code&gt;SEARCH/REPLACE&lt;/code&gt; (&lt;code&gt;Target&lt;/code&gt; / &lt;code&gt;Replacement&lt;/code&gt;) blocks as your primary file-edit primitive:&lt;/strong&gt; Require each &lt;code&gt;search&lt;/code&gt; block to include 2–3 lines of surrounding context so the harness can verify uniqueness (&lt;code&gt;count == 1&lt;/code&gt;) before replacing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Batch coupled edits atomically in a single turn:&lt;/strong&gt; As we prove in &lt;strong&gt;&lt;code&gt;E23&lt;/code&gt; (Atomic Multi-File Batch Edit)&lt;/strong&gt;, letting &lt;code&gt;apply_search_replace&lt;/code&gt; accept a list of edits across multiple files in one tool call prevents broken intermediate test states across module boundaries.&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🏆 &lt;strong&gt;Previous Sequential Post (&lt;code&gt;E13&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e13-parallel-tournament" rel="noopener noreferrer"&gt;Escaping Sunk-Cost Retry Loops in Coding Agents: Single-Session Retries vs. a Parallel Best-of-N Tournament (-67.2% Input Tokens)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🔍 &lt;strong&gt;Rank #5 Companion Post (&lt;code&gt;E14&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e14-progressive-tool-search" rel="noopener noreferrer"&gt;The Hidden "Schema Tax" of MCP Tools: Dumping 32 Tool Schemas vs. On-Demand Tool Search (-92.7% Input Tokens)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🥉 &lt;strong&gt;Rank #3 Next Post (&lt;code&gt;E16&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e16-cli-test-output-token-killer" rel="noopener noreferrer"&gt;Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>coding</category>
      <category>agenticai</category>
    </item>
    <item>
      <title>Escaping Sunk-Cost Retry Loops in Coding Agents: Single-Session Retries vs. a Parallel Best-of-N Tournament (-67.2% Input Tokens)</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Fri, 20 Nov 2026 12:15:05 +0000</pubDate>
      <link>https://dev.to/tappers_/escaping-sunk-cost-retry-loops-in-coding-agents-single-session-retries-vs-a-parallel-best-of-n-3p3i</link>
      <guid>https://dev.to/tappers_/escaping-sunk-cost-retry-loops-in-coding-agents-single-session-retries-vs-a-parallel-best-of-n-3p3i</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E13&lt;/code&gt; (Parallel Worktrees &amp;amp; Best-of-N Selection)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite and throughput benchmarks.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E13&lt;/code&gt; (Backlog Candidate &lt;code&gt;C009&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e13-20261028-vm217-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM217&lt;/code&gt; (&lt;code&gt;ai-lab-e13-tournament&lt;/code&gt;, 0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, UID split &lt;code&gt;labagent13&lt;/code&gt; / &lt;code&gt;labjudge13&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; &lt;strong&gt;Sunk-Cost Single-Session Retry Loop vs. Parallel Best-of-N Branch Tournament + Deterministic Judge&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;When an AI coding agent tackles a legacy codebase containing &lt;strong&gt;both subtle functional bugs and an algorithmic performance bottleneck&lt;/strong&gt; (such as a nested &lt;code&gt;O(n^2)&lt;/code&gt; loop), running sequential retries inside a single chat session triggers two compounding failure modes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Quadratic Context Bloat:&lt;/strong&gt; Every incremental patch and test-failure log stays in the session history, inflating prompt tokens on every retry.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Structural Anchoring (Sunk-Cost Bias):&lt;/strong&gt; Patching inside a single workspace nudges the model toward minimal edits on top of the legacy structure rather than stepping back to redesign the data structure cleanly.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Inspired by parallel &lt;code&gt;git worktree&lt;/code&gt; tournament workflows in Claude Code and Best-of-N test-time compute scaling, we compared two architectures inside isolated VM &lt;code&gt;VM217&lt;/code&gt; on a 3-module &lt;strong&gt;payment ledger reconciler&lt;/strong&gt; (&lt;code&gt;ingest.py&lt;/code&gt;, &lt;code&gt;reconciler.py&lt;/code&gt;, &lt;code&gt;summary.py&lt;/code&gt;) plagued by 4 boundary bugs and an &lt;code&gt;O(n^2)&lt;/code&gt; nested-loop bottleneck (&lt;code&gt;~205 ms&lt;/code&gt; on 2,000 events):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;SunkCostRetry&lt;/code&gt;):&lt;/strong&gt; A single persistent session iteratively patching errors across turns until functional and throughput checks pass.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;ParallelTournament&lt;/code&gt;):&lt;/strong&gt; Spawns &lt;strong&gt;2 independent, clean-context 1-shot branches&lt;/strong&gt; (&lt;code&gt;branch_1_conservative_patch&lt;/code&gt; and &lt;code&gt;branch_2_clean_o_n_hashmap&lt;/code&gt;) and passes both candidates to a &lt;strong&gt;Deterministic Judge&lt;/strong&gt; (functional suite + AST nested-loop detector + &lt;code&gt;N=2,000&lt;/code&gt; latency benchmark) that automatically selects the winning branch.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Both modes passed &lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt; on the hidden grading suite. Yet even counting &lt;strong&gt;both&lt;/strong&gt; tournament branches combined, Mode B cut &lt;strong&gt;LLM calls by &lt;code&gt;-50.0%&lt;/code&gt; (&lt;code&gt;4 ➔ 2&lt;/code&gt; calls)&lt;/strong&gt;, slashed &lt;strong&gt;cumulative input tokens by &lt;code&gt;-67.2%&lt;/code&gt; (&lt;code&gt;13,822 ➔ 4,537&lt;/code&gt; tokens)&lt;/strong&gt;, lowered &lt;strong&gt;API cost by &lt;code&gt;-29.2%&lt;/code&gt; (&lt;code&gt;$0.0232 ➔ $0.0165&lt;/code&gt;)&lt;/strong&gt;, and cut &lt;strong&gt;serial wall-clock time by &lt;code&gt;-22.1%&lt;/code&gt; (&lt;code&gt;37.28s ➔ 29.04s&lt;/code&gt;, or &lt;code&gt;~14.5s&lt;/code&gt; when run concurrently)&lt;/strong&gt;. Better still, the Deterministic Judge caught that Branch 1's conservative patch left the &lt;code&gt;O(n^2)&lt;/code&gt; loop intact (&lt;code&gt;202.952 ms&lt;/code&gt;, score &lt;code&gt;80/100&lt;/code&gt;) and automatically promoted Branch 2's single-pass &lt;code&gt;O(n)&lt;/code&gt; hash-map redesign (&lt;strong&gt;&lt;code&gt;0.962 ms&lt;/code&gt;—&lt;code&gt;211x&lt;/code&gt; faster&lt;/strong&gt;, score &lt;code&gt;100/100&lt;/code&gt;), beating Mode A's final 3,000-event throughput by &lt;strong&gt;&lt;code&gt;-19.1%&lt;/code&gt; (&lt;code&gt;1.280 ms&lt;/code&gt; vs. &lt;code&gt;1.582 ms&lt;/code&gt;)&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Single-Session Retry (&lt;code&gt;SunkCostRetry&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: 2-Branch Tournament (&lt;code&gt;ParallelTournament&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (&lt;code&gt;labjudge13&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Identical 100% accuracy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;3,000-Event Reconciler Latency (&lt;code&gt;labjudge13&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1.582 ms&lt;/code&gt; (&lt;code&gt;O(n)&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1.280 ms&lt;/code&gt; (&lt;code&gt;O(n)&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-19.1% faster (&lt;code&gt;158x&lt;/code&gt; faster than Branch 1)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;4 calls&lt;/code&gt; (Turns 1–4)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2 calls&lt;/code&gt; (Branch 1 + Branch 2)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-50.0% (&lt;code&gt;4 ➔ 2&lt;/code&gt; calls)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative File Edits (&lt;code&gt;file_edits&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;6 file edits&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3 file edits&lt;/code&gt; (winning branch)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-50.0% fewer edits&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;13,822 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;4,537 tokens&lt;/code&gt; (&lt;code&gt;2,242 + 2,295&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-67.2% (zero retry history bloat)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3,432 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;3,480 tokens&lt;/code&gt; (both branches)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;+1.4%&lt;/code&gt; (virtually identical)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;37.28 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;29.04 sec&lt;/code&gt; (&lt;code&gt;~14.5s&lt;/code&gt; parallel)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-22.1% serial (&lt;code&gt;~2.5x&lt;/code&gt; parallel)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0232&lt;/code&gt; (&lt;code&gt;23,238 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0165&lt;/code&gt; (&lt;code&gt;16,453 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-29.2% cheaper than 1 retry loop&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Experiment Architecture: Sunk-Cost Retries vs. Parallel Tournament
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM217 (0 NICs, virtio-serial QGA, grader labjudge13)   |
|  Target Task: 3-Module Payment Ledger Reconciler                        |
|               (ingest.py ➔ reconciler.py ➔ summary.py)                  |
|               Initial State: 4 Boundary Bugs + O(n^2) Loop (~205 ms)    |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SunkCostRetry (Sequential Retries in One Growing Session) |  |
|  |  - Turn 1 (2,377 in): Patches ingest.py     (Score  30 | 205.0ms) |  |
|  |  - Turn 2 (3,052 in): Rewrites reconciler   (Score  85 |   1.5ms) |  |
|  |  - Turn 3 (4,010 in): Fixes summary -0/#    (Score 100 |   2.0ms) |  |
|  |  - Turn 4 (4,383 in): Final integrated pass (Score 100 |   1.0ms) |  |
|  |  --&amp;gt; 4 calls | 13,822 in | $0.0232 | 37.28s | 3k bench: 1.582 ms  |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ParallelTournament (2 Clean Worktrees + Code Judge)       |  |
|  |  - Branch 1 [Conservative Patch] (2,242 in / 1,746 out):          |  |
|  |     Passes functional rules, keeps nested O(n^2) loop (202.95 ms) |  |
|  |     --&amp;gt; Judge Score: 80 / 100 (REJECTED)                          |  |
|  |  - Branch 2 [Clean O(n) Hash-Map] (2,295 in / 1,734 out):         |  |
|  |     Single-pass grouped dict + order list (0.962 ms — 211x faster)|  |
|  |     --&amp;gt; Judge Score: 100 / 100 (JUDGE_SELECT_WINNER)              |  |
|  |  --&amp;gt; 2 calls | 4,537 in  | $0.0165 | 29.04s | 3k bench: 1.280 ms  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Turn-by-Turn &amp;amp; Branch-by-Branch Trace (&lt;code&gt;usage.json&lt;/code&gt; &amp;amp; &lt;code&gt;execution-traces.json&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;Inspecting all 6 LLM calls recorded in &lt;code&gt;VM217&lt;/code&gt; highlights why two parallel 1-shot branches cost &lt;strong&gt;less&lt;/strong&gt; than a single sequential 4-turn session:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Seq&lt;/th&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th&gt;Node / Branch ID&lt;/th&gt;
&lt;th&gt;Input Tokens&lt;/th&gt;
&lt;th&gt;Output Tokens&lt;/th&gt;
&lt;th&gt;Cost (&lt;code&gt;micro-USD&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;
&lt;code&gt;N=2,000&lt;/code&gt; Latency&lt;/th&gt;
&lt;th&gt;AST &lt;code&gt;O(n)&lt;/code&gt; Clean?&lt;/th&gt;
&lt;th&gt;Judge Score&lt;/th&gt;
&lt;th&gt;Status / Bottleneck&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sunk_cost_retry_turn_1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,377&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;556&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3,868&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;205.048 ms&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;False&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;30 / 100&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;reconciler&lt;/code&gt; bounds/conflicts unfixed, &lt;code&gt;summary&lt;/code&gt; format wrong, &lt;code&gt;O(n^2)&lt;/code&gt; slow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sunk_cost_retry_turn_2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3,052&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;877&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5,578&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1.538 ms&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;✅ &lt;code&gt;True&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;85 / 100&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;summary_format_hashtag_or_minus_zero_failed&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sunk_cost_retry_turn_3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,010&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;301&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,137&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2.019 ms&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;✅ &lt;code&gt;True&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;100 / 100&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Errors cleared; stages final submit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sunk_cost_retry_turn_4&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,383&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,698&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;9,655&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0.984 ms&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;✅ &lt;code&gt;True&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;100 / 100&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;SUBMIT&lt;/code&gt; (&lt;strong&gt;1.84x larger input prompt than Turn 1&lt;/strong&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;em&gt;Mode A&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;em&gt;4 sequential turns&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;13,822&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;3,432&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;23,238&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;0.984 ms&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;True&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;100&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;20 / 20&lt;/code&gt; Pass (&lt;code&gt;3,000&lt;/code&gt; events: &lt;code&gt;1.582 ms&lt;/code&gt;)&lt;/em&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;4&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Mode B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;branch_1_conservative_patch&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2,242&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,746&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8,229&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;202.952 ms&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;False&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;80 / 100&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Fixed all functional bugs, but kept &lt;code&gt;for k in seen_keys: for ev in events:&lt;/code&gt; (&lt;code&gt;O(n^2)&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;5&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Mode B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;branch_2_clean_o_n_hashmap&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2,295&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,734&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8,224&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0.962 ms&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;&lt;code&gt;True&lt;/code&gt;&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;100 / 100&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;JUDGE_SELECT_WINNER&lt;/code&gt; (&lt;code&gt;211x&lt;/code&gt; faster than Branch 1)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;em&gt;Mode B&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;em&gt;2 independent 1-shot branches&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;4,537&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;3,480&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;16,453&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;0.962 ms&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;True&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;100&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;em&gt;&lt;code&gt;20 / 20&lt;/code&gt; Pass (&lt;code&gt;3,000&lt;/code&gt; events: *&lt;/em&gt;&lt;code&gt;1.280 ms&lt;/code&gt;*&lt;em&gt;)&lt;/em&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  4. Why a Deterministic Judge Beats an LLM Judge in Best-of-N
&lt;/h2&gt;

&lt;p&gt;Look closely at &lt;code&gt;seq 4&lt;/code&gt; (&lt;code&gt;branch_1_conservative_patch&lt;/code&gt;) vs. &lt;code&gt;seq 5&lt;/code&gt; (&lt;code&gt;branch_2_clean_o_n_hashmap&lt;/code&gt;):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Branch 1&lt;/strong&gt; wrote a surgical, minimal diff that fixed every single functional edge case (&lt;code&gt;chargeback_bps == 300 vs 301&lt;/code&gt;, &lt;code&gt;subtotal == 14999 vs 15000&lt;/code&gt;, &lt;code&gt;bool&lt;/code&gt; rejection, &lt;code&gt;#&lt;/code&gt; separator, and &lt;code&gt;-0&lt;/code&gt; formatting). If you asked an LLM judge to pick the "cleanest, lowest-risk diff" by reading code alone, it could easily pick Branch 1—shipping a &lt;code&gt;202.95 ms&lt;/code&gt; quadratic time bomb into production!&lt;/li&gt;
&lt;li&gt;Because Mode B's Judge was a &lt;strong&gt;0-token deterministic Python harness&lt;/strong&gt; combining unit assertions, an AST nested-loop check, and an &lt;code&gt;N=2,000&lt;/code&gt; micro-benchmark (&lt;code&gt;&amp;lt; 15 ms&lt;/code&gt; threshold), it docked Branch 1 to &lt;code&gt;80 / 100&lt;/code&gt; (&lt;code&gt;quadratic_or_slow_reconciler(nested_for=True,ms=202.952)&lt;/code&gt;) and promoted Branch 2 (&lt;code&gt;0.962 ms&lt;/code&gt;, &lt;code&gt;100 / 100&lt;/code&gt;) with zero human intervention.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: Two Strikes, Then Fork Clean Worktrees
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Cap in-session retries at 2 turns:&lt;/strong&gt; Once an agent fails twice in the same chat thread, prompt tokens start compounding while failed code biases subsequent edits (see &lt;strong&gt;&lt;code&gt;E19&lt;/code&gt; Error-Trace Pruning&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;E28&lt;/code&gt; Loop-Detection Circuit Breaker&lt;/strong&gt;). Reset to clean &lt;code&gt;git worktree&lt;/code&gt; branches with orthogonal strategy prompts (e.g., &lt;em&gt;Conservative Surgical Fix&lt;/em&gt; vs. &lt;em&gt;Clean Data-Structure Redesign&lt;/em&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Score tournament branches with code, not prose:&lt;/strong&gt; Always pair Best-of-N worktrees with a deterministic scoring script (tests + static AST checks + latency/memory micro-benchmarks) so the fastest, cleanest architecture wins automatically.&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Previous Sequential Post (&lt;code&gt;E11&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e11-architecture-guardrail-hooks" rel="noopener noreferrer"&gt;Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🔍 &lt;strong&gt;Rank #5 Companion Post (&lt;code&gt;E14&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e14-progressive-tool-search" rel="noopener noreferrer"&gt;The Hidden "Schema Tax" of MCP Tools: Dumping 32 Tool Schemas vs. On-Demand Tool Search (-92.7% Input Tokens)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;✂️ &lt;strong&gt;Next Sequential Post (&lt;code&gt;E15&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e15-search-replace-blocks" rel="noopener noreferrer"&gt;Why LLMs Fail at Unified Diffs: Standard Diff Hunks vs. Line-Number-Free SEARCH/REPLACE Blocks (2.41x Faster)&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>git</category>
      <category>python</category>
    </item>
    <item>
      <title>Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Thu, 19 Nov 2026 12:15:01 +0000</pubDate>
      <link>https://dev.to/tappers_/why-long-system-prompts-cant-enforce-code-architecture-prompt-rules-vs-deterministic-pre-commit-2ce1</link>
      <guid>https://dev.to/tappers_/why-long-system-prompts-cant-enforce-code-architecture-prompt-rules-vs-deterministic-pre-commit-2ce1</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E11&lt;/code&gt; (Deterministic Guardrails &amp;amp; Sensors)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite and AST policy audits.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E11&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e11-vm215-20261028-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM215&lt;/code&gt; (0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, isolated grader &lt;code&gt;labjudge11&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; &lt;strong&gt;Verbose System-Prompt Style Manuals vs. Upfront AST Violation Diagnostics + Deterministic Pre-Commit AST Hooks&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;To keep AI coding agents from eroding codebase architecture—violating layer boundaries, introducing mutable global state, skipping return type annotations, or bloating files past modular size limits—teams routinely stuff dozens of rules into &lt;code&gt;CLAUDE.md&lt;/code&gt;, &lt;code&gt;.cursorrules&lt;/code&gt;, or system prompts (&lt;strong&gt;Prompt-Only Guidelines&lt;/strong&gt;).&lt;/p&gt;

&lt;p&gt;Yet in production, prompt-only architectural rules suffer from a fundamental flaw: &lt;strong&gt;Silent Soft-Constraint Drift&lt;/strong&gt;. While an LLM focuses on making functional unit tests pass, it probabilistically violates structural constraints (especially line-count budgets and cross-layer invariants)—and because functional smoke tests still pass, the agent submits the violation without ever realizing it broke a rule.&lt;/p&gt;

&lt;p&gt;We tested this head-to-head inside isolated VM &lt;code&gt;VM215&lt;/code&gt; on a 3-module &lt;strong&gt;cross-border freight &amp;amp; customs tariff pipeline&lt;/strong&gt; (&lt;code&gt;intake.py&lt;/code&gt; ➔ &lt;code&gt;tariff.py&lt;/code&gt; ➔ &lt;code&gt;waybill.py&lt;/code&gt;) whose initial buggy fixtures contained &lt;strong&gt;6 functional bugs plus 14 AST architecture violations across 5 strict policy rules&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;BloatedPromptRunner&lt;/code&gt;):&lt;/strong&gt; Places a long, emphatic architectural manual (&lt;code&gt;RULE 1&lt;/code&gt;–&lt;code&gt;RULE 5&lt;/code&gt;) in the system prompt, while &lt;code&gt;SUBMIT&lt;/code&gt; runs only functional smoke checks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;DeterministicHookRunner&lt;/code&gt;):&lt;/strong&gt; Runs a 0-token Python AST linter upfront to inject the exact list of 14 existing AST violations into the initial prompt, and enforces a &lt;strong&gt;hard Pre-Commit AST Hook gate&lt;/strong&gt; on &lt;code&gt;COMMIT_AND_SUBMIT&lt;/code&gt; that mechanically rejects any commit violating &lt;code&gt;RULE 1&lt;/code&gt;–&lt;code&gt;RULE 5&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Both modes passed &lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt; on functional tests in a single turn. However, &lt;strong&gt;Mode A failed overall compliance (&lt;code&gt;4 / 5&lt;/code&gt; architecture rules, &lt;code&gt;False&lt;/code&gt;)&lt;/strong&gt;: it expanded &lt;code&gt;tariff.py&lt;/code&gt; to &lt;strong&gt;&lt;code&gt;74&lt;/code&gt; lines&lt;/strong&gt; (silently breaching &lt;code&gt;RULE_5&lt;/code&gt;'s 65-line cap) and submitted anyway. &lt;strong&gt;Mode B achieved &lt;code&gt;20 / 20&lt;/code&gt; functional accuracy AND &lt;code&gt;5 / 5&lt;/code&gt; (100%) architectural compliance (&lt;code&gt;0&lt;/code&gt; violations)&lt;/strong&gt; on Turn 1 (&lt;code&gt;hook_rejections = 0&lt;/code&gt;), writing &lt;strong&gt;&lt;code&gt;-18.1%&lt;/code&gt; more compact code (&lt;code&gt;113&lt;/code&gt; vs. &lt;code&gt;138&lt;/code&gt; total lines)&lt;/strong&gt; while reducing &lt;strong&gt;output tokens by &lt;code&gt;-10.8%&lt;/code&gt;&lt;/strong&gt;, &lt;strong&gt;latency by &lt;code&gt;-5.6%&lt;/code&gt;&lt;/strong&gt;, and &lt;strong&gt;API cost by &lt;code&gt;-6.6%&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Prompt-Only Manual (&lt;code&gt;BloatedPrompt&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Pre-Commit AST Hook (&lt;code&gt;DeterministicHook&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Functional Suite Score (20 Cases)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Both pass functional tests&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;AST Architecture Policy Compliance (5 Rules)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;4 / 5&lt;/code&gt; (80% — FAILED)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;5 / 5&lt;/code&gt; (100% — PASSED)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Eliminated silent architectural drift&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Remaining AST Violations at Submit&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1 violation&lt;/code&gt; (&lt;code&gt;tariff.py&lt;/code&gt;: &lt;code&gt;74 &amp;gt; 65&lt;/code&gt; lines)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 violations&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100% policy enforcement&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Overall Pass (&lt;code&gt;Functional 20/20 AND Arch 5/5&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;&lt;code&gt;False&lt;/code&gt;&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;&lt;code&gt;True&lt;/code&gt;&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Only Hook mode met full spec&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total Generated Code Size (3 Modules)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;138 lines&lt;/code&gt; (&lt;code&gt;46 + 74 + 18&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;113 lines&lt;/code&gt; (&lt;code&gt;35 + 60 + 18&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-18.1% cleaner, tighter code&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1 call&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1 call&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Single-shot completion&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,535 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,753 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;+218&lt;/code&gt; tokens (14-item AST audit)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,849 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,650 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-10.8% (&lt;code&gt;-199&lt;/code&gt; output tokens)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;15.10 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;14.26 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-5.6% faster&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0088&lt;/code&gt; (&lt;code&gt;8,835 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0083&lt;/code&gt; (&lt;code&gt;8,253 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-6.6% cost reduction&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Experiment Architecture: Prompt Manual vs. Pre-Commit AST Hook
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM215 (0 NICs, virtio-serial QGA, grader labjudge11)   |
|  Target Task: Cross-Border Freight &amp;amp; Tariff Pipeline                    |
|               (intake.py ➔ tariff.py ➔ waybill.py)                      |
|               Initial State: Functional Bugs + 14 AST Rule Violations   |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: BloatedPromptRunner (Verbose Prompt Rules, Smoke-Only)    |  |
|  |  - Prompt: Long prose manual explaining RULE 1 through RULE 5     |  |
|  |  - Turn 1: Writes intake (46L), tariff (74L), waybill (18L)       |  |
|  |  - Gate:   Runs functional smoke test only (PASS) ➔ SUBMIT        |  |
|  |  --&amp;gt; Functional: 20/20 | Architecture: 4/5 FAIL (tariff.py = 74L) |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: DeterministicHookRunner (AST Diagnostic + Hard Gate)      |  |
|  |  - Pre-Scan (0 tokens): audit_architecture_guardrails() injects   |  |
|  |    structured list of all 14 initial AST violations + 65L cap     |  |
|  |  - Turn 1: Writes intake (35L), tariff (60L), waybill (18L)       |  |
|  |  - Gate:   COMMIT_AND_SUBMIT runs Pre-Commit AST Hook (5/5 PASS)  |  |
|  |  --&amp;gt; Functional: 20/20 | Architecture: 5/5 PASS (0 violations)    |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Audit of the 5 AST Architecture Policy Rules (&lt;code&gt;labjudge11&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;The isolated &lt;code&gt;labjudge11&lt;/code&gt; grader ran deterministic Python &lt;code&gt;ast&lt;/code&gt; static analysis across the initial buggy fixtures, Mode A's submission, and Mode B's submission:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Policy Rule ID&lt;/th&gt;
&lt;th&gt;Architectural Constraint (AST Check)&lt;/th&gt;
&lt;th&gt;Initial Fixture&lt;/th&gt;
&lt;th&gt;Mode A (&lt;code&gt;BloatedPrompt&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B (&lt;code&gt;DeterministicHook&lt;/code&gt;)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;RULE_1_ZERO_IMPORTS&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Self-contained pure domain core: zero &lt;code&gt;import&lt;/code&gt; or &lt;code&gt;from ... import&lt;/code&gt; statements allowed&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;3 violations&lt;/code&gt; (&lt;code&gt;math&lt;/code&gt;, &lt;code&gt;decimal&lt;/code&gt;, &lt;code&gt;os&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;✅ &lt;code&gt;0 violations&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;&lt;code&gt;0 violations&lt;/code&gt;&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;RULE_2_EXPLICIT_RETURN_TYPE&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Explicit type contracts: every &lt;code&gt;def&lt;/code&gt; function must declare an AST &lt;code&gt;returns&lt;/code&gt; annotation (&lt;code&gt;-&amp;gt; list&lt;/code&gt;, etc.)&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;3 violations&lt;/code&gt; (missing on all 3 functions)&lt;/td&gt;
&lt;td&gt;✅ &lt;code&gt;0 violations&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;&lt;code&gt;0 violations&lt;/code&gt;&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;RULE_3_NO_MUTABLE_GLOBAL_STATE&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Immutable module globals: top-level &lt;code&gt;list&lt;/code&gt;, &lt;code&gt;dict&lt;/code&gt;, and &lt;code&gt;set&lt;/code&gt; literals prohibited (must use &lt;code&gt;tuple&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;4 violations&lt;/code&gt; (&lt;code&gt;ALLOWED_HS_CLASSES = [...]&lt;/code&gt;, etc.)&lt;/td&gt;
&lt;td&gt;✅ &lt;code&gt;0 violations&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;&lt;code&gt;0 violations&lt;/code&gt;&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;RULE_4_DOMAIN_BOUNDARY&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Strict layer separation: &lt;code&gt;intake.py&lt;/code&gt; cannot reference contract tiers; &lt;code&gt;waybill.py&lt;/code&gt; (view layer) cannot perform arithmetic (&lt;code&gt;*&lt;/code&gt;, &lt;code&gt;//&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;1 violation&lt;/code&gt; (recalculation inside &lt;code&gt;waybill.py&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;✅ &lt;code&gt;0 violations&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;&lt;code&gt;0 violations&lt;/code&gt;&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;RULE_5_DOCSTRING_AND_SIZE&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Every module &amp;amp; function must have a docstring, AND &lt;strong&gt;every file must stay &lt;code&gt;&amp;lt;= 65&lt;/code&gt; lines&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;❌ &lt;code&gt;3 violations&lt;/code&gt; (missing docstrings)&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;&lt;code&gt;1 violation&lt;/code&gt; (&lt;code&gt;tariff.py&lt;/code&gt; hit &lt;code&gt;74&lt;/code&gt; lines &amp;gt; &lt;code&gt;65&lt;/code&gt;)&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;&lt;code&gt;0 violations&lt;/code&gt; (&lt;code&gt;tariff.py&lt;/code&gt; = &lt;code&gt;60&lt;/code&gt; lines)&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  4. Why LLMs Violate Prompt Rules—and Why Hooks Fix Both Sides
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Why Mode A Silently Broke &lt;code&gt;RULE_5&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Inspecting &lt;code&gt;prompt-bundle.json&lt;/code&gt; shows why prose instructions fail on structural budgets:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Token-predicting models have &lt;strong&gt;poor real-time line-counting awareness&lt;/strong&gt; while generating code token-by-token. Mode A followed local syntax rules (&lt;code&gt;tuple&lt;/code&gt; globals, &lt;code&gt;-&amp;gt; dict&lt;/code&gt; return types, docstrings), but spread function signatures and dictionary literals across extra lines in &lt;code&gt;tariff.py&lt;/code&gt;, pushing it to &lt;strong&gt;&lt;code&gt;74&lt;/code&gt; lines&lt;/strong&gt; (&lt;code&gt;+9&lt;/code&gt; over the &lt;code&gt;65&lt;/code&gt;-line limit).&lt;/li&gt;
&lt;li&gt;Because &lt;code&gt;SUBMIT&lt;/code&gt; only executed functional tests, the harness replied &lt;em&gt;"Smoke check passed"&lt;/em&gt; and accepted the commit. Without a deterministic sensor, any rule that doesn't crash unit tests will eventually decay.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Why Mode B Succeeded in 1 Shot With &lt;em&gt;Fewer&lt;/em&gt; Tokens
&lt;/h3&gt;

&lt;p&gt;Mode B combined a &lt;strong&gt;Feed-Forward Guide&lt;/strong&gt; with a &lt;strong&gt;Feedback Sensor&lt;/strong&gt; (a preview of the pattern we explore further in &lt;strong&gt;&lt;code&gt;E39&lt;/code&gt;&lt;/strong&gt;):&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Upfront 0-Token AST Diagnostic:&lt;/strong&gt; Running &lt;code&gt;audit_architecture_guardrails()&lt;/code&gt; before Turn 1 gave the model exact line numbers for all 14 existing violations and highlighted that &lt;code&gt;COMMIT_AND_SUBMIT&lt;/code&gt; mechanically blocks any file exceeding &lt;code&gt;MAX_RULE5_LINES = 65&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tighter Code Generation:&lt;/strong&gt; Knowing a mechanical AST gate stood at commit time, the model formatted &lt;code&gt;intake.py&lt;/code&gt; in &lt;code&gt;35&lt;/code&gt; lines (&lt;code&gt;-11&lt;/code&gt; vs. Mode A) and &lt;code&gt;tariff.py&lt;/code&gt; in &lt;code&gt;60&lt;/code&gt; lines (&lt;code&gt;-14&lt;/code&gt; vs. Mode A)—clearing the hook on the first call (&lt;code&gt;hook_rejections = 0&lt;/code&gt;) and saving &lt;code&gt;10.8%&lt;/code&gt; in output tokens.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: Move Rules Out of &lt;code&gt;CLAUDE.md&lt;/code&gt; and Into Pre-Commit Hooks
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;If a rule can be checked by an AST parser or linter in 5ms, delete the paragraph from your system prompt and make it a Pre-Commit Hook:&lt;/strong&gt; Layer imports (&lt;code&gt;intake&lt;/code&gt; vs. &lt;code&gt;waybill&lt;/code&gt;), mutable globals, missing type annotations, and file-length caps belong in &lt;code&gt;ruff&lt;/code&gt;, &lt;code&gt;eslint&lt;/code&gt;, &lt;code&gt;semgrep&lt;/code&gt;, or a 40-line &lt;code&gt;ast&lt;/code&gt; script—not a 500-line prompt manual.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pair commit hooks with upfront diagnostics and concrete fix hints:&lt;/strong&gt; As we also demonstrate in &lt;strong&gt;&lt;code&gt;E17&lt;/code&gt; (Actionable Fix-Suggestion Linter)&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;E24&lt;/code&gt; (Positive Few-Shot vs. Negative Rules)&lt;/strong&gt;, deterministic hooks work best when violations return an exact &lt;code&gt;line:col&lt;/code&gt; plus a 1-line remediation recipe.&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;⚔️ &lt;strong&gt;Previous Post (&lt;code&gt;E10&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e10-adversarial-critic" rel="noopener noreferrer"&gt;Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🗜️ &lt;strong&gt;Rank #6 Companion Post (&lt;code&gt;E12&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e12-milestone-context-compaction" rel="noopener noreferrer"&gt;How Save-Game Context Compaction Fixes Spec Drift in Multi-Step Coding Agents (-91.9% Input Tokens)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🏆 &lt;strong&gt;Next Sequential Post (&lt;code&gt;E13&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e13-parallel-tournament" rel="noopener noreferrer"&gt;Escaping Sunk-Cost Retry Loops in Coding Agents: Single-Session Retries vs. a Parallel Best-of-N Tournament (-67.2% Input Tokens)&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>architecture</category>
      <category>softwareengineering</category>
    </item>
    <item>
      <title>Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Thu, 19 Nov 2026 00:15:05 +0000</pubDate>
      <link>https://dev.to/tappers_/why-self-review-fails-in-ai-coding-agents-single-agent-self-review-vs-an-adversarial-critic-0-2e3l</link>
      <guid>https://dev.to/tappers_/why-self-review-fails-in-ai-coding-agents-single-agent-self-review-vs-an-adversarial-critic-0-2e3l</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E10&lt;/code&gt; (Verification &amp;amp; Adversarial Sensors)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite and mutation audits.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E10&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e10-vm214-20261028-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM214&lt;/code&gt; (0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, isolated grader &lt;code&gt;labjudge10&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; &lt;strong&gt;Single-Agent Self-Review vs. Adversarial Critic / Maker-Checker (&lt;code&gt;Maker ➔ Read-Only Locked Critic ➔ Finalizer&lt;/code&gt;)&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;One of the most common prompts in agent frameworks is the &lt;strong&gt;Self-Review&lt;/strong&gt; step: after an agent finishes writing code, the harness sends one more message in the same conversation—&lt;em&gt;"Carefully review your implementation, add thorough edge-case tests, and submit when ready."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In practice, asking an agent to review its own code in the same context triggers &lt;strong&gt;Self-Preferential Confirmation Bias&lt;/strong&gt;: the agent inherits every assumption and blind spot from Turn 1, writes confirmatory happy-path checks (or unexecuted test code), and rubber-stamps its own work.&lt;/p&gt;

&lt;p&gt;To test whether separating verification into an &lt;strong&gt;Adversarial Critic (Maker-Checker)&lt;/strong&gt; fixes this blind spot, we ran a controlled A/B experiment on a 3-module &lt;strong&gt;marketplace escrow &amp;amp; dispute reserve pipeline&lt;/strong&gt; (&lt;code&gt;sanitizer.py&lt;/code&gt; ➔ &lt;code&gt;escrow.py&lt;/code&gt; ➔ &lt;code&gt;voucher.py&lt;/code&gt; + &lt;code&gt;adversarial_suite.py&lt;/code&gt;) inside isolated VM &lt;code&gt;VM214&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;SelfReviewRunner&lt;/code&gt;):&lt;/strong&gt; A single agent patches the 3 modules on Turn 1, then self-reviews its code and writes &lt;code&gt;adversarial_suite.py&lt;/code&gt; in the same conversation on Turn 2 before submitting.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;AdversarialMakerCheckerRunner&lt;/code&gt;):&lt;/strong&gt; &lt;strong&gt;Role 1 (&lt;code&gt;Initial Maker&lt;/code&gt;)&lt;/strong&gt; patches the code; &lt;strong&gt;Role 2 (&lt;code&gt;Adversarial Critic&lt;/code&gt;)&lt;/strong&gt; enters with a &lt;strong&gt;fresh context&lt;/strong&gt; whose sole objective is falsification—writing a pure-Python &lt;code&gt;adversarial_suite.py&lt;/code&gt; targeting every boundary invariant in &lt;code&gt;TASK.md&lt;/code&gt;, which the harness immediately locks as &lt;strong&gt;Read-Only&lt;/strong&gt;; &lt;strong&gt;Role 3 (&lt;code&gt;Maker Finalizer&lt;/code&gt;)&lt;/strong&gt; verifies and finalizes the implementation under the locked adversarial suite.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;While both implementations passed the 20 functional cases, &lt;strong&gt;Mode A's self-reviewed audit suite crashed on execution (&lt;code&gt;ModuleNotFoundError: No module named 'pytest'&lt;/code&gt;, &lt;code&gt;returncode=1&lt;/code&gt;) and achieved a &lt;code&gt;0 / 6 (0.0%)&lt;/code&gt; mutation kill rate&lt;/strong&gt;. Mode B's fresh Adversarial Critic produced a 50-line zero-dependency Python falsification suite (&lt;code&gt;returncode=0&lt;/code&gt;) that &lt;strong&gt;killed &lt;code&gt;6 / 6 (100.0%)&lt;/code&gt; boundary mutants&lt;/strong&gt;—buying complete boundary verification for just &lt;strong&gt;&lt;code&gt;+$0.0052&lt;/code&gt; (half a cent)&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;+6.92s&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Single-Agent Self-Review (&lt;code&gt;SelfReview&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Adversarial Maker-Checker (&lt;code&gt;AdversarialCritic&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (20 Cases)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Both implementations pass&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Submitted &lt;code&gt;adversarial_suite.py&lt;/code&gt; Execution&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;FAILED (&lt;code&gt;returncode=1&lt;/code&gt;: missing &lt;code&gt;pytest&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;PASSED (&lt;code&gt;returncode=0&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Eliminated unverified environment assumptions&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Valid Against Golden Reference Implementation&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;False&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;True&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;100% pure-stdlib compliance&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Adversarial Mutant Kill Rate (&lt;code&gt;M1&lt;/code&gt;–&lt;code&gt;M6&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 / 6&lt;/code&gt; (0.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; (100.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+100 percentage points (&lt;code&gt;0% ➔ 100%&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2 calls&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;3 calls&lt;/code&gt; (Maker ➔ Critic ➔ Finalizer)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;+1&lt;/code&gt; call (dedicated Critic pass)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;6,512 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;10,289 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;+3,777&lt;/code&gt; tokens&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,373 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5,003 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;+630&lt;/code&gt; tokens&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;34.05 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;40.97 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;+6.92 sec&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0213&lt;/code&gt; (&lt;code&gt;21,283 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0265&lt;/code&gt; (&lt;code&gt;26,479 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;+$0.0052&lt;/code&gt; (~0.5 cents for 100% mutant kill)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Experiment Architecture: Self-Review vs. Adversarial Maker-Checker
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM214 (0 NICs, offline stdlib-only Python, labjudge10) |
|  Target Task: Marketplace Escrow &amp;amp; Dispute Reserve Pipeline             |
|               (sanitizer.py ➔ escrow.py ➔ voucher.py + suite)           |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SelfReviewRunner (Same Context Reviews Its Own Homework)  |  |
|  |  - Turn 1: Patches sanitizer.py, escrow.py, voucher.py            |  |
|  |  - Turn 2 [Same History]: Self-reviews &amp;amp; rewrites suite using     |  |
|  |    `import pytest` (forgetting stdlib-only rule) + immediately    |  |
|  |    calls SUBMIT without re-running; skips boundary pairs          |  |
|  |  --&amp;gt; Suite crashes (returncode=1) | Mutant Kill Rate: 0/6 (0.0%)  |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: AdversarialMakerCheckerRunner (Fresh Critic + File Lock)  |  |
|  |  - Role 1 [Maker]:  Patches sanitizer.py, escrow.py, voucher.py   |  |
|  |  - Role 2 [Critic]: Fresh context writes 50-line pure-Python      |  |
|  |    falsification suite (100 vs 101 bps, 7999 vs 8000, bool, -0)   |  |
|  |    --&amp;gt; Harness locks adversarial_suite.py as READ-ONLY            |  |
|  |  - Role 3 [Finalizer]: Runs locked suite (returncode=0) &amp;amp; submits |  |
|  |  --&amp;gt; Suite green (returncode=0)   | Mutant Kill Rate: 6/6 (100%)  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Mutation Kill Audit (&lt;code&gt;labjudge10&lt;/code&gt;): &lt;code&gt;0/6&lt;/code&gt; vs. &lt;code&gt;6/6&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;To measure whether each mode's submitted &lt;code&gt;adversarial_suite.py&lt;/code&gt; could actually catch subtle regressions, the isolated &lt;code&gt;labjudge10&lt;/code&gt; grader injected &lt;strong&gt;6 classic off-by-one, type-coercion, and formatting mutants (&lt;code&gt;M1&lt;/code&gt;–&lt;code&gt;M6&lt;/code&gt;)&lt;/strong&gt; into a clean reference implementation and ran both test suites against each mutant:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mutant ID&lt;/th&gt;
&lt;th&gt;Injected Boundary Bug&lt;/th&gt;
&lt;th&gt;Mode A (&lt;code&gt;SelfReview&lt;/code&gt;) Suite&lt;/th&gt;
&lt;th&gt;Mode B (&lt;code&gt;AdversarialCritic&lt;/code&gt;) Suite&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M1_dispute_bps_ge_100_instead_of_gt_100&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Mutates &lt;code&gt;dispute_bps &amp;gt; 100&lt;/code&gt; to &lt;code&gt;&amp;gt;= 100&lt;/code&gt; (wrongly holding reserve at exact &lt;code&gt;100&lt;/code&gt; bps)&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt; (crashed on &lt;code&gt;pytest&lt;/code&gt;; never tested &lt;code&gt;100&lt;/code&gt; vs &lt;code&gt;101&lt;/code&gt; boundary)&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; (asserted exact &lt;code&gt;100&lt;/code&gt; bps ➔ &lt;code&gt;10000&lt;/code&gt; and &lt;code&gt;101&lt;/code&gt; bps ➔ &lt;code&gt;9000&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M2_premier_rebate_strict_gt_8000&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Mutates &lt;code&gt;subtotal &amp;gt;= 8000&lt;/code&gt; to &lt;code&gt;&amp;gt; 8000&lt;/code&gt; (denying rebate on exact &lt;code&gt;8,000&lt;/code&gt; orders)&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt; (only checked &lt;code&gt;subtotal=13500&lt;/code&gt;; missed &lt;code&gt;7999&lt;/code&gt; vs &lt;code&gt;8000&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; (asserted &lt;code&gt;7999&lt;/code&gt; ➔ &lt;code&gt;0&lt;/code&gt; and &lt;code&gt;8000&lt;/code&gt; ➔ &lt;code&gt;800&lt;/code&gt; / &lt;code&gt;1600&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M3_bool_accepted_as_int_in_quantity&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Removes &lt;code&gt;isinstance(qty, bool)&lt;/code&gt; guard so &lt;code&gt;True&lt;/code&gt; is accepted as &lt;code&gt;1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt; (&lt;code&gt;pytest.raises&lt;/code&gt; dependency crashed in offline sandbox)&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; (tested &lt;code&gt;quantity=True&lt;/code&gt;, &lt;code&gt;unit_cents=False&lt;/code&gt;, &lt;code&gt;dispute_bps=True&lt;/code&gt;, &lt;code&gt;holdback_waiver=True&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M4_unit_cents_zero_rejected&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Mutates &lt;code&gt;unit_cents &amp;lt; 0&lt;/code&gt; to &lt;code&gt;&amp;lt;= 0&lt;/code&gt; (rejecting free &lt;code&gt;0&lt;/code&gt;-cent promotional items)&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt; (suite crashed on import)&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; (verified &lt;code&gt;unit_cents=0&lt;/code&gt; passes and &lt;code&gt;-1&lt;/code&gt; raises &lt;code&gt;ValueError&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M5_conflicting_cents_or_bps_ignored&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Silently ignores conflicting &lt;code&gt;unit_cents&lt;/code&gt; or &lt;code&gt;dispute_bps&lt;/code&gt; on duplicate &lt;code&gt;(merchant_id, category)&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt; (conflict test completely omitted from Mode A's suite)&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; (verified both price conflict and bps conflict raise &lt;code&gt;ValueError&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M6_zero_tier_rebate_missing_minus_sign&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Formats zero rebate as &lt;code&gt;TIER_REBATE: 0&lt;/code&gt; instead of mandatory &lt;code&gt;TIER_REBATE: -0&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt; (suite crashed on import)&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; (asserted exact &lt;code&gt;TIER_REBATE: -0&lt;/code&gt; and &lt;code&gt;WAIVER_APPLIED: -0&lt;/code&gt; lines)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  4. Deep Dive: Two Failure Modes of Single-Agent Self-Review
&lt;/h2&gt;

&lt;p&gt;Inspecting &lt;code&gt;self-bundle.json&lt;/code&gt; and &lt;code&gt;execution-traces.json&lt;/code&gt; from &lt;code&gt;VM214&lt;/code&gt; exposes two structural traps that occur whenever an implementing agent reviews its own work:&lt;/p&gt;

&lt;h3&gt;
  
  
  Trap 1: Unverified Last-Mile Edits &amp;amp; Constraint Drift
&lt;/h3&gt;

&lt;p&gt;On Turn 1, Mode A spent its attention fixing the business logic across &lt;code&gt;sanitizer.py&lt;/code&gt;, &lt;code&gt;escrow.py&lt;/code&gt;, and &lt;code&gt;voucher.py&lt;/code&gt;. On Turn 2, when asked to review its work and strengthen &lt;code&gt;adversarial_suite.py&lt;/code&gt;, it forgot the system constraint that the offline sandbox only had Python's standard library (&lt;code&gt;assert&lt;/code&gt; / &lt;code&gt;unittest&lt;/code&gt;), imported &lt;code&gt;pytest&lt;/code&gt;, and immediately emitted &lt;code&gt;"op": "SUBMIT"&lt;/code&gt; in the same turn—submitting a broken test file (&lt;code&gt;returncode=1&lt;/code&gt;) with complete confidence.&lt;/p&gt;

&lt;h3&gt;
  
  
  Trap 2: Representative-Value Confirmation Bias
&lt;/h3&gt;

&lt;p&gt;Even setting aside the &lt;code&gt;pytest&lt;/code&gt; import error, inspecting the assertions inside Mode A's &lt;code&gt;adversarial_suite.py&lt;/code&gt; revealed pure &lt;strong&gt;confirmatory testing&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Instead of testing the exact boundary pair &lt;code&gt;dispute_bps=100&lt;/code&gt; (no reserve) vs. &lt;code&gt;dispute_bps=101&lt;/code&gt; (10% reserve), Mode A only tested &lt;code&gt;dispute_bps=101&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Instead of testing &lt;code&gt;subtotal=7999&lt;/code&gt; vs. &lt;code&gt;subtotal=8000&lt;/code&gt;, Mode A only tested a comfortable middle value (&lt;code&gt;subtotal=13500&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;It completely omitted any test for conflicting &lt;code&gt;unit_cents&lt;/code&gt; / &lt;code&gt;dispute_bps&lt;/code&gt; across duplicate &lt;code&gt;(merchant_id, category)&lt;/code&gt; entries (&lt;code&gt;M5&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;By contrast, Mode B's &lt;strong&gt;Adversarial Critic (&lt;code&gt;role_2_adversarial_critic&lt;/code&gt;)&lt;/strong&gt; had zero emotional or contextual attachment to the Maker's code. Given a compact 55-line budget and an explicit mandate to break flawed implementations using pure Python, it wrote a dense 50-line (&lt;code&gt;3,532&lt;/code&gt;-character) falsification suite (&lt;code&gt;bad(fn, *a, **kw)&lt;/code&gt; helper + exact boundary pairs) that caught all 6 mutants on the first try.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: Never Let the Maker Own the Test File
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Replace "Review your own code" prompts with a fresh Adversarial Critic context:&lt;/strong&gt; If a task has non-trivial boundary math (&lt;code&gt;&amp;lt;&lt;/code&gt; vs. &lt;code&gt;&amp;lt;=&lt;/code&gt;), type edge cases, or strict formatting contracts, spawn a separate context whose prompt rewards falsification rather than completion.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Enforce Read-Only locks on Critic tests:&lt;/strong&gt; When handing control back to the Maker (&lt;code&gt;role_3_maker_finalizer&lt;/code&gt;) to fix any failing assertions, physically block write permissions on &lt;code&gt;adversarial_suite.py&lt;/code&gt; (see also &lt;strong&gt;Blind TDD (&lt;code&gt;E06&lt;/code&gt;)&lt;/strong&gt; and &lt;strong&gt;Cryptographic Test Locks (&lt;code&gt;E38&lt;/code&gt;)&lt;/strong&gt;) so the Maker cannot weaken the tests to pass.&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🗺️ &lt;strong&gt;Previous Post (&lt;code&gt;E09&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e09-dependency-map-vs-raw-grep" rel="noopener noreferrer"&gt;Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Next Companion Post (&lt;code&gt;E11&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e11-architecture-guardrail-hooks" rel="noopener noreferrer"&gt;Why Long System Prompts Can't Enforce Code Architecture: Prompt Rules vs. Deterministic Pre-Commit AST Hooks&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🧪 &lt;strong&gt;Related Verification Deep Dive (&lt;code&gt;E06&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e06-blind-tdd-worktrees" rel="noopener noreferrer"&gt;Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>testing</category>
      <category>agenticai</category>
    </item>
    <item>
      <title>Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Wed, 18 Nov 2026 12:15:04 +0000</pubDate>
      <link>https://dev.to/tappers_/stop-letting-your-coding-agent-blindly-grep-decoy-files-raw-grep-vs-a-0-token-ast-dependency-map-2emj</link>
      <guid>https://dev.to/tappers_/stop-letting-your-coding-agent-blindly-grep-decoy-files-raw-grep-vs-a-0-token-ast-dependency-map-2emj</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E09&lt;/code&gt; (Guides &amp;amp; Context Hygiene)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E09&lt;/code&gt; (Backlog Candidate &lt;code&gt;C005&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e09-20261028-vm213-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM213&lt;/code&gt; (Debian 13 Trixie, 0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, POSIX UID separation)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-2.5-flash&lt;/code&gt; (&lt;code&gt;temperature=0.0&lt;/code&gt;, &lt;code&gt;thinkingBudget=0&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; &lt;em&gt;"Context Engine / AST Dependency Graph Pre-Injection vs. Monolithic Grep Waste"&lt;/em&gt; — &lt;strong&gt;Free-Form Grep Exploration vs. 0-Token Deterministic AST Symbol &amp;amp; Call-Graph Mini-Map&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;In any realistic production repository, core domain terms—like &lt;code&gt;volume_discount&lt;/code&gt;, &lt;code&gt;error_pct&lt;/code&gt;, &lt;code&gt;rebate&lt;/code&gt;, &lt;code&gt;base_cost&lt;/code&gt;, or tier names (&lt;code&gt;STARTER&lt;/code&gt;, &lt;code&gt;GROWTH&lt;/code&gt;, &lt;code&gt;SCALE&lt;/code&gt;)—don't just live in the active execution pipeline. They also appear in legacy migration adapters, Prometheus metrics exporters, compliance archivers, and webhook formatters.&lt;/p&gt;

&lt;p&gt;When you hand an AI coding agent &lt;code&gt;grep&lt;/code&gt; and file-read tools and tell it to explore on its own (&lt;strong&gt;Mode A: &lt;code&gt;RawGrepExploration&lt;/code&gt;&lt;/strong&gt;), keyword searches light up every file that shares domain vocabulary. The agent ends up reading unreferenced auxiliary files, polluting its context window, and burning ~60% of its tokens before finishing the actual fix.&lt;/p&gt;

&lt;p&gt;To measure how much exploration waste can be eliminated deterministically, we benchmarked two harness modes on an &lt;strong&gt;8-module multi-tenant API quota &amp;amp; usage billing repository&lt;/strong&gt; (3 active pipeline modules containing 7 coupled bugs + 5 unreferenced auxiliary/legacy decoy modules) inside isolated VM &lt;code&gt;VM213&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;RawGrepExploration&lt;/code&gt;):&lt;/strong&gt; Standard repository-wide &lt;code&gt;grep&lt;/code&gt; and iterative file inspection across matching modules.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;ASTDependencyMap&lt;/code&gt;):&lt;/strong&gt; Before calling the LLM, a millisecond Python &lt;code&gt;ast.parse()&lt;/code&gt; pre-scan (0 tokens) extracts function signatures, constants, and the live call chain (&lt;code&gt;ACTIVE_PIPELINE&lt;/code&gt; vs. &lt;code&gt;AUXILIARY_UNREFERENCED&lt;/code&gt;) into an &lt;strong&gt;11-line AST mini-map&lt;/strong&gt; injected into Turn 1.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Both modes achieved a &lt;strong&gt;perfect &lt;code&gt;20 / 20&lt;/code&gt; (100%) pass&lt;/strong&gt; on the hidden grading suite. However, pre-injecting the 11-line AST dependency map &lt;strong&gt;eliminated 100% of decoy file reads (&lt;code&gt;5 ➔ 0&lt;/code&gt; files)&lt;/strong&gt;, cut &lt;strong&gt;LLM calls by &lt;code&gt;-66.7%&lt;/code&gt; (&lt;code&gt;3 ➔ 1&lt;/code&gt; single-shot pass)&lt;/strong&gt;, reduced &lt;strong&gt;input tokens by &lt;code&gt;-59.1%&lt;/code&gt; (&lt;code&gt;9,508 ➔ 3,893&lt;/code&gt;)&lt;/strong&gt;, cut &lt;strong&gt;output tokens by &lt;code&gt;-64.5%&lt;/code&gt; (&lt;code&gt;4,992 ➔ 1,774&lt;/code&gt;)&lt;/strong&gt;, lowered &lt;strong&gt;API cost by &lt;code&gt;-63.0%&lt;/code&gt; (&lt;code&gt;$0.0259 ➔ $0.0096&lt;/code&gt;)&lt;/strong&gt;, and ran &lt;strong&gt;&lt;code&gt;3.15x&lt;/code&gt; faster (&lt;code&gt;42.30s ➔ 13.45s&lt;/code&gt;, &lt;code&gt;-68.2%&lt;/code&gt;)&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Free-Form Grep (&lt;code&gt;RawGrepExploration&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: 0-Token AST Map (&lt;code&gt;ASTDependencyMap&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (20 Cases)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Identical 100% accuracy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total Files Inspected (&lt;code&gt;files_inspected_count&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8 / 8 files&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3 / 8 files&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-62.5% (&lt;code&gt;-5&lt;/code&gt; files opened)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Unreferenced Decoy Files Read (&lt;code&gt;decoy_files_inspected&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5 / 5 decoys&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 / 5 decoys&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-100.0% (zero decoy waste)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3 calls&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1 call&lt;/code&gt; (1-shot)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-66.7% (&lt;code&gt;-2&lt;/code&gt; exploration turns)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative File Edits (&lt;code&gt;file_edits&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8 edits&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3 edits&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-62.5% (&lt;code&gt;-5&lt;/code&gt; redundant edits)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;9,508 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3,893 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-59.1% (&lt;code&gt;-5,615&lt;/code&gt; tokens)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,992 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,774 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-64.5% (&lt;code&gt;-3,218&lt;/code&gt; tokens)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;42.30 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;13.45 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-68.2% (&lt;code&gt;3.15x&lt;/code&gt; faster)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0259&lt;/code&gt; (&lt;code&gt;25,852 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0096&lt;/code&gt; (&lt;code&gt;9,573 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-63.0% cost reduction&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Experiment Architecture &amp;amp; Repository Layout
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM213 (0 NICs, virtio-serial QGA, offline Debian 13)   |
|  Target Repo: 8-Module Multi-Tenant API Quota &amp;amp; Billing Codebase        |
|   - 3 Active Pipeline Files: ingest.py ➔ quota_engine.py ➔ receipt.py   |
|   - 5 Unreferenced Decoys:   legacy_v1_adapter, audit_exporter,         |
|                              webhook_notifier, metrics_collector,       |
|                              csv_archiver                               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: RawGrepExploration (Keyword Grep + Iterative Inspection)  |  |
|  |  - Turn 1: Repo grep hits domain terms ➔ reads 5 files (3 decoys) |  |
|  |  - Turn 2: Reads remaining 3 files (2 decoys + receipt.py)        |  |
|  |  - Turn 3: Final boundary &amp;amp; non-mutation patch pass               |  |
|  |  --&amp;gt; 3 calls | 8/8 files read (5 decoys) | 9,508 in | 42.30s      |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: ASTDependencyMap (0-Token ast.parse() Call-Graph Map)     |  |
|  |  - Pre-Scan (0 tokens, &amp;lt;5ms): build_ast_dependency_map()          |  |
|  |    Marks 3 files ACTIVE_PIPELINE &amp;amp; 5 files AUXILIARY_UNREFERENCED |  |
|  |  - Turn 1: Opens 0 decoys ➔ patches all 7 bugs in 3 active files  |  |
|  |  --&amp;gt; 1 call  | 3/8 files read (0 decoys) | 3,893 in | 13.45s      |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Why Keyword &lt;code&gt;grep&lt;/code&gt; Gets Trapped: Active Modules vs. Vocabulary Decoys
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Module&lt;/th&gt;
&lt;th&gt;AST Classification&lt;/th&gt;
&lt;th&gt;Key Symbols&lt;/th&gt;
&lt;th&gt;Bug / Why &lt;code&gt;grep&lt;/code&gt; Gets Baited&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ingest.py&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;ACTIVE_PIPELINE&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;parse_events(raw_events)&lt;/code&gt; &lt;code&gt;[L6-L37]&lt;/code&gt;, &lt;code&gt;ALLOWED_ENDPOINTS&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Bugs 1–3:&lt;/strong&gt; Mutates input dict &amp;amp; leaks extra keys; accepts &lt;code&gt;bool&lt;/code&gt; via &lt;code&gt;isinstance(x, int)&lt;/code&gt;; rejects free-tier &lt;code&gt;base_cost == 0&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;quota_engine.py&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;ACTIVE_PIPELINE&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;settle_quota(events, tier, prepay_credit)&lt;/code&gt; &lt;code&gt;[L6-L67]&lt;/code&gt;, &lt;code&gt;ALLOWED_TIERS&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Bugs 4–6:&lt;/strong&gt; Penalizes &lt;code&gt;error_pct &amp;gt;= 5&lt;/code&gt; instead of &lt;code&gt;&amp;gt; 5&lt;/code&gt;; checks &lt;code&gt;subtotal &amp;gt; 10000&lt;/code&gt; instead of &lt;code&gt;&amp;gt;= 10000&lt;/code&gt;; ignores conflicting rates on duplicate keys&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;receipt.py&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;ACTIVE_PIPELINE&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;format_receipt(summary)&lt;/code&gt; &lt;code&gt;[L4-L20]&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Bug 7:&lt;/strong&gt; Outputs &lt;code&gt;0&lt;/code&gt; instead of mandatory signed zero &lt;code&gt;-0&lt;/code&gt; when discount/credit is zero&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;legacy_v1_adapter.py&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;AUXILIARY_UNREFERENCED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;convert_legacy_v1_record&lt;/code&gt;, &lt;code&gt;estimate_legacy_rebate&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Contains &lt;code&gt;rebate&lt;/code&gt;, &lt;code&gt;error_pct&lt;/code&gt;, &lt;code&gt;base_cost&lt;/code&gt;, &lt;code&gt;STARTER_V1&lt;/code&gt;, &lt;code&gt;GROWTH_V1&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;audit_exporter.py&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;AUXILIARY_UNREFERENCED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;compute_audit_digest&lt;/code&gt;, &lt;code&gt;build_compliance_envelope&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Contains &lt;code&gt;volume_discount&lt;/code&gt;, &lt;code&gt;credit_applied&lt;/code&gt;, &lt;code&gt;STARTER&lt;/code&gt; (offline SOX/SOC2 exporter)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;webhook_notifier.py&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;AUXILIARY_UNREFERENCED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;should_trigger_overage_webhook&lt;/code&gt;, &lt;code&gt;format_webhook_payload&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Contains &lt;code&gt;STARTER&lt;/code&gt;, &lt;code&gt;GROWTH&lt;/code&gt;, &lt;code&gt;SCALE&lt;/code&gt;, &lt;code&gt;volume_discount&lt;/code&gt;, &lt;code&gt;error_pct_max&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;metrics_collector.py&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;AUXILIARY_UNREFERENCED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;render_prometheus_metrics&lt;/code&gt;, &lt;code&gt;aggregate_endpoint_error_rates&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Contains &lt;code&gt;error_pct&lt;/code&gt;, &lt;code&gt;STARTER&lt;/code&gt;, &lt;code&gt;subtotal&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;csv_archiver.py&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;AUXILIARY_UNREFERENCED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;settlement_lines_to_csv&lt;/code&gt;, &lt;code&gt;parse_archived_csv_line&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Contains &lt;code&gt;base_cost&lt;/code&gt;, &lt;code&gt;error_pct&lt;/code&gt;, &lt;code&gt;line_total&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  The Actual 11-Line 0-Token AST Mini-Map Injected in Mode B
&lt;/h3&gt;

&lt;p&gt;Using only Python's standard &lt;code&gt;ast&lt;/code&gt; library (&lt;code&gt;build_ast_dependency_map()&lt;/code&gt;), the harness generated this compact structural map in milliseconds before making a single API call:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# AST Symbol &amp;amp; Pipeline Dependency Map (0-Token Deterministic Pre-Scan)
Active Pipeline Chain: ingest.parse_events(raw_events) -&amp;gt; quota_engine.settle_quota(events, tier, prepay_credit) -&amp;gt; receipt.format_receipt(summary)

- ingest.py [ACTIVE_PIPELINE]: funcs=[parse_events(raw_events) [L6-L37]] consts=[ALLOWED_ENDPOINTS]
- quota_engine.py [ACTIVE_PIPELINE]: funcs=[settle_quota(events, tier, prepay_credit) [L6-L67]] consts=[ALLOWED_TIERS]
- receipt.py [ACTIVE_PIPELINE]: funcs=[format_receipt(summary) [L4-L20]] consts=[]
- legacy_v1_adapter.py [AUXILIARY_UNREFERENCED]: funcs=[convert_legacy_v1_record(v1_row) [L23-L39]; estimate_legacy_rebate(gross_amount, error_pct, legacy_tier) [L42-L47]; summarize_legacy_migration_batch(rows) [L50-L58]] consts=[LEGACY_ENDPOINT_MAP, LEGACY_TIER_LIMITS]
- audit_exporter.py [AUXILIARY_UNREFERENCED]: funcs=[compute_audit_digest(tenant_id, endpoint, calls, line_total) [L12-L15]; build_compliance_envelope(settlement_dict, region, auditor_id) [L18-L46]; serialize_compliance_ndjson(envelopes) [L49-L51]] consts=[AUDIT_SCHEMA_VERSION, COMPLIANCE_REGIONS]
- webhook_notifier.py [AUXILIARY_UNREFERENCED]: funcs=[should_trigger_overage_webhook(tier, grand_total, error_pct_max) [L10-L17]; format_webhook_payload(tenant_id, tier, subtotal, volume_discount, grand_total) [L20-L35]] consts=[ALERT_THRESHOLDS]
- metrics_collector.py [AUXILIARY_UNREFERENCED]: funcs=[render_prometheus_metrics(settlement_list) [L4-L18]; aggregate_endpoint_error_rates(raw_events) [L21-L34]] consts=[]
- csv_archiver.py [AUXILIARY_UNREFERENCED]: funcs=[settlement_lines_to_csv(summary) [L14-L20]; parse_archived_csv_line(line_text) [L23-L35]] consts=[CSV_COLUMNS]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Turn-by-Turn Token &amp;amp; Cost Breakdown (&lt;code&gt;usage.json&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;Inspecting all 4 recorded model invocations (&lt;code&gt;seq 0&lt;/code&gt;–&lt;code&gt;2&lt;/code&gt; for Mode A, &lt;code&gt;seq 3&lt;/code&gt; for Mode B) shows exactly how blind &lt;code&gt;grep&lt;/code&gt; multiplies both input and output tokens:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Seq&lt;/th&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th&gt;Node ID&lt;/th&gt;
&lt;th&gt;Input Tokens&lt;/th&gt;
&lt;th&gt;Output Tokens&lt;/th&gt;
&lt;th&gt;Cost (&lt;code&gt;micro-USD&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Execution Behavior&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Mode A&lt;/strong&gt; (&lt;code&gt;RawGrep&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;raw_grep_turn_1_explore_and_patch&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5,161&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,477&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;9,410&lt;/code&gt; (&lt;code&gt;$0.0094&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Runs repo-wide &lt;code&gt;grep&lt;/code&gt;, loads 5 matching files (&lt;strong&gt;including 3 decoys&lt;/strong&gt;: &lt;code&gt;legacy_v1_adapter&lt;/code&gt;, &lt;code&gt;audit_exporter&lt;/code&gt;, &lt;code&gt;webhook_notifier&lt;/code&gt;), and makes partial edits to &lt;code&gt;ingest.py&lt;/code&gt; &amp;amp; &lt;code&gt;quota_engine.py&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Mode A&lt;/strong&gt; (&lt;code&gt;RawGrep&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;raw_grep_turn_2_refine_receipt_and_bounds&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,630&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,775&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;8,629&lt;/code&gt; (&lt;code&gt;$0.0086&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Loads the remaining 3 files (&lt;strong&gt;including 2 decoys&lt;/strong&gt;: &lt;code&gt;metrics_collector&lt;/code&gt;, &lt;code&gt;csv_archiver&lt;/code&gt;) plus &lt;code&gt;receipt.py&lt;/code&gt;; patches &lt;code&gt;-0&lt;/code&gt; formatting and discount bounds.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Mode A&lt;/strong&gt; (&lt;code&gt;RawGrep&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;raw_grep_turn_3_final_patch&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,717&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,740&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;7,813&lt;/code&gt; (&lt;code&gt;$0.0078&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Re-edits the 3 active files to finalize &lt;code&gt;bool&lt;/code&gt; rejection, input non-mutation, and first-seen ordering (8 cumulative edits).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;em&gt;Mode A Total&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;em&gt;3 turns (5 decoys read)&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;9,508&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;4,992&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;25,852&lt;/code&gt; (&lt;code&gt;$0.0259&lt;/code&gt;)&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;20 / 20&lt;/code&gt; Pass in &lt;code&gt;42.30s&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Mode B&lt;/strong&gt; (&lt;code&gt;ASTDepMap&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ast_dep_map_single_shot_fix&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3,893&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,774&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;9,573&lt;/code&gt; (&lt;code&gt;$0.0096&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Sees the 11-line AST mini-map upfront, &lt;strong&gt;ignores all 5 unreferenced decoys&lt;/strong&gt;, loads only the 3 active modules, and fixes all 7 bugs in a single shot.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;em&gt;Mode B Total&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;em&gt;1 turn (0 decoys read)&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;3,893&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;1,774&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;9,573&lt;/code&gt; (&lt;code&gt;$0.0096&lt;/code&gt;)&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;20 / 20&lt;/code&gt; Pass in &lt;code&gt;13.45s&lt;/code&gt; (&lt;code&gt;3.15x&lt;/code&gt; faster)&lt;/em&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Notice that Mode B's &lt;strong&gt;entire run (&lt;code&gt;$0.00957&lt;/code&gt;) cost almost the exact same as Turn 1 alone of Mode A (&lt;code&gt;$0.00941&lt;/code&gt;)&lt;/strong&gt;, because Mode B never wasted turns reading legacy adapters or re-patching partially understood pipelines.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Practical Takeaway: Use Static Analysis for Navigation, LLMs for Reasoning
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Never pay frontier LLM token prices to reconstruct an import graph:&lt;/strong&gt; Python's &lt;code&gt;ast&lt;/code&gt;, TypeScript's compiler API, or &lt;code&gt;tree-sitter&lt;/code&gt; can build an exact symbol and reachability map in &lt;code&gt;&amp;lt; 10 ms&lt;/code&gt; for &lt;code&gt;$0.00&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Label reachable vs. unreferenced modules explicitly:&lt;/strong&gt; Tagging modules as &lt;code&gt;[ACTIVE_PIPELINE]&lt;/code&gt; vs. &lt;code&gt;[AUXILIARY_UNREFERENCED]&lt;/code&gt; (and pairing this map with &lt;strong&gt;Tree-sitter Interface Skeletons (&lt;code&gt;E21&lt;/code&gt;)&lt;/strong&gt; for large dependencies) prevents legacy and observability code from hijacking the agent's attention.&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;📐 &lt;strong&gt;Previous Post (&lt;code&gt;E08&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e08-spec-first-sdd" rel="noopener noreferrer"&gt;Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9%&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;⚔️ &lt;strong&gt;Next Companion Post (&lt;code&gt;E10&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e10-adversarial-critic" rel="noopener noreferrer"&gt;Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🥈 &lt;strong&gt;Rank #2 Deep Dive (&lt;code&gt;E21&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e21-treesitter-skeleton-context" rel="noopener noreferrer"&gt;Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>contextengineering</category>
      <category>python</category>
    </item>
    <item>
      <title>Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9%</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Wed, 18 Nov 2026 00:15:05 +0000</pubDate>
      <link>https://dev.to/tappers_/vibe-coding-vs-spec-first-sdd-how-locking-an-executable-spec-eliminates-rework-and-cuts-input-4i2l</link>
      <guid>https://dev.to/tappers_/vibe-coding-vs-spec-first-sdd-how-locking-an-executable-spec-eliminates-rework-and-cuts-input-4i2l</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E08&lt;/code&gt; (Spec-Driven Development)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E08&lt;/code&gt; (Backlog Candidate &lt;code&gt;C004&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e08-20261028-vm212-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM212&lt;/code&gt; (&lt;code&gt;ai-e08-212&lt;/code&gt;, 0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, &lt;code&gt;systemd&lt;/code&gt; UID separation)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-2.5-flash&lt;/code&gt; (&lt;code&gt;temperature: 0.0&lt;/code&gt;, &lt;code&gt;thinkingBudget: 0&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; Spec-Driven Development (SDD) — &lt;strong&gt;Iterative Vibe Coding vs. Read-Only Executable Spec Lock (&lt;code&gt;spec_plan.py&lt;/code&gt;) + Single-Pass Implementation&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Vibe Coding&lt;/strong&gt;—prompting an LLM with a high-level feature request, eyeballing the generated code, and sending follow-up prompts to patch edge cases—feels fast on Turn 1. By Turn 3, however, iterative prompt rework inflates input tokens and routinely introduces regressions or missed edge cases.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Spec-First Software Driven Development (SDD)&lt;/strong&gt; enforces a strict two-step harness gate: before a single line of production code can be edited, the agent must formalize all boundary conditions into an &lt;strong&gt;executable contract (&lt;code&gt;spec_plan.py&lt;/code&gt;)&lt;/strong&gt;, which the harness locks &lt;strong&gt;read-only (&lt;code&gt;0444&lt;/code&gt; + SHA-256 verification)&lt;/strong&gt; before unlocking production file edits.&lt;/p&gt;

&lt;p&gt;We tested both approaches inside isolated Proxmox VM &lt;code&gt;VM212&lt;/code&gt; on a 3-module cloud usage &amp;amp; SLA billing pipeline (&lt;code&gt;parser.py&lt;/code&gt;, &lt;code&gt;calculator.py&lt;/code&gt;, &lt;code&gt;statement.py&lt;/code&gt;) containing 7 initial bugs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;VibeCoding&lt;/code&gt;):&lt;/strong&gt; Jumps straight into coding from a high-level prompt on Turn 1, followed by two iterative rework prompts (Turn 2: SLA &amp;amp; conflict rules; Turn 3: &lt;code&gt;bool&lt;/code&gt; rejection &amp;amp; credit caps).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;SpecFirstSDD&lt;/code&gt;):&lt;/strong&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;role_1_spec_architect&lt;/code&gt; (Turn 1):&lt;/strong&gt; Translates &lt;code&gt;TASK.md&lt;/code&gt; into an executable contract &lt;code&gt;spec_plan.py&lt;/code&gt; covering all 10 domain invariants. The harness verifies it catches the initial bugs (&lt;code&gt;spec_caught_initial_gaps: true&lt;/code&gt;) and locks &lt;code&gt;spec_plan.py&lt;/code&gt; read-only (&lt;code&gt;0444&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;role_2_plan_implementer&lt;/code&gt; (Turn 2):&lt;/strong&gt; Implements all 3 production modules in a single pass against the locked contract (&lt;code&gt;rework_turns = 0&lt;/code&gt;).&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Despite 3 turns and 6 file edits, &lt;strong&gt;Mode A (&lt;code&gt;VibeCoding&lt;/code&gt;)&lt;/strong&gt; still missed empty-list/non-list input rejection (&lt;code&gt;19 / 20&lt;/code&gt; on the hidden grading suite) and left &lt;code&gt;spec_plan.py&lt;/code&gt; at &lt;strong&gt;&lt;code&gt;0 / 6&lt;/code&gt; (0.0%) mutant kill rate&lt;/strong&gt;. &lt;strong&gt;Mode B (&lt;code&gt;SpecFirstSDD&lt;/code&gt;)&lt;/strong&gt; achieved &lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%) accuracy&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; (100%) mutant kill rate&lt;/strong&gt; with &lt;strong&gt;zero rework turns&lt;/strong&gt;, while cutting &lt;strong&gt;cumulative input tokens by &lt;code&gt;-32.9%&lt;/code&gt; (&lt;code&gt;9,375 ➔ 6,284&lt;/code&gt;)&lt;/strong&gt;, &lt;strong&gt;LLM calls by &lt;code&gt;-33.3%&lt;/code&gt; (&lt;code&gt;3 ➔ 2&lt;/code&gt;)&lt;/strong&gt;, &lt;strong&gt;API cost by &lt;code&gt;-6.9%&lt;/code&gt;&lt;/strong&gt;, and &lt;strong&gt;execution time by &lt;code&gt;-6.0%&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Iterative Vibe Coding (&lt;code&gt;VibeCoding&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Read-Only Spec-First SDD (&lt;code&gt;SpecFirstSDD&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (20 Cases)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;19 / 20&lt;/code&gt; (95.0% — &lt;strong&gt;❌ failed &lt;code&gt;parser_02&lt;/code&gt;&lt;/strong&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100% Perfect Pass)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+5.0 pt (100% compliance)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Spec Mutant Kill Rate (6 Mutants)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 / 6&lt;/code&gt; (0.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; (100.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+100.0 pt contract rigor&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Rework Turns (&lt;code&gt;rework_turns&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;2 rework turns&lt;/code&gt; (Turn 2 &amp;amp; Turn 3)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 rework turns&lt;/code&gt; (1-shot pass)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-100% rework eliminated&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total File Edits (&lt;code&gt;file_edits&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;6 edits&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;4 edits&lt;/code&gt; (&lt;code&gt;spec_plan.py&lt;/code&gt; + 3 modules)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-33.3% fewer edits&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3 calls&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2 calls&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-33.3% fewer calls&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;9,375 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6,284 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-32.9% (&lt;code&gt;-3,091&lt;/code&gt; tokens)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,879 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3,169 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+10.1% (executable spec investment)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;28.12 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;26.43 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-6.0% faster&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0178&lt;/code&gt; (&lt;code&gt;17,829 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0166&lt;/code&gt; (&lt;code&gt;16,598 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-6.9% cheaper&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Experiment Architecture &amp;amp; Turn-by-Turn Token Breakdown
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM212 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Cloud SLA &amp;amp; Tiered Billing Pipeline (7 Bugs)     |
|               (parser.py ➔ calculator.py ➔ statement.py)                |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: VibeCoding (Code First, Fix Edge Cases via Follow-Ups)    |  |
|  |  - Turn 1: Initial 3-file vibe patch    (1,759 in / 1,448 out)    |  |
|  |  - Turn 2: Rework #1 (SLA &amp;lt; 99 &amp;amp; tiers) (3,382 in /   718 out)    |  |
|  |  - Turn 3: Rework #2 (bool &amp;amp; credit)    (4,234 in /   713 out)    |  |
|  |  --&amp;gt; 9,375 in | 6 file edits | 19/20 Pass (missed [] check!)      |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: SpecFirstSDD (Lock Executable Contract ➔ Implement Once)  |  |
|  |  - Turn 1 [Spec Architect]: Writes `spec_plan.py` (1,936 in)      |  |
|  |  - Harness Gate: Verifies red phase &amp;amp; locks `spec_plan.py` (0444) |  |
|  |  - Turn 2 [Plan Implementer]: Patches 3 files in 1 pass (4,348 in)|  |
|  |  --&amp;gt; 6,284 in | 0 rework     | 20/20 Pass | 6/6 Mutants Killed    |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Code-Level Postmortem: Why Follow-Up Vibe Prompts Leak Edge Cases
&lt;/h2&gt;

&lt;p&gt;Inspecting &lt;code&gt;vibe-bundle.json&lt;/code&gt; (Mode A) vs. &lt;code&gt;sdd-bundle.json&lt;/code&gt; (Mode B) shows why reactive prompting struggles with completeness.&lt;/p&gt;

&lt;p&gt;On Turn 3 of &lt;strong&gt;Mode A (&lt;code&gt;VibeCoding&lt;/code&gt;)&lt;/strong&gt;, when prompted to &lt;em&gt;"reject &lt;code&gt;None&lt;/code&gt;, invalid inputs, and &lt;code&gt;bool&lt;/code&gt; types"&lt;/em&gt;, the model patched &lt;code&gt;parser.py&lt;/code&gt; narrowly around &lt;code&gt;None&lt;/code&gt;—forgetting to reject empty lists &lt;code&gt;[]&lt;/code&gt; or non-list iterables like strings:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# ❌ Mode A (`VibeCoding`, Turn 3 final `parser.py`): Narrow patch missed `[]` and strings!&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;parse_usage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw_records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;raw_records&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"invalid_records"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;raw_records&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because Mode A never upgraded &lt;code&gt;spec_plan.py&lt;/code&gt; beyond its initial smoke check, &lt;code&gt;parse_usage([])&lt;/code&gt; silently returned &lt;code&gt;[]&lt;/code&gt; and Mode A declared victory—failing &lt;code&gt;parser_02_empty_and_non_list_rejected&lt;/code&gt; in the hidden grader.&lt;/p&gt;

&lt;p&gt;In &lt;strong&gt;Mode B (&lt;code&gt;SpecFirstSDD&lt;/code&gt;)&lt;/strong&gt;, &lt;code&gt;role_1_spec_architect&lt;/code&gt; wrote and locked explicit boundary assertions &lt;em&gt;before&lt;/em&gt; any production file was touched:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# ✅ Mode B (`SpecFirstSDD`, Turn 1 locked `spec_plan.py` excerpt):&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;verify_spec_plan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;orig&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"project_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;" p1 "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"service"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;" compute "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"units"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unit_rate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"SLA_pct"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"extra"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;parse_usage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;orig&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;orig&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"project_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;" p1 "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"service"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;" compute "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"units"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unit_rate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"SLA_pct"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"extra"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"project_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"P1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"service"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"COMPUTE"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"units"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unit_rate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"SLA_pct"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parse_usage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parse_usage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parse_usage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"project_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"P"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"service"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"COMPUTE"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"units"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unit_rate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"SLA_pct"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# Exact off-by-one gates: SLA 99 (no penalty) vs 98 (10% penalty), PRO 4999 (0%) vs 5000 (10%)&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;calculate_charges&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"project_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"P"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"service"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"GPU"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"units"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unit_rate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"SLA_pct"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"PRO"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"tier_discount"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;calculate_charges&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"project_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"P"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"service"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"GPU"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"units"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4999&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unit_rate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"SLA_pct"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"PRO"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"tier_discount"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With &lt;code&gt;spec_plan.py&lt;/code&gt; locked at &lt;code&gt;chmod 0444&lt;/code&gt; (verified in &lt;code&gt;inode-manifest.json&lt;/code&gt;), &lt;code&gt;role_2_plan_implementer&lt;/code&gt; had an unambiguous, machine-checked target and nailed all 20 hidden cases on Turn 2.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Mutation Audit (&lt;code&gt;M1&lt;/code&gt;–&lt;code&gt;M6&lt;/code&gt;) Summary (&lt;code&gt;labjudge08&lt;/code&gt;)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mutant ID&lt;/th&gt;
&lt;th&gt;Injected Off-by-One / Type Defect&lt;/th&gt;
&lt;th&gt;Mode A (&lt;code&gt;VibeCoding&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B (&lt;code&gt;SpecFirstSDD&lt;/code&gt;)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M1&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;SLA penalty mutated from &lt;code&gt;&amp;lt; 99&lt;/code&gt; to &lt;code&gt;&amp;lt;= 99&lt;/code&gt; (penalizes 99% SLA)&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M2&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;PRO&lt;/code&gt; tier threshold mutated from &lt;code&gt;&amp;gt;= 5000&lt;/code&gt; to &lt;code&gt;&amp;gt; 5000&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M3&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Boolean &lt;code&gt;units=True&lt;/code&gt; accepted as integer &lt;code&gt;1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M4&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Promotional &lt;code&gt;unit_rate == 0&lt;/code&gt; erroneously rejected&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M5&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Conflicting &lt;code&gt;unit_rate&lt;/code&gt; / &lt;code&gt;SLA_pct&lt;/code&gt; for same key ignored&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M6&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Zero discount formatted as &lt;code&gt;0&lt;/code&gt; instead of &lt;code&gt;-0&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;6 Contract Mutants&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 / 6&lt;/code&gt; (0.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; (100.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: Lock the Spec Read-Only Before Unlocking Edits
&lt;/h2&gt;

&lt;p&gt;Even on a fast, lightweight model (&lt;code&gt;gemini-2.5-flash&lt;/code&gt;), &lt;strong&gt;2 turns of Spec-First SDD beat 3 turns of Vibe Coding on accuracy (&lt;code&gt;100%&lt;/code&gt; vs. &lt;code&gt;95%&lt;/code&gt;), mutation kill rate (&lt;code&gt;100%&lt;/code&gt; vs. &lt;code&gt;0%&lt;/code&gt;), speed, and token cost (&lt;code&gt;-32.9%&lt;/code&gt; input tokens)&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;In your own agent workflows:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Require a &lt;strong&gt;Phase 1 Read-Only / Spec-Lock step&lt;/strong&gt; that turns vague requirements into executable assertions (types, empty inputs, exact &lt;code&gt;&amp;lt;&lt;/code&gt; vs. &lt;code&gt;&amp;lt;=&lt;/code&gt; boundaries).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lock that contract file (&lt;code&gt;chmod 0444&lt;/code&gt; or pre-commit hash check, see &lt;code&gt;E38&lt;/code&gt;)&lt;/strong&gt; so the implementation pass cannot weaken the spec to make tests green.&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;⚙️ &lt;strong&gt;Previous (&lt;code&gt;E07&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e07-code-mode-workflows" rel="noopener noreferrer"&gt;Eliminating Goal Drift in Multi-File Coding Agents: In-Chat Loops vs. Code-Mode Dynamic Workflows&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🥇 &lt;strong&gt;Rank #1 Deep Dive (&lt;code&gt;E20&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e20-episodic-gotchas-memory" rel="noopener noreferrer"&gt;Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item &lt;code&gt;MEMORY.md&lt;/code&gt;&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>softwareengineering</category>
      <category>coding</category>
    </item>
    <item>
      <title>Eliminating Goal Drift in Multi-File Coding Agents: In-Chat Loops vs. Code-Mode Dynamic Workflows</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Tue, 17 Nov 2026 12:15:06 +0000</pubDate>
      <link>https://dev.to/tappers_/eliminating-goal-drift-in-multi-file-coding-agents-in-chat-loops-vs-code-mode-dynamic-workflows-5fm5</link>
      <guid>https://dev.to/tappers_/eliminating-goal-drift-in-multi-file-coding-agents-in-chat-loops-vs-code-mode-dynamic-workflows-5fm5</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E07&lt;/code&gt; (Dynamic Code-Mode Orchestration)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E07&lt;/code&gt; (Backlog Candidate &lt;code&gt;C003b&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e07-20261028-vm211-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM211&lt;/code&gt; (&lt;code&gt;ai-e07-211&lt;/code&gt;, 0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, &lt;code&gt;systemd&lt;/code&gt; UID separation)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt; (&lt;code&gt;thinkingLevel: LOW&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; Anthropic Claude Code Dynamic Workflows (&lt;em&gt;"Stop Babysitting Your Agents / Code-Mode Orchestration"&lt;/em&gt;, Sid Bidasaria &amp;amp; Thariq Shihipar) — &lt;strong&gt;In-Chat Iterative Loop vs. Dynamic Python Workflow Generation&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;When an AI coding agent audits and refactors multiple files turn-by-turn inside a single growing chat thread (&lt;strong&gt;In-Chat Iterative Loop&lt;/strong&gt;), two failure modes reliably appear in later turns:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Quadratic Prompt Bloat:&lt;/strong&gt; Every file patch emitted on Turn 1, Turn 2, and Turn 3 stays in chat history and gets re-billed on every subsequent turn.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Goal Drift &amp;amp; Late-Turn Laziness:&lt;/strong&gt; By Turn 4, accumulated code diffs crowd out fine-grained specification edge cases from Turn 1, causing the agent to skip boundary checks and write a superficial verification check just to finish.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Anthropic's &lt;strong&gt;Code-Mode Dynamic Workflows&lt;/strong&gt; proposes flipping the control model: instead of iterating step-by-step in chat, have the agent first synthesize a &lt;strong&gt;deterministic Python audit &amp;amp; orchestration script (&lt;code&gt;workflow.py&lt;/code&gt;)&lt;/strong&gt; encoding the full contract, run it in Python at zero token cost, and pass the structured audit output to a fresh worker context to fix all modules in a single batch.&lt;/p&gt;

&lt;p&gt;We tested both architectures head-to-head on a 3-module financial settlement &amp;amp; risk-scoring pipeline (&lt;code&gt;validator.py&lt;/code&gt;, &lt;code&gt;settlement.py&lt;/code&gt;, &lt;code&gt;manifest.py&lt;/code&gt;, plus &lt;code&gt;workflow.py&lt;/code&gt;) inside isolated Proxmox VM &lt;code&gt;VM211&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;ChatLoop&lt;/code&gt;):&lt;/strong&gt; Patches &lt;code&gt;validator.py&lt;/code&gt; (Turn 1), &lt;code&gt;settlement.py&lt;/code&gt; (Turn 2), &lt;code&gt;manifest.py&lt;/code&gt; (Turn 3), and &lt;code&gt;workflow.py&lt;/code&gt; (Turn 4) sequentially inside one persistent chat thread.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;CodeModeWorkflow&lt;/code&gt;):&lt;/strong&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;role_1_workflow_architect&lt;/code&gt; (Turn 1):&lt;/strong&gt; Writes a comprehensive 39-line programmatic audit script (&lt;code&gt;workflow.py&lt;/code&gt;) directly from &lt;code&gt;TASK.md&lt;/code&gt;, which the Python runtime executes immediately to catch initial bugs (&lt;code&gt;audit_caught_initial_bugs: true&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;role_2_code_mode_fixer&lt;/code&gt; (Turn 2, fresh context):&lt;/strong&gt; Receives the locked &lt;code&gt;workflow.py&lt;/code&gt; and fixes all 3 modules in one pass until &lt;code&gt;workflow.py&lt;/code&gt; passes 100%.&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Mode A suffered late-turn &lt;strong&gt;Goal Drift&lt;/strong&gt;, dropping two validation boundaries (&lt;code&gt;18 / 20&lt;/code&gt; on the hidden suite) and writing a lazy 1-case smoke check on Turn 4 (&lt;strong&gt;&lt;code&gt;0 / 6&lt;/code&gt; mutant kill rate&lt;/strong&gt;). Mode B achieved &lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%) accuracy&lt;/strong&gt; and a &lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; (100%) mutant kill rate&lt;/strong&gt;, while cutting &lt;strong&gt;LLM calls by &lt;code&gt;-50.0%&lt;/code&gt; (&lt;code&gt;4 ➔ 2&lt;/code&gt;)&lt;/strong&gt;, &lt;strong&gt;cumulative input tokens by &lt;code&gt;-48.5%&lt;/code&gt; (&lt;code&gt;12,509 ➔ 6,445&lt;/code&gt;)&lt;/strong&gt;, &lt;strong&gt;wall-clock time by &lt;code&gt;-9.5%&lt;/code&gt;&lt;/strong&gt;, and &lt;strong&gt;API cost by &lt;code&gt;-2.1%&lt;/code&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: In-Chat Iterative Loop (&lt;code&gt;ChatLoop&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Code-Mode Dynamic Workflow (&lt;code&gt;CodeModeWorkflow&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (20 Cases)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;18 / 20&lt;/code&gt; (90.0% — &lt;strong&gt;❌ 2 boundary bugs&lt;/strong&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100% Perfect Pass)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+10.0 pt (Zero Goal Drift)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Workflow Mutant Kill Rate (6 Mutants)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 / 6&lt;/code&gt; (0.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; (100.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+100.0 pt audit coverage&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Initial Bugs Caught Deterministically&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No (untested until Turn 4)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Yes (&lt;code&gt;audit_caught_initial_bugs: true&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0-token Python audit gate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4 calls&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2 calls&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-50.0% fewer calls&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;12,509 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6,445 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-48.5% prompt reduction&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,880 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3,003 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+59.7% (invested in rich audit code)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;26.42 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;23.92 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-9.5% faster&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0164&lt;/code&gt; (&lt;code&gt;16,433 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0161&lt;/code&gt; (&lt;code&gt;16,096 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-2.1% cheaper with 100% quality&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Architecture &amp;amp; Turn-by-Turn Token Progression
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM211 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Financial Risk &amp;amp; Settlement Pipeline             |
|               (validator.py ➔ settlement.py ➔ manifest.py)              |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: ChatLoop (Sequential File-by-File Chat Iteration)         |  |
|  |  - Turn 1: patch validator.py   (2,248 in / 425 out)              |  |
|  |  - Turn 2: patch settlement.py  (2,720 in / 814 out)              |  |
|  |  - Turn 3: patch manifest.py    (3,591 in / 303 out)              |  |
|  |  - Turn 4: write workflow.py    (3,950 in / 338 out — lazy check!)|  |
|  |  --&amp;gt; 12,509 in | 18/20 Hidden Cases | 0/6 Mutants Killed          |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: CodeModeWorkflow (Dynamic Script Generation + Fresh Fixer)|  |
|  |  - Turn 1 [Workflow Architect]: Generates 39-line `workflow.py`   |  |
|  |            encoding all boundary &amp;amp; anomaly invariants (2,067 in)  |  |
|  |  - Python Runtime [0 tokens]:   Runs `workflow.py` &amp;amp; locks file   |  |
|  |  - Turn 2 [Code-Mode Fixer]:    Fresh context fixes all 3 modules |  |
|  |            against locked `workflow.py` in 1 batch (4,378 in)     |  |
|  |  --&amp;gt; 6,445 in  | 20/20 Hidden Cases | 6/6 Mutants Killed          |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Code-Level Postmortem: Late-Turn Laziness vs. Programmatic Rigor
&lt;/h2&gt;

&lt;h3&gt;
  
  
  3.1 Mode A (&lt;code&gt;ChatLoop&lt;/code&gt;): Goal Drift &amp;amp; Turn-4 Laziness
&lt;/h3&gt;

&lt;p&gt;By Turn 4 (&lt;code&gt;3,950&lt;/code&gt; input tokens of accumulated patches), Mode A had missed two critical edge cases in its implementation:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;validator_02_empty_and_non_list_rejected&lt;/code&gt;:&lt;/strong&gt; Checked &lt;code&gt;if not isinstance(raw_batches, list):&lt;/code&gt; but forgot to reject empty lists &lt;code&gt;[]&lt;/code&gt; with &lt;code&gt;ValueError&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;settlement_08_invalid_tier_or_bool_reserve_rejected&lt;/code&gt;:&lt;/strong&gt; Silently defaulted unknown tiers like &lt;code&gt;"GOLD"&lt;/code&gt; to &lt;code&gt;rebate = 0&lt;/code&gt; and omitted negative/boolean validation on &lt;code&gt;reserve_cents&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And when asked on Turn 4 to write &lt;code&gt;workflow.py&lt;/code&gt; to verify the pipeline, the fatigued chat context produced a minimal happy-path script that tested zero negative or boundary conditions:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# ❌ Mode A (`ChatLoop`, Turn 4 `workflow.py`): Superficial happy-path check (0/6 mutants killed)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_workflow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"tenant_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"  t-1 "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"tx_count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"amount_cents"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"risk_score"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;75&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"tenant_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"T-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"tx_count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"amount_cents"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"risk_score"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"tenant_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"T-2"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"EU"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"tx_count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"amount_cents"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"risk_score"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;batches&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;validator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;validate_batches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;st&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;settlement&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compute_settlement&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batches&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tier&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"ENTERPRISE"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reserve_cents&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;txt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;manifest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;render_manifest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="s2"&gt;"SETTLEMENT MANIFEST (ENTERPRISE)"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;txt&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="s2"&gt;"REBATE: -"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;txt&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="s2"&gt;"RESERVE: -500"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;txt&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  3.2 Mode B (&lt;code&gt;CodeModeWorkflow&lt;/code&gt;): Full Contract Encoded in Python on Turn 1
&lt;/h3&gt;

&lt;p&gt;In Mode B, &lt;code&gt;role_1_workflow_architect&lt;/code&gt; started with a clean &lt;code&gt;2,067-token&lt;/code&gt; prompt dedicated solely to turning &lt;code&gt;TASK.md&lt;/code&gt; into executable Python assertions:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# ✅ Mode B (`CodeModeWorkflow`, Turn 1 `workflow.py` excerpt — 6/6 mutants killed)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_workflow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;orig&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"tenant_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;" t1 "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;" us "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"tx_count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"amount_cents"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"risk_score"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;70&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"extra"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;orig_copy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;orig&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;validate_batches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;orig&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;orig&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;orig_copy&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="s2"&gt;"extra"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;validate_batches&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;validate_batches&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"not_list"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;val&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tenant_id"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tenant_id"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tenant_id"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"XX"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                   &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tx_count"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tx_count"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"amount_cents"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"amount_cents"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                   &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"risk_score"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"risk_score"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;101&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"risk_score"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;orig_copy&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;val&lt;/span&gt;
        &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;validate_batches&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;compute_settlement&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"INVALID"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;compute_settlement&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"STANDARD"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;compute_settlement&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"STANDARD"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# Exact boundary checks: risk_score 69 vs 70, anomaly |r1-r2| 50 vs 51, ENTERPRISE 99999 vs 100000, REBATE: -0&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  4. Mutation Audit (&lt;code&gt;M1&lt;/code&gt;–&lt;code&gt;M6&lt;/code&gt;) Summary (&lt;code&gt;labjudge07&lt;/code&gt;)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mutant ID&lt;/th&gt;
&lt;th&gt;Boundary Mutation Injected&lt;/th&gt;
&lt;th&gt;Mode A (&lt;code&gt;ChatLoop&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B (&lt;code&gt;CodeModeWorkflow&lt;/code&gt;)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M1&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Risk surcharge &lt;code&gt;&amp;gt;= 70&lt;/code&gt; mutated to strict &lt;code&gt;&amp;gt; 70&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M2&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ENTERPRISE&lt;/code&gt; rebate &lt;code&gt;&amp;gt;= 100000&lt;/code&gt; mutated to &lt;code&gt;&amp;gt; 100000&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M3&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;isinstance(tx, bool)&lt;/code&gt; check removed from &lt;code&gt;tx_count&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M4&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Zero-amount batch (&lt;code&gt;amount_cents == 0&lt;/code&gt;) rejected&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M5&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Risk anomaly check (&lt;code&gt;abs(r1 - r2) &amp;gt; 50&lt;/code&gt;) omitted&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M6&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Zero rebate formatted as &lt;code&gt;REBATE: 0&lt;/code&gt; instead of &lt;code&gt;REBATE: -0&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ Survived&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;6 Boundary Mutants&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 / 6&lt;/code&gt; (0.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; (100.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: Offload Multi-Module Audits From Chat to Code
&lt;/h2&gt;

&lt;p&gt;Whenever an agent task touches &lt;strong&gt;3+ files or repeated cross-file checks&lt;/strong&gt;:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Have the agent write a Python audit/orchestration script on Turn 1&lt;/strong&gt; instead of stepping through files manually inside chat history.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pass the script's structured output—not the prior chat log—to a fresh fixer context&lt;/strong&gt;, cutting input tokens nearly in half while eliminating late-turn Goal Drift.&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🧪 &lt;strong&gt;Previous (&lt;code&gt;E06&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e06-blind-tdd-worktrees" rel="noopener noreferrer"&gt;Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;📐 &lt;strong&gt;Next (&lt;code&gt;E08&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e08-spec-first-sdd" rel="noopener noreferrer"&gt;Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9%&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>python</category>
      <category>automation</category>
    </item>
    <item>
      <title>Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Tue, 17 Nov 2026 00:15:05 +0000</pubDate>
      <link>https://dev.to/tappers_/why-ai-agents-write-weak-tests-for-their-own-code-single-agent-tdd-vs-two-agent-blind-tdd-0-vs-1fah</link>
      <guid>https://dev.to/tappers_/why-ai-agents-write-weak-tests-for-their-own-code-single-agent-tdd-vs-two-agent-blind-tdd-0-vs-1fah</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E06&lt;/code&gt; (Verification &amp;amp; Mutation Testing)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E06&lt;/code&gt; (Backlog Candidate &lt;code&gt;C003&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e06-vm210-20261028-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM210&lt;/code&gt; (&lt;code&gt;ai-e06-210&lt;/code&gt;, 0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, &lt;code&gt;systemd&lt;/code&gt; UID separation)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt; (&lt;code&gt;thinkingLevel: LOW&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; &lt;em&gt;"Two Agents, Two Worktrees: Blind Tester Writes the Tests, Implementer Makes Them Pass"&lt;/em&gt; — &lt;strong&gt;Single-Context Self-TDD vs. Two-Agent Blind Worktree TDD&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;Telling a coding agent &lt;em&gt;"Use TDD—write unit tests and fix the code"&lt;/em&gt; inside a single conversation has a fatal flaw: &lt;strong&gt;Self-Validation Blindness&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;When the exact same LLM context sees the existing buggy code and writes both &lt;code&gt;test_suite.py&lt;/code&gt; and the production modules, the model unconsciously anchors on its own implementation assumptions—writing a weak, happy-path-only test that rubber-stamps its own bugs.&lt;/p&gt;

&lt;p&gt;To measure this effect rigorously, we benchmarked a 3-module SaaS subscription proration pipeline (&lt;code&gt;normalizer.py&lt;/code&gt;, &lt;code&gt;proration.py&lt;/code&gt;, &lt;code&gt;invoice.py&lt;/code&gt;) inside isolated VM &lt;code&gt;VM210&lt;/code&gt;, evaluating both &lt;strong&gt;hidden 20-case functional accuracy&lt;/strong&gt; and &lt;strong&gt;Mutation Testing (&lt;code&gt;M1&lt;/code&gt;–&lt;code&gt;M6&lt;/code&gt; boundary mutant kill rate)&lt;/strong&gt; on the agent-authored &lt;code&gt;test_suite.py&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;SingleContextTddRunner&lt;/code&gt;):&lt;/strong&gt; One agent receives &lt;code&gt;TASK.md&lt;/code&gt;, the 3 buggy implementation files, and &lt;code&gt;test_suite.py&lt;/code&gt;, updating both tests and code together until its own &lt;code&gt;test_suite.py&lt;/code&gt; passes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;BlindWorktreeTddRunner&lt;/code&gt;):&lt;/strong&gt; Physically splits the task across two isolated workspaces:

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Worktree 1 (&lt;code&gt;role_1_blind_tester&lt;/code&gt;):&lt;/strong&gt; Sees &lt;strong&gt;only &lt;code&gt;TASK.md&lt;/code&gt;&lt;/strong&gt; (zero bytes of implementation code) and writes &lt;code&gt;test_suite.py&lt;/code&gt;. The harness verifies that &lt;code&gt;test_suite.py&lt;/code&gt; fails against the initial buggy code (&lt;strong&gt;TDD Red Phase&lt;/strong&gt;), then locks &lt;code&gt;test_suite.py&lt;/code&gt; read-only.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Worktree 2 (&lt;code&gt;role_2_implementer&lt;/code&gt;):&lt;/strong&gt; Receives the locked &lt;code&gt;test_suite.py&lt;/code&gt; and edits only the 3 production modules until all assertions pass (&lt;strong&gt;TDD Green Phase&lt;/strong&gt;).&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Mode A wrote a 28-line happy-path test that &lt;strong&gt;killed &lt;code&gt;0 / 6 (0.0%)&lt;/code&gt; boundary mutants&lt;/strong&gt;, declared 100% success, and &lt;strong&gt;leaked a type-coercion bug (&lt;code&gt;19 / 20&lt;/code&gt; on the hidden suite)&lt;/strong&gt;. Mode B's Blind Tester wrote a ruthless 52-line specification test suite that &lt;strong&gt;killed &lt;code&gt;6 / 6 (100.0%)&lt;/code&gt; mutants&lt;/strong&gt;, forcing the Implementer to pass &lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt; on the hidden grading suite for less than one extra cent (&lt;code&gt;+$0.0098&lt;/code&gt;).&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Single-Context Self-TDD (&lt;code&gt;SingleContextTDD&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Two-Agent Blind TDD (&lt;code&gt;BlindWorktreeTDD&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (20 Cases)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;19 / 20&lt;/code&gt; (95.0% — &lt;strong&gt;❌ leaked bug&lt;/strong&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100% Perfect Pass)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+5.0 pt (Zero leaked defects)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Test Suite Mutant Kill Rate (6 Mutants)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 / 6&lt;/code&gt; (0.0% — all 6 survived!)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; (100.0% — all 6 killed!)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+100.0 pt mutation coverage&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;TDD Red Phase Verified on Initial Bugs&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No (simultaneous edit)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Yes (&lt;code&gt;red_phase_caught_initial_bugs: true&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;True Red ➔ Green enforcement&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Agent's Self-Reported Test Status&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;PASS&lt;/code&gt; &lt;em&gt;(False Positive!)&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;&lt;code&gt;PASS&lt;/code&gt; *(Verified True Positive)&lt;/strong&gt;*&lt;/td&gt;
&lt;td&gt;Eliminated rubber-stamp green&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1 call&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2 calls&lt;/code&gt; (1 Blind Tester + 1 Implementer)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+1 call for independent verification&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,035 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;7,493 tokens&lt;/code&gt; (&lt;code&gt;2,448&lt;/code&gt; + &lt;code&gt;5,045&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Lean 2-worktree isolation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;14.70 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;25.61 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;+10.91 sec for 100% defect-free code&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0080&lt;/code&gt; (&lt;code&gt;7,965 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0178&lt;/code&gt; (&lt;code&gt;17,835 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+$0.0098 (&amp;lt; 1 cent) for 100% kill rate&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Architecture Comparison: Self-TDD Collusion vs. Blind Worktree Isolation
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM210 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Subscription Proration Pipeline                  |
|               (normalizer.py ➔ proration.py ➔ invoice.py)               |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SingleContextTddRunner (Same Agent Writes Tests &amp;amp; Code)   |  |
|  |  - Sees existing buggy `str(item.get("account_id"))`              |  |
|  |  - Writes 1 happy-path test + misses non-string rejection bug     |  |
|  |  - Self-test: PASS! ➔ Hidden Judge: 19/20 FAIL | Mutants: 0/6     |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: BlindWorktreeTddRunner (2 Worktrees, Physically Split)    |  |
|  |  - Worktree 1 [Blind Tester]: Sees ONLY TASK.md (0 impl files!)   |  |
|  |      Writes 52-line adversarial `test_suite.py`                   |  |
|  |      Red Phase Check: FAILS on initial buggy code (Verified!)     |  |
|  |  - Worktree 2 [Implementer]:  `test_suite.py` LOCKED read-only;   |  |
|  |      Patches normalizer/proration/invoice until Green             |  |
|  |  - Hidden Judge: 20/20 PASS (100%) | Mutants Killed: 6/6 (100%)   |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Code-Level Postmortem: Implementation Anchoring Caught on Tape
&lt;/h2&gt;

&lt;p&gt;Comparing &lt;code&gt;single-bundle.json&lt;/code&gt; (Mode A) and &lt;code&gt;blind-bundle.json&lt;/code&gt; (Mode B) demonstrates &lt;strong&gt;Implementation Anchoring&lt;/strong&gt; in action.&lt;/p&gt;

&lt;p&gt;In the initial buggy &lt;code&gt;normalizer.py&lt;/code&gt;, account IDs were coerced using &lt;code&gt;str(...)&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Initial buggy fixture in normalizer.py:&lt;/span&gt;
&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"account_id"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"account_id"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because &lt;strong&gt;Mode A&lt;/strong&gt; saw that line in its prompt, it carried the &lt;code&gt;str(...)&lt;/code&gt; coercion habit straight into its "fixed" implementation—silently converting integer &lt;code&gt;account_id: 999&lt;/code&gt; into &lt;code&gt;"999"&lt;/code&gt; instead of raising &lt;code&gt;ValueError&lt;/code&gt; as required by &lt;code&gt;TASK.md&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# ❌ Mode A (`SingleContextTDD`): Anchored on str() coercion &amp;amp; wrote 0 negative tests!&lt;/span&gt;
&lt;span class="n"&gt;acc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"account_id"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;upper&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Worse still, Mode A's &lt;code&gt;test_suite.py&lt;/code&gt; tested only a single valid string &lt;code&gt;"acc-1"&lt;/code&gt; with a &lt;code&gt;SAVE10&lt;/code&gt; coupon—never testing non-string IDs, &lt;code&gt;bool&lt;/code&gt; values, &lt;code&gt;9,999&lt;/code&gt; vs. &lt;code&gt;10,000&lt;/code&gt; discount boundaries, or &lt;code&gt;DISCOUNT: -0&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;In contrast, &lt;strong&gt;Mode B's Blind Tester&lt;/strong&gt; never saw the &lt;code&gt;str(...)&lt;/code&gt; bug. Reading &lt;code&gt;TASK.md&lt;/code&gt; cold, it immediately built a table-driven adversarial helper &lt;code&gt;bad(fn, *a, **k)&lt;/code&gt; probing booleans (&lt;code&gt;[True, False, 1.5]&lt;/code&gt;), integer account IDs, &lt;code&gt;days_active&lt;/code&gt; bounds (&lt;code&gt;0&lt;/code&gt; and &lt;code&gt;31&lt;/code&gt;), &lt;code&gt;unit_price == 0&lt;/code&gt; free trials, exact &lt;code&gt;9,999&lt;/code&gt; vs. &lt;code&gt;10,000&lt;/code&gt; thresholds, compounding &lt;code&gt;SAVE10&lt;/code&gt;/&lt;code&gt;VIP20&lt;/code&gt; order, and &lt;code&gt;DISCOUNT: -0&lt;/code&gt;—forcing Worktree 2's Implementer to enforce strict &lt;code&gt;isinstance(account_id, str)&lt;/code&gt; checks.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Mutation Testing Audit (&lt;code&gt;M1&lt;/code&gt;–&lt;code&gt;M6&lt;/code&gt;) &amp;amp; 20-Case Grading Breakdown
&lt;/h2&gt;

&lt;p&gt;Our hidden evaluator (&lt;code&gt;labjudge06&lt;/code&gt;) injected 6 subtle boundary bugs (&lt;strong&gt;Mutants &lt;code&gt;M1&lt;/code&gt;–&lt;code&gt;M6&lt;/code&gt;&lt;/strong&gt;) into a reference implementation to measure whether each agent's &lt;code&gt;test_suite.py&lt;/code&gt; could actually catch regressions:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mutant ID&lt;/th&gt;
&lt;th&gt;Injected Edge-Case Bug&lt;/th&gt;
&lt;th&gt;Mode A &lt;code&gt;test_suite.py&lt;/code&gt; (&lt;code&gt;Single TDD&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B &lt;code&gt;test_suite.py&lt;/code&gt; (&lt;code&gt;Blind TDD&lt;/code&gt;)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M1&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Off-by-one volume discount (&lt;code&gt;&amp;gt; 10000&lt;/code&gt; instead of &lt;code&gt;&amp;gt;= 10000&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt; &lt;em&gt;(only tested 9,000)&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; &lt;em&gt;(tested 9,999 &amp;amp; 10,000)&lt;/em&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M2&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Coupon applied to &lt;code&gt;gross_subtotal&lt;/code&gt; instead of &lt;code&gt;after_volume&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; &lt;em&gt;(verified 10,000 + &lt;code&gt;SAVE10&lt;/code&gt; = 900)&lt;/em&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M3&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Python &lt;code&gt;bool&lt;/code&gt; (&lt;code&gt;seats=True&lt;/code&gt;) accepted as integer &lt;code&gt;1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; &lt;em&gt;(tested &lt;code&gt;[True, False, 1.5]&lt;/code&gt;)&lt;/em&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M4&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Free trial (&lt;code&gt;unit_price == 0&lt;/code&gt;) erroneously rejected&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; &lt;em&gt;(verified &lt;code&gt;unit_price: 0&lt;/code&gt;)&lt;/em&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M5&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Upper bound &lt;code&gt;days_active == 31&lt;/code&gt; (&amp;gt; 30 max) accepted&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; &lt;em&gt;(verified &lt;code&gt;0&lt;/code&gt; and &lt;code&gt;31&lt;/code&gt; raise &lt;code&gt;ValueError&lt;/code&gt;)&lt;/em&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;M6&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Zero discount formatted as &lt;code&gt;DISCOUNT: 0&lt;/code&gt; instead of &lt;code&gt;-0&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌ &lt;strong&gt;Survived&lt;/strong&gt; &lt;em&gt;(only tested &lt;code&gt;-1400&lt;/code&gt;)&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;✅ &lt;strong&gt;Killed&lt;/strong&gt; &lt;em&gt;(explicitly asserted &lt;code&gt;DISCOUNT: -0&lt;/code&gt;)&lt;/em&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Mutation Kill Score&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 / 6&lt;/code&gt; (0.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; (100.0%)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the &lt;strong&gt;20 hidden grading cases&lt;/strong&gt;, Mode A failed &lt;code&gt;normalizer_03_blank_account_or_unknown_plan_rejected&lt;/code&gt; (&lt;code&gt;19 / 20&lt;/code&gt;), while Mode B passed all &lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: Fold Implementation (&lt;code&gt;E05&lt;/code&gt;), Isolate Verification (&lt;code&gt;E06&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;Together, &lt;strong&gt;&lt;code&gt;E05&lt;/code&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;E06&lt;/code&gt;&lt;/strong&gt; resolve the biggest debate in multi-agent engineering:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Never split implementation roles&lt;/strong&gt; (&lt;code&gt;Architect ➔ Coder A ➔ Coder B&lt;/code&gt;) via lossy text summaries (&lt;code&gt;E05&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Always isolate the Test Author&lt;/strong&gt; into a &lt;strong&gt;Blind Worktree&lt;/strong&gt; that sees &lt;em&gt;only&lt;/em&gt; the specification (&lt;code&gt;SPEC.md&lt;/code&gt; / issue contract) and zero lines of implementation (&lt;code&gt;E06&lt;/code&gt;), then lock the resulting test files read-only before launching your folded implementation loop.&lt;/li&gt;
&lt;/ul&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🪆 &lt;strong&gt;Previous (&lt;code&gt;E05&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e05-subagent-folding" rel="noopener noreferrer"&gt;Why Role-Based Subagents Slow Down Coding Agents: Subagent Folding + Progressive Skill Disclosure (2.66x Faster)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;⚙️ &lt;strong&gt;Next (&lt;code&gt;E07&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e07-code-mode-workflows" rel="noopener noreferrer"&gt;Eliminating Goal Drift in Multi-File Coding Agents: In-Chat Loops vs. Code-Mode Dynamic Workflows&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>tdd</category>
      <category>testing</category>
    </item>
    <item>
      <title>Why Role-Based Subagents Slow Down Coding Agents: Subagent Folding + Progressive Skill Disclosure (2.66x Faster)</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Mon, 16 Nov 2026 12:51:36 +0000</pubDate>
      <link>https://dev.to/tappers_/why-role-based-subagents-slow-down-coding-agents-subagent-folding-progressive-skill-disclosure-59bf</link>
      <guid>https://dev.to/tappers_/why-role-based-subagents-slow-down-coding-agents-subagent-folding-progressive-skill-disclosure-59bf</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E05&lt;/code&gt; (Multi-Agent vs. Folded Loop)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E05&lt;/code&gt; (Backlog Candidate &lt;code&gt;C002&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e05-vm209-20261028-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM209&lt;/code&gt; (&lt;code&gt;ai-e05-209&lt;/code&gt;, 0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, &lt;code&gt;systemd&lt;/code&gt; UID separation)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt; (&lt;code&gt;thinkingLevel: LOW&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; Anthropic Engineering (&lt;em&gt;"Building agents: When to fold subagents into your main agent loop"&lt;/em&gt;, Diane Sarkis) — &lt;strong&gt;Role-Based Subagent Swarm vs. Single Folded Loop + Progressive Skill Disclosure&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;When developers try to scale an AI coding agent, a tempting pattern is to mirror a human org chart: split the workflow into &lt;strong&gt;Role-Based Subagents (&lt;code&gt;Architect&lt;/code&gt; ➔ &lt;code&gt;Core Implementer&lt;/code&gt; ➔ &lt;code&gt;Ledger Implementer&lt;/code&gt; ➔ &lt;code&gt;QA Fixer&lt;/code&gt;)&lt;/strong&gt; and pass natural-language handoff summaries down the chain.&lt;/p&gt;

&lt;p&gt;On tightly coupled codebases, this creates three severe bottlenecks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The "Telephone Game" Effect (Handoff Context Loss):&lt;/strong&gt; When an Architect's 1,400-character plan is compressed or summarized for downstream workers, subtle edge-case invariants (like formatting zero discounts as &lt;code&gt;-0&lt;/code&gt; instead of &lt;code&gt;0&lt;/code&gt;) get dropped.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Downstream Rework Loops:&lt;/strong&gt; Implementers guess at missing details, causing integration failures that force QA Fixer retries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Static Skill Preloading Bloat:&lt;/strong&gt; Because the router doesn't know which domain rules each subagent will need, all company skill docs (SOAP/XML, Kubernetes, GDPR, UI tokens, SaaS billing) get stuffed into subagent system prompts upfront.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;To test Anthropic's &lt;strong&gt;Subagent Folding + Progressive Skill Disclosure&lt;/strong&gt; architecture, we benchmarked a 3-module SaaS usage-billing pipeline (&lt;code&gt;normalizer.py&lt;/code&gt;, &lt;code&gt;billing.py&lt;/code&gt;, &lt;code&gt;ledger.py&lt;/code&gt;) inside isolated VM &lt;code&gt;VM209&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;RoleSubagentSwarmRunner&lt;/code&gt;):&lt;/strong&gt; Splits the task across 4 role-based subagents with 3 handoffs and preloads all 5 internal company skill documents into the prompt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;FoldedSkillLoopRunner&lt;/code&gt;):&lt;/strong&gt; Folds planning, coding, and verification into &lt;strong&gt;one main agent loop&lt;/strong&gt; with zero handoffs. The initial prompt shows only a &lt;strong&gt;5-line Skill Catalog Index&lt;/strong&gt;; the agent dynamically calls &lt;code&gt;load_skill("saas_billing_spec")&lt;/code&gt; on Turn 1 and patches all 3 files in one coherent pass on Turn 2.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Mode A suffered a classic &lt;strong&gt;Telephone Game bug&lt;/strong&gt;: handoff truncation dropped the &lt;code&gt;DISCOUNT: -0&lt;/code&gt; formatting invariant, causing &lt;code&gt;ledger.py&lt;/code&gt; to fail QA and trigger an extra rework turn (&lt;code&gt;5&lt;/code&gt; LLM calls, &lt;code&gt;36.36s&lt;/code&gt;). Mode B completed with &lt;strong&gt;zero handoffs and zero rework in 2 calls (&lt;code&gt;13.68s&lt;/code&gt;)&lt;/strong&gt;—running &lt;strong&gt;&lt;code&gt;2.66x&lt;/code&gt; faster (&lt;code&gt;-62.4%&lt;/code&gt; time)&lt;/strong&gt; while cutting &lt;strong&gt;input tokens by &lt;code&gt;-67.0%&lt;/code&gt; (&lt;code&gt;14,815 ➔ 4,889&lt;/code&gt;)&lt;/strong&gt;, &lt;strong&gt;output tokens by &lt;code&gt;-39.9%&lt;/code&gt; (&lt;code&gt;2,397 ➔ 1,440&lt;/code&gt;)&lt;/strong&gt;, and &lt;strong&gt;API cost by &lt;code&gt;-54.9%&lt;/code&gt; (&lt;code&gt;$0.0201 ➔ $0.0091&lt;/code&gt;)&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Role-Based Subagents (&lt;code&gt;RoleSubagentSwarm&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Folded Loop + Skill Disclosure (&lt;code&gt;FoldedSkillLoop&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (20 Cases)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;20 / 20&lt;/code&gt; (100% &lt;em&gt;after rework&lt;/em&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100% &lt;em&gt;1st-pass&lt;/em&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Zero rework needed&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Subagent Handoffs&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;3 handoffs&lt;/code&gt; (with summary loss)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 handoffs&lt;/code&gt; (single shared context)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Eliminated Telephone Game loss&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;QA Failure Rework Retries (&lt;code&gt;failed_check_retries&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1 retry&lt;/code&gt; (&lt;code&gt;DISCOUNT: -0&lt;/code&gt; bug)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 retries&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-100% rework&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5 calls&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2 calls&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-60.0% fewer calls&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total File Edits (&lt;code&gt;file_edits&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;4 edits&lt;/code&gt; (&lt;code&gt;ledger.py&lt;/code&gt; edited twice)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3 edits&lt;/code&gt; (1 per file)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-25.0% fewer edits&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 1 Input Tokens&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;4,095 tokens&lt;/code&gt; (5 skills preloaded)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,220 tokens&lt;/code&gt; (5-line index only)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-70.2% initial prompt bloat&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;14,815 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;4,889 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-67.0% reduction (~3x leaner)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,397 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,440 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-39.9% (no handoff prose or rework)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;36.36 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;13.68 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-62.4% (2.66x faster)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0201&lt;/code&gt; (&lt;code&gt;20,101 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0091&lt;/code&gt; (&lt;code&gt;9,067 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-54.9% cost reduction&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Architecture Comparison: Role Handoffs vs. Folded Skill Loop
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM209 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module SaaS Usage Billing Pipeline                      |
|               (normalizer.py ➔ billing.py ➔ ledger.py)                  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: RoleSubagentSwarmRunner (4 Roles + 5 Preloaded Skills)    |  |
|  |  - Turn 1 [Architect]:   Writes 1,437-char plan ➔ truncated (220c)|  |
|  |  - Turn 2 [Impl Core]:   Patches normalizer.py &amp;amp; billing.py       |  |
|  |  - Turn 3 [Impl Ledger]: Patches ledger.py (misses `-0` rule!)    |  |
|  |  - Turn 4 [QA Fixer]:    run_tests ➔ FAILS (`zero_discount_-0`)   |  |
|  |  - Turn 5 [QA Fixer]:    Re-patches ledger.py ➔ PASS &amp;amp; SUBMIT     |  |
|  |  --&amp;gt; 5 calls | 14,815 in | 2,397 out | 36.36s                     |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: FoldedSkillLoopRunner (Single Loop + Progressive Skills)  |  |
|  |  - Turn 1: Sees 5-line Skill Index ➔ calls `load_skill` for only  |  |
|  |            `saas_billing_spec` (1,220 in / 21 out)                |  |
|  |  - Turn 2: Patches all 3 modules coherently in one context        |  |
|  |            + passes verifier on 1st try ➔ AUTO_SUBMIT (3,669 in)  |  |
|  |  --&amp;gt; 2 calls | 4,889 in  | 1,440 out | 13.68s (2.66x faster!)     |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Code-Level Postmortem: Catching the "Telephone Game" Bug in the Act
&lt;/h2&gt;

&lt;p&gt;Inspecting &lt;code&gt;actions.json&lt;/code&gt; and &lt;code&gt;execution-traces.json&lt;/code&gt; from &lt;code&gt;VM209&lt;/code&gt; shows exactly how role-based handoffs break coupled invariants:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;On Turn 1 (&lt;code&gt;seq 0&lt;/code&gt;), &lt;code&gt;role_1_architect&lt;/code&gt;&lt;/strong&gt; read all files and all 5 preloaded skill docs (&lt;code&gt;4,095&lt;/code&gt; input tokens) and authored a detailed &lt;code&gt;1,437-character&lt;/code&gt; &lt;code&gt;handoff_plan&lt;/code&gt;. However, the router's summary budget (&lt;code&gt;220 chars&lt;/code&gt;) truncated the tail of the note—cutting off the &lt;code&gt;ledger.py&lt;/code&gt; formatting clause requiring &lt;code&gt;DISCOUNT: -0&lt;/code&gt; when the discount is zero.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On Turn 3 (&lt;code&gt;seq 2&lt;/code&gt;), &lt;code&gt;role_3_implementer_ledger&lt;/code&gt;&lt;/strong&gt; received only the truncated handoff summary and guessed a "reasonable" Python conditional:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# ❌ Mode A — Turn 3 (`role_3_implementer_ledger`): Guessed from truncated handoff!&lt;/span&gt;
&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"DISCOUNT: -{disc}"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;disc&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"DISCOUNT: {disc}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;On Turn 4 (&lt;code&gt;seq 3&lt;/code&gt;), &lt;code&gt;role_4_qa_fixer&lt;/code&gt;&lt;/strong&gt; ran integration tests (&lt;code&gt;3,293&lt;/code&gt; input tokens) and immediately hit &lt;code&gt;{"error": "zero_discount_must_format_as_-0", "passed": false}&lt;/code&gt;, forcing &lt;strong&gt;Turn 5 (&lt;code&gt;seq 4&lt;/code&gt;, &lt;code&gt;4,213&lt;/code&gt; input tokens)&lt;/strong&gt; to re-emit &lt;code&gt;ledger.py&lt;/code&gt; from scratch.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;In &lt;strong&gt;Mode B (&lt;code&gt;FoldedSkillLoop&lt;/code&gt;)&lt;/strong&gt;, the single agent loaded &lt;code&gt;saas_billing_spec&lt;/code&gt; directly into its working context (&lt;code&gt;1,220&lt;/code&gt; tokens on Turn 1) without loading the 4 irrelevant enterprise skills (SOAP/XML, K8s Helm, GDPR logging, React UI tokens), and wrote &lt;code&gt;f"DISCOUNT: -{disc}"&lt;/code&gt; accurately on the very first pass.&lt;/p&gt;

&lt;h3&gt;
  
  
  Turn-by-Turn Token Breakdown (&lt;code&gt;usage.json&lt;/code&gt;)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th&gt;Seq&lt;/th&gt;
&lt;th&gt;Role / Action&lt;/th&gt;
&lt;th&gt;Input Tokens&lt;/th&gt;
&lt;th&gt;Output Tokens&lt;/th&gt;
&lt;th&gt;Cost (&lt;code&gt;micro-USD&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;What Happened&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;role_1_architect&lt;/code&gt;: &lt;code&gt;handoff_plan&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,095&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;412&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,617&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;5 skills preloaded; 1,437-char plan truncated&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;role_2_implementer_core&lt;/code&gt;: &lt;code&gt;patch_core&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,992&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,228&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;6,099&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Implements &lt;code&gt;normalizer.py&lt;/code&gt; &amp;amp; &lt;code&gt;billing.py&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;role_3_implementer_ledger&lt;/code&gt;: &lt;code&gt;patch_ledger&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,222&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;402&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,424&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Handoff loss:&lt;/strong&gt; writes &lt;code&gt;DISCOUNT: 0&lt;/code&gt; bug&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;role_4_qa_fixer&lt;/code&gt;: &lt;code&gt;run_tests&lt;/code&gt; (&lt;strong&gt;FAIL&lt;/strong&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3,293&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,500&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Catches &lt;code&gt;zero_discount_must_format_as_-0&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;role_4_qa_fixer&lt;/code&gt;: &lt;code&gt;fix_and_submit&lt;/code&gt; (&lt;strong&gt;PASS&lt;/strong&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,213&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;347&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,461&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Re-patches &lt;code&gt;ledger.py&lt;/code&gt; to recover&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;em&gt;Mode A Sum&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;em&gt;5 calls (3 handoffs, 1 rework)&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;14,815&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;2,397&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;20,101&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;Extra output spent on prose + duplicate patch&lt;/em&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;folded_turn_1&lt;/code&gt;: &lt;code&gt;load_skill("saas_billing_spec")&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,220&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;21&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;994&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Loads only the 1 relevant skill from 5-line index&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;6&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;folded_turn_2&lt;/code&gt;: &lt;code&gt;patch_pipeline&lt;/code&gt; ➔ &lt;code&gt;AUTO_SUBMIT&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3,669&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,419&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8,073&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Patches all 3 modules in 1 shot; 0 rework&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;em&gt;Mode B Sum&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;em&gt;2 calls (0 handoffs, 0 rework)&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;4,889&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;1,440&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;9,067&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;2.66x faster, -54.9% cheaper&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  4. All 20 Hidden Grading Suite Results (&lt;code&gt;labjudge05&lt;/code&gt;)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Test Case IDs&lt;/th&gt;
&lt;th&gt;Invariants Verified&lt;/th&gt;
&lt;th&gt;Mode A (&lt;code&gt;Role Swarm&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B (&lt;code&gt;Folded Loop&lt;/code&gt;)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Stage 1 (&lt;code&gt;normalizer.py&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;normalizer_01&lt;/code&gt; to &lt;code&gt;normalizer_06&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Trim/uppercase &lt;code&gt;tenant_id&lt;/code&gt; &amp;amp; &lt;code&gt;metric&lt;/code&gt;, allow only &lt;code&gt;API_CALLS&lt;/code&gt;/&lt;code&gt;STORAGE_GB&lt;/code&gt;, reject &lt;code&gt;units &amp;lt;= 0&lt;/code&gt; &amp;amp; &lt;code&gt;bool&lt;/code&gt;/&lt;code&gt;float&lt;/code&gt;, non-mutation&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;6 / 6&lt;/code&gt; PASS&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; PASS&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Stage 2 (&lt;code&gt;billing.py&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;billing_01&lt;/code&gt; to &lt;code&gt;billing_08&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;First-seen order, rate conflict check, &lt;code&gt;9999 vs 10000&lt;/code&gt; &lt;code&gt;PRO&lt;/code&gt;/&lt;code&gt;ENTERPRISE&lt;/code&gt; tier discount before &lt;code&gt;credit_note&lt;/code&gt;, floor &lt;code&gt;// 10&lt;/code&gt;, credit cap&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;8 / 8&lt;/code&gt; PASS&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;8 / 8&lt;/code&gt; PASS&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Stage 3 &amp;amp; E2E (&lt;code&gt;ledger.py&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;e2e_01&lt;/code&gt; to &lt;code&gt;e2e_06&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;&lt;code&gt;DISCOUNT: -0&lt;/code&gt; signed zero&lt;/strong&gt;, aggregated units, free-tier ledger, floor tax E2E, line order, error propagation&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;6 / 6&lt;/code&gt; PASS &lt;em&gt;(fixed on Turn 5)&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; PASS *(1st try)&lt;/strong&gt;*&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;20 Hidden Cases&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: When to Split (&lt;code&gt;E04&lt;/code&gt;) vs. When to Fold (&lt;code&gt;E05&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;Combining &lt;strong&gt;&lt;code&gt;E04&lt;/code&gt; (Loops vs. Graphs)&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;E05&lt;/code&gt; (Subagent Folding)&lt;/strong&gt; gives a crisp architectural rule for multi-agent design:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Split by Module + Deterministic Verifier (&lt;code&gt;E04&lt;/code&gt;):&lt;/strong&gt; Only isolate steps into separate contexts when each step produces a &lt;strong&gt;code artifact verified by a deterministic test suite&lt;/strong&gt; and has a fixed interface contract.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fold Sequential Roles (&lt;code&gt;E05&lt;/code&gt;):&lt;/strong&gt; Never split &lt;em&gt;Architect ➔ Coder ➔ Reviewer&lt;/em&gt; across separate subagents connected by natural-language summaries on coupled files. Keep them in &lt;strong&gt;one folded loop&lt;/strong&gt;, and replace bloated system prompts with &lt;strong&gt;Progressive Skill Disclosure&lt;/strong&gt; (a 5-to-10 line index + on-demand &lt;code&gt;load_skill&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🕸️ &lt;strong&gt;Previous (&lt;code&gt;E04&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e04-loops-vs-graphs" rel="noopener noreferrer"&gt;Single Loops vs. Directed Graphs for AI Coding Agents: Cutting Input Tokens by 74.4% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🧪 &lt;strong&gt;Next (&lt;code&gt;E06&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e06-blind-tdd-worktrees" rel="noopener noreferrer"&gt;Why AI Agents Write Weak Tests for Their Own Code: Single-Agent TDD vs. Two-Agent Blind TDD (0% vs. 100% Mutant Kill Rate)&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>contextengineering</category>
      <category>multiagent</category>
    </item>
    <item>
      <title>Single Loops vs. Directed Graphs for AI Coding Agents: Cutting Input Tokens by 74.4% Without Losing Accuracy</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Mon, 16 Nov 2026 12:51:30 +0000</pubDate>
      <link>https://dev.to/tappers_/single-loops-vs-directed-graphs-for-ai-coding-agents-cutting-input-tokens-by-744-without-losing-19jp</link>
      <guid>https://dev.to/tappers_/single-loops-vs-directed-graphs-for-ai-coding-agents-cutting-input-tokens-by-744-without-losing-19jp</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E04&lt;/code&gt; (Orchestration &amp;amp; Control Flow)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E04&lt;/code&gt; (Backlog Candidate &lt;code&gt;C001&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e04-vm208-20261027-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM208&lt;/code&gt; (&lt;code&gt;ai-e04-208&lt;/code&gt;, 0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, &lt;code&gt;systemd&lt;/code&gt; UID separation)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt; (&lt;code&gt;thinkingLevel: LOW&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; &lt;em&gt;"Loops and Graphs: How to Stop Babysitting Agents and Only Approve the Last Step"&lt;/em&gt; (@hanakoxbt) — &lt;strong&gt;Single Free-Form ReAct Loop vs. Stage-Gated Directed Graph&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;When building an AI coding agent, the default instinct is to give a single free-form &lt;strong&gt;ReAct Loop (&lt;code&gt;Single Loop&lt;/code&gt;)&lt;/strong&gt; access to every tool (&lt;code&gt;inspect_all&lt;/code&gt;, &lt;code&gt;edit_file&lt;/code&gt;, &lt;code&gt;run_tests&lt;/code&gt;, &lt;code&gt;SUBMIT&lt;/code&gt;) and let the LLM figure out what to inspect, which file to patch next, when to run tests, and when to stop.&lt;/p&gt;

&lt;p&gt;On multi-module tasks, that design suffers from two compounding taxes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Orchestration Overhead:&lt;/strong&gt; Over half of the LLM calls are spent deciding &lt;em&gt;how to sequence steps&lt;/em&gt; rather than writing code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Snowballing Context History:&lt;/strong&gt; Every file read, test log, and previously generated code patch stays in the chat history, inflating prompt tokens on every subsequent turn.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Inspired by the &lt;em&gt;"Loops and Graphs"&lt;/em&gt; pattern—which separates &lt;strong&gt;inner local Generator + Verifier loops&lt;/strong&gt; from an &lt;strong&gt;outer deterministic Directed Graph&lt;/strong&gt;—we ran a controlled A/B experiment on a 3-module order &amp;amp; invoice pipeline (&lt;code&gt;parser.py&lt;/code&gt;, &lt;code&gt;pricing.py&lt;/code&gt;, &lt;code&gt;report.py&lt;/code&gt;) containing 6 coupled bugs inside isolated VM &lt;code&gt;VM208&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;SingleLoopRunner&lt;/code&gt;):&lt;/strong&gt; A single free-form agent loop with full tool access across all 3 modules and persistent conversation history.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;DirectedGraphRunner&lt;/code&gt;):&lt;/strong&gt; A 4-node deterministic &lt;strong&gt;Directed Graph&lt;/strong&gt; (&lt;code&gt;node_1_parser&lt;/code&gt; ➔ &lt;code&gt;node_2_pricing&lt;/code&gt; ➔ &lt;code&gt;node_3_report&lt;/code&gt; ➔ &lt;code&gt;node_4_terminal_approval_gate&lt;/code&gt;). Each node receives a fresh, minimal context for its single target module, runs a deterministic stage verifier immediately upon edit, and resets chat history before advancing to the next node.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Both modes achieved a &lt;strong&gt;perfect &lt;code&gt;20 / 20&lt;/code&gt; (100%) pass&lt;/strong&gt; on the hidden grading suite and generated virtually the exact same volume of Python code (&lt;strong&gt;&lt;code&gt;1,188&lt;/code&gt; vs. &lt;code&gt;1,189&lt;/code&gt; output tokens&lt;/strong&gt;—a 1-token difference!). Yet moving workflow orchestration into a stage-gated Directed Graph cut &lt;strong&gt;LLM calls by &lt;code&gt;-57.1%&lt;/code&gt; (&lt;code&gt;7 ➔ 3&lt;/code&gt; turns)&lt;/strong&gt;, &lt;strong&gt;cumulative input tokens by &lt;code&gt;-74.4%&lt;/code&gt; (&lt;code&gt;23,322 ➔ 5,966&lt;/code&gt; tokens)&lt;/strong&gt;, &lt;strong&gt;execution time by &lt;code&gt;-48.0%&lt;/code&gt; (&lt;code&gt;1.92x&lt;/code&gt; faster)&lt;/strong&gt;, and &lt;strong&gt;API cost by &lt;code&gt;-59.3%&lt;/code&gt; (&lt;code&gt;$0.0220 ➔ $0.0089&lt;/code&gt;)&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Single Free-Form Loop (&lt;code&gt;SingleLoop&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Stage-Gated Directed Graph (&lt;code&gt;DirectedGraph&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (20 Cases)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Identical 100% accuracy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;7 calls&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3 calls&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-57.1% (4 orchestration turns eliminated)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Actual File Edits (&lt;code&gt;file_edits&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3 edits&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3 edits&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Identical coding work&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;23,322 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;5,966 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-74.4% (~4x context compression)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,188 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,189 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Identical (&lt;code&gt;+1&lt;/code&gt; token)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;36.69 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;19.07 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-48.0% (1.92x faster)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0220&lt;/code&gt; (&lt;code&gt;21,950 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0089&lt;/code&gt; (&lt;code&gt;8,930 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-59.3% cost reduction&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Experiment Architecture: Single Loop vs. Directed Graph
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM208 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Order &amp;amp; Invoice Pipeline (6 Coupled Bugs)        |
|               (parser.py ➔ pricing.py ➔ report.py)                      |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SingleLoopRunner (Traditional Free-Form ReAct Loop)       |  |
|  |  - Turn 1: inspect_all        (1,652 in /   8 out)                |  |
|  |  - Turn 2: run_tests [Fail]   (2,848 in /   8 out)                |  |
|  |  - Turn 3: edit parser.py     (2,883 in / 357 out)                |  |
|  |  - Turn 4: edit pricing.py    (3,324 in / 591 out)                |  |
|  |  - Turn 5: edit report.py     (4,000 in / 209 out)                |  |
|  |  - Turn 6: run_tests [Pass]   (4,297 in /   8 out)                |  |
|  |  - Turn 7: SUBMIT             (4,318 in /   7 out)                |  |
|  |  --&amp;gt; 7 calls | 23,322 in | 1,188 out | 36.69s | 20/20 Pass        |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: DirectedGraphRunner (Stage-Gated Graph + Context Reset)   |  |
|  |  - Node 1 [Fresh]: patch parser.py  + verify_parser  (1,947 in)   |  |
|  |  - Node 2 [Reset]: patch pricing.py + verify_pricing (2,174 in)   |  |
|  |  - Node 3 [Reset]: patch report.py  + verify_report  (1,845 in)   |  |
|  |  - Node 4 [Gate]:  Terminal Approval Gate (AUTO_SUBMIT, 0 tokens) |  |
|  |  --&amp;gt; 3 calls | 5,966 in  | 1,189 out | 19.07s | 20/20 Pass        |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  The 6 Coupled Bugs Across the 3-Stage Pipeline
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Stage 1 (&lt;code&gt;parser.py&lt;/code&gt; — &lt;code&gt;parse_items&lt;/code&gt;):&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Allowed &lt;code&gt;qty == 0&lt;/code&gt; and Python &lt;code&gt;bool&lt;/code&gt; (&lt;code&gt;isinstance(True, int) == True&lt;/code&gt;) instead of rejecting non-positive/boolean quantities with &lt;code&gt;ValueError&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Failed to normalize SKU strings (&lt;code&gt;.strip().upper()&lt;/code&gt;) and mutated caller input dictionaries in-place.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stage 2 (&lt;code&gt;pricing.py&lt;/code&gt; — &lt;code&gt;compute_invoice&lt;/code&gt;):&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Ignored conflicting &lt;code&gt;unit_price&lt;/code&gt; values for the same SKU instead of raising &lt;code&gt;ValueError&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Applied &lt;code&gt;WELCOME500&lt;/code&gt; coupon &lt;em&gt;before&lt;/em&gt; checking the volume discount and used &lt;code&gt;&amp;gt; 10000&lt;/code&gt; instead of inclusive &lt;code&gt;&amp;gt;= 10000&lt;/code&gt;—wiping out the 10% volume discount on exact &lt;code&gt;10,000&lt;/code&gt; orders (and silently ignoring invalid coupon codes).&lt;/li&gt;
&lt;li&gt;Used Python banker's rounding (&lt;code&gt;round()&lt;/code&gt;) instead of integer floor truncation (&lt;code&gt;// 10&lt;/code&gt;) for discounts and tax.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stage 3 (&lt;code&gt;report.py&lt;/code&gt; — &lt;code&gt;format_summary&lt;/code&gt;):&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;Hardcoded item quantity as &lt;code&gt;x1&lt;/code&gt; and read the wrong key (&lt;code&gt;invoice.get("discount", 0)&lt;/code&gt; instead of &lt;code&gt;discount_total&lt;/code&gt;), omitting the mandatory signed zero format (&lt;code&gt;DISCOUNT: -0&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  3. Turn-by-Turn Token Anatomy: Where Did 74.4% of Input Tokens Go?
&lt;/h2&gt;

&lt;p&gt;Inspecting all 10 recorded LLM calls (&lt;code&gt;seq 0&lt;/code&gt;–&lt;code&gt;6&lt;/code&gt; for Mode A, &lt;code&gt;seq 7&lt;/code&gt;–&lt;code&gt;9&lt;/code&gt; for Mode B) reveals why free-form loops waste tokens even when the model makes zero coding mistakes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th&gt;Call Seq&lt;/th&gt;
&lt;th&gt;Node / Action&lt;/th&gt;
&lt;th&gt;Input Tokens&lt;/th&gt;
&lt;th&gt;Output Tokens&lt;/th&gt;
&lt;th&gt;Context State&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;inspect_all&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1,652&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Spec only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;run_tests&lt;/code&gt; (initial fail)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,848&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;+ All 3 initial files appended&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;edit_file&lt;/code&gt; (&lt;code&gt;parser.py&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,883&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;357&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;+ Initial test failure trace appended&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;edit_file&lt;/code&gt; (&lt;code&gt;pricing.py&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;3,324&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;591&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;+ Full patched &lt;code&gt;parser.py&lt;/code&gt; in history&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;edit_file&lt;/code&gt; (&lt;code&gt;report.py&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;209&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;+ Full patched &lt;code&gt;pricing.py&lt;/code&gt; in history&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;run_tests&lt;/code&gt; (green)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4,297&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;+ Full patched &lt;code&gt;report.py&lt;/code&gt; in history&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode A&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;6&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;SUBMIT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;4,318&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;7&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2.6x larger than Turn 1 just to say &lt;code&gt;SUBMIT&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;em&gt;Mode A Total&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;em&gt;7 turns&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;23,322&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;&lt;code&gt;1,188&lt;/code&gt;&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;Snowballing history across all 7 turns&lt;/em&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;7&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;node_1_parser&lt;/code&gt; + auto-verify&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,947&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;366&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Fresh context (&lt;code&gt;parser.py&lt;/code&gt; only)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;node_2_pricing&lt;/code&gt; + auto-verify&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2,174&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;600&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Context reset&lt;/strong&gt; (&lt;code&gt;pricing.py&lt;/code&gt; only)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Mode B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;9&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;node_3_report&lt;/code&gt; + auto-verify ➔ &lt;code&gt;AUTO_SUBMIT&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,845&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;223&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Context reset&lt;/strong&gt; (&lt;code&gt;report.py&lt;/code&gt; only)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;em&gt;Mode B Total&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;em&gt;3 turns (+ 0-token gate)&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;5,966&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;em&gt;&lt;code&gt;1,189&lt;/code&gt;&lt;/em&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;Flat ~2,000-token context on every node&lt;/em&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Notice the smoking gun on Call &lt;code&gt;6&lt;/code&gt;: Mode A paid &lt;strong&gt;&lt;code&gt;4,318&lt;/code&gt; input tokens&lt;/strong&gt; just to emit a 7-token &lt;code&gt;{"op": "SUBMIT"}&lt;/code&gt; JSON object. Meanwhile, Mode B kept every single node under &lt;code&gt;2,200&lt;/code&gt; input tokens and transitioned to &lt;code&gt;AUTO_SUBMIT&lt;/code&gt; deterministically in Python the instant &lt;code&gt;verify_report_stage&lt;/code&gt; passed.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. All 20 Hidden Grading Suite Results (&lt;code&gt;labjudge04&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;Both modes achieved &lt;code&gt;20 / 20&lt;/code&gt; on the hidden evaluation suite executed by the isolated &lt;code&gt;labjudge04&lt;/code&gt; user:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Test Case IDs&lt;/th&gt;
&lt;th&gt;Invariants Verified&lt;/th&gt;
&lt;th&gt;Mode A (&lt;code&gt;SingleLoop&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B (&lt;code&gt;DirectedGraph&lt;/code&gt;)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Stage 1 (&lt;code&gt;parser.py&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;parser_01&lt;/code&gt; to &lt;code&gt;parser_06&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Trim/uppercase SKU, non-mutation, reject &lt;code&gt;[]&lt;/code&gt;/&lt;code&gt;bool&lt;/code&gt;/&lt;code&gt;float&lt;/code&gt;/&lt;code&gt;qty &amp;lt;= 0&lt;/code&gt;, allow &lt;code&gt;unit_price == 0&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;6 / 6&lt;/code&gt; PASS&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; PASS&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Stage 2 (&lt;code&gt;pricing.py&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;pricing_01&lt;/code&gt; to &lt;code&gt;pricing_08&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;First-seen SKU order, price conflict check, &lt;code&gt;9999 vs 10000&lt;/code&gt; boundary, volume-before-coupon order, floor &lt;code&gt;// 10&lt;/code&gt;, coupon cap, unknown coupon rejection&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;8 / 8&lt;/code&gt; PASS&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;8 / 8&lt;/code&gt; PASS&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Stage 3 &amp;amp; E2E (&lt;code&gt;report.py&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;e2e_01&lt;/code&gt; to &lt;code&gt;e2e_06&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;DISCOUNT: -0&lt;/code&gt; signed zero, multi-SKU summary, free-item invoice, floor tax E2E, line order, cross-stage &lt;code&gt;ValueError&lt;/code&gt; propagation&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;6 / 6&lt;/code&gt; PASS&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; PASS&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;20 Hidden Cases&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: Separate Reasoning (Loops) From Control Flow (Graphs)
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Don't ask an LLM to be a state machine:&lt;/strong&gt; Let the LLM write patches inside small, bounded loops, and move deterministic transitions ("load file ➔ run stage test ➔ advance on green ➔ submit") into regular Python/TypeScript graph edges.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use deterministic stage verifiers as context compressors:&lt;/strong&gt; Once a module's contract verifier passes, drop the raw editing history before moving to the next module. (Note: As we show in &lt;strong&gt;&lt;code&gt;E05&lt;/code&gt;&lt;/strong&gt;, only split into graph nodes when each stage has a &lt;strong&gt;deterministic code verifier&lt;/strong&gt;—never split into role-based subagents passing lossy natural-language summaries!)&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🪆 &lt;strong&gt;Next Companion Post (&lt;code&gt;E05&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e05-subagent-folding" rel="noopener noreferrer"&gt;Why Role-Based Subagents Slow Down Coding Agents: Subagent Folding + Progressive Skill Disclosure (2.66x Faster)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🥇 &lt;strong&gt;Rank #1 Deep Dive (&lt;code&gt;E20&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e20-episodic-gotchas-memory" rel="noopener noreferrer"&gt;Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item &lt;code&gt;MEMORY.md&lt;/code&gt;&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>harness</category>
      <category>architecture</category>
      <category>agenticai</category>
    </item>
    <item>
      <title>Benchmarking the Claude Code /advisor Pattern: Solo Executor Loops vs. Upfront Advisor Blueprint Routing</title>
      <dc:creator>tappers</dc:creator>
      <pubDate>Sun, 01 Nov 2026 12:53:43 +0000</pubDate>
      <link>https://dev.to/tappers_/benchmarking-the-claude-code-advisor-pattern-solo-executor-loops-vs-upfront-advisor-blueprint-46mg</link>
      <guid>https://dev.to/tappers_/benchmarking-the-claude-code-advisor-pattern-solo-executor-loops-vs-upfront-advisor-blueprint-46mg</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;AI Labo Benchmark Series — Experiment &lt;code&gt;E18&lt;/code&gt; (Ranked #7 out of 41 Patterns)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;View All 41 Benchmark Results &amp;amp; Master Ranking&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;&lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;Read How Our Proxmox &amp;amp; systemd Anti-Cheating Sandbox Works&lt;/a&gt;&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Metadata &amp;amp; Experiment Identity
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID:&lt;/strong&gt; &lt;code&gt;E18&lt;/code&gt; (Backlog Candidate &lt;code&gt;C014&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run ID:&lt;/strong&gt; &lt;code&gt;e18-20261028-vm222-01&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environment:&lt;/strong&gt; Proxmox Isolated VM &lt;code&gt;VM222&lt;/code&gt; (&lt;code&gt;ai-e18-222&lt;/code&gt;, 0 NICs, &lt;code&gt;virtio-serial&lt;/code&gt; QGA relay, &lt;code&gt;systemd&lt;/code&gt; UID separation)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; &lt;code&gt;gemini-3.8-flash&lt;/code&gt; (&lt;code&gt;thinkingLevel: LOW&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pattern Under Test:&lt;/strong&gt; Claude Code &lt;code&gt;/advisor&lt;/code&gt; &amp;amp; Upfront Architectural Routing — &lt;em&gt;"Solo Executor Trial-and-Error Loop vs. Upfront Advisor Blueprint + Executor One-Shot"&lt;/em&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  1. Executive Summary (TL;DR)
&lt;/h2&gt;

&lt;p&gt;When an AI coding agent is tasked with fixing a bug that spans multiple coupled modules, running a single worker loop without upfront architectural guidance (&lt;strong&gt;Solo Executor Loop&lt;/strong&gt;) often leads to &lt;strong&gt;myopic Whack-a-Mole patching&lt;/strong&gt;: the agent edits Module 1, runs a trial, discovers a downstream contract mismatch in Module 2, patches Module 2, and repeats—accumulating trial logs and regenerating code across multiple turns.&lt;/p&gt;

&lt;p&gt;Anthropic's Claude Code &lt;code&gt;/advisor&lt;/code&gt; pattern (pairing a high-level architectural Advisor memo with a fast Executor) proposes a cleaner division of labor: consult the &lt;strong&gt;Advisor&lt;/strong&gt; once at the start of the task to produce a concise cross-module &lt;strong&gt;Architectural Blueprint Memo&lt;/strong&gt;, then let the &lt;strong&gt;Executor&lt;/strong&gt; implement all coupled files in a single pass.&lt;/p&gt;

&lt;p&gt;To test this pattern empirically, we benchmarked our 3-module payment settlement pipeline (&lt;code&gt;validator.py&lt;/code&gt;, &lt;code&gt;settlement.py&lt;/code&gt;, &lt;code&gt;statement.py&lt;/code&gt;) inside isolated Proxmox VM &lt;code&gt;VM222&lt;/code&gt; across two modes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mode A (&lt;code&gt;SoloExecutorLoopRunner&lt;/code&gt;):&lt;/strong&gt; No Advisor consultation (&lt;code&gt;advisor_calls = 0&lt;/code&gt;). The Executor runs an initial trial (&lt;code&gt;run_solo_trial&lt;/code&gt;), applies piecemeal patches (&lt;code&gt;stage_executor_patch&lt;/code&gt; on Turn 2 and Turn 3), and finally submits all 3 modules on Turn 4 (&lt;code&gt;4&lt;/code&gt; LLM calls, &lt;code&gt;5&lt;/code&gt; file edits).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mode B (&lt;code&gt;AdvisorExecutorRoutingRunner&lt;/code&gt;):&lt;/strong&gt; Calls &lt;code&gt;consult_advisor&lt;/code&gt; on Turn 1 (&lt;code&gt;advisor_calls = 1&lt;/code&gt;) to retrieve a &lt;strong&gt;&lt;code&gt;1,143-character&lt;/code&gt; 5-point cross-module execution blueprint&lt;/strong&gt;, allowing the Executor to implement and submit all 3 modules in one shot on Turn 2 (&lt;code&gt;2&lt;/code&gt; LLM calls, &lt;code&gt;3&lt;/code&gt; file edits).&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Key Result
&lt;/h3&gt;

&lt;p&gt;Injecting a &lt;strong&gt;&lt;code&gt;1,143-character&lt;/code&gt; upfront Advisor blueprint&lt;/strong&gt; eliminated &lt;strong&gt;all unplanned partial-patch retries (&lt;code&gt;2 ➔ 0&lt;/code&gt;)&lt;/strong&gt;, halved the number of LLM calls (&lt;code&gt;4 ➔ 2&lt;/code&gt;), cut cumulative input tokens by &lt;strong&gt;&lt;code&gt;-84.9%&lt;/code&gt; (&lt;code&gt;27,699&lt;/code&gt; ➔ &lt;code&gt;4,172&lt;/code&gt; tokens, &lt;code&gt;6.64x&lt;/code&gt; efficiency)&lt;/strong&gt;, reduced cumulative output tokens by &lt;strong&gt;&lt;code&gt;-45.9%&lt;/code&gt; (&lt;code&gt;2,897&lt;/code&gt; ➔ &lt;code&gt;1,568&lt;/code&gt; tokens)&lt;/strong&gt;, lowered API cost by &lt;strong&gt;&lt;code&gt;-71.5%&lt;/code&gt; (&lt;code&gt;$0.0316&lt;/code&gt; ➔ &lt;code&gt;$0.0090&lt;/code&gt;)&lt;/strong&gt;, and sped up execution by &lt;strong&gt;&lt;code&gt;1.78x&lt;/code&gt; (&lt;code&gt;33.45s&lt;/code&gt; ➔ &lt;code&gt;18.78s&lt;/code&gt;)&lt;/strong&gt;—with both modes passing &lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt; on the hidden grading suite.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Mode A: Solo Executor Loop (&lt;code&gt;SoloExecutorLoop&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B: Advisor-Executor Routing (&lt;code&gt;AdvisorExecutorRouting&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Delta / Improvement&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Hidden Grading Suite Score (20 Cases)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Both passed all 20 cases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Unplanned Partial-Patch Retries (&lt;code&gt;unplanned_retries&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;2 retries&lt;/code&gt; (Turn 2, Turn 3)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;0 retries&lt;/code&gt; (One-shot implementation)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-100% (Zero rework)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total File Edits (&lt;code&gt;file_edits&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5 edits&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;3 edits&lt;/code&gt; (1 per module)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-40.0% reduction&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;LLM Calls (&lt;code&gt;model_calls&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;4 calls&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2 calls&lt;/code&gt; (1 Advisor + 1 Executor)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-50.0% fewer calls&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Tool Output (&lt;code&gt;tool_output_chars&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;22,720 chars&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,245 chars&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-94.5% (18.25x less log noise)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 1 Input / Output Tokens&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1,909&lt;/code&gt; in / &lt;code&gt;10&lt;/code&gt; out&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,846&lt;/code&gt; in / &lt;code&gt;8&lt;/code&gt; out&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Equivalent initial invocation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 2 Input / Output Tokens&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;5,000&lt;/code&gt; in / &lt;code&gt;488&lt;/code&gt; out (&lt;code&gt;validator.py&lt;/code&gt; patch)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;2,326&lt;/code&gt; in / &lt;code&gt;1,560&lt;/code&gt; out (All 3 modules done!)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Mode B finishes on Turn 2&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 3 Input / Output Tokens&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;8,527&lt;/code&gt; in / &lt;code&gt;795&lt;/code&gt; out (&lt;code&gt;settlement.py&lt;/code&gt; patch)&lt;/td&gt;
&lt;td&gt;— &lt;em&gt;(Completed on Turn 2)&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;Avoided intermediate retry turn&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Turn 4 Input / Output Tokens&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;12,263&lt;/code&gt; in / &lt;code&gt;1,604&lt;/code&gt; out (Final submit)&lt;/td&gt;
&lt;td&gt;— &lt;em&gt;(Completed on Turn 2)&lt;/em&gt;
&lt;/td&gt;
&lt;td&gt;Avoided &lt;code&gt;12,263&lt;/code&gt;-token final prompt&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Input Tokens (&lt;code&gt;promptTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;27,699 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;4,172 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-84.9% (6.64x token efficiency)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cumulative Output Tokens (&lt;code&gt;candidatesTokenCount&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2,897 tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;1,568 tokens&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-45.9% (No duplicate code output)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Elapsed Execution Time (&lt;code&gt;elapsed_seconds&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;33.45 sec&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;18.78 sec&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-43.9% (1.78x faster)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Estimated API Cost (&lt;code&gt;estimated_micro_usd&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;$0.0316&lt;/code&gt; (&lt;code&gt;31,640 micro-USD&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;$0.0090&lt;/code&gt; (&lt;code&gt;9,010 micro-USD&lt;/code&gt;)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;-71.5% (3.51x cheaper)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  2. Experiment Architecture: Solo Trial Loop vs. Advisor Blueprint Routing
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+-------------------------------------------------------------------------+
| Proxmox Isolated VM222 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Bugfix                       |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: SoloExecutorLoopRunner (No Upfront Design Blueprint)      |  |
|  |  - Turn 1 (1,909 in):  run_solo_trial (+7,500 chars trial log)    |  |
|  |  - Turn 2 (5,000 in):  stage_executor_patch (validator.py only)   |  |
|  |  - Turn 3 (8,527 in):  stage_executor_patch (settlement.py only)  |  |
|  |  - Turn 4 (12,263 in): submit_settlement_bundle (All 3 modules)   |  |
|  |  --&amp;gt; Total: 27,699 in / 2,897 out / 5 file edits / 33.45 sec      |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: AdvisorExecutorRoutingRunner (Claude Code /advisor Style) |  |
|  |  - Turn 1 (1,846 in / 8 out):                                     |  |
|  |      Calls `consult_advisor` --&amp;gt; Receives 1,143-char 5-point      |  |
|  |      cross-module `one_shot_execution_blueprint`                  |  |
|  |  - Turn 2 (2,326 in / 1,560 out):                                 |  |
|  |      Executor implements &amp;amp; submits all 3 modules in 1 shot!       |  |
|  |  --&amp;gt; Total: 4,172 in / 1,568 out / 3 file edits / 18.78 sec       |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  3. Why Advisor-Executor Routing Beats Both Solo Loops and Role Subagents
&lt;/h2&gt;

&lt;h3&gt;
  
  
  3.1 The Dual Token Penalty of Solo Trial-and-Error (&lt;code&gt;Mode A&lt;/code&gt;)
&lt;/h3&gt;

&lt;p&gt;Analyzing &lt;code&gt;usage.json&lt;/code&gt; and &lt;code&gt;execution-traces.json&lt;/code&gt; shows why piecemeal patching without an upfront architectural plan is so expensive:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Input Token Snowballing:&lt;/strong&gt; Every time Mode A staged a partial single-file patch (&lt;code&gt;validator.py&lt;/code&gt; on Turn 2, &lt;code&gt;settlement.py&lt;/code&gt; on Turn 3), the harness ran a trial diagnostic that appended roughly &lt;code&gt;7,500 characters&lt;/code&gt; of output to the chat history (&lt;code&gt;1,909&lt;/code&gt; ➔ &lt;code&gt;5,000&lt;/code&gt; ➔ &lt;code&gt;8,527&lt;/code&gt; ➔ &lt;code&gt;12,263&lt;/code&gt; input tokens).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Duplicate Output Token Generation (&lt;code&gt;+84.8%&lt;/code&gt; extra output):&lt;/strong&gt; Because Mode A generated &lt;code&gt;validator.py&lt;/code&gt; on Turn 2 (&lt;code&gt;488&lt;/code&gt; output tokens), &lt;code&gt;settlement.py&lt;/code&gt; on Turn 3 (&lt;code&gt;795&lt;/code&gt; output tokens), and then re-emitted all 3 modules in the final Turn 4 bundle (&lt;code&gt;1,604&lt;/code&gt; output tokens), it generated &lt;strong&gt;2,897 output tokens&lt;/strong&gt; compared to Mode B's &lt;strong&gt;1,568 output tokens&lt;/strong&gt;. Since LLM output tokens cost 3x to 5x more per token than input tokens, avoiding duplicate code generation directly slashes API cost and latency.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  3.2 How &lt;code&gt;E18&lt;/code&gt; (Advisor-Executor) Differs from &lt;code&gt;E05&lt;/code&gt; (Role Subagents)
&lt;/h3&gt;

&lt;p&gt;A natural question is: &lt;em&gt;"In &lt;code&gt;E05&lt;/code&gt; (Subagent Folding), you showed that splitting a task across Architect ➔ Coder ➔ QA subagents was 2.66x slower. Why did &lt;code&gt;E18&lt;/code&gt; (Advisor-Executor) succeed?"&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The difference lies in &lt;strong&gt;who holds the context and how the handoff works&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;In &lt;strong&gt;&lt;code&gt;E05&lt;/code&gt; (Role Subagent Handoff — Anti-Pattern)&lt;/strong&gt;, the Architect subagent ran in an isolated context and passed a &lt;strong&gt;lossy natural-language summary&lt;/strong&gt; to the Coder, hiding the original codebase and dropping fine-grained rules (like signed zero &lt;code&gt;-0&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;In &lt;strong&gt;&lt;code&gt;E18&lt;/code&gt; (Advisor Blueprint Injection — Golden Pattern)&lt;/strong&gt;, the Executor retains &lt;strong&gt;direct visibility into the actual source files&lt;/strong&gt;, and the Advisor injects a &lt;strong&gt;concise &lt;code&gt;1,143-character&lt;/code&gt; checklist of cross-module invariants&lt;/strong&gt; (&lt;code&gt;one_shot_execution_blueprint&lt;/code&gt;) directly into Turn 1 before the Executor writes code. Nothing is lost in translation, and the Executor completes all 3 files in a single pass.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  4. All 20 Hidden Grading Suite Results (&lt;code&gt;labjudge18&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;Both modes passed all 20 hidden unit and E2E test cases verified by &lt;code&gt;labjudge18&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;#&lt;/th&gt;
&lt;th&gt;Test Case ID&lt;/th&gt;
&lt;th&gt;Invariant / Boundary Verified&lt;/th&gt;
&lt;th&gt;Mode A (&lt;code&gt;Solo Loop&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Mode B (&lt;code&gt;Advisor-Executor&lt;/code&gt;)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1–6&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;validator_01&lt;/code&gt; to &lt;code&gt;validator_06&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Trim/uppercase, non-mutation, &lt;code&gt;JPY&lt;/code&gt;, &lt;code&gt;bool&lt;/code&gt; rejection, &lt;code&gt;0&lt;/code&gt; cents &amp;amp; &lt;code&gt;1000&lt;/code&gt; bps bounds&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;6 / 6&lt;/code&gt; PASS&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;6 / 6&lt;/code&gt; PASS&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7–15&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;settlement_01&lt;/code&gt; to &lt;code&gt;settlement_09&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;First-seen order, conflict check, &lt;code&gt;300 vs 301&lt;/code&gt; bps, &lt;code&gt;14999 vs 15000&lt;/code&gt; tier, rebate cap&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;9 / 9&lt;/code&gt; PASS&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;9 / 9&lt;/code&gt; PASS&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;16–20&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;e2e_01&lt;/code&gt; to &lt;code&gt;e2e_05&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;#&lt;/code&gt; separator, signed zero &lt;code&gt;-0&lt;/code&gt;, &lt;code&gt;$0&lt;/code&gt; auth statement, line order, error propagation&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;5 / 5&lt;/code&gt; PASS&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;5 / 5&lt;/code&gt; PASS&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;20 Hidden Cases&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;20 / 20&lt;/code&gt; (100%)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  5. Practical Takeaway: Inject an Architectural Blueprint Before First Edit
&lt;/h2&gt;

&lt;p&gt;When using Claude Code (&lt;code&gt;/advisor&lt;/code&gt;), Antigravity (&lt;code&gt;/plan&lt;/code&gt;), or your own custom agent harness on multi-file tasks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Never let an Executor start editing File 1 before mapping File 2 and File 3's contracts:&lt;/strong&gt;
Piecemeal file-by-file trial loops double your output token generation and bloat input history with intermediate failure logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Generate a 5-bullet Cross-Module Blueprint on Turn 1:&lt;/strong&gt;
Have your Advisor model (or planning step) output a compact &lt;code&gt;&amp;lt;1,500-character&lt;/code&gt; contract checklist specifying the exact data types, boundary inequalities (&lt;code&gt;&amp;gt;&lt;/code&gt; vs. &lt;code&gt;&amp;gt;=&lt;/code&gt;), and formatting invariants across all affected files before unlocking file edit tools.&lt;/li&gt;
&lt;/ol&gt;




&lt;h3&gt;
  
  
  🔗 Continue Reading the Series
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;🚀 &lt;strong&gt;Master Benchmark Hub (All 41 Patterns Ranked):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/ai-agent-harness-patterns-benchmark" rel="noopener noreferrer"&gt;I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🛡️ &lt;strong&gt;Methodology &amp;amp; Sandbox Architecture:&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/air-gapped-proxmox-systemd-agent-sandbox" rel="noopener noreferrer"&gt;How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox &amp;amp; systemd Sandbox&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🥇 &lt;strong&gt;Rank #1 Deep Dive (&lt;code&gt;E20&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e20-episodic-gotchas-memory" rel="noopener noreferrer"&gt;Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item &lt;code&gt;MEMORY.md&lt;/code&gt;&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🥈 &lt;strong&gt;Rank #2 Deep Dive (&lt;code&gt;E21&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e21-treesitter-skeleton-context" rel="noopener noreferrer"&gt;Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🥉 &lt;strong&gt;Rank #3 Deep Dive (&lt;code&gt;E16&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e16-cli-test-output-token-killer" rel="noopener noreferrer"&gt;Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🌿 &lt;strong&gt;Rank #4 Deep Dive (&lt;code&gt;E19&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e19-error-trace-pruning" rel="noopener noreferrer"&gt;Why Keeping Failed Code Attempts in Chat History Poisons Your Coding Agent: Error-Trace Pruning vs. Accumulated History&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🔍 &lt;strong&gt;Rank #5 Deep Dive (&lt;code&gt;E14&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e14-progressive-tool-search" rel="noopener noreferrer"&gt;The Hidden "Schema Tax" of MCP Tools: Dumping 32 Tool Schemas vs. On-Demand Tool Search (-92.7% Input Tokens)&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🗜️ &lt;strong&gt;Rank #6 Deep Dive (&lt;code&gt;E12&lt;/code&gt;):&lt;/strong&gt; &lt;a href="https://tappers.hashnode.dev/e12-milestone-context-compaction" rel="noopener noreferrer"&gt;How Save-Game Context Compaction Fixes Spec Drift in Multi-Step Coding Agents (-91.9% Input Tokens)&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>agents</category>
      <category>architecture</category>
    </item>
  </channel>
</rss>
