DEV Community: tappers

Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md

tappers — Sat, 31 Oct 2026 13:24:37 +0000

💡 AI Labo Benchmark Series — Experiment E20 (Ranked #1 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

🚀 View All 41 Benchmark Results & Master Ranking

🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

Experiment ID: E20 (Backlog Candidate C016)
Run ID: e20-20261028-vm224-01
Sandbox Environment: Proxmox Isolated VM VM224 (ai-e20-224, 0 NICs, virtio-serial QGA relay, systemd UID separation)
Model: gemini-3.8-flash (thinkingLevel: LOW)
Pattern Under Test: Letta / MemGPT & Claude Code CLAUDE.md / MEMORY.md Pattern — "Episodic Gotchas Memory vs. Cold-Start Exploration"

1. Executive Summary (TL;DR)

When running multiple tasks sequentially on the same repository, starting every agent session from scratch (Cold-Start Exploration) forces the model to rediscover non-obvious domain invariants and repository quirks ("gotchas") through repeated test failures—burning tens of thousands of tokens on stack traces and diluting model attention.

To test whether a compact Episodic Gotchas Memory (MEMORY.md) solves this, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) containing 5 subtle repository gotchas (7 domain invariants) across two modes:

Mode A (Cold-Start Exploration): No MEMORY.md injected; the agent explores the codebase and diagnoses failures via standard test/audit feedback.
Mode B (Episodic Gotchas Memory): A 5-item MEMORY.md (+442 tokens) distilled from prior session failures is injected into the initial prompt.

Key Result

Injecting just +442 tokens of MEMORY.md reduced repository trap hits from 2 ➔ 0, achieving a 1-turn 20 / 20 (100%) pass on the hidden grading suite. Compared to Mode A (17 / 20 pass across 3 turns), Mode B cut cumulative input tokens by -95.5% (46,560 ➔ 2,078 tokens), API cost by -83.7% ($0.0454 ➔ $0.0074), and wall-clock time by -54.8% (2.21x faster).

Metric	Mode A: Cold-Start Exploration (No `MEMORY.md`)	Mode B: Episodic Gotchas Memory (`MEMORY.md` Injected)	Delta / Improvement
Hidden Grading Suite Score (20 Cases)	`17 / 20` (85.0% — missed 3 edge cases)	`20 / 20` (100% Perfect Pass)	+15.0 pt (Zero defects)
Repository Gotcha Traps Triggered	`2 times` (Turn 1, Turn 2)	`0 times` (Avoided on Turn 1)	-100.0% (Zero traps)
LLM Calls (`model_calls`)	`3 calls`	`1 call` (1-shot completion)	-66.7% (3.00x faster convergence)
Total File Edits (`file_edits`)	`5 edits`	`3 edits` (1 per module)	-40.0% reduction
Tool Output Volume (`tool_output_chars`)	`56,605 chars`	`101 chars`	-99.8% (560x less noise)
Turn 1 Input / Output Tokens	`1,636` in / `11` out	`2,078` in / `1,560` out	`MEMORY.md` overhead is only `+442` in
Turn 2 Input / Output Tokens	`15,153` in / `1,220` out	— (Completed on Turn 1)	Avoided Turn 1 failure log bloat
Turn 3 Input / Output Tokens	`29,771` in / `1,569` out	— (Completed on Turn 1)	Avoided Turn 2 failure log bloat
Cumulative Input Tokens (`promptTokenCount`)	`46,560 tokens`	`2,078 tokens`	-95.5% (22.41x token efficiency)
Cumulative Output Tokens (`candidatesTokenCount`)	`2,800 tokens`	`1,560 tokens`	-44.3% reduction
Elapsed Execution Time (`elapsed_seconds`)	`30.16 sec`	`13.64 sec`	-54.8% (2.21x faster)
Estimated API Cost (`estimated_micro_usd`)	`$0.0454` (`45,421 micro-USD`)	`$0.0074` (`7,409 micro-USD`)	-83.7% (6.13x cheaper)

2. Experiment Architecture & Repository Gotchas Under Test

+-------------------------------------------------------------------------+
| Proxmox Isolated VM224 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Bugfix                       |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: ColdStartRunner (No MEMORY.md — Cold-Start Exploration)   |  |
|  |  - Turn 1: probe_repo_gotchas   (1,636 in / 28,317 chars trap log)|  |
|  |  - Turn 2: stage_partial_fix    (15,153 in / 27,975 chars log)    |  |
|  |  - Turn 3: submit_settlement    (29,771 in --> 17 / 20 passed)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: EpisodicGotchasMemoryRunner (5-Item MEMORY.md [+442 tok]) |  |
|  |  - Turn 1: Reads 5-item MEMORY.md & submits all 3 modules         |  |
|  |            (2,078 in / 1,560 out --> 20 / 20 passed on 1st try!)  |  |
|  +-------------------------------------------------------------------+  |
|                                    |                                    |
|                                    v                                    |
|  +-------------------------------------------------------------------+  |
|  | Independent Grader (labjudge20): 20 Hidden Unit & E2E Test Cases  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

The 5 Non-Obvious Repository Gotchas in the Initial Codebase

The initial workspace inside VM224 contains three payment modules with subtle domain bugs and Python traps that are easy to miss on a cold read:

validator.py (Transaction Validation & Normalization):
- ALLOWED_CURRENCIES = ("USD", "EUR") is missing "JPY".
- Empty list [] rejection is missing, and because bool is a subclass of int in Python (isinstance(True, int) == True), isinstance(v, bool) must be explicitly rejected before checking isinstance(v, int).
- The bounds check if cnt <= 0 or cents <= 0 or not (0 <= bps < 1000): erroneously rejects unit_cents == 0 ($0 card authorization probes) and chargeback_bps == 1000 (inclusive upper bound).
- It mutates caller dictionaries in-place (item["merchant_id"] = ...; out.append(item)) and leaks extra unvalidated keys (extra) into the output.
settlement.py (Batch Aggregation, Reserve & Tier Discounts):
- Iterates with for k in sorted(order):, destroying first-seen (merchant_id, currency) ordering.
- Applies the 10% chargeback reserve at chargeback_bps >= 300 instead of strictly > 300 (at exactly 300 bps, reserve must be 0).
- Uses subtotal > 15000 instead of inclusive >= 15000, omits the 10% discount for ENTERPRISE below 15000, and forgets to cap rebate_applied = min(fee_rebate, after_discount).
statement.py (Canonical Statement Formatter):
- Uses : instead of # between merchant ID and currency ({merchant_id}#{currency}:).
- Formats zero discounts as "0" instead of preserving the mandatory leading minus sign (TIER_DISCOUNT: -0, REBATE_APPLIED: -0).

3. Inside the Prompts: What Mode B Saw vs. What Mode A Endured

3.1 The Exact 5-Item `MEMORY.md` Injected into Mode B (`+442 tokens`)

In Mode B (input-03.json), the harness prepended this exact 5-bullet # MEMORY.md section to the initial prompt:

# MEMORY.md — Repository Episodic Gotchas & Non-Obvious Invariants (Distilled from Session #1)
1. [GOTCHA-01 / validator.py] `ALLOWED_CURRENCIES` MUST include `"JPY"` (`("USD", "EUR", "JPY")`), empty `records` list MUST raise `ValueError`, and `isinstance(v, bool)` MUST be rejected before `isinstance(v, int)` because `bool` is a subclass of `int` in Python.
2. [GOTCHA-02 / validator.py] Zero `unit_cents == 0` and boundary `chargeback_bps == 1000` are VALID (`tx_count > 0`, `unit_cents >= 0`, `0 <= chargeback_bps <= 1000`). Always return a brand-new 5-key dict without mutating caller dicts.
3. [GOTCHA-03 / settlement.py] Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, NEVER `sorted(order)`). The 10% reserve applies STRICTLY when `chargeback_bps > 300` (at `300` bps, `reserve_cents == 0`).
4. [GOTCHA-04 / settlement.py] Tier discount threshold `subtotal >= 15000` is INCLUSIVE: `GROWTH` gets 10% (`>= 15000`), `ENTERPRISE` gets 20% (`>= 15000`) or 10% (`< 15000`), and `rebate_applied = min(fee_rebate, after_discount)`.
5. [GOTCHA-05 / statement.py] Header delimiter between merchant and currency is `#` (`{merchant_id}#{currency}: tx=...`) and discount/rebate lines MUST always carry `-` even when zero (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).

3.2 The 56,605-Character Failure Trace That Flooded Mode A (`Cold-Start`)

Without MEMORY.md, Mode A called probe_repo_gotchas on Turn 1 and stage_partial_fix on Turn 2, accumulating 28,317 characters on Turn 1 and 27,975 characters on Turn 2 (totaling 56,605 characters of verbose audit stack traces) inside its conversation history:

=== COLD-START REPOSITORY GOTCHA TRAP LOG (STAGE #1) ===
Status: FAILED (4 repository gotchas triggered due to missing cross-session MEMORY.md)
[COLD_START_GOTCHA_AUDIT_001] stage=1 invariant_probe=GOTCHA_02 trace_digest=0000000000000000000000000000000000000000000000000000000000000001
  File '/var/lib/ai-lab-e20-vm224-01/work_cold_start/settlement.py', line 21, in verify_repo_invariants
    AssertionError: Cold-start session rediscovered repository gotcha #2 via trial-and-error failure.
... (28,000+ chars of repetitive invariant audit traces) ...
TRIGGERED_GOTCHA [validator.py] (validator_contract_violation): Failed JPY currency, empty list check, bool rejection, zero unit_cents, 1000 bps, or mutated input dict.
TRIGGERED_GOTCHA [settlement.py] (settlement_order_or_reserve_boundary): Lines reordered by sorted(order) or 10% reserve charged at chargeback_bps == 300.
TRIGGERED_GOTCHA [settlement.py] (settlement_tier_discount_or_rebate_cap): Missed >=15000 inclusive discount boundary, ENTERPRISE 10% below 15000, or min(fee_rebate, after_discount) cap.
TRIGGERED_GOTCHA [statement.py] (statement_delimiter_or_signed_zero): Used ':' instead of '#' between merchant and currency or omitted '-' sign when discount/rebate is 0.

4. Code-Level Postmortem: Why Mode A Failed 3 out of 20 Test Cases

Comparing the final submitted code bundles (cold-start-bundle.json vs. episodic-memory-bundle.json) reveals the exact mechanism of Attention Dilution:

# ❌ Mode A (Cold-Start, Turn 3 Final Submission — from cold-start-bundle.json)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            **item,
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
        })

# ✅ Mode B (Episodic Gotchas Memory, Turn 1 Final Submission — from episodic-memory-bundle.json)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")

        out.append({
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })

By Turn 3, Mode A was carrying 29,771 input tokens of noisy logs. While it managed to fix settlement.py, statement.py, "JPY", bool rejection, and chargeback_bps <= 1000, its diluted attention missed two fine-grained details in validator.py:

Left cents <= 0 instead of cents < 0: Rejecting unit_cents == 0 ($0 auth probes) caused both unit test validator_05_zero_unit_cents_allowed_negative_rejected and E2E test e2e_03_zero_amount_auth_statement to fail.
Used `{item, ...} spread instead of constructing a clean 5-key dict:** Extra unvalidated keys from the input dict leaked into the normalized output, failing **validator_01_normalize_and_non_mutation`**.

Mode B, reading [GOTCHA-02 / validator.py] inside a clean 2,078-token Turn 1 prompt, nailed every single invariant on the first try.

All 20 Hidden Test Case Results (Mode A vs. Mode B)

#	Test Case ID	Invariant / Boundary Verified	Mode A (`Cold-Start`)	Mode B (`MEMORY.md`)
1	`validator_01_normalize_and_non_mutation`	Trim, uppercase, strip extra keys (strict 5-key dict), no caller mutation	❌ FAIL (`**item` leaked keys)	✅ PASS
2	`validator_02_empty_and_non_list_rejected`	Reject `[]`, `None`, and non-dict elements with `ValueError`	✅ PASS	✅ PASS
3	`validator_03_blank_or_non_str_merchant_and_unknown_currency_rejected`	Reject blank merchant ID / unknown currency; allow `JPY`	✅ PASS	✅ PASS
4	`validator_04_tx_count_and_chargeback_bps_bounds_0_to_1000`	Allow `chargeback_bps` `0` and `1000` (inclusive); reject out-of-range	✅ PASS	✅ PASS
5	`validator_05_zero_unit_cents_allowed_negative_rejected`	Allow `unit_cents == 0` (`$0` auth); reject negative amounts	❌ FAIL (`cents <= 0`)	✅ PASS
6	`validator_06_bool_rejected_on_all_int_fields`	Reject Python `bool` (`True`/`False`) on all integer fields	✅ PASS	✅ PASS
7	`settlement_01_first_seen_order_and_aggregation`	Preserve first-seen `(merchant_id, currency)` order and sum `tx_count`	✅ PASS	✅ PASS
8	`settlement_02_conflicting_unit_cents_or_bps_rejected`	Reject conflicting `unit_cents` or `chargeback_bps` for same key	✅ PASS	✅ PASS
9	`settlement_03_exact_chargeback_bps_300_vs_301_boundary`	Exact boundary: `300 bps` (0 reserve) vs. `301 bps` (10% reserve)	✅ PASS	✅ PASS
10	`settlement_04_exact_15000_growth_discount_boundary`	`GROWTH` tier boundary: `14999` (0%) vs. `15000` (10% discount)	✅ PASS	✅ PASS
11	`settlement_05_enterprise_tiered_10_vs_20_discount`	`ENTERPRISE` tier: `< 15000` (10%) vs. `>= 15000` (20%)	✅ PASS	✅ PASS
12	`settlement_06_floor_truncation_reserve_and_fee`	Integer floor division (`//`) for reserve and wire fee	✅ PASS	✅ PASS
13	`settlement_07_rebate_capped_at_after_discount`	Cap rebate via `min(fee_rebate, after_discount)`	✅ PASS	✅ PASS
14	`settlement_08_invalid_tier_or_bool_rebate_rejected`	Reject unknown tier names and boolean `fee_rebate`	✅ PASS	✅ PASS
15	`settlement_09_lowercase_tier_and_multi_currency_grouping`	Normalize lowercase tier names and group multi-currency lines	✅ PASS	✅ PASS
16	`e2e_01_growth_below_15000_minus_zero_and_hash_separator`	E2E: `#` delimiter and `TIER_DISCOUNT: -0` / `REBATE_APPLIED: -0`	✅ PASS	✅ PASS
17	`e2e_02_enterprise_reserve_and_rebate_statement`	E2E: Full `ENTERPRISE` reserve + 20% discount + rebate statement	✅ PASS	✅ PASS
18	`e2e_03_zero_amount_auth_statement`	E2E: End-to-end processing of `unit_cents == 0` (`$0` auth)	❌ FAIL (Rejected in Stage 1)	✅ PASS
19	`e2e_04_line_order_preserved_in_statement`	E2E: Preserve first-seen line order in rendered statement	✅ PASS	✅ PASS
20	`e2e_05_error_propagation_across_stages`	E2E: Propagate Stage 2 conflicting unit price `ValueError`	✅ PASS	✅ PASS
Total	20 Hidden Cases	—	`17 / 20` (85.0%)	`20 / 20` (100%)

5. Practical Takeaway: How to Structure Your Own `MEMORY.md`

A 442-token investment saved 44,482 input tokens (a 100x+ token ROI) while eliminating 3 subtle edge-case bugs.

To get the same result in your own repositories with Claude Code, Cursor, or custom agents:

Don't turn CLAUDE.md or MEMORY.md into a 500-line textbook. (As we proved in E35, monolithic instruction files hurt performance.)
Instead, maintain a 10-to-20 line MEMORY.md containing only [GOTCHA-ID / filename] 1-line invariant entries distilled from actual test failures or PR review catches:

# MEMORY.md (Repository Gotchas — Keep under 20 lines)
- [GOTCHA-01 / validator.py] `isinstance(x, bool)` is True for `int` in Python; always reject `bool` before checking `isinstance(x, int)`.
- [GOTCHA-02 / validator.py] `unit_cents == 0` ($0 auth) and `chargeback_bps == 1000` are valid; construct a new 5-key dict (never `**item` or mutate in-place).
- [GOTCHA-03 / settlement.py] Keep first-seen order (`for k in order:`); apply 10% reserve strictly when `chargeback_bps > 300`.
- [GOTCHA-04 / statement.py] Use `#` between merchant and currency (`{mid}#{cur}:`), and format zero discount/rebate with a leading minus (`-0`).

🔗 Continue Reading the Series

🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox

How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox

tappers — Sat, 31 Oct 2026 13:18:48 +0000

💡 Methodology & Eval Infrastructure Deep Dive

This article is the Methodology & Architecture companion to our AI Labo Agent Harness Engineering Benchmark Series (41 Patterns Tested).
Here we explain how we designed an isolated evaluation infrastructure combining Proxmox VE (0-NIC VMs), QEMU Guest Agent (virtio-serial), a budget-enforcing Mac relay, and Linux UID / systemd sandboxing to prevent test peeking, reward hacking, and API key leaks—along with a candid look at the current security boundaries and remaining limitations of our setup.

TL;DR

When benchmarking autonomous AI coding agents fairly and safely, engineers inevitably run into a three-way dilemma (trilemma):

Eval Cheating / Reward Hacking: If the AI agent and the grading test suite share the same user privileges and filesystem view, a capable model may inspect the test cases to hardcode edge-case branches or modify the test files (pytest.skip(), weakening assertions) instead of fixing the underlying bug.
API Key Exposure & Runaway Spend: Passing GEMINI_API_KEY or OPENAI_API_KEY directly into the guest environment creates a risk of credential leakage via logs/prompts and runaway billing if an agent enters an infinite retry loop.
Network Isolation vs. Cloud LLM Access: Stripping network interfaces (NICs) from the sandbox VM prevents unauthorized external traffic, but also cuts off direct HTTPS access to cloud LLM APIs.

To solve these challenges simultaneously, we combined a mini-PC running Proxmox VE with a macOS controller over a 1-to-1 Tailscale link to build the AI Labo Evaluation Sandbox:

Network-Isolated VMs (0 NICs) + Inspected Relay Proxy: Experiment VMs have zero network interfaces. The sole external data path is structured JSON passed over QEMU Guest Agent (virtio-serial) to a macOS relay that loads the API key from the OS Keychain into memory once and enforces strict call, token, and USD budget caps (zero API keys inside the VM).
OS-Level Separation via Linux UIDs and systemd (ProtectSystem=strict / InaccessiblePaths): While the agent (labexp) runs, ProtectSystem=strict mounts the filesystem read-only (except for pre-created target .py files bound via ReadWritePaths), and InaccessiblePaths blocks access to the grading suite (evaluation/).
0.8s Cached ISO Generation + Disposable Full-Clone VMs: Pre-verified .deb packages (verified via InRelease ➔ Packages SHA-256 chains) and experiment payloads are burned into read-only iso9660 CD-ROM images in 0.8 seconds, mounted onto a fresh full clone of a stopped Debian 13 base VM, and executed end-to-end in 60–90 seconds per experiment.

1. Why Naive Single-User Execution Distorts Agent Benchmarks

Anyone who has evaluated coding agents at scale has seen Reward Hacking firsthand: the agent reports that all tests pass on Turn 1, but git diff reveals that instead of fixing the bug, it modified the test runner or weakened an assertion.

Running the Agent (Generator) and the Grader (Verifier) under the same user and writable filesystem introduces four structural failure modes:

Failure Mode in Naive Setups	Root Cause	AI Labo Mitigation
1. Test Peeking (Eval Cheating)	If hidden grading tests are readable by the agent's UID, file-search tools can read test inputs and hardcode answers.	Separate UID (`labjudge`, `mode 0750`) + `systemd` `InaccessiblePaths` denies access (`EACCES`) to the grading directory during agent execution.
2. Tampering with Tests or Configs	A writable working directory allows the agent to overwrite test files, runners, or config files.	`ProtectSystem=strict` mounts the filesystem read-only; `ReadWritePaths` permits in-place writes only to the existing inodes of target `.py` files.
3. API Key Leaks & Unbounded Spend	Storing API keys in guest environment variables risks exposure in logs and unthrottled API loops.	Zero NICs and zero API keys inside the VM. The Mac relay enforces `maxOutputTokens`, cumulative token limits, and micro-USD budget caps.
4. Cross-Experiment Environment Drift	Reusing the same environment across runs leaves behind temp files, caches, or altered state.	Every experiment full-clones a stopped Debian 13 Golden Base VM (`VM9300`) and mounts immutable `iso9660` CD-ROMs.

💡 Why Proxmox VMs + systemd Instead of Docker Containers?
You can achieve similar logical separation in Docker by splitting agent and grader containers, passing --network none and --read-only, and keeping secrets on a host proxy.
We chose Proxmox VMs + systemd for two reasons:

Kernel Isolation: Avoiding shared host kernels between the controller/host and disposable guest environments.

Full-Clone & ISO Reproducibility: LVM full clones combined with immutable iso9660 virtual CD-ROMs make it trivial to pin and audit exact VM configuration SHA-256 digests before and after every run.

2. System Architecture (Mac × Tailscale × Proxmox × 0-NIC VM)

The infrastructure is split across a macOS Controller/Relay and a Proxmox VE Host connected via a 1-to-1 peer-restricted Tailscale VPN:

+-------------------------------------------------------------------------+
| 1. Controller & Relay Host (macOS)                                      |
|  - OS Keychain / 0400 Secret --> Loaded once into memory at startup     |
|  - Experiment Orchestrator   --> Drives Phase 1 (ISO) & Phase 2 (Run)   |
|  - Private API Relay         --> Enforces call, token & micro-USD caps  |
|  - Evidence Store            --> Verifies & saves 7 artifacts + SHA256  |
+------------------------------------+------------------------------------+
                                     |
         1-to-1 Peer-Restricted VPN  |  (SSH Control + Peer-Pinned HTTP)
                                     v
+-------------------------------------------------------------------------+
| 2. Virtualization Host (Proxmox VE Mini-PC)                             |
|  - Host Controller (driver.py)  --> Allowlisted qm/pvesh + Watchdogs    |
|  - Local ISO Builder            --> Verified .deb cache -> iso9660 CDs  |
|  - Golden Base VM (VM9300)      --> Stopped Debian 13 clone source      |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | 3. Disposable Sandbox VM (Full Clone per Experiment / 0 NICs)     |  |
|  |  - serial0 (/dev/ttyS0) : One-way boot & QGA readiness signal     |  |
|  |  - virtio-serial (QGA)  : Bidirectional RPC via `qm guest exec`   |  |
|  |  - ide2 & ide3 (CD-ROM) : Read-only offline packages & payload    |  |
|  |                                                                   |  |
|  |  +-----------------------------+  +----------------------------+  |  |
|  |  | Agent Sandbox (UID: labexp) |  | Grader Sandbox (labjudge)  |  |  |
|  |  | - PrivateNetwork=yes        |  | - Runs AFTER agent exits   |  |  |
|  |  | - ProtectSystem=strict      |  | - Evaluates 20 hidden cases|  |  |
|  |  | - InaccessiblePaths=/opt/.. |  |   + 6 boundary mutants     |  |  |
|  |  |   /evaluation (EACCES)      |  | - Writes grader-output.json|  |  |
|  |  +-----------------------------+  +----------------------------+  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

Full Component Matrix (15 Components, Privileges, and Security Roles)

Environment	Component	Execution Identity	Primary Role	Security & Integrity Meaning
External Cloud	Cloud LLM API	Managed Service	`countTokens` & `generateContent`	Accepts HTTPS only from the Mac relay; unreachable from the guest VM.
Mac Controller	OS Keychain / Secret	`Security.framework` / `0400`	Stores LLM API key	Never written to `.env` or VM disks; read once into relay memory at startup.
Mac Controller	Orchestrator	Local user	Runs Phase 1 (ISO) & Phase 2 (VM run)	Invokes the Proxmox host driver over SSH and verifies final evidence hashes.
Mac Controller	Private API Relay	Local user (`serve_once.py`)	Proxies LLM calls & enforces budgets	Enforces max calls, input/output token caps (`maxOutputTokens`), and micro-USD caps.
Mac Controller	Evidence Storage	Local filesystem	Stores 7 evidence files + SHA256 manifest	Preserves tamper-evident records of prompts, actions, token usage, and test results.
Private VPN	Tailscale 1-to-1 Link	Encrypted Mesh VPN	Mac ⇔ Proxmox control & relay traffic	Pins source peer IP (`100.x.x.x`) so LAN or external hosts cannot hit the relay.
Proxmox Host	Host Controller	`root` (`driver.py`)	Clones VM, drives QGA, runs watchdogs	Executes only allowlisted `qm` / `pvesh` commands with stage watchdogs.
Proxmox Host	Local ISO Builder	`builder` (`builder.py`)	Packs verified `.deb` cache & code into ISOs	Verifies `.deb` SHA-256 hashes from signed `InRelease` ➔ `Packages` metadata.
Proxmox Host	Golden Base VM	Stopped (`VM9300`)	Clean Debian 13 clone source	Kept stopped with pinned config SHA-256 to eliminate cross-run drift.
Proxmox Host	Audit Broker VM	Isolated VM (`VM100`)	Holds ledger hash pins	Verified before and after every experiment to ensure host state integrity.
Sandbox VM (0 NIC)	`serial0` (`/dev/ttyS0`)	Kernel / Bootstrap	One-way boot progress notification	Emulated 16550 UART used only during initial boot to signal when offline QGA installation finishes.
Sandbox VM (0 NIC)	`virtio-serial` (QGA)	`root` (`guest_rpc.py`)	Bidirectional host-guest RPC channel	Uses `org.qemu.guest_agent.0` (`.qga` socket) via `qm guest exec` to exchange JSON.
Sandbox VM (0 NIC)	Virtual CD-ROMs × 2	Read-only (`iso9660`)	Delivers `.deb` deps (`ide2`) & payload (`ide3`)	Immutable media inside the guest; `install.py` verifies payload SHA-256 before extraction.
Sandbox VM (0 NIC)	Agent Sandbox	Unprivileged `labexp`	Runs the AI agent under test (Mode A / B)	`ProtectSystem=strict` mounts `/` read-only; `InaccessiblePaths` blocks access to `evaluation/`.
Sandbox VM (0 NIC)	Grader Sandbox	Unprivileged `labjudge`	Runs independent grading suite after agent exits	Starts only after `agent.service` stops; evaluates snapshots against 20 hidden test cases.

3. Core Mechanism 1: OS-Level Separation via Linux UIDs and `systemd`

Inside each experiment VM, the Agent (labexp) and the Independent Grader (labjudge) run under separate Linux UIDs and separate systemd mount namespaces.

Before starting the agent, the root installer (guest/install.py) sets strict filesystem ownership and permissions:

Grading directory (/opt/.../evaluation): Owned by root:labjudge with mode 0750 (files 0640), preventing labexp from reading it even at the basic DAC level.
Work directories (work_cold_start/, work_episodic_memory/): Owned by root:root with mode 0555 (read/traverse only). Only the 3 target Python files inside each directory (validator.py, settlement.py, statement.py) are pre-created with labexp ownership (0644).

Below are the actual systemd unit files from experiment E20 (VM224):

Agent Execution Unit (`ai-lab-e20-01-agent.service`)

[Unit]
Description=AI Lab E20 Episodic Gotchas Memory Isolated Agent Runner (VM224)

[Service]
Type=oneshot
User=labexp20
Group=labexp20
WorkingDirectory=/var/lib/ai-lab-e20-vm224-01
ExecStart=/usr/bin/python3 -I -B /opt/ai-lab-e20-vm224-01/guest/run_guest.py
Restart=no
KillMode=control-group
TimeoutStartSec=240
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateNetwork=yes
ProtectSystem=strict
ProtectHome=yes
CapabilityBoundingSet=
AmbientCapabilities=
# Under ProtectSystem=strict, ONLY outbox and the 6 pre-created target .py files are writable:
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/outbox
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_cold_start/validator.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_cold_start/settlement.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_cold_start/statement.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_episodic_memory/validator.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_episodic_memory/settlement.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_episodic_memory/statement.py

# Grading code (evaluation/), initial fixtures, and evidence dirs are overlaid with 0000 mounts:
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/root
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/evidence
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/grade
InaccessiblePaths=/opt/ai-lab-e20-vm224-01/evaluation
InaccessiblePaths=/opt/ai-lab-e20-vm224-01/fixtures

Independent Grader Unit (`ai-lab-e20-01-grade.service`)

Only after agent.service terminates and the root RPC handler (guest_rpc.py) copies a snapshot of the edited files into grade/ does the grader service launch as labjudge20:

[Unit]
Description=AI Lab E20 Episodic Gotchas Memory Isolated Grader (VM224)

[Service]
Type=oneshot
User=labjudge20
Group=labjudge20
WorkingDirectory=/var/lib/ai-lab-e20-vm224-01/grade
ExecStart=/usr/bin/python3 -I -B /opt/ai-lab-e20-vm224-01/evaluation/grade_comparison.py
Restart=no
KillMode=control-group
TimeoutStartSec=30
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateNetwork=yes
ProtectSystem=strict
ProtectHome=yes
CapabilityBoundingSet=
AmbientCapabilities=
# The grader can only write to the single grader-output.json file:
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/grader-output.json
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/root
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/evidence
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/inbox
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/outbox
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/work_cold_start
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/work_episodic_memory

This layered setup enforces three protections during agent execution:

Blocking Access to Grading Contents (DAC 0750 + InaccessiblePaths): InaccessiblePaths bind-mounts an empty directory with 0000 permissions over the target path in the unit's mount namespace. While listing /opt/ai-lab-.../ still shows the directory entry evaluation, any attempt to read or traverse into it fails immediately with Permission denied (EACCES).
Preventing File Creation, Deletion, or Rename (ProtectSystem=strict + Root-Owned 0555 Workdir): The primary mechanism blocking file creation or deletion is ProtectSystem=strict, which mounts the entire filesystem read-only and bind-mounts only the 6 listed .py files as writable (backed up by root-owned 0555 directory permissions so labexp cannot chmod the parent directory). Running Python with -I (isolated mode) also disables user site-packages and PYTHONPATH injection.
Structured JSON Actions Only: In our benchmark suite (E04–E44), the agent does not execute arbitrary shell commands; both the Mac relay (serve_once.py) and guest harness (pipeline_checks.py) validate structured JSON tool calls against a strict schema, verify ast.parse() syntax, and overwrite existing file inodes in-place via os.open(..., os.O_WRONLY | os.O_TRUNC | os.O_NOFOLLOW).

💡 How We Use Mutation Testing (Mutation Kill Rate)
In experiments where the agent submits both production code and a test suite (test_suite.py in E06 Blind TDD, E07, E08, E10), passing functional tests isn't enough—we also need to know if the agent wrote weak, self-serving tests.
Our grader (judges.py) runs the agent's test_suite.py against 6 deliberate boundary mutants of the reference implementation (e.g., changing >= 10000 to > 10000, removing isinstance(x, bool) checks, omitting -0 formatting) to measure the exact Mutation Kill Rate (0/6 to 6/6).

4. Core Mechanism 2: Dual Serial Channels (`serial0` vs. `virtio-serial`) & Budget-Gated Relay

Because the experiment VM has zero network interfaces (verified at boot by asserting that /sys/class/net contains only lo), it is a network-isolated VM whose sole external channel is an inspected host-mediated relay.

We use two distinct serial mechanisms between the Proxmox host and the guest VM for different phases of the lifecycle:

serial0 (/dev/ttyS0 inside guest — emulated 16550 UART):
- Role: Used exclusively for one-way boot progress notifications during the initial 25-second offline bootstrap (before qemu-guest-agent is installed from the CD-ROM). The host controller (driver.py) monitors /var/run/qemu-server/<vmid>.serial0 until it receives the common_prepared_QGA_acceptance_pending JSON marker.
virtio-serial (org.qemu.guest_agent.0 / .qga socket — QEMU Guest Agent):
- Role: Used for all subsequent bidirectional RPC (qm guest exec <vmid>), including payload installation, polling the agent's Outbox, delivering LLM responses to the Inbox, and retrieving the 7 final evidence files.

End-to-End Execution Lifecycle (60–90 Seconds per Experiment)

Step	Phase & Actor	Operation	Typical Duration / Security Gate
1	Phase 1: ISO Build (Mac ➔ Proxmox)	Packs SHA256-verified `.deb` cache and experiment payload into read-only `iso9660` image.	0.8s (with warm package cache)
2	Phase 2: Pre-Flight (Mac & Proxmox)	Mac loads API key from Keychain into memory; Proxmox verifies `VM9300` & broker ledger hashes.	Zero API keys written to disk
3	Clone & Offline Boot (Proxmox ➔ VM)	Full-clones `VM9300`, attaches 2 CD-ROMs (`ide2`, `ide3`), verifies `0 NICs`, boots VM, installs QGA offline.	25s (notifies host via `serial0`)
4	Workspace Setup (Proxmox ➔ QGA)	`install.py` verifies payload SHA-256, sets `0750` on `evaluation/` and `0555` on `work/`, starts `agent.service`.	Enforces UID & `systemd` locks
5	Agent Turn Loop (VM ⇔ PVE ⇔ Mac ⇔ LLM)	Polls `Outbox` via QGA ➔ Mac relay checks `countTokens` & `maxOutputTokens` ➔ calls Gemini API ➔ delivers action via `deliver_and_request`.	15–40s (1-trip QGA batching)
6	Independent Grading (Proxmox ➔ QGA ➔ VM)	After `agent.service` stops, copies snapshot to `grade/` and runs `grade.service` as `labjudge`.	Runs 20 hidden tests + 6 mutants
7	Evidence & Shutdown (VM ➔ PVE ➔ Mac)	Collects 7 JSON artifacts + `manifest.json`, verifies SHA-256 on Mac, gracefully shuts down VM (`qm shutdown`).	Tamper-evident archive locked

External Token Accounting & Pre-Call Budget Enforcement (`usage.json`)

Because the Mac relay (serve_once.py) mediates every LLM call outside the VM, the agent cannot tamper with token accounting. Before each generateContent call, the relay calls countTokens to verify that cumulative_inputs + prompt_tokens <= MAX_INPUT_TOKENS and cumulative_outputs + maxOutputTokens <= MAX_OUTPUT_TOKENS.

Upon receiving the response, the relay extracts promptTokenCount, candidatesTokenCount, and thoughtsTokenCount (reasoning tokens) from usageMetadata, computing billed_out = candidatesTokenCount + thoughtsTokenCount and recording exact costs in micro-USD (1e-6 USD) against a hard session cap (e.g., 350,000 micro-USD = $0.35).

5. Known Limitations & Future Security Roadmap

While this architecture is highly effective for benchmarking structured-action coding agents, evaluating untrusted, adversarial agents with arbitrary code/shell execution requires addressing several remaining security boundaries. We document these limitations openly for anyone building on this design:

Limitation 1: [Critical] The Grader (`labjudge`) Directly `exec`s Agent-Submitted Code in the Same Process

Current Behavior & Risk: During agent execution (labexp), InaccessiblePaths prevents the agent from reading evaluation/judges.py. However, once the agent finishes, judges.py runs as labjudge and calls exec(compile(code_bytes, ...), ns) on the submitted validator.py inside the grader process. If an adversarial agent planted a Trojan payload inside validator.py that executes upon import/call—such as reading /opt/.../evaluation/judges.py, inspecting the caller stack frame, or writing a fake passing JSON directly to /var/lib/.../grader-output.json (which is writable by labjudge via ReadWritePaths)—that payload would execute with labjudge privileges during grading. (In our 41 experiments, all submitted code bundles are archived in mode-a/b-bundle.json and verified to contain only pure domain logic, but the architectural boundary itself is porous against active malice.)
Planned Remediation (Decoupling Untrusted Runner from Trusted Scorer):
1. Split grading into two separate processes/UIDs: an Untrusted Test Runner (which imports the candidate code with zero access to judges.py or grader-output.json) and a Trusted Scorer (root or a separate UID).
2. Pass test input vectors to the Untrusted Runner over a pipe (stdin/stdout), compare the returned JSON outputs against expected values in the Trusted Scorer, and have only the Trusted Scorer write grader-output.json.

Limitation 2: Denylist (`InaccessiblePaths`) Blind Spots & Mount Visibility

Current Behavior & Risk: InaccessiblePaths is a denylist mechanism. If any unredacted copy of the experiment payload remains accessible elsewhere (for example, on the mounted iso9660 CD-ROM path if not unmounted or restricted before agent launch), a shell-capable agent could read grading assets from the mount source. Additionally, the current unit file can be hardened further:
- Adding ProtectProc=invisible to hide /proc command-line arguments of other processes (such as root's guest_rpc.py).
- Adding RestrictAddressFamilies=none (since PrivateNetwork=yes blocks TCP/IP and abstract sockets, but filesystem UNIX domain sockets remain reachable unless address families are restricted).
- Adding MemoryMax=, TasksMax=, and LimitFSIZE= to prevent fork bombs or disk-filling DoS writes to outbox/.
Planned Remediation: Transition from a denylist (InaccessiblePaths) to an allowlist root filesystem (TemporaryFileSystem=/ + minimal BindReadOnlyPaths=) and audit the unit with systemd-analyze security.

Limitation 3: Tamper-Evident vs. Tamper-Proof Evidence

The SHA-256 manifest (manifest.json) makes the 7 evidence files tamper-evident across transit and storage, but it is not cryptographically tamper-proof against a host administrator who recalculates both the file and manifest hashes.

6. Two Real-World Operational Gotchas We Hit (and How We Fixed Them)

While running 40+ consecutive VM benchmarks, we encountered two non-obvious operational traps worth sharing:

Gotcha 1: QGA Timeouts When Passing Non-ASCII Strings Directly to `qm guest exec -c`

What We Observed

When invoking /usr/sbin/qm guest exec <vmid> -- /usr/bin/python3 -c <script_source> from the Proxmox host, early builds of E21 that included Japanese UTF-8 characters inside Python docstrings consistently failed with VM qga command 'guest-exec' failed - got timeout (exit code 255), leaving guest-ping unresponsive afterward.

Analysis & Best Practice Workaround

While virtio-serial itself is a byte stream and Proxmox's QGA client prepends guest-sync-delimited (\xff delimiter resynchronization), passing large multi-kilobyte scripts containing multi-byte UTF-8 characters directly as command-line arguments to Proxmox's /usr/sbin/qm guest exec CLI (which is implemented in Perl under the hood) triggered encoding/buffer handling stalls.

Our Fix: We enforced a unit-tested 100% Pure ASCII (all(b < 128)) policy on the bootstrap scripts passed via -c (guest_rpc.py and install.py) and passed all experiment payloads as Base64-encoded JSON (payload_b64) over stdin. Across 40+ subsequent VM runs, we observed zero QGA timeouts.
General Recommendation: Avoid passing raw multi-line script strings to qm guest exec -- python3 -c "...". Instead, pass a tiny Base64 decoder one-liner (python3 -c "import base64,sys;exec(base64.b64decode(sys.stdin.read()))") or write the file via guest-file-write first.

Gotcha 2: False-Positive LLM Safety Filter Blocks (`PROHIBITED_CONTENT`) During Infra Debugging

What We Observed

While using an AI coding assistant on our development machine to debug Proxmox socket states and systemd sandbox rules (not inside the benchmarked VMs themselves), the Gemini API occasionally blocked responses with:

This request was blocked by Gemini's filters. They can occasionally trigger by mistake on safe coding, security, or biology-related queries.

Analysis & Workaround

This is a known false positive (promptFeedback.blockReason: "PROHIBITED_CONTENT") documented on the Google AI Developers Forum that cannot be disabled via standard safety threshold sliders. Because the safety classifier evaluates the full conversation history, accumulating multiple inline snippets referencing raw hypervisor sockets (*.qmp, *.qga) and namespace boundary checks (InaccessiblePaths, runuser) likely resembled VM-escape or privilege-escalation exploit development.
Switching our debugging workflow to invoke only standard /usr/sbin/qm guest exec CLI wrappers and pre-committed scripts eliminated these false positives completely (and zero filter blocks ever occurred inside the 41 benchmark runs).

7. The 7 Tamper-Evident Artifacts Collected per Run

Every metric reported in our benchmark series (E04–E44) is extracted directly from the 7 JSON files collected from the VM and verified against manifest.json:

run-summary.json: High-level comparison of Mode A vs. Mode B (tests passed, turns, elapsed seconds, tool output chars).
usage.json: External relay ledger recording promptTokenCount, candidatesTokenCount, thoughtsTokenCount, and estimated_micro_usd per call.
execution-traces.json: Turn-by-turn log of tool calls, output character counts, and sensor/hook triggers.
independent-tests.json: Individual pass/fail results for all 20 hidden test cases (plus mutation kill audits where applicable).
mode-a-bundle.json: Full snapshot of the final Python files submitted by the baseline agent.
mode-b-bundle.json: Full snapshot of the final Python files submitted by the harnessed agent.
actions.json: Normalized JSON action payloads returned by the model on every turn.

🔗 Explore the Benchmark Results

Ready to see what we discovered when we ran 41 agent harness patterns through this sandbox? Check out the master benchmark index and our #1 ranked pattern deep dive:

🚀 Master Benchmark Hub: I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md

I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy

tappers — Sat, 31 Oct 2026 13:18:44 +0000

💡 Living Benchmark Index (Last Updated: October 31, 2026 | Verified in Isolated Sandbox VMs: 41 Patterns)

Across X/Twitter, Hacker News, and engineering blogs from OpenAI, Anthropic, Google, Martin Fowler (Thoughtworks), and Databricks, dozens of "Agent Harness Engineering" patterns have been proposed to make coding agents (Claude Code, Cursor, Codex, custom agents) faster, cheaper, and more reliable.

To find out which patterns actually work—and which ones make things worse, we built an isolated Proxmox VM benchmark infrastructure (AI Labo) and ran controlled A/B experiments across 41 harness patterns (E04–E44) using identical models, identical initial codebases, a hidden 20-case grading suite, and mutation testing.

0. Introduction: What Is "Harness Engineering"?

Every engineer deploying AI coding agents in production eventually hits the same wall:
"Why does the exact same frontier model (Claude, GPT, or Gemini) write a flawless patch in one turn today, and tomorrow burn 80,000 tokens editing the wrong files before breaking the test suite?"

When we started this benchmark series (E04–E24), we weren't setting out to define a new discipline—we were simply testing practical agent workflow ideas from developer communities bottom-up. As the data accumulated, we realized that our findings converged directly on what OpenAI, Anthropic, Google, and Thoughtworks call "Harness Engineering". We then expanded the suite to cover industry-standard patterns (E25–E42) and cross-model / cross-generation scaling laws (E43–E44).

At its core, Harness Engineering boils down to a single equation:

Agent = Model (Probabilistic Reasoning Core) + Harness (Deterministic Control System)

An LLM is a non-deterministic token predictor. Turning it into a reliable engineering system requires wrapping it in a Harness composed of three control layers:

🧭 Guides (Feed-Forward Control — Before Writing Code): Steering the agent onto the right path before it generates a patch (e.g., directory-pointer AGENTS.md, AST call-graph injection, two-phase read-only plan locks, spec-first gates).
🛡️ Sensors (Feedback Control — After Writing Code): Deterministically catching errors, regressions, and reward-hacking after an edit so the agent can self-correct (e.g., AST pre-commit hooks, actionable fix-hint linters, cryptographic test locks, loop-detection circuit breakers).
🧹 Context Hygiene (Attention & Memory Management): Ruthlessly stripping noise that dilutes model attention (e.g., CLI test output token killers, failed-attempt rollback pruning, Tree-sitter skeleton context, episodic MEMORY.md gotchas).

🛡️ Methodology & Anti-Cheating Sandbox Architecture
All 41 A/B experiments were executed inside disposable Debian 13 Proxmox VMs with zero network interfaces (0 NICs), controlled via QEMU Guest Agent (virtio-serial) and a budget-enforcing Mac relay, with Linux UID + systemd (ProtectSystem=strict / InaccessiblePaths) isolation separating the AI agent (labexp) from the hidden grading suite (labjudge).
👉 For a full technical deep dive into our evaluation infrastructure—including actual systemd unit files and security boundary limitations—read our companion post: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox.

1. Top 7 High-Impact Patterns (Ranked by Token Reduction & Accuracy)

Among all 41 verified patterns (E04–E44), these 7 patterns delivered the most dramatic improvements in input token reduction, first-turn pass rate (zero rework), and defect/mutant detection:

Rank	Pattern Name (Experiment ID)	Input Token Reduction	API Cost Reduction	Speedup	Key Empirical Finding
🥇 #1	Episodic Gotchas Memory (`MEMORY.md`) (`E20`: Injecting 5 repo-specific gotchas)	-95.5% (`46,560` ➔ `2,078`)	-83.7%	2.21x faster	Cold-start exploration hit 2 repo traps and stalled at `17/20`; injecting a 442-token `MEMORY.md` achieved a 1-turn `20/20` pass.
🥈 #2	Tree-sitter Skeleton Context (`E21`: Signature-only dependency context)	-94.9% (`45,270` ➔ `2,316`)	-82.6%	2.34x faster	Feeding full dependency implementations (78k chars) degraded accuracy to `17/20`; passing only function signatures & type contracts hit `20/20` in 1 turn.
🥉 #3	CLI Test Output Token Killer (`E16`: Structured JSON failure-diff filter)	-94.1% (`92,385` ➔ `5,445`)	-87.0%	1.54x faster	Stripping `PASS` lines and ANSI noise from raw `pytest` output (`117,391` ➔ `1,604` chars) enabled a clean 1-shot fix.
#4	Error-Trace Pruning (Failed-Attempt Rollback) (`E19`: Pruning failed patches & stack traces)	-93.0% (`59,944` ➔ `4,210`)	-80.3%	1.57x faster	Leaving failed code patches in history triggers self-imitation bias; pruning failed diffs into a 1-line lesson improved token efficiency by 14.2x.
#5	Progressive Tool Schema Search (`E14`: On-demand MCP tool loading)	-92.7% (`79,773` ➔ `5,844`)	-84.8%	1.94x faster	Eliminated the "schema tax" of injecting 32 MCP tool schemas every turn and reduced decoy tool misfires from `2 ➔ 0`.
#6	Milestone Context Compaction (`E12`: Save-game JSON state reset)	-91.9% (`35,869` ➔ `2,922`)	-72.2%	1.10x faster	Accumulating full chat history caused spec drift (`18/20`); compacting state to JSON at milestones achieved `20/20`.
#7	Advisor-Executor Model Routing (`E18`: Upfront architectural guidance memo)	-84.9% (`27,699` ➔ `4,172`)	-71.5%	1.78x faster	Injecting a cross-module design memo on Turn 1 eliminated 2 unplanned rework retries (`2 ➔ 0`).

2. Why Does the "Harness" Matter More Than the Prompt?

Across all 41 experiments, five structural anti-patterns consistently caused token explosions and accuracy failures—and five deterministic harness mechanisms solved them:

❌ Common Agent Anti-Pattern (Token Bloat & Drift)	✨ Verified Harness Solution (AI Labo)	Measured Impact
1. Dumping all tools & full history into a single free loop	Stage-Gated Directed Graphs (`E04`) & Progressive Tool Search (`E14`)	Input tokens `-74%` to `-93%`
2. Splitting tightly coupled tasks across role-based subagents	Subagent Folding + Progressive Skill Disclosure (`E05`)	2.66x faster, zero handoff loss
3. Letting the same agent write code, write tests, and self-review	Blind TDD (`E06`) / Adversarial Critic (`E10`) / Reproduction-First Gate (`E22`)	Mutant kill rate `0%` ➔ `100%`
4. Appending raw CLI test logs & failed patches to chat history	CLI Token Killer (`E16`) / Trace Pruning (`E19`) / AST Skeletons (`E09`, `E21`)	Input tokens `-93%` to `-95%`, 1-shot pass
5. Stuffing 20+ negative prohibitions into a monolithic `CLAUDE.md`	Deterministic Pre-Commit AST Hooks (`E11`) + Positive Few-Shot (`E24`)	0 architectural rule violations (`100%`)

Key Insight 1: Stage-Gated Graphs Work, but Role-Based Subagents Backfire (`E04` vs. `E05`)

In E04 (Loops vs. Graphs), replacing a single free-form loop with a stage-gated Directed Graph (resetting context after each module's deterministic verifier passes) cut input tokens by -74.4% and execution time by -48.0%.
Conversely, in E05 (Subagent Folding), splitting the exact same pipeline into role-based subagents (Architect ➔ Implementer ➔ QA) passing natural-language handoff summaries caused a "Telephone Game" effect: subtle formatting invariants (like signed zero -0) were lost in summarization, triggering rework that made the multi-agent setup 2.66x slower and 2.2x more expensive than a folded single loop with on-demand skill loading.

Key Insight 2: Self-Testing and Self-Review Yield a 0% Mutation Kill Rate (`E06`, `E08`, `E10`, `E22`)

In E06 (Blind TDD) and E10 (Adversarial Critic), when a single agent context wrote both the implementation and the unit tests (or reviewed its own code), the agent unconsciously wrote weak "happy-path" tests tailored to pass its own bugs—resulting in a 0/6 (0%) mutation kill rate against boundary mutants.
Isolating the test author into a Blind Tester (E06) that sees only SPEC.md (never the implementation), enforcing Spec-First SDD (E08), using an Adversarial Critic (E10) rewarded for finding counterexamples, or locking production files until a failing reproduction test is recorded (Reproduction-First Gate E22) pushed test mutation kill rates and hidden-suite accuracy to 100%.

Key Insight 3: 90% of Input Tokens Are Wasted on Exploration, Unused Schemas, and Raw Logs (`E09`, `E14`, `E16`, `E19`, `E21`)

Across every experiment, the actual fixed Python code generated by the LLM was only 1,200 to 1,800 output tokens.
Why did baseline agents consume 30,000 to 92,000 input tokens? Because of blind grep exploration (E09), 30 unused MCP tool schemas (E14), 76 passing test lines in raw terminal output (E16), failed code diffs left in context (E19), and full source bodies of dependency files (E21).
Filtering these deterministically in the harness cuts API costs by 80%–95% while simultaneously preventing Lost-in-the-Middle attention degradation.

Key Insight 4: A Cheap Model + Great Harness Beats an Unharnessed Frontier Model (`E43` & `E44`)

In E43 (Cross-Model Giant Killing), we pitted an unharnessed frontier model (gemini-4-argon) in a raw loop against a model costing 1/8th as much (gemini-3.1-flash-lite) equipped with our combined golden harness (E09 AST Map + E20 MEMORY.md + E21 Skeleton + E11 Hook + E16 Token Killer + E39 Remediation Sensor).
The harnessed lightweight model achieved 20/20 (100%) in 1 turn (5.6s, $0.0029), beating the unharnessed frontier model (15/20 in 3 turns, 39.1s, $0.1538)—delivering better accuracy at 6.95x the speed and -98.1% lower cost.
In E44 (Model Evolution vs. Harness Boundary), testing across four model generations (2.5-flash ➔ 3.5-flash ➔ 3.8-flash ➔ 4-argon) revealed that while syntax-assist harnesses (like diff format helpers) gradually become less critical as models improve, repository-knowledge & physical guardrail harnesses (MEMORY.md, output filters, circuit breakers, cryptographic test locks) remain indispensable even for the latest frontier models.

3. Master Benchmark Table: All 41 Patterns (`E04`–`E44`)

(Note: Each experiment represents a controlled n=1 A/B case study on a 3-module domain pipeline evaluated against 20 hidden test cases. Detailed deep-dive posts are linked as they are published.)

ID	Harness Pattern	Baseline (Mode A)	Harnessed (Mode B)	Empirical A/B Result (Hidden 20-Case Suite)
🕸️ E04	Loops vs. Graphs	Single free-form ReAct loop with all tools and full history	Stage-gated Directed Graph with per-stage deterministic verifiers and context resets	Both 20/20 Pass \| Calls: `7 ➔ 3` (-57.1%) \
🪆 E05	Subagent Folding	Role-based subagents (Architect/Coder/QA) passing handoff summaries	Single folded loop with on-demand Progressive Skill Disclosure	Both 20/20 Pass, Folded wins \| Calls: `5 ➔ 2` (-60.0%) \
🧪 E06	Blind TDD Worktrees	Single agent writing both tests and implementation in one context	Agent 1 (Blind Tester) writes tests from spec only; Agent 2 implements in separate worktree	Accuracy: `19/20` ➔ `20/20` \
⚙️ E07	Code-Mode Dynamic Workflows	Iterating multi-module audits inside a single chat history	Agent writes a deterministic orchestrator script invoking fresh worker contexts	Accuracy: `18/20` ➔ `20/20` \
📐 E08	Spec-First SDD	Vibe coding directly from a high-level prompt with iterative fixes	Locking a strict contract (`SPEC.md`) and change plan before unlocking code edits	Accuracy: `19/20` ➔ `20/20` (0 rework) \
🗺️ E09	AST Dependency Map vs. Raw Grep	Giving the agent `grep` and file-read tools to explore on its own	Pre-injecting a 0-token deterministic AST symbol & call-graph mini-map	Both 20/20 Pass \
⚔️ E10	Adversarial Critic (Maker-Checker)	Asking the implementing agent to self-review before submitting	Separate Adversarial Critic agent rewarded for writing falsifying counterexample tests	Both 20/20 impl \| Test Mutant Kill Rate: Self-Review `0/6 (0%)` ➔ Adversarial Critic `6/6 (100%)`
🛡️ E11	Architecture Guardrail Hooks	Listing coding rules & line limits in a long system prompt	Enforcing AST architecture rules & line limits via a deterministic Pre-Commit Hook	Both 20/20 functional \| Arch Rule Compliance: Prompt-only `4/5` (violated line cap) ➔ AST Hook `5/5 (100%)`
🗜️ E12	Milestone Context Compaction	Accumulating all file reads and logs across multi-step tasks	Resetting chat history at milestones into a compact JSON state object	Accuracy: `18/20` ➔ `20/20` \
🏆 E13	Parallel Best-of-N Tournament	Retrying sequential fixes inside a single stuck session	Running 2 parallel lightweight branches + deterministic benchmark judge selection	Both 20/20 Pass \| Judge selected `O(n)` `0.96ms` branch over `202.95ms` `O(n²)` branch \
🔍 E14	Progressive Tool Schema Search	Injecting all 32 MCP tool JSON schemas on every turn	Providing a lightweight `search_tools` meta-tool to load schemas on demand	Both 20/20 Pass \
✂️ E15	Search-Replace Blocks vs. Unified Diff	Outputting standard Unified Diffs (`@@ -l,s +l,s @@`) or full rewrites	Outputting line-number-free `SEARCH/REPLACE` anchor blocks	Both 20/20 Pass \
🧹 E16	CLI Test Output Token Killer	Returning raw `pytest` stdout (76 PASS lines + ANSI + deep traces)	Wrapper strips passing tests and returns only compact JSON failure diffs	Both 20/20 Pass \
💡 E17	Actionable Fix-Suggestion Linter	Returning bare compiler/linter error codes (`E402 Invalid state`)	Injecting a concrete 1-line `💡 Fix Hint` code recipe directly into the linter error	Both 20/20 Pass \
🧭 E18	Advisor-Executor Model Routing	Running a single worker prompt across all planning and editing turns	Injecting a concise architectural memo from an Advisor on Turn 1 before Executor edits	Both 20/20 Pass \| Rework retries: `2 ➔ 0` \
🌿 E19	Error-Trace Pruning (Failed Rollback)	Keeping failed code patches and long error traces in chat history	Pruning failed patches on test failure and leaving only a 1-line negative lesson	Both 20/20 Pass \| Input tokens: `59,944 ➔ 4,210` (-93.0%, 14.2x efficiency) \| Time: -36.3% \| Cost: -80.3%
🧠 E20	Episodic Gotchas Memory (`MEMORY.md`)	Starting every task cold and rediscovering repo traps via test failures	Injecting a 5-item (`+442` token) `MEMORY.md` of repo gotchas distilled from prior runs	Accuracy: `17/20` ➔ `20/20` \
🦴 E21	Tree-sitter Skeleton Context	Loading full implementation bodies of peripheral dependency files	Folding dependency bodies into function signatures, types, and docstrings only	Accuracy: `17/20` ➔ `20/20` \
🚦 E22	Reproduction-First Gate	Letting the agent immediately patch production code upon reading an issue	Locking production files until a minimal bug-reproducing test fails (Red) first	Accuracy: `17/20` ➔ `20/20` \
📦 E23	Atomic Multi-File Batch Edit	Editing one file at a time and running tests in broken intermediate states	Applying coupled multi-file patch sets atomically in a single turn before testing	Accuracy: `17/20` ➔ `20/20` \
✨ E24	Positive Few-Shot vs. Negative Rules	Listing 20+ negative prohibitions ("NEVER do X") in the system prompt	Replacing negative rules with a single 20-line affirmative Golden Code Example	Accuracy: `16/20` ➔ `20/20` \
⚡ E25	Prefix-Stable Prompt Caching	Mutating timestamps and task tables at the top of the system prompt	Freezing an append-only prefix and appending state diffs only at the tail	Accuracy: `17/20` ➔ `20/20` \
🔒 E26	Two-Phase Read-Only Plan Lock	Allowing interleaved Read and Write tool calls from Turn 1	Physically locking Write tools during Phase 1 exploration until plan approval	Accuracy: `17/20` ➔ `20/20` \
🗂️ E27	Structured Hypothesis Table	Debugging complex multi-bug failures via unstructured monologue	Requiring a structured `THINK.md` Hypothesis Elimination Matrix before editing	Accuracy: `19/20` ➔ `20/20` \
🔌 E28	Loop-Detection Circuit Breaker	Allowing unguarded retry loops on repeated test failures	Detecting oscillating edits (`A➔B➔A`), rolling back, and forcing a strategy shift	Both 20/20 Pass \
🎯 E29	Root-Cause Error Clustering	Dumping 50 cascading downstream type/runtime errors raw	Clustering cascading errors via dependency graph into 1 root-cause signature error	Both 20/20 Pass \| Calls: `3 ➔ 1` \
⏩ E30	Dependency-Scoped Incremental Tests	Running the entire repository test suite after every single file edit	Running only AST-impacted tests per edit and gating full suite at final submit	Accuracy: `16/20` ➔ `20/20` \
🌳 E31	Parameterized AST Codemod Tool	Manually editing 15+ call sites across files via text replacement	Providing a deterministic AST transformation primitive (`ast_transform_calls`)	Both 20/20 Pass \
🔍 E32	Pre-Submit Blast-Radius & Diff Audit	Accepting `SUBMIT` immediately once functional tests pass	Auditing `git diff` for debug leftovers, out-of-scope symbol edits, and entropy	Accuracy: `16/20` ➔ `20/20` \
🎲 E33	Property-Based Fuzzing Sensor	Validating patches against fixed hand-written example unit tests only	Running an invariant property fuzzer that feeds minimal falsifying counterexamples	Both 20/20 Pass \| Calls: `3 ➔ 1` \
📸 E34	Runtime Variable-Snapshot Injection	Returning standard stack traces (file, line number, exception message)	Injecting live `locals()` variable snapshots at the failing frame into the trace	Accuracy: `16/20` ➔ `20/20` \
📖 E35	Directory-Pointer `AGENTS.md`	Stuffing a 500-line monolithic encyclopedia `AGENTS.md` into initial context	Using a 30-line pointer `AGENTS.md` that routes the agent to scoped docs on demand	Accuracy: `19/20` ➔ `20/20` \
🤝 E36	Shift-Handover Harness	Passing unstructured chat summary text across multi-session resets	Enforcing a 3-artifact mechanical handover (`feature_list.json`, `init.sh`, `progress.txt`)	Accuracy: `19/20` ➔ `20/20` \
🪜 E37	Hypothesis-Pivot Ladder	Allowing blind micro-retries when the exact same error repeats	Locking current approach on repeated error signature, restoring via git, forcing 3 new hypotheses	Accuracy: `19/20` ➔ `20/20` \
🔐 E38	Cryptographic Test-Tampering Lock	Asking the agent via prompt not to modify or weaken existing tests	Enforcing SHA-256 hash locks on test files to physically reject test tampering	Accuracy: `19/20` ➔ `20/20` \
🧭 E39	Guides & Sensors Remediation	Returning standard linter/compiler errors on architectural layer violations	Pairing upfront boundary maps (Guides) with DI remediation code recipes on violation (Sensors)	Accuracy: `19/20` ➔ `20/20` \
🐾 E40	Behavioral Action-Sequence Gate	Judging only end-to-end test outcomes regardless of investigation steps	Blocking symbol edits unless `find_references` / `grep` was executed on that symbol first	Both 20/20 Pass \
🪶 E41	Zero-Tax Minimal Primitives (Pi-Style)	Heavy wrapper with 15+ custom tool schemas and verbose instructions	Stripping tools down to 2 minimal primitives (`bash` and `file_edit`)	Accuracy: `19/20` ➔ `20/20` \
🧹 E42	Post-Green Entropy Sweep Pass	Stopping immediately as soon as functional tests turn green	Triggering an automated 1-turn cleanup pass (deduping helpers, naming alignment) after green	Accuracy: `19/20` ➔ `20/20` \
⚔️ E43	Cross-Model Giant Killing	Unharnessed frontier model (`gemini-4-argon`) in a raw loop	1/8th-cost `gemini-3.1-flash-lite` equipped with combined Golden Harnesses (`E09+E11+E16+E20+E21+E39`)	Accuracy: Frontier Raw `15/20` vs. Flash-Lite Harnessed `20/20` \
🧬 E44	Model Evolution vs. Harness Boundary	Syntax-only harnesses across model generations (`2.5` ➔ `3.5` ➔ `3.8` ➔ `4-Argon`)	Repo-knowledge & physical guardrail harnesses (`E16+E20+E28+E35+E38`) across generations	Both 20/20 Pass \

4. A 5-Point Checklist You Can Apply to Your Agent Harness Tomorrow

Never return raw CLI test or linter stdout to the LLM (E16 / E17) Wrap your test runner to strip passing tests and ANSI codes. Return only a structured JSON of failed assertions plus a 1-line 💡 Fix Hint showing the expected helper or pattern.
Replace negative prompt prohibitions with Pre-Commit Hooks and 1 Golden Example (E11 / E24) Listing 20 "DO NOT do X" rules in CLAUDE.md triggers the Pink Elephant effect. Show one 20-line affirmative code example and enforce constraints mechanically via AST save/commit hooks.
Maintain a 5-to-15 line MEMORY.md of repository-specific gotchas (E20 / E35) Instead of a 500-line encyclopedia, keep AGENTS.md as a 30-line directory pointer and maintain a tiny MEMORY.md listing only non-obvious domain invariants that previously caused test failures.
Use line-number-free SEARCH/REPLACE blocks + Atomic Multi-File Batches (E15 / E23) Unified diffs (@@ -l,s +l,s @@) frequently fail due to LLM line-counting errors. Apply multi-file interface changes atomically in a single turn before running verification tests.
Separate the Test Author context from the Implementer context (E06 / E22) Never trust an agent to grade its own homework in the same context. Lock production files until a reproduction test fails (E22), or generate tests in a blind worktree that only sees the spec (E06).

🔗 Explore the Series

🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md

DEV Community: tappers

Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md

Metadata & Experiment Identity

1. Executive Summary (TL;DR)

Key Result

2. Experiment Architecture & Repository Gotchas Under Test

The 5 Non-Obvious Repository Gotchas in the Initial Codebase

3. Inside the Prompts: What Mode B Saw vs. What Mode A Endured

3.1 The Exact 5-Item MEMORY.md Injected into Mode B (+442 tokens)

3.2 The 56,605-Character Failure Trace That Flooded Mode A (Cold-Start)

4. Code-Level Postmortem: Why Mode A Failed 3 out of 20 Test Cases

All 20 Hidden Test Case Results (Mode A vs. Mode B)

5. Practical Takeaway: How to Structure Your Own MEMORY.md

🔗 Continue Reading the Series

How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox

TL;DR

1. Why Naive Single-User Execution Distorts Agent Benchmarks

2. System Architecture (Mac × Tailscale × Proxmox × 0-NIC VM)

Full Component Matrix (15 Components, Privileges, and Security Roles)

3. Core Mechanism 1: OS-Level Separation via Linux UIDs and systemd

Agent Execution Unit (ai-lab-e20-01-agent.service)

Independent Grader Unit (ai-lab-e20-01-grade.service)

4. Core Mechanism 2: Dual Serial Channels (serial0 vs. virtio-serial) & Budget-Gated Relay

End-to-End Execution Lifecycle (60–90 Seconds per Experiment)

External Token Accounting & Pre-Call Budget Enforcement (usage.json)

5. Known Limitations & Future Security Roadmap

Limitation 1: [Critical] The Grader (labjudge) Directly execs Agent-Submitted Code in the Same Process

Limitation 2: Denylist (InaccessiblePaths) Blind Spots & Mount Visibility

Limitation 3: Tamper-Evident vs. Tamper-Proof Evidence

6. Two Real-World Operational Gotchas We Hit (and How We Fixed Them)

Gotcha 1: QGA Timeouts When Passing Non-ASCII Strings Directly to qm guest exec -c

What We Observed

Analysis & Best Practice Workaround

Gotcha 2: False-Positive LLM Safety Filter Blocks (PROHIBITED_CONTENT) During Infra Debugging

What We Observed

Analysis & Workaround

7. The 7 Tamper-Evident Artifacts Collected per Run

🔗 Explore the Benchmark Results

I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy

0. Introduction: What Is "Harness Engineering"?

1. Top 7 High-Impact Patterns (Ranked by Token Reduction & Accuracy)

2. Why Does the "Harness" Matter More Than the Prompt?

Key Insight 1: Stage-Gated Graphs Work, but Role-Based Subagents Backfire (E04 vs. E05)

Key Insight 2: Self-Testing and Self-Review Yield a 0% Mutation Kill Rate (E06, E08, E10, E22)

Key Insight 3: 90% of Input Tokens Are Wasted on Exploration, Unused Schemas, and Raw Logs (E09, E14, E16, E19, E21)

Key Insight 4: A Cheap Model + Great Harness Beats an Unharnessed Frontier Model (E43 & E44)

3. Master Benchmark Table: All 41 Patterns (E04–E44)

4. A 5-Point Checklist You Can Apply to Your Agent Harness Tomorrow

🔗 Explore the Series

3.1 The Exact 5-Item `MEMORY.md` Injected into Mode B (`+442 tokens`)

3.2 The 56,605-Character Failure Trace That Flooded Mode A (`Cold-Start`)

5. Practical Takeaway: How to Structure Your Own `MEMORY.md`

3. Core Mechanism 1: OS-Level Separation via Linux UIDs and `systemd`

Agent Execution Unit (`ai-lab-e20-01-agent.service`)

Independent Grader Unit (`ai-lab-e20-01-grade.service`)

4. Core Mechanism 2: Dual Serial Channels (`serial0` vs. `virtio-serial`) & Budget-Gated Relay

External Token Accounting & Pre-Call Budget Enforcement (`usage.json`)

Limitation 1: [Critical] The Grader (`labjudge`) Directly `exec`s Agent-Submitted Code in the Same Process

Limitation 2: Denylist (`InaccessiblePaths`) Blind Spots & Mount Visibility

Gotcha 1: QGA Timeouts When Passing Non-ASCII Strings Directly to `qm guest exec -c`

Gotcha 2: False-Positive LLM Safety Filter Blocks (`PROHIBITED_CONTENT`) During Infra Debugging

Key Insight 1: Stage-Gated Graphs Work, but Role-Based Subagents Backfire (`E04` vs. `E05`)

Key Insight 2: Self-Testing and Self-Review Yield a 0% Mutation Kill Rate (`E06`, `E08`, `E10`, `E22`)

Key Insight 3: 90% of Input Tokens Are Wasted on Exploration, Unused Schemas, and Raw Logs (`E09`, `E14`, `E16`, `E19`, `E21`)

Key Insight 4: A Cheap Model + Great Harness Beats an Unharnessed Frontier Model (`E43` & `E44`)

3. Master Benchmark Table: All 41 Patterns (`E04`–`E44`)