DEV Community

tappers
tappers

Posted on Edited on Originally published at tappers.hashnode.dev

How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox

#ai #security #proxmox #sandbox

💡 Methodology & Eval Infrastructure Deep Dive

This article is the Methodology & Architecture companion to our AI Labo Agent Harness Engineering Benchmark Series (41 Patterns Tested).
Here we explain how we designed an isolated evaluation infrastructure combining Proxmox VE (0-NIC VMs), QEMU Guest Agent (virtio-serial), a budget-enforcing Mac relay, and Linux UID / systemd sandboxing to prevent test peeking, reward hacking, and API key leaks—along with a candid look at the current security boundaries and remaining limitations of our setup.

TL;DR

When benchmarking autonomous AI coding agents fairly and safely, engineers inevitably run into a three-way dilemma (trilemma):

  1. Eval Cheating / Reward Hacking: If the AI agent and the grading test suite share the same user privileges and filesystem view, a capable model may inspect the test cases to hardcode edge-case branches or modify the test files (pytest.skip(), weakening assertions) instead of fixing the underlying bug.
  2. API Key Exposure & Runaway Spend: Passing GEMINI_API_KEY or OPENAI_API_KEY directly into the guest environment creates a risk of credential leakage via logs/prompts and runaway billing if an agent enters an infinite retry loop.
  3. Network Isolation vs. Cloud LLM Access: Stripping network interfaces (NICs) from the sandbox VM prevents unauthorized external traffic, but also cuts off direct HTTPS access to cloud LLM APIs.

To solve these challenges simultaneously, we combined a mini-PC running Proxmox VE with a macOS controller over a 1-to-1 Tailscale link to build the AI Labo Evaluation Sandbox:

  • Network-Isolated VMs (0 NICs) + Inspected Relay Proxy: Experiment VMs have zero network interfaces. The sole external data path is structured JSON passed over QEMU Guest Agent (virtio-serial) to a macOS relay that loads the API key from the OS Keychain into memory once and enforces strict call, token, and USD budget caps (zero API keys inside the VM).
  • OS-Level Separation via Linux UIDs and systemd (ProtectSystem=strict / InaccessiblePaths): While the agent (labexp) runs, ProtectSystem=strict mounts the filesystem read-only (except for pre-created target .py files bound via ReadWritePaths), and InaccessiblePaths blocks access to the grading suite (evaluation/).
  • 0.8s Cached ISO Generation + Disposable Full-Clone VMs: Pre-verified .deb packages (verified via InRelease ➔ Packages SHA-256 chains) and experiment payloads are burned into read-only iso9660 CD-ROM images in 0.8 seconds, mounted onto a fresh full clone of a stopped Debian 13 base VM, and executed end-to-end in 60–90 seconds per experiment.

1. Why Naive Single-User Execution Distorts Agent Benchmarks

Anyone who has evaluated coding agents at scale has seen Reward Hacking firsthand: the agent reports that all tests pass on Turn 1, but git diff reveals that instead of fixing the bug, it modified the test runner or weakened an assertion.

Running the Agent (Generator) and the Grader (Verifier) under the same user and writable filesystem introduces four structural failure modes:

Failure Mode in Naive Setups Root Cause AI Labo Mitigation
1. Test Peeking (Eval Cheating) If hidden grading tests are readable by the agent's UID, file-search tools can read test inputs and hardcode answers. Separate UID (labjudge, mode 0750) + systemd InaccessiblePaths denies access (EACCES) to the grading directory during agent execution.
2. Tampering with Tests or Configs A writable working directory allows the agent to overwrite test files, runners, or config files. ProtectSystem=strict mounts the filesystem read-only; ReadWritePaths permits in-place writes only to the existing inodes of target .py files.
3. API Key Leaks & Unbounded Spend Storing API keys in guest environment variables risks exposure in logs and unthrottled API loops. Zero NICs and zero API keys inside the VM. The Mac relay enforces maxOutputTokens, cumulative token limits, and micro-USD budget caps.
4. Cross-Experiment Environment Drift Reusing the same environment across runs leaves behind temp files, caches, or altered state. Every experiment full-clones a stopped Debian 13 Golden Base VM (VM9300) and mounts immutable iso9660 CD-ROMs.

💡 Why Proxmox VMs + systemd Instead of Docker Containers?
You can achieve similar logical separation in Docker by splitting agent and grader containers, passing --network none and --read-only, and keeping secrets on a host proxy.
We chose Proxmox VMs + systemd for two reasons:

  1. Kernel Isolation: Avoiding shared host kernels between the controller/host and disposable guest environments.
  2. Full-Clone & ISO Reproducibility: LVM full clones combined with immutable iso9660 virtual CD-ROMs make it trivial to pin and audit exact VM configuration SHA-256 digests before and after every run.

2. System Architecture (Mac × Tailscale × Proxmox × 0-NIC VM)

The infrastructure is split across a macOS Controller/Relay and a Proxmox VE Host connected via a 1-to-1 peer-restricted Tailscale VPN:

+-------------------------------------------------------------------------+
| 1. Controller & Relay Host (macOS)                                      |
|  - OS Keychain / 0400 Secret --> Loaded once into memory at startup     |
|  - Experiment Orchestrator   --> Drives Phase 1 (ISO) & Phase 2 (Run)   |
|  - Private API Relay         --> Enforces call, token & micro-USD caps  |
|  - Evidence Store            --> Verifies & saves 7 artifacts + SHA256  |
+------------------------------------+------------------------------------+
                                     |
         1-to-1 Peer-Restricted VPN  |  (SSH Control + Peer-Pinned HTTP)
                                     v
+-------------------------------------------------------------------------+
| 2. Virtualization Host (Proxmox VE Mini-PC)                             |
|  - Host Controller (driver.py)  --> Allowlisted qm/pvesh + Watchdogs    |
|  - Local ISO Builder            --> Verified .deb cache -> iso9660 CDs  |
|  - Golden Base VM (VM9300)      --> Stopped Debian 13 clone source      |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | 3. Disposable Sandbox VM (Full Clone per Experiment / 0 NICs)     |  |
|  |  - serial0 (/dev/ttyS0) : One-way boot & QGA readiness signal     |  |
|  |  - virtio-serial (QGA)  : Bidirectional RPC via `qm guest exec`   |  |
|  |  - ide2 & ide3 (CD-ROM) : Read-only offline packages & payload    |  |
|  |                                                                   |  |
|  |  +-----------------------------+  +----------------------------+  |  |
|  |  | Agent Sandbox (UID: labexp) |  | Grader Sandbox (labjudge)  |  |  |
|  |  | - PrivateNetwork=yes        |  | - Runs AFTER agent exits   |  |  |
|  |  | - ProtectSystem=strict      |  | - Evaluates 20 hidden cases|  |  |
|  |  | - InaccessiblePaths=/opt/.. |  |   + 6 boundary mutants     |  |  |
|  |  |   /evaluation (EACCES)      |  | - Writes grader-output.json|  |  |
|  |  +-----------------------------+  +----------------------------+  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

Full Component Matrix (15 Components, Privileges, and Security Roles)

Environment Component Execution Identity Primary Role Security & Integrity Meaning
External Cloud Cloud LLM API Managed Service countTokens & generateContent Accepts HTTPS only from the Mac relay; unreachable from the guest VM.
Mac Controller OS Keychain / Secret Security.framework / 0400 Stores LLM API key Never written to .env or VM disks; read once into relay memory at startup.
Mac Controller Orchestrator Local user Runs Phase 1 (ISO) & Phase 2 (VM run) Invokes the Proxmox host driver over SSH and verifies final evidence hashes.
Mac Controller Private API Relay Local user (serve_once.py) Proxies LLM calls & enforces budgets Enforces max calls, input/output token caps (maxOutputTokens), and micro-USD caps.
Mac Controller Evidence Storage Local filesystem Stores 7 evidence files + SHA256 manifest Preserves tamper-evident records of prompts, actions, token usage, and test results.
Private VPN Tailscale 1-to-1 Link Encrypted Mesh VPN Mac ⇔ Proxmox control & relay traffic Pins source peer IP (100.x.x.x) so LAN or external hosts cannot hit the relay.
Proxmox Host Host Controller root (driver.py) Clones VM, drives QGA, runs watchdogs Executes only allowlisted qm / pvesh commands with stage watchdogs.
Proxmox Host Local ISO Builder builder (builder.py) Packs verified .deb cache & code into ISOs Verifies .deb SHA-256 hashes from signed InRelease ➔ Packages metadata.
Proxmox Host Golden Base VM Stopped (VM9300) Clean Debian 13 clone source Kept stopped with pinned config SHA-256 to eliminate cross-run drift.
Proxmox Host Audit Broker VM Isolated VM (VM100) Holds ledger hash pins Verified before and after every experiment to ensure host state integrity.
Sandbox VM (0 NIC) serial0 (/dev/ttyS0) Kernel / Bootstrap One-way boot progress notification Emulated 16550 UART used only during initial boot to signal when offline QGA installation finishes.
Sandbox VM (0 NIC) virtio-serial (QGA) root (guest_rpc.py) Bidirectional host-guest RPC channel Uses org.qemu.guest_agent.0 (.qga socket) via qm guest exec to exchange JSON.
Sandbox VM (0 NIC) Virtual CD-ROMs × 2 Read-only (iso9660) Delivers .deb deps (ide2) & payload (ide3) Immutable media inside the guest; install.py verifies payload SHA-256 before extraction.
Sandbox VM (0 NIC) Agent Sandbox Unprivileged labexp Runs the AI agent under test (Mode A / B) ProtectSystem=strict mounts / read-only; InaccessiblePaths blocks access to evaluation/.
Sandbox VM (0 NIC) Grader Sandbox Unprivileged labjudge Runs independent grading suite after agent exits Starts only after agent.service stops; evaluates snapshots against 20 hidden test cases.

3. Core Mechanism 1: OS-Level Separation via Linux UIDs and systemd

Inside each experiment VM, the Agent (labexp) and the Independent Grader (labjudge) run under separate Linux UIDs and separate systemd mount namespaces.

Before starting the agent, the root installer (guest/install.py) sets strict filesystem ownership and permissions:

  • Grading directory (/opt/.../evaluation): Owned by root:labjudge with mode 0750 (files 0640), preventing labexp from reading it even at the basic DAC level.
  • Work directories (work_cold_start/, work_episodic_memory/): Owned by root:root with mode 0555 (read/traverse only). Only the 3 target Python files inside each directory (validator.py, settlement.py, statement.py) are pre-created with labexp ownership (0644).

Below are the actual systemd unit files from experiment E20 (VM224):

Agent Execution Unit (ai-lab-e20-01-agent.service) 

[Unit]
Description=AI Lab E20 Episodic Gotchas Memory Isolated Agent Runner (VM224)

[Service]
Type=oneshot
User=labexp20
Group=labexp20
WorkingDirectory=/var/lib/ai-lab-e20-vm224-01
ExecStart=/usr/bin/python3 -I -B /opt/ai-lab-e20-vm224-01/guest/run_guest.py
Restart=no
KillMode=control-group
TimeoutStartSec=240
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateNetwork=yes
ProtectSystem=strict
ProtectHome=yes
CapabilityBoundingSet=
AmbientCapabilities=
# Under ProtectSystem=strict, ONLY outbox and the 6 pre-created target .py files are writable:
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/outbox
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_cold_start/validator.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_cold_start/settlement.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_cold_start/statement.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_episodic_memory/validator.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_episodic_memory/settlement.py
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/work_episodic_memory/statement.py

# Grading code (evaluation/), initial fixtures, and evidence dirs are overlaid with 0000 mounts:
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/root
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/evidence
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/grade
InaccessiblePaths=/opt/ai-lab-e20-vm224-01/evaluation
InaccessiblePaths=/opt/ai-lab-e20-vm224-01/fixtures
Enter fullscreen mode Exit fullscreen mode

Independent Grader Unit (ai-lab-e20-01-grade.service)

Only after agent.service terminates and the root RPC handler (guest_rpc.py) copies a snapshot of the edited files into grade/ does the grader service launch as labjudge20:

[Unit]
Description=AI Lab E20 Episodic Gotchas Memory Isolated Grader (VM224)

[Service]
Type=oneshot
User=labjudge20
Group=labjudge20
WorkingDirectory=/var/lib/ai-lab-e20-vm224-01/grade
ExecStart=/usr/bin/python3 -I -B /opt/ai-lab-e20-vm224-01/evaluation/grade_comparison.py
Restart=no
KillMode=control-group
TimeoutStartSec=30
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateNetwork=yes
ProtectSystem=strict
ProtectHome=yes
CapabilityBoundingSet=
AmbientCapabilities=
# The grader can only write to the single grader-output.json file:
ReadWritePaths=/var/lib/ai-lab-e20-vm224-01/grader-output.json
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/root
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/evidence
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/inbox
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/outbox
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/work_cold_start
InaccessiblePaths=/var/lib/ai-lab-e20-vm224-01/work_episodic_memory
Enter fullscreen mode Exit fullscreen mode

This layered setup enforces three protections during agent execution:

  1. Blocking Access to Grading Contents (DAC 0750 + InaccessiblePaths): InaccessiblePaths bind-mounts an empty directory with 0000 permissions over the target path in the unit's mount namespace. While listing /opt/ai-lab-.../ still shows the directory entry evaluation, any attempt to read or traverse into it fails immediately with Permission denied (EACCES).
  2. Preventing File Creation, Deletion, or Rename (ProtectSystem=strict + Root-Owned 0555 Workdir): The primary mechanism blocking file creation or deletion is ProtectSystem=strict, which mounts the entire filesystem read-only and bind-mounts only the 6 listed .py files as writable (backed up by root-owned 0555 directory permissions so labexp cannot chmod the parent directory). Running Python with -I (isolated mode) also disables user site-packages and PYTHONPATH injection.
  3. Structured JSON Actions Only: In our benchmark suite (E04–E44), the agent does not execute arbitrary shell commands; both the Mac relay (serve_once.py) and guest harness (pipeline_checks.py) validate structured JSON tool calls against a strict schema, verify ast.parse() syntax, and overwrite existing file inodes in-place via os.open(..., os.O_WRONLY | os.O_TRUNC | os.O_NOFOLLOW).

💡 How We Use Mutation Testing (Mutation Kill Rate)
In experiments where the agent submits both production code and a test suite (test_suite.py in E06 Blind TDD, E07, E08, E10), passing functional tests isn't enough—we also need to know if the agent wrote weak, self-serving tests.
Our grader (judges.py) runs the agent's test_suite.py against 6 deliberate boundary mutants of the reference implementation (e.g., changing >= 10000 to > 10000, removing isinstance(x, bool) checks, omitting -0 formatting) to measure the exact Mutation Kill Rate (0/6 to 6/6).

4. Core Mechanism 2: Dual Serial Channels (serial0 vs. virtio-serial) & Budget-Gated Relay

Because the experiment VM has zero network interfaces (verified at boot by asserting that /sys/class/net contains only lo), it is a network-isolated VM whose sole external channel is an inspected host-mediated relay.

We use two distinct serial mechanisms between the Proxmox host and the guest VM for different phases of the lifecycle:

  1. serial0 (/dev/ttyS0 inside guest — emulated 16550 UART):
    • Role: Used exclusively for one-way boot progress notifications during the initial 25-second offline bootstrap (before qemu-guest-agent is installed from the CD-ROM). The host controller (driver.py) monitors /var/run/qemu-server/<vmid>.serial0 until it receives the common_prepared_QGA_acceptance_pending JSON marker.
  2. virtio-serial (org.qemu.guest_agent.0 / .qga socket — QEMU Guest Agent):
    • Role: Used for all subsequent bidirectional RPC (qm guest exec <vmid>), including payload installation, polling the agent's Outbox, delivering LLM responses to the Inbox, and retrieving the 7 final evidence files.

End-to-End Execution Lifecycle (60–90 Seconds per Experiment)

Step Phase & Actor Operation Typical Duration / Security Gate
1 Phase 1: ISO Build
(Mac ➔ Proxmox)		 Packs SHA256-verified .deb cache and experiment payload into read-only iso9660 image. 0.8s (with warm package cache)
2 Phase 2: Pre-Flight
(Mac & Proxmox)		 Mac loads API key from Keychain into memory; Proxmox verifies VM9300 & broker ledger hashes. Zero API keys written to disk
3 Clone & Offline Boot
(Proxmox ➔ VM)		 Full-clones VM9300, attaches 2 CD-ROMs (ide2, ide3), verifies 0 NICs, boots VM, installs QGA offline. 25s (notifies host via serial0)
4 Workspace Setup
(Proxmox ➔ QGA)		 install.py verifies payload SHA-256, sets 0750 on evaluation/ and 0555 on work/, starts agent.service. Enforces UID & systemd locks
5 Agent Turn Loop
(VM ⇔ PVE ⇔ Mac ⇔ LLM)		 Polls Outbox via QGA ➔ Mac relay checks countTokens & maxOutputTokens ➔ calls Gemini API ➔ delivers action via deliver_and_request. 15–40s (1-trip QGA batching)
6 Independent Grading
(Proxmox ➔ QGA ➔ VM)		 After agent.service stops, copies snapshot to grade/ and runs grade.service as labjudge. Runs 20 hidden tests + 6 mutants
7 Evidence & Shutdown
(VM ➔ PVE ➔ Mac)		 Collects 7 JSON artifacts + manifest.json, verifies SHA-256 on Mac, gracefully shuts down VM (qm shutdown). Tamper-evident archive locked

External Token Accounting & Pre-Call Budget Enforcement (usage.json)

Because the Mac relay (serve_once.py) mediates every LLM call outside the VM, the agent cannot tamper with token accounting. Before each generateContent call, the relay calls countTokens to verify that cumulative_inputs + prompt_tokens <= MAX_INPUT_TOKENS and cumulative_outputs + maxOutputTokens <= MAX_OUTPUT_TOKENS.

Upon receiving the response, the relay extracts promptTokenCount, candidatesTokenCount, and thoughtsTokenCount (reasoning tokens) from usageMetadata, computing billed_out = candidatesTokenCount + thoughtsTokenCount and recording exact costs in micro-USD (1e-6 USD) against a hard session cap (e.g., 350,000 micro-USD = $0.35).

5. Known Limitations & Future Security Roadmap

While this architecture is highly effective for benchmarking structured-action coding agents, evaluating untrusted, adversarial agents with arbitrary code/shell execution requires addressing several remaining security boundaries. We document these limitations openly for anyone building on this design:

Limitation 1: [Critical] The Grader (labjudge) Directly execs Agent-Submitted Code in the Same Process

  • Current Behavior & Risk: During agent execution (labexp), InaccessiblePaths prevents the agent from reading evaluation/judges.py. However, once the agent finishes, judges.py runs as labjudge and calls exec(compile(code_bytes, ...), ns) on the submitted validator.py inside the grader process. If an adversarial agent planted a Trojan payload inside validator.py that executes upon import/call—such as reading /opt/.../evaluation/judges.py, inspecting the caller stack frame, or writing a fake passing JSON directly to /var/lib/.../grader-output.json (which is writable by labjudge via ReadWritePaths)—that payload would execute with labjudge privileges during grading. (In our 41 experiments, all submitted code bundles are archived in mode-a/b-bundle.json and verified to contain only pure domain logic, but the architectural boundary itself is porous against active malice.)
  • Planned Remediation (Decoupling Untrusted Runner from Trusted Scorer):
    1. Split grading into two separate processes/UIDs: an Untrusted Test Runner (which imports the candidate code with zero access to judges.py or grader-output.json) and a Trusted Scorer (root or a separate UID).
    2. Pass test input vectors to the Untrusted Runner over a pipe (stdin/stdout), compare the returned JSON outputs against expected values in the Trusted Scorer, and have only the Trusted Scorer write grader-output.json.

Limitation 2: Denylist (InaccessiblePaths) Blind Spots & Mount Visibility

  • Current Behavior & Risk: InaccessiblePaths is a denylist mechanism. If any unredacted copy of the experiment payload remains accessible elsewhere (for example, on the mounted iso9660 CD-ROM path if not unmounted or restricted before agent launch), a shell-capable agent could read grading assets from the mount source. Additionally, the current unit file can be hardened further:
    • Adding ProtectProc=invisible to hide /proc command-line arguments of other processes (such as root's guest_rpc.py).
    • Adding RestrictAddressFamilies=none (since PrivateNetwork=yes blocks TCP/IP and abstract sockets, but filesystem UNIX domain sockets remain reachable unless address families are restricted).
    • Adding MemoryMax=, TasksMax=, and LimitFSIZE= to prevent fork bombs or disk-filling DoS writes to outbox/.
  • Planned Remediation: Transition from a denylist (InaccessiblePaths) to an allowlist root filesystem (TemporaryFileSystem=/ + minimal BindReadOnlyPaths=) and audit the unit with systemd-analyze security.

Limitation 3: Tamper-Evident vs. Tamper-Proof Evidence

  • The SHA-256 manifest (manifest.json) makes the 7 evidence files tamper-evident across transit and storage, but it is not cryptographically tamper-proof against a host administrator who recalculates both the file and manifest hashes.

6. Two Real-World Operational Gotchas We Hit (and How We Fixed Them)

While running 40+ consecutive VM benchmarks, we encountered two non-obvious operational traps worth sharing:

Gotcha 1: QGA Timeouts When Passing Non-ASCII Strings Directly to qm guest exec -c

What We Observed

When invoking /usr/sbin/qm guest exec <vmid> -- /usr/bin/python3 -c <script_source> from the Proxmox host, early builds of E21 that included Japanese UTF-8 characters inside Python docstrings consistently failed with VM qga command 'guest-exec' failed - got timeout (exit code 255), leaving guest-ping unresponsive afterward.

Analysis & Best Practice Workaround

While virtio-serial itself is a byte stream and Proxmox's QGA client prepends guest-sync-delimited (\xff delimiter resynchronization), passing large multi-kilobyte scripts containing multi-byte UTF-8 characters directly as command-line arguments to Proxmox's /usr/sbin/qm guest exec CLI (which is implemented in Perl under the hood) triggered encoding/buffer handling stalls.

  • Our Fix: We enforced a unit-tested 100% Pure ASCII (all(b < 128)) policy on the bootstrap scripts passed via -c (guest_rpc.py and install.py) and passed all experiment payloads as Base64-encoded JSON (payload_b64) over stdin. Across 40+ subsequent VM runs, we observed zero QGA timeouts.
  • General Recommendation: Avoid passing raw multi-line script strings to qm guest exec -- python3 -c "...". Instead, pass a tiny Base64 decoder one-liner (python3 -c "import base64,sys;exec(base64.b64decode(sys.stdin.read()))") or write the file via guest-file-write first.

Gotcha 2: False-Positive LLM Safety Filter Blocks (PROHIBITED_CONTENT) During Infra Debugging

What We Observed

While using an AI coding assistant on our development machine to debug Proxmox socket states and systemd sandbox rules (not inside the benchmarked VMs themselves), the Gemini API occasionally blocked responses with:

This request was blocked by Gemini's filters. They can occasionally trigger by mistake on safe coding, security, or biology-related queries.

Analysis & Workaround

This is a known false positive (promptFeedback.blockReason: "PROHIBITED_CONTENT") documented on the Google AI Developers Forum that cannot be disabled via standard safety threshold sliders. Because the safety classifier evaluates the full conversation history, accumulating multiple inline snippets referencing raw hypervisor sockets (*.qmp, *.qga) and namespace boundary checks (InaccessiblePaths, runuser) likely resembled VM-escape or privilege-escalation exploit development.
Switching our debugging workflow to invoke only standard /usr/sbin/qm guest exec CLI wrappers and pre-committed scripts eliminated these false positives completely (and zero filter blocks ever occurred inside the 41 benchmark runs).

7. The 7 Tamper-Evident Artifacts Collected per Run

Every metric reported in our benchmark series (E04–E44) is extracted directly from the 7 JSON files collected from the VM and verified against manifest.json:

  1. run-summary.json: High-level comparison of Mode A vs. Mode B (tests passed, turns, elapsed seconds, tool output chars).
  2. usage.json: External relay ledger recording promptTokenCount, candidatesTokenCount, thoughtsTokenCount, and estimated_micro_usd per call.
  3. execution-traces.json: Turn-by-turn log of tool calls, output character counts, and sensor/hook triggers.
  4. independent-tests.json: Individual pass/fail results for all 20 hidden test cases (plus mutation kill audits where applicable).
  5. mode-a-bundle.json: Full snapshot of the final Python files submitted by the baseline agent.
  6. mode-b-bundle.json: Full snapshot of the final Python files submitted by the harnessed agent.
  7. actions.json: Normalized JSON action payloads returned by the model on every turn.

🔗 Explore the Benchmark Results

Ready to see what we discovered when we ran 41 agent harness patterns through this sandbox? Check out the master benchmark index and our #1 ranked pattern deep dive:

Top comments (0)