If Your Eval Shares Your Laptop, Discard the Score

You cannot rank an agent from a dirty machine.
Local shells leak secrets, caches, and leftover files.
Treat any laptop score as contaminated until you isolate it.

Laptop evals are not measurements

A real measurement needs a known starting state.
Your laptop is a graveyard of prior experiments.
Your laptop cannot be a starting state.
It is only folklore from prior runs.

You export an API key for just this run.
The next run inherits that key without asking.
Then you praise a model for finding credentials.

You install a CLI during last week's spike.
Today's agent shells out and then discovers it.
You call that autonomy, but it is residue.

Why "same prompt" still lies

People defend laptop evals with a ritual phrase.
They say the prompt file never changed once.
That claim does not rescue a dirty host.

The prompt is one input among many inputs.
The shell, the PATH, and the home directory vote.
Cached wheels and editor swap files vote too.

You would not benchmark a database on leftover WAL.
You would not score a compiler against dirty objects.
Stop giving language models a special exemption here.

What actually contaminates the score

Walk your last eval through this list.
If any item is true, throw the score away.

Environment leaks

• PATH still contains last week's tool binaries
• ~/.config holds tokens from a different vendor
• leftover OPENAI_* or ANTHROPIC_* variables
• Docker sockets that expose your real containers
• SSH agents that can reach production hosts

An agent that just works on your PATH is cheating.
It did not earn the tool and only inherited it.

Filesystem memory

• ./tmp from a previous loop still exists
• node_modules or .venv already solved imports
• git hooks rewrite commits behind the model's back
• editor swap files contain the target answer
• shell history teaches yesterday's exact commands

Green output after a dirty tree is not success.
It is only playback of yesterday's work.

Hidden retries

You rerun the same prompt after a failure.
The model sees a warmer cache or a fixed file.
You log one attempt while the machine lived three.

Retries belong in the trace, not in your head.
If you cannot show them, you did not measure them.

The opinion, without hedging

You should stop ranking models on developer laptops.
The ranking is a story about your machine.
It is not a story about the model.

Isolation is not a luxury for careful teams.
Isolation is the actual fixture for the score.
If you cannot throw the machine away, you cannot trust it.

A public leaderboard built on laptops is theater.
Readers cannot see your PATH or your extra CLIs.
They only see a number that you already liked.

Pin the protocol, not the vibes

Before you open a chat pane, write four pins.

1. Prompt text, stored as a file, hashed
2. Tool allowlist, stored as a file, hashed
3. Budget: max steps, max output bytes
4. Success predicate: a command that exits 0 or 1

If any pin is missing, you are demoing, not evaluating.
Demos can live on laptops, but evals cannot.

Hash the pins before the first model call.
Print those hashes in the trace header every time.
If a hash moves, you started a different experiment.

Artifact: a throwaway isolation harness

The harness below is a starter, not a product.
Run it on a machine you can destroy.
Label every result untrusted until that is true.

#!/usr/bin/env python3
"""Isolation harness for agent evals. Starter you can run."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
import time
from pathlib import Path

ROOT = Path(os.environ.get("EVAL_ROOT", "/tmp/eval-root"))
TRACE = ROOT / "trace.jsonl"
MAX_STEPS = int(os.environ.get("EVAL_MAX_STEPS", "8"))
MAX_BYTES = int(os.environ.get("EVAL_MAX_BYTES", "32768"))


def die(msg: str, code: int = 2) -> None:
    print(msg, file=sys.stderr)
    raise SystemExit(code)


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256(path.read_bytes()).hexdigest()
    return digest[:16]


def require_clean_machine() -> None:
    forbidden = [
        Path.home() / ".ssh",
        Path.home() / ".config",
        Path("/var/run/docker.sock"),
    ]
    for path in forbidden:
        if path.exists():
            die(f"contaminated path present: {path}")

    extra_keys = [
        key for key in os.environ
        if key.endswith("_API_KEY") or key.endswith("_TOKEN")
    ]
    if extra_keys:
        die(f"secret-shaped env vars: {extra_keys}")

    if os.geteuid() == 0 and Path("/home").exists():
        # Root on a throwaway box is fine.
        # Root on a laptop with user homes is not.
        homes = [p for p in Path("/home").iterdir() if p.is_dir()]
        if homes:
            die(f"user homes visible from eval host: {homes}")


def pin_inputs() -> dict:
    prompt = Path("prompt.txt")
    tools = Path("tools.json")
    predicate = Path("success.sh")
    for path in (prompt, tools, predicate):
        if not path.is_file():
            die(f"missing pin file: {path}")
    return {
        "prompt_sha": sha256_file(prompt),
        "tools_sha": sha256_file(tools),
        "predicate_sha": sha256_file(predicate),
        "max_steps": MAX_STEPS,
        "max_bytes": MAX_BYTES,
    }


def record(event: dict) -> None:
    ROOT.mkdir(parents=True, exist_ok=True)
    event["ts"] = time.time()
    with TRACE.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(event) + "\n")


def run_step(step: int, cmd: list[str]) -> dict:
    started = time.time()
    proc = subprocess.run(
        cmd,
        cwd=ROOT,
        capture_output=True,
        text=True,
        timeout=30,
    )
    event = {
        "step": step,
        "cmd": cmd,
        "exit": proc.returncode,
        "stdout": (proc.stdout or "")[:MAX_BYTES],
        "stderr": (proc.stderr or "")[:MAX_BYTES],
        "elapsed_s": round(time.time() - started, 3),
    }
    record(event)
    return event


def main() -> None:
    require_clean_machine()
    pins = pin_inputs()
    record({"type": "start", "pins": pins, "cwd": str(ROOT)})

    # Replace ["true"] with your agent runner.
    # Keep the budget gate outside the model.
    for step in range(1, MAX_STEPS + 1):
        event = run_step(step, ["true"])
        if event["exit"] != 0:
            record({"type": "abort", "reason": "tool_failed", "step": step})
            raise SystemExit(1)

    pred = subprocess.run(["bash", "success.sh"], cwd=ROOT)
    record({"type": "finish", "success": pred.returncode == 0})
    raise SystemExit(0 if pred.returncode == 0 else 1)


if __name__ == "__main__":
    main()

Save it as isolate_eval.py beside the pin files.
The agent never chooses the budget. You do.
That split is the whole point of the fixture.

Companion files

tools.json should be an allowlist, not a wish list.

{
  "allow": ["python3", "rg", "git"],
  "deny": ["ssh", "docker", "curl", "sudo"]
}

success.sh must be boring and binary.

#!/usr/bin/env bash
set -euo pipefail
test -f /tmp/eval-root/REPORT.md
rg -q "STATUS: PASS" /tmp/eval-root/REPORT.md

If the predicate needs a human, it is not a predicate.
It is a review, so log it as a review.

prompt.txt should name the task and the stop rule.
Do not hide extra hints in shell aliases nearby.
If the hint matters, put the hint in the file.

How you actually run it

# On a throwaway host, not your laptop.
install -d /tmp/eval-root
export EVAL_ROOT=/tmp/eval-root
export EVAL_MAX_STEPS=8
export EVAL_MAX_BYTES=32768
python3 isolate_eval.py; echo exit:$?
sha256sum prompt.txt tools.json success.sh
wc -l /tmp/eval-root/trace.jsonl

Copy the traces off the box before you wipe it.
Then destroy the box so residue cannot accumulate.
A kept eval machine becomes a laptop with extra steps.

Decision table: keep the score or burn it

Condition	Verdict	Why
Home directory mounted into the eval	Burn	Hidden files coach the agent
Any *_API_KEY in the environment	Burn	Tool use is no longer earned
Prompt edited between retries	Burn	You measured two systems
Budget enforced outside the model	Keep	The loop cannot flatter itself
Success is a command, not a vibe	Keep	Humans drift; scripts do not
Trace is complete and hashed	Keep	Replay is possible later
Host still exists after the run	Burn	Residue will leak into the next score

Print this table next to your CI job.
If a cell says burn, do not publish a leaderboard.

A CI sketch that refuses dirty runners

Laptop habits sneak into CI through shared runners.
Treat the runner like the laptop until proven empty.
Fail the job when the isolation checks fail.

# proposal: isolation gate, not a full agent platform
name: isolated-eval
on: [push]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Refuse a dirty environment
        run: |
          python3 isolate_eval.py
      - name: Upload traces only
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-trace
          path: /tmp/eval-root/trace.jsonl

Do not install your favorite CLIs "to help the agent".
That help is contamination with a nicer name.
If the task needs a tool, pin it in tools.json.

Common objections, answered bluntly

"I know what is on my machine."
You do not, not after six months of tooling.
Memory is not an inventory system.

"Isolation slows iteration."
Good. Ranking should be slower than chatting.
Speed without pins is just a better demo.

"The model should see a realistic developer box."
Then say you measured a developer box, not a model.
Do not mix those claims in the same sentence.

"We already log prompts."
Prompts without host state are incomplete evidence.
Log the hashes, the allowlist, and the budget too.

Where a free remote box fits

You need a machine that starts empty and dies full.
A local VM helps. A remote throwaway host helps more.
The point is disposal, not brand.

MonkeyCode offers free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Use those only as the disposable fixture for this harness.
Do not treat them as a production cluster or a quota promise.

Wire the harness to whatever model endpoint you already trust.
Keep the allowlist and the budget on your side of the cable.
The model proposes actions. Your host accepts or refuses.

If you need a box you can wipe after the trace, that free server option is one place to park the harness.

Limitations you should not skip

This harness does not prove model quality.
It only proves the score was not born on a dirty laptop.
That is a lower bound, not a medal.

It does not sandbox syscalls or kernel objects.
A determined agent can still wreck the throwaway host.
That is acceptable if the host is truly disposable.

It does not pin model weights or provider routing.
If the endpoint drifts, your hashes will not save you.
Log provider headers when the API gives them to you.

Network egress is still a hole in many setups.
Deny curl and ssh in the allowlist, then verify.
An allowlist you never test is documentation.

Shared CI runners can still be dirty in quieter ways.
Cached actions, leftover workspaces, and privileged sockets remain.
Read the runner's filesystem before you trust the first score.

Who should not use this approach

• You are pairing, not measuring. Use the laptop.
• You lack permission to run untrusted code on a host.
• Your agent never shells out. A unit test is enough.
• You need guaranteed uptime, GPUs, or a named SLA.
• You cannot write a binary success predicate yet.

If you are in that last group, stop ranking models.
Write the predicate first. Then isolate. Then rank.

What you should do on Monday

Pick one agent task you already brag about.
Write prompt.txt, tools.json, and success.sh.
Run the harness on a machine you can delete.

If the score collapses, believe the collapse.
Your laptop was the missing teammate all along.
Ship the isolation fixture before you ship the leaderboard.