A Warm Cache Is Not a Control: A Myth-Busting FAQ

Consider a composite afternoon on a checkout service, where two coding-agent sessions both claimed to fix the same failing tax calculation. The first session finished quickly because node_modules already existed and a previous TypeScript build still sat in dist. The second session looked slower in the chat log, yet it rebuilt from a cleaned tree and changed the production discount path. Reviewers compared clocks and patch size as if both runs were instruments rather than two disks wearing one repository name.

The myth under that comparison is simple enough to print on a sticky note beside the merge button. A warm cache is not a control, and a complimentary session is not a frozen stack. Until the tree, the lockfile, and the runtime are labeled, a green replay is commentary rather than measurement. This FAQ walks through the claims teams repeat, the evidence those claims skip, and a small protocol that makes the skip visible.

Why does a green replay keep lying about speed?

A coding loop reports success when tests pass, but tests do not disclose whether inputs were cold, pinned, or even unique. Package managers reuse tarballs, compilers keep incremental objects, and containers may still hold anonymous volumes from the last occupant. When leftovers remain, both latency and patch shape drift, because the agent is solving a different disk than the transcript describes. Treating that drift as model quality is how a lucky cache gets promoted into a capability story.

If you want a minimum control, record a digest of the worktree before the first prompt, not after the celebration. The helper below is a proposed local script, not a vendor benchmark, and it does not isolate a noisy neighbor on a shared host. It only makes the missing control visible enough that a reviewer can refuse the comparison.

#!/usr/bin/env python3
"""workspace_digest.py — proposed helper, unexecuted against any vendor."""
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

TRACKED = [
    "package-lock.json",
    "pnpm-lock.yaml",
    "yarn.lock",
    "Cargo.lock",
    "go.sum",
    "poetry.lock",
    "requirements.txt",
]


def git_one(args: list[str]) -> str:
    result = subprocess.run(
        ["git", *args], capture_output=True, text=True, check=False
    )
    return (result.stdout or "").strip()


def sha256_file(path: Path) -> str | None:
    if not path.is_file():
        return None
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()


def main() -> int:
    root = Path(".").resolve()
    card = {
        "cwd": str(root),
        "git_head": git_one(["rev-parse", "HEAD"]),
        "git_dirty": bool(git_one(["status", "--porcelain"])),
        "lock_hashes": {},
        "cache_hints": {
            "node_modules": (root / "node_modules").is_dir(),
            "dist": (root / "dist").is_dir(),
            "target": (root / "target").is_dir(),
            "pytest_cache": (root / ".pytest_cache").is_dir(),
        },
    }
    for name in TRACKED:
        digest = sha256_file(root / name)
        if digest:
            card["lock_hashes"][name] = digest
    json.dump(card, sys.stdout, indent=2, sort_keys=True)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it twice, once before cleaning and once after, and the diff usually explains why the first agent looked brilliant. The commands are ordinary, which is the point: controls should be boring enough to replay on Monday without a ceremony.

mkdir -p eval_cards
python3 workspace_digest.py > eval_cards/warm.json
git status --porcelain
# proposed cold-start: this deletes local build products on purpose
rm -rf node_modules dist .turbo .pytest_cache
python3 workspace_digest.py > eval_cards/cold.json
diff -u eval_cards/warm.json eval_cards/cold.json

Myth: the transcript is the experiment

Chat logs feel like lab notebooks because they are linear, timestamped, and full of confident summaries. They omit the filesystem, the served model identity, the queue delay, and every file the agent never opened. An experiment needs those omissions written down, or the next person cannot tell a method from a mood. A useful correction is to treat the transcript as commentary and the digest card as the specimen label.

When a free remote runtime enters the picture, the gap grows, because you do not own the noisy neighbor or the image that booted. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option that can host a practice loop, without turning that box into a published fixture. Treat the option as rehearsal space, then carry the same card into the environment you actually ship.

Myth: clearing chat history resets the machine

Teams often start a fresh agent thread and assume the disk followed the UI like a well-trained dog. Package caches, Docker layers, and language-server indexes do not read that thread identifier. If the previous run installed a yanked package or wrote golden files, the new thread inherits a private universe with a public smile. The corrected model is closer to continuous integration than to conversation: reset is a command you can hash, not a button you can click.

A small wrapper makes that reset explicit for the next person who reviews the patch. It remains a proposal, and it will be the wrong hammer for monorepos that keep expensive toolchains on purpose. Read it as a checklist encoded in bash, then delete the lines that would destroy a cache you actually need.

#!/usr/bin/env bash
# reset_for_agent.sh — proposed, review before pointing at a real repo
set -euo pipefail
mkdir -p eval_cards
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
python3 workspace_digest.py > "eval_cards/before_${stamp}.json"
case "${1:-}" in
  node) rm -rf node_modules dist coverage ;;
  py)
    find . -name '__pycache__' -type d -prune -exec rm -rf {} +
    rm -rf .pytest_cache .mypy_cache
    ;;
  go) go clean -cache -testcache ;;
  *) echo "usage: $0 node|py|go" >&2; exit 2 ;;
esac
python3 workspace_digest.py > "eval_cards/after_reset_${stamp}.json"

Myth: two HTTP successes are two equal models

Free model access is useful for iteration, but an endpoint that answers is not a version pin. Routers can change backends, clients can inject hidden system text, and temperature defaults can differ between wrappers that share a marketing name. If the story to leadership is that the model improved overnight, you need the identifier you actually hit. Until that identifier sits in the card, you are comparing weather reports from unnamed stations.

Do not invent a scoreboard from a handful of complimentary runs, because cache, queue, and routing variance usually dwarf the refactor you wanted to celebrate. The honest output of a free session is a patch plus a card, not a ranking dressed as science. If two cards disagree on dirtiness, lockfile hash, or cache presence, the patches are not peers, regardless of how similar the prose in the chat looks.

The comparator below is labeled proposed code. It fails closed when either card is missing a lock digest, which is the behavior you want during review.

#!/usr/bin/env python3
"""compare_cards.py — proposed checker for unlabeled evals."""
from __future__ import annotations

import json
import sys
from pathlib import Path


def load_card(path: Path) -> dict:
    with path.open() as handle:
        return json.load(handle)


def main() -> int:
    if len(sys.argv) != 3:
        print("usage: compare_cards.py a.json b.json", file=sys.stderr)
        return 2
    left = load_card(Path(sys.argv[1]))
    right = load_card(Path(sys.argv[2]))
    problems: list[str] = []
    if left.get("git_head") != right.get("git_head"):
        problems.append("git HEAD differs; the patches are not on the same specimen")
    if left.get("git_dirty") or right.get("git_dirty"):
        problems.append("at least one tree is dirty; uncommitted files are hidden inputs")
    if not left.get("lock_hashes") or not right.get("lock_hashes"):
        problems.append("a lock digest is missing; refuse the comparison")
    elif left.get("lock_hashes") != right.get("lock_hashes"):
        problems.append("lockfile hashes differ; dependency resolution is not a control")
    if left.get("cache_hints") != right.get("cache_hints"):
        problems.append("cache presence differs; wall-clock and patch shape are confounded")
    if problems:
        print("not comparable:")
        for item in problems:
            print(f"- {item}")
        return 1
    print("cards match on head, dirtiness, locks, and cache hints")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

python3 compare_cards.py eval_cards/warm.json eval_cards/cold.json
echo $?

A nonzero exit is the useful outcome. It tells review that the faster session was not a better method, only a warmer disk, and it does so without a dashboard or a slogan.

Myth: wall-clock on a shared box is a benchmark

Shared servers exist to remove laptop friction, not to underwrite latency publications. A neighbor compile, a cold image pull, and a bursting disk can move a run by more than one extra tool call from the agent. Borrowed time can still be useful if you log start and end, note whether caches were dropped, and then refuse to publish the clock as evidence. The control you actually own is ordinal and local: did this patch survive a cold tree on the branch you intend to merge?

That boundary also marks who should skip the workflow entirely. Do not place regulated customer data on a complimentary server, and do not argue a production rollout from a warm laptop cache. Do not treat a free model slot as a contractual version, because availability is not a pin. Teams that need isolation, audit logs, or latency objectives should run the same card inside owned CI, where fixtures have owners.

A corrected mental model

Think of every agent session as a field sample, not as laboratory grade equipment with a calibration sticker. Field samples stay valuable when they are labeled: dirty or clean tree, lockfile hash, cache present or absent, runtime owned or complimentary. They collapse into folklore when the only remaining artifact is a screenshot of tests going green. The checkout team in the opening scene did not need a larger model so much as they needed to stop comparing a warm node_modules against a rebuilt tree.

Keep the card next to the patch, and let CI fail when the card would have warned you that the trees were not peers. The method is the reset you can replay on a second machine, not the session that happened to be fast. Once that habit lands, complimentary runtimes become what they always were: convenient desks, not frozen stacks.