Postmortem: A Warm Workspace Certified the Previous Commit

A green log from a reused workspace is not merge evidence. The log can describe tests from an older tree. The gate must bind every result to a workspace generation.

Incident conclusion

The patch never compiled on a clean checkout. A warm job directory still held the previous commit's bytecode.

Pytest imported that leftover bytecode and reported success. A later summary treated the report as proof.

Timeline

1. At 09:10 UTC the agent received a failing import report.
2. At 09:14 UTC a draft patch edited two modules and a test.
3. At 09:16 UTC the job applied that patch on a reused checkout.
4. At 09:18 UTC pytest exited zero without rebuilding stale bytecode.
5. At 09:21 UTC the model wrote a summary quoting that green log.
6. At 09:30 UTC the merge gate accepted the summary plus the log.
7. At 11:02 UTC a clean runner failed on the same commit.

Times above are a reconstructed lab timeline, not production telemetry. Treat the clock times as labels for ordering only.

What the clean runner showed

The clean runner had no leftover .pyc files. The editable install pointed at a different path.

The import error returned exactly as first reported. The warm job had never executed the patched module.

Contributing factors

• The job script reused one checkout across unrelated agent runs.
• Pytest imported bytecode that predated the applied patch.
• An editable install from the prior commit remained on sys.path.
• The merge gate scored the written summary instead of a fresh tree.
• No file recorded which commit the test process actually loaded.

Where the free options entered

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The draft patch came from MonkeyCode's free model access. The warm pytest run used MonkeyCode's free server option.

Neither free option created the stale checkout itself. The job script failed to wipe the tree between runs. Free access only made a cheap retry look sufficient.

Why the warm run stayed green

Pytest can import a .pyc even when the source changed. That happens when the bytecode timestamp still looks newer.

An editable install can also shadow the patched checkout. The process then executes the previous commit's files.

Durable fix

Bind evidence to a generation stamp before any summary is trusted. Stamp the tree after the patch is applied and before tests. Reject logs, bytecode, and editable installs that predate that stamp.

Run the same check on a second clean directory. The stamp remains a local gate, not a product feature. The checker does not rate model quality or speed.

It does not claim a quota, a machine size, or a retention window. Operators must verify those limits outside this postmortem.

Generation checker

The script below is a proposed local gate. It has not been run as a published benchmark.

#!/usr/bin/env python3
"""Reject test evidence that predates this workspace generation."""

from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

STAMP = '.workspace_generation.json'

def git(root: Path, *args: str) -> str:
    return subprocess.check_output(
        ['git', '-C', str(root), *args], text=True
    ).strip()

def tree_id(root: Path) -> str:
    head = git(root, 'rev-parse', 'HEAD')
    dirty = git(root, 'status', '--porcelain')
    digest = hashlib.sha256()
    digest.update(head.encode())
    digest.update(b'\0')
    digest.update(dirty.encode())
    return digest.hexdigest()

def write_stamp(root: Path) -> None:
    payload = {
        'generation': os.urandom(16).hex(),
        'tree_id': tree_id(root),
        'head': git(root, 'rev-parse', 'HEAD'),
    }
    path = root / STAMP
    path.write_text(json.dumps(payload, indent=2) + '\n', encoding='utf-8')

def refuse_stale(root: Path, evidence: Path) -> None:
    path = root / STAMP
    if not path.is_file():
        raise SystemExit('missing workspace generation stamp')
    payload = json.loads(path.read_text(encoding='utf-8'))
    if payload.get('tree_id') != tree_id(root):
        raise SystemExit('tree changed after the generation stamp')
    stamped = path.stat().st_mtime
    if evidence.stat().st_mtime + 1 < stamped:
        raise SystemExit('evidence predates the workspace generation')
    for pyc in root.rglob('*.pyc'):
        if '.git' in pyc.parts:
            continue
        if pyc.stat().st_mtime + 1 < stamped:
            raise SystemExit(f'stale bytecode: {pyc}')

def main() -> None:
    root = Path(sys.argv[1]).resolve()
    action = sys.argv[2]
    if action == 'stamp':
        write_stamp(root)
        return
    if action == 'check':
        refuse_stale(root, Path(sys.argv[3]).resolve())
        return
    raise SystemExit('usage: workspace_gen.py ROOT stamp|check EVIDENCE')

if __name__ == '__main__':
    main()

Job commands

Treat the block below as a proposed job script. Run the lines in order and stop on failure. Do not summarize a log that failed the check.

git apply --check patch.diff
git apply patch.diff
find . -type d -name __pycache__ -prune -exec rm -rf {} +
find . -type f -name '*.pyc' -delete
python3 -m venv /tmp/job-venv
/tmp/job-venv/bin/pip install -e '.[test]'
python3 workspace_gen.py . stamp
export PYTHONDONTWRITEBYTECODE=1
/tmp/job-venv/bin/python -B -m pytest -q --junitxml=/tmp/job-junit.xml
python3 workspace_gen.py . check /tmp/job-junit.xml

The find deletes belong to the job script, not a product feature. The venv path is ephemeral by operator choice. A second clean clone should repeat the same commands.

Bytecode controls

Always export PYTHONDONTWRITEBYTECODE=1 before the pytest process starts. Pass -B to Python so leftover bytecode is ignored. Delete every __pycache__ directory after applying the patch.

Editable origin check

Confirm the imported origin still lives inside this checkout. Replace the placeholder your_package with the real import name. Fail the job when the origin resolves outside the root.

/tmp/job-venv/bin/python - <<'PY'
import importlib.util
import pathlib
spec = importlib.util.find_spec('your_package')
if spec is None or not spec.origin:
    raise SystemExit('package origin missing')
origin = pathlib.Path(spec.origin).resolve()
root = pathlib.Path('.').resolve()
if root not in origin.parents and origin != root:
    raise SystemExit('editable origin outside checkout: ' + str(origin))
print(origin)
PY

Decision rules

• Accept a generation file only when the tree id matches.
• Reject a missing stamp or a tree id that differs.
• Accept a JUnit file only when its mtime follows the stamp.
• Reject a JUnit file whose mtime predates the stamp.
• Accept bytecode only when none of it predates the stamp.
• Reject the run when any .pyc predates the stamp.
• Accept an editable install only when it points at this checkout.
• Reject an editable install that points at another checkout.
• Accept a model summary only after the check passes.
• Reject a summary written from an unchecked log.

Each row above is a hard stop for the gate. A passing summary cannot override a failed row.

Test plan

1. Stamp a clean tree, run pytest, and expect the check to pass.
2. Copy an older JUnit file over the new log and expect rejection.
3. Plant a .pyc with an old mtime and expect rejection.
4. Change one tracked file after the stamp and expect rejection.
5. Remove the stamp file and expect an immediate rejection.
6. Repeat steps 1 through 5 on a freshly cloned directory.

Label every step as unexecuted until the operator runs it. Do not publish pass rates from a plan.

Contrast with transcript scoring

A transcript can quote a green line from memory. The generation file cannot be satisfied by quoted text.

The check reads the checkout that produced the log. The merge gate should call the check, not the summary.

Blast radius

In the reconstruction the bad commit reached main. That same commit broke the next clean build.

No customer data was involved in this reconstructed case. The cost was a reverted patch and a wasted review cycle.

Limitations

Mtime checks fail open when clocks move backward. Prefer the generation id, and treat mtime as a secondary hint.

The checker does not prove the test process loaded the stamped files. A process can still import a package from another path.

The dirty-tree hash ignores untracked files unless git status shows them. Untracked bytecode outside the scanned tree stays invisible.

This gate does not find logic bugs that pass on a clean tree. Free model access and the free server option are availability claims only. This article does not state quotas, model names, hardware, duration, or permanence.

Those product details were not verified for this article. Ask the operator for current docs before citing any limit.

Who should not use this

Skip this gate if you need a security boundary. It does not isolate secrets or prove runner identity.

Skip it as a substitute for an ephemeral CI runner you already trust. Teams with per-job virtual machines still need a tree bind. The bytecode scan alone is not their durable control.

Do not use a free server run as a release certification. No uptime promise was verified for this workflow.

Corrective actions

Corrective actions belong to the job runner and the gate. They do not require a new model or a larger machine.

1. The job runner must wipe the checkout before apply.
2. The merge gate must call the generation check.
3. Block every summary until that check exits zero.
4. Record the generation id beside the stored JUnit file.

After the repro

Reproduce the warm checkout, then refuse the log unless the stamp matches. Readers with MonkeyCode's free model access and free server option can run this repro. Keep the generation check on a machine you control before merge.