Postmortem: A Reused Sandbox Served Yesterday's Output as Today's Input

The failure was environmental, not model-level. Two agent runs shared one workspace directory. Run B read run A's output and published it as today's numbers.

Nothing crashed. Every exit code was zero. That is why it took 26 hours to find.

What the pipeline did

• A nightly job fetched a market extract.
• An agent summarized the extract and wrote report.json.
• A publisher read report.json and posted it downstream.

The workspace pool keyed directories by job name. Every run of the same job landed in the same folder.

Timeline (UTC, day of incident)

1. 02:57 - Run A started, fetched the 2026-09-13 extract, wrote outputs/report.json.
2. 03:04 - Run A finished with exit code 0.
3. 03:05 - Run B started with the 2026-09-14 extract.
4. 03:06 - Run B's summarizer step timed out on the model call and was retried by the loop.
5. 03:07 - The retry reused the existing workspace, found report.json, and skipped generation.
6. 03:12 - The publisher posted run A's numbers under run B's date.
7. 05:00 - The dashboard showed a day-over-day change of exactly 0.0 percent.
8. 09:30 - An analyst flagged the flat series as suspicious.
9. 11:40 - Logs showed two different input fingerprints in one directory.
10. 12:15 - Publishing was disabled and a replay started.
11. 12:50 - Replay with run-scoped workspaces produced the correct report.
12. Next day 05:00 - A stale-read test was added to CI.

Contributing factors

• Workspace identity was the job name, never the run.
• The summarizer treated "file exists" as "work is done".
• The retry path had no idempotency key tied to the run.
• Readers trusted file mtime, and mtime said the file was fresh.
• No alarm watched input fingerprints per directory.
• Zero exit codes masked the substitution.

The mtime detail matters most. The file really was written minutes before it was read. Freshness was real. Provenance was wrong.

The artifact: a run-scoped workspace guard

The durable fix has three parts. Unique run directories, an ownership check on every read and write, and a fencing token at the sink.

Layout

/srv/agent/runs/<run_id>/
  manifest.json
  inputs/
  outputs/

run_id is generated once per scheduled execution and passed to every process in that run.

Guard

"""workspace_guard.py - refuses cross-run reads and reused directories."""
import json
import os
import time
from pathlib import Path


class StaleOutput(RuntimeError):
    """An output file was produced by a different run."""


class RunWorkspace:
    def __init__(self, root, run_id: str) -> None:
        if not run_id:
            raise ValueError("run_id must be non-empty")
        self.run_id = run_id
        self.root = Path(root)
        self.dir = self.root / "runs" / run_id
        self.manifest = self.dir / "manifest.json"

    def open(self, input_fingerprint: str) -> Path:
        if self.dir.exists():
            raise RuntimeError("run dir already exists: %s" % self.dir)
        (self.dir / "inputs").mkdir(parents=True, exist_ok=False)
        (self.dir / "outputs").mkdir(parents=True, exist_ok=False)
        self.manifest.write_text(json.dumps({
            "run_id": self.run_id,
            "started_at": time.time(),
            "host": os.uname().nodename,
            "input_fingerprint": input_fingerprint,
        }, indent=2, sort_keys=True))
        return self.dir

    def write_output(self, name: str, payload: bytes) -> Path:
        self._assert_owner()
        target = self.dir / "outputs" / ("%s.%s" % (name, self.run_id))
        target.write_bytes(payload)
        return target

    def read_output(self, name: str, producer_run: str) -> bytes:
        if producer_run != self.run_id:
            raise StaleOutput("cross-run read: %s -> %s" % (producer_run, self.run_id))
        return (self.dir / "outputs" / ("%s.%s" % (name, producer_run))).read_bytes()

    def _assert_owner(self) -> None:
        record = json.loads(self.manifest.read_text())
        if record["run_id"] != self.run_id:
            raise StaleOutput("manifest says %s" % record["run_id"])
        if record["started_at"] < time.time() - 86400:
            raise StaleOutput("manifest is older than the run window")

The output filename carries the run id. Cross-run reads now fail loudly instead of succeeding quietly.

Fencing token at the sink

A reused directory is a routing bug. A late duplicate write is a storage bug. Both need a fence.

UPDATE leases
   SET fencing_token = fencing_token + 1,
       holder = :holder,
       expires_at = now() + interval '5 minutes'
 WHERE job_name = :job
   AND (holder = :holder OR expires_at < now())
RETURNING fencing_token;

INSERT INTO daily_report (report_date, payload, fencing_token)
VALUES (:date, :payload, :fence)
ON CONFLICT (report_date) DO UPDATE
   SET payload = EXCLUDED.payload,
       fencing_token = EXCLUDED.fencing_token
 WHERE daily_report.fencing_token < EXCLUDED.fencing_token;

A writer holding an old fence updates zero rows. The publisher logs the miss and exits cleanly. No silent overwrite.

Reproduce it in a few minutes

mkdir -p /tmp/agent-demo && cd /tmp/agent-demo
mkdir tests
# save the guard above as workspace_guard.py
python -m pytest -q tests/test_workspace_guard.py

# tests/test_workspace_guard.py
import pytest
from workspace_guard import RunWorkspace, StaleOutput


def test_stale_output_is_rejected(tmp_path):
    a = RunWorkspace(tmp_path, "run-a")
    a.open(input_fingerprint="fp-1")
    a.write_output("report.json", b'{"rows": 3}')

    b = RunWorkspace(tmp_path, "run-b")
    b.open(input_fingerprint="fp-2")

    with pytest.raises(StaleOutput):
        b.read_output("report.json", producer_run="run-a")


def test_run_dir_cannot_be_reopened(tmp_path):
    a = RunWorkspace(tmp_path, "run-a")
    a.open(input_fingerprint="fp-1")
    with pytest.raises(RuntimeError):
        a.open(input_fingerprint="fp-1")


def test_duplicate_delivery_is_idempotent(tmp_path):
    a = RunWorkspace(tmp_path, "run-a")
    a.open(input_fingerprint="fp-1")
    first = a.write_output("report.json", b'{"rows": 3}')
    second = a.write_output("report.json", b'{"rows": 3}')
    assert first == second

The third test is the loop-retry case. A retry that lands in the same run reuses the same path. The bytes get written twice. The effect happens once.

When a cheap shared sandbox is acceptable

Situation	Reuse OK?	Minimum guard
Throwaway analysis of public data	Yes	Run-scoped directory
Job that reads yesterday's artifact	Yes, with care	Manifest ownership check
Job that writes to a production table	No	Fencing token plus idempotency key
Long agent loop with reconnects	Yes	Lease, fence, deterministic output path
Workload holding customer PII	No	Dedicated ephemeral host

Limitations

• The guard is a convention, not a security boundary. A process that ignores it can still read anything.
• started_at uses wall-clock time, so clock skew across hosts weakens that check.
• Fencing tokens need a store with compare-and-set semantics. Plain object storage will not do.
• Deterministic output paths collide if two runs share a run_id. Generate it with a UUID or a timestamp plus random suffix.
• Free-tier quotas, servers, and limits change. Any plan built on them needs a re-check before it becomes a dependency.

Who should not use this

Teams that cannot legally share a host with other tenants, workloads needing a fixed egress IP during a transaction, and anything where a stale write is a compliance event rather than a bug.

Rollout checklist

1. Generate RUN_ID in the scheduler, not inside the job.
2. Pass it to every child process through the environment.
3. Fail the run when the workspace directory already exists.
4. Stamp the run id into every output filename.
5. Add a lease with a monotonic fencing token at the sink.
6. Keep the stale-read test in CI permanently.

Where the free tier fit

The reproduction harness above does not need a large machine. It needs a shell, Python, and one agent call. That is the shape of job where free model access and a free server option are genuinely useful. The harness for this postmortem ran on MonkeyCode, using its free model access for the summarizer step and its free server option for the throwaway runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding agent project. The operator states the free tier includes free model access, a free server option, and a token allowance advertised at roughly ten million tokens at the time of writing. Treat those numbers as volatile. Check the current terms on the project page before you build a dependency on them. Quotas move; the guard above does not.

If the harness is useful, run it once in a free session and delete the workspace afterward.