Postmortem: Preemption Split a Multi-File Agent Apply

A killed agent apply can leave a repository half-patched. Tests then pass on an incomplete, unreviewed tree. The durable fix is atomic staging plus a checksum manifest.

This postmortem records a controlled lab reproduction only. The failure is common on shared CI runners. Agent jobs write several files, then lose the process.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. Those two facts matter only as a constrained sandbox. The sandbox does not remove the need for atomic apply. Teams still need a durable, crash-safe write protocol.

The failure in one line

The agent planned a four-file field rename. Preemption arrived after only two writes. The remaining two files still described the old contract. Unit tests covered only the files that landed.

CI stayed green on that torn tree. Reviewers saw a small, tidy two-file diff. The broken contract sat in files never reached.

Scope of the reproduction

The lab used a tiny Python service with four modules. One module defined a request schema. One module validated the inbound payload. One module stored the record. One module exposed the HTTP handler.

The intended patch renamed a required field. Every module had to change together. A partial apply was worse than no apply.

The reproduction did not use production traffic. It used a local git worktree and a kill signal. The same pattern appears on preemptible free servers.

Timeline

The times below are relative to job start. They come from the lab log.

1. T+0s — Worktree created from main at a1f3c0.
2. T+4s — Agent read all four modules into context.
3. T+11s — Agent emitted a four-file unified diff.
4. T+13s — Apply started; order was schema, then store.
5. T+14s — schema.py and store.py were overwritten on disk.
6. T+15s — SIGTERM arrived. The apply process exited 143.
7. T+16s — validate.py and handler.py still used user_id.
8. T+22s — pytest -q reported eight passed, zero failed.
9. T+30s — A later job committed the dirty tree.
10. T+41s — Review text treated the two-file rename as complete.

The green test run was the damaging event. It converted a crash into a merge candidate.

Why tests stayed green

The test suite mocked the HTTP layer. It constructed Store objects directly. It never sent a payload through validate.py.

The two written files agreed with each other. The two untouched files still agreed with each other. No test crossed the cut line between them.

Green tests did not prove a complete patch. They proved the suite did not span the rename.

Contributing factors

Several conditions had to line up at once.

• The apply wrote files in place, one at a time.
• The job had no write lease or heartbeat file.
• The test graph did not cover the module cut.
• The next agent treated a dirty tree as valid.
• Review looked at the diff, not the planned file set.
• Free-server style preemption made long applies unsafe.

None of these conditions is exotic or rare. Shared CI runners preempt jobs as well. Laptops sleep. Containers hit memory limits hard. The write protocol must assume sudden death.

Reproduction commands

The commands below recreate the split tree. Run them in an empty directory. They are a lab fixture, not a product tutorial.

mkdir -p svc && cd svc
git init -q

cat > schema.py << 'PY'
REQUIRED = "user_id"
PY

cat > store.py << 'PY'
from schema import REQUIRED

def save(record):
    assert REQUIRED in record
    return record[REQUIRED]
PY

cat > validate.py << 'PY'
from schema import REQUIRED

def check(payload):
    return REQUIRED in payload
PY

cat > handler.py << 'PY'
from validate import check
from store import save

def handle(payload):
    if not check(payload):
        return {"ok": False}
    return {"ok": True, "id": save(payload)}
PY

cat > test_store.py << 'PY'
from store import save

def test_save():
    assert save({"user_id": "u1"}) == "u1"
PY

git add . && git commit -qm base

The next script mimics a half-finished field rename. It writes two files, then stops.

cat > schema.py << 'PY'
REQUIRED = "account_id"
PY

cat > store.py << 'PY'
from schema import REQUIRED

def save(record):
    assert REQUIRED in record
    return record[REQUIRED]
PY

# SIGTERM analogue: do not touch validate.py or handler.py
python -m pytest -q
echo "exit=$?"

Expected lab result: tests pass. handler.py still requires user_id. A live request with account_id fails at validation.

Artifact: an atomic apply protocol

The durable fix is not retry-harder logic. Retry on a dirty tree repeats the split. The apply must be all-or-nothing.

The protocol has five rules.

1. Stage every file under .apply-staging/<job-id>/.
2. Write a manifest with path, mode, and sha256.
3. Heartbeat a lease file every few seconds.
4. Swap only when the manifest is complete.
5. Roll back if the process dies mid-swap.

Incomplete staging is discarded on sight. The original tree stays internally consistent. The next job starts from a known commit.

Manifest shape

{
  "job_id": "job-lab-43",
  "base_sha": "a1f3c0",
  "files": [
    {"path": "schema.py", "sha256": "...", "mode": "0644"},
    {"path": "store.py", "sha256": "...", "mode": "0644"},
    {"path": "validate.py", "sha256": "...", "mode": "0644"},
    {"path": "handler.py", "sha256": "...", "mode": "0644"}
  ],
  "status": "complete"
}

A manifest with status other than complete cannot swap. A missing file hash is a hard failure. No partial commit is allowed through this gate.

Staging and swap script

The script below is a lab helper. It is not production software. Review it before any real use.

#!/usr/bin/env bash
set -euo pipefail

JOB_ID="${1:?job id}"
STAGE=".apply-staging/${JOB_ID}"
MANIFEST="${STAGE}/manifest.json"
LEASE=".apply-staging/${JOB_ID}.lease"

if [[ ! -f "$MANIFEST" ]]; then
  echo "missing manifest" >&2
  exit 2
fi

python - "$MANIFEST" "$STAGE" << 'PY'
import json, hashlib, sys, pathlib
manifest_path, stage = sys.argv[1], pathlib.Path(sys.argv[2])
man = json.loads(pathlib.Path(manifest_path).read_text())
if man.get("status") != "complete":
    raise SystemExit("manifest not complete")
for item in man["files"]:
    p = stage / item["path"]
    if not p.is_file():
        raise SystemExit(f"missing {item['path']}")
    digest = hashlib.sha256(p.read_bytes()).hexdigest()
    if digest != item["sha256"]:
        raise SystemExit(f"hash mismatch {item['path']}")
print("manifest ok")
PY

date -u +%s > "$LEASE"

SWAP=".apply-staging/${JOB_ID}.swap"
rm -rf "$SWAP"
mkdir -p "$SWAP"

python - "$MANIFEST" "$STAGE" "$SWAP" << 'PY'
import json, pathlib, shutil, sys
man = json.loads(pathlib.Path(sys.argv[1]).read_text())
stage, swap = pathlib.Path(sys.argv[2]), pathlib.Path(sys.argv[3])
for item in man["files"]:
    src = stage / item["path"]
    dest = swap / item["path"]
    dest.parent.mkdir(parents=True, exist_ok=True)
    shutil.copy2(src, dest)
print("swap tree ready")
PY

python - "$MANIFEST" "$SWAP" << 'PY'
import json, pathlib, sys
man = json.loads(pathlib.Path(sys.argv[1]).read_text())
swap = pathlib.Path(sys.argv[2])
for item in man["files"]:
    src = swap / item["path"]
    dest = pathlib.Path(item["path"])
    dest.parent.mkdir(parents=True, exist_ok=True)
    src.replace(dest)
print("swap done")
PY

A crash before swap done leaves main untouched. A crash during the final replace is still residual risk. For that last window, keep the job single-writer. Do not start a second apply on the same worktree.

Test plan for the protocol

Run these checks after every protocol change.

1. Kill the apply before the manifest is complete. Expect no file in main to change.
2. Kill the apply after staging, before swap. Expect no file in main to change.
3. Flip one staged hash. Expect the checker to abort.
4. Drop one staged file. Expect the checker to abort.
5. Complete a four-file rename. Expect handler.py and schema.py to agree.
6. Run pytest plus one HTTP-level check that crosses modules.
7. Start a second job against a live lease. Expect the second job to refuse.
8. Expire the lease, then retry. Expect a clean apply from base_sha.

The HTTP-level check is mandatory here. Store-only tests hid the original split.

# test_handler_contract.py
from handler import handle

def test_new_field_roundtrip():
    result = handle({"account_id": "a1"})
    assert result["ok"] is True

def test_old_field_rejected():
    result = handle({"user_id": "u1"})
    assert result["ok"] is False

Add this file before trusting any agent rename. Cross-module tests are the real gate.

Decision table

Use this table before an agent is allowed to edit.

Condition	Action
Manifest missing or not complete	Abort. Do not swap.
File hash mismatch	Abort. Delete staging.
Lease owned by another job_id	Abort. Wait or fail.
base_sha != git rev-parse HEAD	Abort. Rebase or replan.
Cross-module test absent for the change	Abort. Do not merge.
Apply killed after complete swap	Continue. Tree should be consistent.
Apply killed before complete swap	Discard staging. Tree should match base.

The table is the policy. The scripts only enforce it.

What a free server option changes

Shared compute raises the preemption rate. Laptop CI hosts do the same. MonkeyCode's free server option is one place this shows up. Free model access does not change the write rules.

A longer model reply still needs an atomic apply. A cheaper sandbox still needs a lease. Product access is not a substitute for a manifest.

Teams that already enforce the protocol can use that sandbox as one more preemptible host. The protocol remains useful without that product. Remove every product mention and the failure still stands.

Limitations

The swap is not a true filesystem transaction. replace still runs per file. A crash in the final loop can tear the tree. On that path, restore from base_sha.

The lab ignores permissions, symlinks, and binary files. It ignores submodules. It ignores generated lockfiles. Those need extra manifest fields.

The heartbeat is a file timestamp, not a cluster lock. It will not coordinate two machines. Use one worktree per job.

The reproduction used SIGTERM. OOM kills and disk-full errors look similar. They are not identical. Disk-full can corrupt the staging dir itself. Treat staging as disposable.

This postmortem does not claim runtime numbers. It does not name models. It does not claim quotas, hardware, or duration.

Who should not use this approach

• Do not use in-place multi-file writes on preemptible hosts.
• Do not merge agent diffs that lack a file manifest.
• Do not trust unit tests that never cross the changed modules.
• Do not run two apply jobs in one worktree.
• Skip this protocol for single-file edits already wrapped in git commit.
• Skip it if the repo cannot restore HEAD from a known sha.

Security-sensitive trees need more than this lab script. Add code owners, secret scanning, and a human review gate. This article does not replace those controls.

Durable fix

The durable fix has three parts.

First, agents write only to staging. Second, a complete manifest authorizes a swap. Third, tests must cross every module in the planned set.

With those three, preemption becomes a retry, not a corrupt merge. The tree either matches the old contract or the new one. It never matches both.

That is the whole incident. Partial applies are silent failures. Make them loud, then make them impossible.