Postmortem: Stale Workspace Index Rewrote a Deleted Path

Stale workspace indexes cause coding agents to patch deleted paths. The apply looks successful in the agent tool log. Git then rejects the commit or ships a silent resurrection.

This postmortem reconstructs that failure as a labeled incident. It covers timeline, contributing factors, and a durable fix. The artifact is a freshness gate plus a decision table.

What failed

The agent never confirmed the worktree matched HEAD. It trusted a file catalog built during an earlier job. That catalog still listed a helper the branch had deleted.

A resurrected file can pass tests on a dirty tree. Reviewers may merge a path that the product no longer owns. Later refactors then collide with the ghost module.

Labeled scope

The write-up is a reconstructed incident, not a scored outage. No customer names, quotas, or hardware claims appear here. Operators should replace sample hashes with their own logs.

Warm checkouts reduce cold start on a shared free server. They also retain indexes, patch caches, and dirty paths. Cleanup that only drops a lock leaves those artifacts.

Shared queues such as MonkeyCode's free server option make reuse visible. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access still requires a worktree freshness check.

Timeline

T+0 — prior job exits

Job A finished a refactor and left the workspace mounted. The process wrote a catalog under .agent/index.json. Cleanup removed the lock file but not the snapshot.

T+1 — next job is scheduled

Job B claimed the same checkout for a rename ticket. The scheduler marked the tree idle after lock release. No git fetch or hard reset ran before the prompt.

T+2 — the agent plans a rename

The agent loaded .agent/index.json as the file catalog. The catalog listed src/helpers/parse_date.py as still present. HEAD on this branch had deleted that path already.

T+3 — the apply

The agent created src/helpers/parse_dt.py from the stale path. It also patched imports in a module that still existed. The deleted original was copied forward under the new name.

T+4 — tests on the dirty tree

Tests ran against the dirty worktree, not a clean clone. The ghost module satisfied imports the deleted path once provided. The suite passed for the wrong reason and stayed green.

T+5 — detection

A later git status showed an untracked resurrection plus extra edits. CI on a clean runner failed with a missing-module import. The merge stopped only because CI used a fresh checkout.

Contributing factors

Several independent mistakes had to line up at once.

1. The scheduler treated lock release as proof of a clean tree.
2. The agent treated an on-disk index as an authoritative catalog.
3. No step compared git ls-files with planned patch paths.
4. Tests executed in the same dirty workspace as the apply.
5. Reuse was tuned for latency rather than workspace isolation.

None of these looks severe in isolation. Together they turn a rename into a file resurrection. The model quality is not the primary control point.

Durable fix

The durable fix is a freshness gate before every apply. The gate records HEAD, porcelain status, and deleted-path overlap. Any mismatch fails the job before the model writes files.

The gate does not grade the patch. It only proves the tree is the tree the prompt described. That single invariant stops this class of resurrection.

Proposed command flow

The following commands are a labeled, unexecuted example. They belong in the job wrapper, not inside the model prompt. Run them as the same user that will apply the patch.

git fetch --prune origin
git rev-parse HEAD
git rev-parse @{upstream}
git status --porcelain
git ls-files -d

git ls-files -d lists paths deleted in the worktree versus the index. A shared workspace should have an empty list before planning. A non-empty list means the last job did not reset.

Proposed freshness gate

The script below is proposed sample code, not production telemetry. It expects a git repo root and an optional index file. Planned paths are read from stdin, one path per line.

#!/usr/bin/env bash
# freshness_gate.sh — proposed sample, not a measured benchmark
set -euo pipefail

ROOT="${1:-.}"
INDEX_FILE="${ROOT}/.agent/index.json"
cd "$ROOT"

if ! git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
  echo "FAIL: not a git worktree"
  exit 9
fi

HEAD="$(git rev-parse HEAD)"
echo "head=${HEAD}"

UPSTREAM="$(git rev-parse --abbrev-ref --symbolic-full-name @{upstream} 2>/dev/null || true)"
if [[ -n "${UPSTREAM}" ]]; then
  git fetch --quiet --prune
  REMOTE="$(git rev-parse "${UPSTREAM}")"
  if [[ "${HEAD}" != "${REMOTE}" ]]; then
    echo "FAIL: HEAD ${HEAD} != upstream ${REMOTE}"
    exit 10
  fi
fi

PORCELAIN="$(git status --porcelain)"
if [[ -n "${PORCELAIN}" ]]; then
  echo "FAIL: dirty worktree"
  printf '%s\n' "${PORCELAIN}"
  exit 11
fi

if [[ -f "${INDEX_FILE}" ]]; then
  INDEX_HEAD="$(python3 -c 'import json,sys
p=sys.argv[1]
d=json.load(open(p, encoding="utf-8"))
print(d.get("head") or d.get("sha") or "")' "${INDEX_FILE}")"
  if [[ -z "${INDEX_HEAD}" ]]; then
    echo "FAIL: index missing head field"
    exit 12
  fi
  if [[ "${INDEX_HEAD}" != "${HEAD}" ]]; then
    echo "FAIL: index head ${INDEX_HEAD} != ${HEAD}"
    exit 12
  fi
fi

DELETED="$(git ls-files -d)"
if [[ -n "${DELETED}" ]]; then
  echo "FAIL: deleted paths still in worktree view"
  printf '%s\n' "${DELETED}"
  exit 13
fi

while IFS= read -r path; do
  [[ -z "${path}" ]] && continue
  if ! git cat-file -e "HEAD:${path}" 2>/dev/null; then
    if git log -1 --diff-filter=D --pretty=%H -- "${path}" | grep -q .; then
      echo "FAIL: planned path deleted at HEAD: ${path}"
      exit 14
    fi
  fi
done

echo "OK"

Save it as freshness_gate.sh and keep it executable. Feed the planned paths after the model emits a patch plan. Do not feed paths after the writes already landed.

chmod +x freshness_gate.sh
printf '%s\n' src/helpers/parse_dt.py src/app.py | ./freshness_gate.sh /workspace/repo
echo $?

Exit 0 means the tree and catalog agree. Exit 10 through 14 mean a specific freshness fault. The wrapper should skip the apply and reset the checkout.

Proposed reset on failure

A failed gate should discard the warm tree. The commands below are again a labeled example. They assume the job owns this worktree exclusively.

git reset --hard HEAD
git clean -fd
rm -f .agent/index.json
git checkout --force "${EXPECTED_SHA}"

EXPECTED_SHA must come from the ticket, not from the leftover index. After reset, rebuild the catalog from git ls-files. Only then load the model prompt for this job.

Decision table

Use the table before choosing reuse, reset, or reject.

Observation	Meaning	Action
HEAD matches upstream and porcelain is empty	Tree is current and clean	Allow plan and apply
HEAD differs from upstream	Warm checkout lagged the branch	Fetch and hard reset
Porcelain is non-empty	Prior job leaked dirty files	Reset, clean, rebuild catalog
Index head field != HEAD	Catalog belongs to another job	Delete index, rebuild from git ls-files
Planned path missing at HEAD	Agent is about to resurrect a delete	Reject apply, return the path to the operator
Tests would run in the same tree	Green results can be contaminated	Run tests in a second, clean clone

The table is the policy for the wrapper. The script is only the sensor for that policy. Humans still decide whether a missing path is an intentional add.

Proposed test plan

These checks are unexecuted examples for a local clone. They do not report production pass rates. Copy them into a throwaway repo before trusting the wrapper.

1. Create a repo, commit a helper file, then delete it on main.
2. Write .agent/index.json with the old head and deleted path.
3. Run the gate with the new helper path and expect a failure exit.
4. Reset the index head to current HEAD but keep a dirty resurrection file.
5. Run the gate again and expect exit eleven from porcelain.
6. Hard reset, rebuild a matching index, and expect exit zero with empty stdin.
7. Pass an intentional new path that never existed and confirm allow.
8. Pass a path deleted at HEAD and confirm the gate rejects apply.

A minimal fixture script can automate steps one through three.

#!/usr/bin/env bash
# proposed fixture — labeled example
set -euo pipefail
TMP="$(mktemp -d)"
trap 'rm -rf "${TMP}"' EXIT
cd "${TMP}"
git init -q
git config user.email "[e-mail]"
git config user.name "dev"
mkdir -p src/helpers
echo 'def parse_date(): ...' > src/helpers/parse_date.py
git add src/helpers/parse_date.py
git commit -qm "add helper"
OLD="$(git rev-parse HEAD)"
git rm -q src/helpers/parse_date.py
git commit -qm "drop helper"
mkdir -p .agent
printf '{"head":"%s","files":["src/helpers/parse_date.py"]}\n' "${OLD}" > .agent/index.json
echo "fixture_head=$(git rev-parse HEAD)"
echo "stale_index_head=${OLD}"

Point freshness_gate.sh at the temp fixture after it runs. The stale head field should fail closed. That is the regression this incident needs.

What the gate does not fix

The script does not prove semantic correctness of any patch. It cannot see files ignored by git or written outside the repo. A race remains if two jobs pass the gate together.

A separate exclusive lock is still required around the apply window. Freshness without isolation will lose that race. Isolation without freshness will still resurrect deleted paths.

The gate also cannot repair a prompt that names the wrong file. If the ticket itself cites a deleted module, reject that ticket. Do not let the model helpfully recreate the module.

Intentional re-adds of a deleted path need an operator override. Exit 14 is correct until that override exists. Record the override beside the ticket hash, not inside the catalog.

Who should not use this approach

Teams with fully ephemeral single-job containers gain little here. The extra fetch will hurt air-gapped runners without a git remote. Do not treat the gate as a substitute for review.

Do not run git clean -fd on a workspace that holds operator edits. The reset path is for disposable agent checkouts only. Interactive developer trees need a different, slower protocol.

Close

Deleted paths come back when catalogs outlive the commit that removed them. Fail closed on HEAD drift, dirty porcelain, and stale index hashes. Rebuild the file list from git, then let the model plan.

Install the gate, then keep shared workspaces disposable after each job.