Postmortem: Two Agent Jobs Wrote Into the Same Checkout

Two agent jobs can destroy a clean git history.
A shared working checkout is the usual cause.
This reconstructed postmortem maps that failure path.
It also ships a durable isolation workflow.

The incident below is a labeled reconstruction only.
No production customer data appears in this writeup.
The failure class is common on reused remote boxes.
Treat every command here as an unexecuted example.

Core finding

Agents do not honor human workspace habits.
They write files wherever the process starts.
Two jobs on one tree produce a mixed patch.
Green tests on a mixed tree are not proof.

Git worktrees plus an exclusive lock stop this.
Commit trailers make the isolation auditable later.
Skip either control and collisions return fast.

Incident summary

A team ran two coding-agent jobs on one host.
Both jobs used the same clone of a service repo.
Job A refactored a rate limiter module.
Job B added retries around an outbound client.

The host was a reused remote development box.
Prior uncommitted files remained on local disk.
Neither job created a dedicated git worktree.
Both jobs treated git status output as noise.

The merged patch compiled without a linker error.
Unit tests passed on the mixed working tree.
Reviewers saw a single green check on the PR.
Production then logged contradictory timeout values.

Reconstructed timeline

Times below are illustrative, not live SEV stamps.
The sequence is the artifact, not the clock.

1. 09:12 — Job A checks out main at c0ffee1.
2. 09:14 — Job A leaves generated fixtures untracked.
3. 09:31 — Job B starts inside the same directory.
4. 09:33 — Job B rebases without a clean index.
5. 09:47 — Job A rewrites limiter.py and nearby tests.
6. 09:52 — Job B rewrites limiter.py for retries.
7. 10:05 — The test runner executes the mixed files.
8. 10:08 — The suite reports twelve passing tests.
9. 10:19 — Job A commits, capturing Job B hunks.
10. 10:41 — The pull request reaches main after review.
11. 11:16 — Staging shows duplicate retry wrappers.
12. 11:40 — New agent jobs freeze on that host.

The clock is not the real lesson.
The missing lock is the real lesson.

Impact

The mixed patch changed timeout policy twice.
Callers saw both shorter and longer budgets.
On-call spent a morning bisecting noisy traces.
The revert was larger than either intended diff.

No external breach belonged to this incident class.
No secret leaked through the agent session log.
The bill was engineering time and review trust.

Contributing factors

Several ordinary choices stacked into one failure.
None of them required a novel toolchain bug.

• The clone lived at a fixed path, /srv/app.
• Agents inherited that path from a shell profile.
• No git worktree wrapper sat in front of jobs.
• git status --porcelain was never a hard gate.
• Tests targeted functions, not file provenance.
• Review UI collapsed both jobs into one diff.
• The remote box outlived any single agent job.

None of these choices is exotic in daily work.
Together they erase authorship of a hunk.

Detection gaps

pytest cannot see a dirty index by itself.
Coverage cannot see a foreign hunk either.
A green eval does not name its tree.

The team watched token spend and request latency.
Nobody watched pwd and .git identity together.
The dashboard stayed calm during the mix.

After-the-fact debugging workflow

Use this order on a suspect clone.
Do not start a third agent job first.

1. Print every attached worktree and its HEAD.
2. Snapshot porcelain status before any cleanup.
3. List files with two authors in one short window.
4. Recover intended hunks from reflog, not memory.

# unexecuted example: inspect a suspect clone
git worktree list --porcelain
git status --porcelain=v2
git diff --name-only HEAD
git log --since='6 hours ago' --format='%h %an %s' -- limiter.py
git blame -L 1,40 limiter.py
git reflog --date=iso | head -n 40

worktree list answers occupancy of the repo.
Porcelain status answers dirty-tree questions next.
Blame answers mixed authorship last.

A mixed file often keeps tests green.
Look for one constant edited twice.
Look for imports that serve two features.

# unexecuted example: conflict signature in git blame
12f3a11 (job-a  2026-09-03 09:47:01) TIMEOUT_MS = 150
9bb0c02 (job-b  2026-09-03 09:52:44) TIMEOUT_MS = 400

Two nearby assignments are the usual scar.
That scar is the postmortem evidence.

Durable fix

The durable fix is mechanical, not cultural.
Each job gets a worktree and a lock.
A dirty tree fails the job at start.
A commit trailer records job identity.

The controls below are executable examples.
They are not claimed production metrics.
Operators should test them on a throwaway clone.

1. Refuse a dirty shared tree

#!/usr/bin/env bash
# file: scripts/agent-preflight.sh
# proposal: run before any agent job
set -euo pipefail

repo_root="$(git rev-parse --show-toplevel)"
cd "$repo_root"

if [[ -n "$(git status --porcelain)" ]]; then
  echo "refuse: working tree is dirty" >&2
  git status --porcelain >&2
  exit 12
fi

head_sha="$(git rev-parse HEAD)"
echo "preflight_ok head=${head_sha}"

A job that starts dirty must stop immediately.
Cleaning belongs to humans, not the agent.
Exit code 12 is a contract for wrappers.

2. One worktree per job

#!/usr/bin/env bash
# file: scripts/agent-worktree.sh
# proposal: unique path, exclusive flock
set -euo pipefail

job_id="${1:?usage: agent-worktree.sh <job-id>}"
base_ref="${2:-HEAD}"

repo_root="$(git rev-parse --show-toplevel)"
safe_id="$(printf '%s' "$job_id" | tr -cd 'a-zA-Z0-9._-')"
work_dir="${repo_root}/../app-jobs/${safe_id}"
lock_file="${repo_root}/.git/agent-jobs.lock"

mkdir -p "$(dirname "$work_dir")"

exec 9>"$lock_file"
if ! flock -n 9; then
  echo "refuse: another agent job holds the clone lock" >&2
  exit 13
fi

if [[ -e "$work_dir" ]]; then
  echo "refuse: worktree path already exists: $work_dir" >&2
  exit 14
fi

git worktree add --detach "$work_dir" "$base_ref"
echo "worktree=$work_dir"
echo "job_id=$safe_id"

flock -n fails fast on occupancy collision.
The worktree path encodes the job id.
Detached HEAD avoids surprise branch moves.
Keep the lock held for the whole job.

3. Stamp the commit with job identity

#!/usr/bin/env bash
# file: scripts/agent-commit.sh
# proposal: trailers, no invented model names
set -euo pipefail

job_id="${AGENT_JOB_ID:?set AGENT_JOB_ID}"
host_label="${AGENT_HOST_LABEL:-local}"
model_label="${AGENT_MODEL_LABEL:-unspecified}"

if [[ -z "$(git status --porcelain)" ]]; then
  echo "refuse: nothing to commit" >&2
  exit 15
fi

git add -A
git commit -m "agent job ${job_id}" \
  -m "Agent-Job-Id: ${job_id}" \
  -m "Agent-Host: ${host_label}" \
  -m "Agent-Model-Label: ${model_label}"

Trailers survive rebase better than PR body text.
CI can grep them without a vendor API.
Leave the model label unspecified when unknown.
Do not invent a model name for the trailer.

4. CI gate for missing trailers

# file: .github/workflows/agent-trailer.yml
# proposal: policy check, not a vendor score
name: agent-trailer
on:
  pull_request:
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: require Agent-Job-Id on agent commits
        run: |
          set -euo pipefail
          commits="$(git log --format=%H origin/main..HEAD)"
          if [[ -z "$commits" ]]; then
            echo "no commits in range"
            exit 0
          fi
          while read -r sha; do
            [[ -z "$sha" ]] && continue
            msg="$(git log -1 --format=%B "$sha")"
            if grep -q '^agent job ' <<<"$msg"; then
              grep -q '^Agent-Job-Id: ' <<<"$msg"
              grep -q '^Agent-Host: ' <<<"$msg"
            fi
          done <<<"$commits"

The workflow is a labeled proposal only.
Teams should match their real default branch.
The check encodes policy, not model quality.

Decision table

Condition	Start job?	Required control
Dirty index on shared clone	No	agent-preflight.sh
Second job, same path	No	flock plus new worktree
Clean tree, unique worktree	Yes	Trailers on commit
Job on ephemeral VM, one clone	Yes	Preflight still required
Human WIP in the same clone	No	Separate worktree for the agent
Missing model label	Yes, with unspecified	Do not invent a model name

The table is the review artifact.
Arguments about taste do not override it.

Test plan

Run this on a disposable clone only.
Do not point it at a production checkout.

1. Create a tiny repo with one Python file.
2. Run agent-preflight.sh on a clean tree. Expect exit 0.
3. Touch a file and rerun preflight. Expect exit 12.
4. Clean the tree. Start worktree job-a.
5. Start job-b under the same lock. Expect exit 13 or 14.
6. Commit in job-a with trailers. Inspect git log -1.
7. Mix two edits by hand in one tree. Confirm tests can pass.
8. Reset. Repeat with worktrees. Confirm diffs no longer mix.

Step 7 is the teaching step.
Green tests on mixed files are easy to produce.
That is why the lock sits in front.

# file: limiter.py
TIMEOUT_MS = 200

def allow(budget_ms: int) -> bool:
    return budget_ms >= TIMEOUT_MS

# file: test_limiter.py
from limiter import allow, TIMEOUT_MS

def test_allow_boundary():
    assert allow(TIMEOUT_MS) is True
    assert allow(TIMEOUT_MS - 1) is False

These tests stay green under many mixed edits.
They never assert a single author for TIMEOUT_MS.
Do not treat them as collision detectors.

Where a free remote server fits

Some teams overflow agent work off the laptop.
A free remote server makes that overflow cheap.
Free model access makes extra jobs tempting.
Cheap overflow increases concurrent jobs quickly.

MonkeyCode offers free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Concurrent jobs need isolation before extra capacity.
The scripts above do not depend on a vendor.
Remove the product name and the git rules remain.
Host labels should stay generic in trailers.
Use unspecified until an operator records a real id.

Limitations

flock is local to one kernel only.
NFS home directories can ignore advisory locks.
Worktrees still share object storage on disk.
They do not provide multi-tenant security.

This workflow does not replace human review.
It does not pin a model checkpoint.
It does not prove tests match production.
It does not make a free server dedicated hardware.
It does not publish quotas, SKUs, or uptime claims.

Do not store production secrets on a shared box.
Do not point the agent at credential files.
Do not treat trailers as cryptographic proof.

Windows users need Git worktree support first.
WSL is a simpler path for these scripts.
Validate flock before trusting the lock file.

Who should skip this

Teams that spawn one ephemeral VM per job can skip worktrees.
They should still keep the dirty-tree preflight.
Solo developers with one laptop job can skip the lock.
They should still refuse dirty trees at start.

This approach is wrong as a substitute for design review.
It is wrong as a billing dashboard.
It is wrong as a claim of model quality.
It is wrong on a box that already holds production secrets.

What remains after the fix

The mixed patch cannot form the same way.
A second job fails at lock or path.
CI rejects unlabeled agent commits.
Bisect can name a job id again.

The remaining risk is human bypass.
Someone will export AGENT_JOB_ID by habit.
Periodic audit of trailers still matters.

Remote free compute stays useful after this control.
It is overflow capacity, not a shared brain.
Isolation is the price of that overflow.
Paste agent-preflight.sh before the next remote job.