Postmortem: Readiness Probe Replayed a One-Shot File Delete

A readiness probe replayed a one-shot file delete. The second delete hit a recycled workspace tree. Green logs hid the duplicate mutation until tests failed. Durable replay protection belongs in the tool gateway.

This reconstructed postmortem walks a probe-replay delete. The timeline is a worked example, not a dated outage report. The failure class is common on ephemeral agent runners.

Incident summary

The worker stored the last mutating tool request. A platform probe called the resume endpoint without a body. Empty JSON meant repeat the last mutation on that worker.

The delete ran again after a workspace recycle. The intended target lived under job A's tree. The recycled volume mounted job B's checkout instead.

Job B lost a generated fixture during the second delete. CI then failed on a missing path, not on the original cleanup. Agent logs still showed one successful delete_file call.

Scope of the worked example

• One sibling CI job lost tmp/gen_schema.sql during recycle.
• No production database table was dropped in this sketch.
• Agent logs still showed a single successful delete.
• Detection lagged until a later queue consumed job B.

This was a control-plane bug, not a model-quality bug. The language model proposed a valid cleanup path. The gateway executed that cleanup more than once.

Timeline

Times are runner-local and truncated to minutes. Treat the clock as a teaching sequence.

1. 09:14 — The agent plans cleanup of tmp/gen_schema.sql.
2. 09:15 — delete_file returns {"ok": true, "id": "t-441"}.
3. 09:15 — The worker caches the raw HTTP request for resume.
4. 09:16 — Readiness handling fails over to /internal/replay-last.
5. 09:16 — The probe posts {} because empty JSON is allowed.
6. 09:16 — The worker maps {} onto replay last mutating call.
7. 09:17 — The workspace volume recycles between two probe ticks.
8. 09:17 — Replay resolves the same relative path in a new tree.
9. 09:18 — Job B's tests fail on a missing fixture file.
10. 09:41 — On-call finds one tool id with two parent spans.

Contributing factors

Several small engineering choices lined up in production. None of those choices looked severe in isolation.

• Resume support stored a full mutating request, not a cursor.
• Empty JSON was a legal alias for repeat last.
• Health checks shared the agent HTTP app, not a sidecar.
• Workspace volumes were ephemeral and reused by relative path.
• Tool ids were unique per call, not per execution attempt.
• Success logs omitted the replay source header entirely.

Why the original job stayed green

The original job still had a green delete result. Job B failed later in another CI queue. No assertion checked that a tool id ran once.

Log compaction also dropped the probe access line. Relative paths made the second delete look identical. Both trees contained tmp/gen_schema.sql for different reasons.

Hash-based path guards were never enabled on temp files. Temp cleanup looked harmless in code review. Harmless relative deletes become cross-job deletes after recycle.

Reproduction workflow

Do not reproduce this on a shared production runner. Use a disposable workspace and a fake probe.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Those two options fit a throwaway replay drill. They are not a claim about capacity, hardware, or model names.

Label: the commands below are a proposed local drill. They are not a production incident runbook.

# proposed drill — isolated directories only
mkdir -p /tmp/job-a/tmp /tmp/job-b/tmp
echo "fixture-a" > /tmp/job-a/tmp/gen_schema.sql
echo "fixture-b" > /tmp/job-b/tmp/gen_schema.sql

# worker sees JOB_ROOT from the environment
export JOB_ROOT=/tmp/job-a
python proposed_worker.py delete_file tmp/gen_schema.sql

# simulate recycle, then empty-body replay
export JOB_ROOT=/tmp/job-b
python proposed_worker.py replay_last --body '{}'

# inspect both trees after the replay attempt
wc -l /tmp/job-a/tmp/gen_schema.sql /tmp/job-b/tmp/gen_schema.sql

Expected failure without the fix is a missing job B fixture. Expected result with the fix is a 409 replay_conflict. Stop the drill if either path lives outside /tmp.

Proposed idempotency envelope

Label: proposed Python gateway, unexecuted in this article. Reviewers should treat it as a sketch.

# proposed_worker.py — sketch for review, not a library
from dataclasses import dataclass
from hashlib import sha256
import json, os

MUTATING = {"delete_file", "write_file", "apply_patch"}

@dataclass
class Envelope:
    tool: str
    path: str
    idem: str
    body_hash: str

class ReplayGuard:
    def __init__(self):
        self.seen = {}  # idem -> Envelope

    def accept(self, tool, path, body, headers):
        if tool not in MUTATING:
            return {"ok": True, "skipped": "read_only"}
        idem = headers.get("Idempotency-Key")
        if not idem:
            return {"ok": False, "error": "missing_idempotency_key"}
        if body in ({}, None):
            return {"ok": False, "error": "empty_body_not_replayable"}
        digest = sha256(
            json.dumps(body, sort_keys=True).encode()
        ).hexdigest()
        prior = self.seen.get(idem)
        if prior:
            same = prior.body_hash == digest and prior.path == path
            if same:
                return {"ok": True, "replayed": True, "id": idem}
            return {"ok": False, "error": "replay_conflict"}
        self.seen[idem] = Envelope(tool, path, idem, digest)
        return {"ok": True, "committed": True, "id": idem}

def resolve(path):
    root = os.path.realpath(os.environ["JOB_ROOT"])
    full = os.path.realpath(os.path.join(root, path))
    if full != root and not full.startswith(root + os.sep):
        raise ValueError("path_escapes_root")
    return full

Empty bodies and missing keys now fail closed. Same key with a different body fails closed. Same key with the same body is a no-op success.

Proposed test plan

Label: proposed checks, not recorded CI output. Each case should fail closed without a live cluster.

1. First mutating call with key k1 commits once.
2. Second call with k1 and the same body returns replayed.
3. Second call with k1 and a new path returns replay_conflict.
4. Call with {} body returns empty_body_not_replayable.
5. Call without Idempotency-Key is rejected.
6. After JOB_ROOT swap, replay still refuses empty bodies.
7. Read-only tools still run without keys.
8. Path escape outside JOB_ROOT raises path_escapes_root.

# proposed
python -m pytest proposed_test_replay_guard.py -q

# proposed_test_replay_guard.py
from proposed_worker import ReplayGuard

def test_empty_body_does_not_repeat_delete():
    g = ReplayGuard()
    headers = {"Idempotency-Key": "k1"}
    body = {"path": "tmp/gen_schema.sql"}
    first = g.accept("delete_file", body["path"], body, headers)
    second = g.accept("delete_file", body["path"], {}, headers)
    assert first["committed"] is True
    assert second["ok"] is False
    assert second["error"] == "empty_body_not_replayable"

def test_same_key_new_path_conflicts():
    g = ReplayGuard()
    headers = {"Idempotency-Key": "k1"}
    a = g.accept("delete_file", "tmp/a.sql", {"path": "tmp/a.sql"}, headers)
    b = g.accept("delete_file", "tmp/b.sql", {"path": "tmp/b.sql"}, headers)
    assert a["committed"] is True
    assert b["error"] == "replay_conflict"

Decision table

Signal	Treat as	Action
Missing idempotency key	Unsafe client	Reject mutation
Empty JSON body	Probe or reconnect noise	Reject mutation
Same key, same hash	Network retry	Return prior success
Same key, new hash	Client mix-up	Conflict, do not write
Volume recycle plus relative path	Cross-job risk	Resolve realpath under JOB_ROOT
Health check on the app port	Replay hazard	Move probes to a sidecar

Durable fix

The sketched control plane removes /internal/replay-last from the public app. Health checks now hit a sidecar that cannot see tool state. Mutating tools require an idempotency key from the orchestrator.

Empty bodies no longer alias to the last call. Workspace mounts now include the job id in the root. Relative temp deletes cannot cross into a sibling checkout.

Tool logs record execution_id, idempotency_key, and replayed. Operators can grep one execution across retries without collapsing spans. Duplicate deletes then show up as conflicts, not as quiet success.

Limitations

This envelope does not order concurrent distinct keys. Two legal deletes can still race on one path. It also does not persist self.seen across process death.

A restarted worker can accept a key again. Hashing the JSON body does not canonicalize equivalent patches. Keyed retries must send byte-identical JSON bodies.

The path guard requires a trusted JOB_ROOT. Attackers who set JOB_ROOT bypass the realpath check. Operators must pin JOB_ROOT outside agent-writable env files.

Who should not use this approach

• Teams without mutating tools can skip the extra key.
• Multi-tenant runners that share one JOB_ROOT need isolation first.
• Workflows that treat empty bodies as a documented resume API must redesign that API.
• Regulated delete paths still need human approval beyond idempotency.

Do not copy this sketch onto a cluster with an existing replay log. Unify on one store before adding another in-memory dictionary. An extra dictionary will hide the real conflict stream.

What remains

Probe replay is a platform footgun around agents. The language model did not invent the second delete. The HTTP gateway executed the extra delete against disk.

Treat mutating tool gateways like payment APIs here. Keep production volumes off this class of experiment. Replay drills belong on disposable roots, then stop.