Fingerprint Repeated Failures in a Ninety-Minute Spike

Late on a Thursday, a platform engineer watched an assistant rerun the same failing integration test while the scroll kept moving. The terminal looked busy, and the wall clock still had nearly an hour left on the ninety-minute spike. Each invocation printed the same missing column name, yet the process treated every retry as a fresh attempt. The room had the energy of progress and the substance of a stuck elevator between floors.

Time-boxed spikes fail in a quiet way when busy motion is mistaken for evidence of learning. A coding agent can burn the entire clock inside one failing command without any memory of prior failures. The useful analogy is a locksmith who tries the same key on the same door, then bills for the motion of the wrist. The spike needs a budget for identical failures, not only a calendar alarm on the wall.

The working rule fits inside a small shell wrapper that the coding agent never owns or edits. One hypothesis is written before any agent starts, and the wall clock is set to ninety minutes from that moment. A command fingerprint log sits outside the repository tree the agent is allowed to modify during the spike. If the same command returns a non-zero status three times, the spike ends even when minutes remain on the clock.

The hypothesis file is a contract, not a diary of hopeful notes written after the first failure. It names one change, one observable, and one kill condition that does not depend on the agent's self-report. A spike that cannot state those three items in a short file is not ready for a clock. The sample below is labeled as a template, because the numbers inside it are not measurements from a published trial.

# HYPOTHESIS.md
# Template for a ninety-minute spike. Not a result from a live run.

Statement: Adding an explicit NOT NULL migration will make the users.email
integration test pass without changing application query code.

Observable: pytest tests/test_users_email.py -q exits 0, and git diff --stat
stays inside db/migrations/ plus the test file.

Kill: The same failing command fingerprint is recorded three times, or the
ninety-minute deadline file expires, whichever arrives first.

Ship: Observable holds and the retry budget is unused or only partly used.

Public conversation around agent loops often celebrates another retry as a sign of professional grit. That reading is generous when the process is only a while loop with no memory of earlier exits. A fingerprint log is a cheap way to see the loop as a loop, then stop it with written evidence. The wrapper below is the mechanism, and the hypothesis file is the reason that mechanism may halt work.

The wrapper records a fingerprint for every command the operator or the agent launches through one entry point. The fingerprint is a hash of the working directory, the argument vector, and the integer exit status. A passing command with the same arguments does not consume the failure budget, because success is not a stuck loop. A failing command that matches an earlier failing fingerprint increments a counter that lives beside the log.

#!/usr/bin/env python3
"""Fingerprint a command for duplicate-failure accounting.
Template for a local lab. Not a measured benchmark.
"""
from __future__ import annotations

import hashlib
import json
import os
import sys


def fingerprint(cwd: str, argv: list[str], exit_code: int) -> str:
    payload = json.dumps(
        {
            "cwd": os.path.realpath(cwd),
            "argv": argv,
            "exit": int(exit_code),
        },
        separators=(",", ":"),
        ensure_ascii=True,
    )
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]


if __name__ == "__main__":
    if len(sys.argv) < 4:
        sys.stderr.write("usage: fingerprint.py <cwd> <exit_code> <argv...>\n")
        sys.exit(2)
    cwd = sys.argv[1]
    code = int(sys.argv[2])
    argv = sys.argv[3:]
    print(fingerprint(cwd, argv, code))

Coarse fingerprints create false kills, and that failure mode is more common than it first appears. Hashing only the binary name treats pytest foo and pytest bar as one command, which collapses unrelated work. Hashing the full argument vector plus the working directory keeps those runs apart while still catching a stuck loop. Environment variables are left out on purpose, because leaking them into a log can copy secrets into the evidence bundle.

The start script uses a portable Python one-liner for the deadline so GNU date extensions do not silently fail on a laptop. Every later command must go through spike-run, including test runners that the agent would otherwise call directly. If the agent learns the real pytest path and bypasses the wrapper, the budget is theater and the spike should be killed. That bypass is why the control directory lives outside the workspace, and why the agent account should not own it.

#!/usr/bin/env python3
"""Start a ninety-minute spike clock. Unexecuted lab template."""
from __future__ import annotations

import os
import sys
import time
from pathlib import Path

SPIKE_DIR = Path(os.environ.get("SPIKE_DIR", str(Path.home() / "spike-control")))
SPIKE_DIR.mkdir(parents=True, exist_ok=True)
hypothesis = SPIKE_DIR / "HYPOTHESIS.md"
if not hypothesis.is_file() or hypothesis.stat().st_size == 0:
    sys.stderr.write("start_spike.py: write HYPOTHESIS.md before starting the clock\n")
    sys.exit(2)

deadline = SPIKE_DIR / "deadline_epoch"
deadline.write_text(str(int(time.time()) + 90 * 60) + "\n")
(SPIKE_DIR / "command_log.jsonl").write_text("")
(SPIKE_DIR / "fail_counts").write_text("")
print(f"spike started in {SPIKE_DIR}")
print(f"deadline_epoch={deadline.read_text().strip()}")

#!/usr/bin/env bash
# spike-run — template wrapper. Unexecuted example for a local lab.
set -euo pipefail

SPIKE_DIR="${SPIKE_DIR:-$HOME/spike-control}"
DEADLINE_FILE="$SPIKE_DIR/deadline_epoch"
LOG_FILE="$SPIKE_DIR/command_log.jsonl"
COUNT_FILE="$SPIKE_DIR/fail_counts"
HYPOTHESIS="${HYPOTHESIS:-$SPIKE_DIR/HYPOTHESIS.md}"
MAX_DUP="${MAX_DUP:-3}"
FINGERPRINT_PY="${FINGERPRINT_PY:-$SPIKE_DIR/fingerprint.py}"

if [[ ! -f "$HYPOTHESIS" ]]; then
  echo "spike-run: hypothesis file missing" >&2
  exit 2
fi
if [[ ! -f "$DEADLINE_FILE" ]]; then
  echo "spike-run: spike not started" >&2
  exit 2
fi

now="$(date +%s)"
deadline="$(tr -d '[:space:]' < "$DEADLINE_FILE")"
if (( now >= deadline )); then
  echo "spike-run: clock expired; write ship-or-kill evidence" >&2
  exit 3
fi

if [[ $# -lt 1 ]]; then
  echo "usage: spike-run <command> [args...]" >&2
  exit 2
fi

set +e
"$@"
status=$?
set -e

fp="$(python3 "$FINGERPRINT_PY" "$PWD" "$status" "$@")"
python3 - "$LOG_FILE" "$PWD" "$status" "$fp" "$@" <<'PY'
import json, sys, time
log_path, cwd, status, fp = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
record = {
    "ts": int(time.time()),
    "cwd": cwd,
    "argv": sys.argv[5:],
    "exit": status,
    "fp": fp,
}
with open(log_path, "a", encoding="utf-8") as handle:
    handle.write(json.dumps(record, ensure_ascii=True) + "\n")
PY

if (( status != 0 )); then
  count="$(awk -v k="$fp" '$1==k {c=$2} END{print c+0}' "$COUNT_FILE")"
  count=$((count + 1))
  tmp="$(mktemp)"
  awk -v k="$fp" -v c="$count" 'NF && $1!=k {print} END{print k, c}' "$COUNT_FILE" > "$tmp"
  mv "$tmp" "$COUNT_FILE"
  if (( count >= MAX_DUP )); then
    echo "spike-run: duplicate failure budget exhausted for $fp ($count)" >&2
    echo "spike-run: kill the spike and keep the log" >&2
    exit 4
  fi
fi

exit "$status"

The operator starts the clock from a control directory, then routes every test and every migration command through spike-run. A sample session is shown next as an unexecuted lab sequence, not as output captured from a production incident. Shipping means the hypothesis observable moved in the recorded direction and the retry budget was not exhausted. Killing means the branch is discarded while the log, the hypothesis, and the decision paragraph are kept as evidence.

export SPIKE_DIR="$HOME/spike-control"
mkdir -p "$SPIKE_DIR"
cp fingerprint.py start_spike.py "$SPIKE_DIR/"
install -m 0755 spike-run "$HOME/bin/spike-run"
# Write HYPOTHESIS.md first, then start the ninety-minute clock.
python3 "$SPIKE_DIR/start_spike.py"
cd /path/to/repo
spike-run git status --porcelain
spike-run pytest tests/test_users_email.py -q
spike-run git diff --stat

When the clock expires, the wrapper refuses new commands and requires that ship-or-kill paragraph in the control directory. The leftover is evidence, and the discarded branch is the cost of keeping the main line honest during a spike. Some teams prefer to run the wrapper on a disposable host so laptop secrets never enter the spike environment. Disclosure: This article was prepared as part of MonkeyCode's product outreach, and the product mention below is limited to that context.

MonkeyCode offers free model access and a free server option that can host the wrapper without turning a personal workstation into the experiment. The duplicate-command budget does not depend on that product, and any isolated shell host with a clock can enforce the same rule. After the clock or the budget fires, the operator writes a short evidence file beside the log rather than merging by inertia. The file is boring on purpose, because drama in a kill report usually hides a missing fingerprint or a missing observable.

The example that follows is a filled template for a killed spike, not a claim about a particular vendor or model. A shipped spike uses the same shape, with the observable line copied from the test runner output rather than from chat. Reviewers can read the fingerprint counter without reconstructing a chat transcript after the fact. That is the whole point of putting memory in a file the agent cannot edit.

# EVIDENCE.md
# Filled template for a killed spike. Not a vendor benchmark.

Hypothesis: NOT NULL migration makes tests/test_users_email.py pass.
Clock: 90 minutes. Stopped at minute 41 by duplicate-failure cap.
Fingerprint: 9c1e0b7a4d22f310  argv=[pytest, tests/test_users_email.py, -q]  exit=1
Count: 3 matching failures. Same missing column name in captured stdout.
Diff: agent also touched app/models/user.py, which the hypothesis forbade.
Decision: KILL. Discard the branch. Keep this file, HYPOTHESIS.md, and command_log.jsonl.

This approach has limits that matter more than the small elegance of a sixteen-character command hash. It does not catch semantic loops in which the agent changes flags while chasing the same broken idea. Three matching failures is a chosen constant for a ninety-minute window, not a measured optimum from a study. Remote clocks that drift, and wrappers that the agent can edit, will both mint a false sense of control.

Teams in live incident response should not use this budget, because repeating the same health check can be the work. Flaky suites that talk to shared staging systems will trip the cap even when the application code is not stuck. Researchers who are mapping an unknown codebase may need repeated failing commands as a form of reconnaissance. The method is for a time-boxed product spike with one hypothesis, a human owner, and a willingness to discard the branch.

Operators who already keep spike logs can drop the fingerprint file beside those logs and read it at minute ninety. The cheap extra file is the difference between a busy session and a documented loop that can be shown to a reviewer. Nothing here asks a model to grade its own homework, and nothing here requires a particular vendor to be present. A ninety-minute spike that cannot survive a duplicate-failure cap was not about to ship a trustworthy change anyway.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.