Three weeks after the merge, the pager went off. The anomaly trace pointed at a function nobody remembered writing. Git blame showed a commit from an AI assistant. The only comment said "refactor config parsing for clarity." Nobody could explain why the change existed. Nobody could prove it was safe. The diff was correct. The reasoning was gone.
That is the provenance problem. Human developers leave reasoning everywhere. PR descriptions explain trade-offs. Comments capture dead ends. Commit messages record intent. AI assistants leave a single artifact: the patch. The code works. The why evaporates.
The fix is not better prompts. The fix is a file.
The Evidence File
Every AI-generated patch deserves a companion document. Call it an evidence file. It captures four things. The exact task description. The environment snapshot. The test results, including a control run. The human reviewer's verdict.
Think of it as a lab notebook. A chemist records every step. The temperature, the reagent, the observation. Nobody would trust an experiment with no notebook. Yet we merge AI patches with no record. The evidence file is the notebook.
The evidence file turns a mysterious diff into an auditable record. It answers the question "why does this exist?" without a séance.
The Control Run
A green test suite proves nothing. The suite passed before the patch. It passed after the patch. The agent's contribution could be zero. You need a control.
Revert the patch. Run the suite again. If no test fails, the patch is dead code. If a test fails, the patch is doing something. That one command separates signal from noise.
The control run is the part most teams skip. It feels redundant. The patch passed the tests, so why revert it? Because passing tests are the baseline. The suite passes on every commit. The question is whether the patch changes anything. The control answers that.
#!/usr/bin/env bash
# evidence.sh — generate an audit trail for one AI patch
set -euo pipefail
TASK="$1"; REPO="$2"; OUT="evidence-$(date +%Y%m%d-%H%M)"
mkdir -p "$OUT"
echo "=== snapshot ===" > "$OUT/env.txt"
git -C "$REPO" rev-parse HEAD >> "$OUT/env.txt"
node --version >> "$OUT/env.txt" 2>/dev/null || true
npm ls --depth=0 >> "$OUT/env.txt" 2>/dev/null || true
cp "$TASK" "$OUT/task.md"
# Agent invocation is environment-specific. Substitute your own command.
your-agent --task "$TASK" --repo "$REPO" --output "$OUT/agent.patch"
cd "$REPO"
git apply --check "$OUT/agent.patch" || exit 1
git apply "$OUT/agent.patch"
# Test run with the patch applied.
npm test > "$OUT/with-patch.txt" 2>&1 || true
# Control: revert and test again.
git apply -R "$OUT/agent.patch"
npm test > "$OUT/without-patch.txt" 2>&1 || true
cat > "$OUT/verdict.json" <<EOF
{
"task": "$(basename "$TASK")",
"commit": "$(git rev-parse HEAD)",
"patch_applies": true,
"tests_pass_with_patch": $(grep -qE 'failing|✗' "$OUT/with-patch.txt" && echo false || echo true),
"tests_fail_without_patch": $(grep -qE 'failing|✗' "$OUT/without-patch.txt" && echo true || echo false)
}
EOF
echo "Evidence written to $OUT/"
Reading the Verdict
The evidence file produces three outcomes. Use the table.
| Signal | Verdict | Action |
|---|---|---|
| Tests pass with patch, fail without | Ship | Merge and monitor |
| Tests pass both ways | Kill | The patch is dead weight |
| Tests fail with patch | Kill | Send back with the log |
| Patch does not apply | Kill | Rebase or reject |
The table is coarse by design. A binary verdict beats a nuanced shrug. The control run is the star. Without it, the first row is meaningless.
Archive the evidence file next to the task. Name it with the date and the task ID. Store it in the repository or a shared drive. Six months later, someone will ask why the patch exists. The file answers without a meeting.
The Budget Question
Evidence costs tokens. Each trial burns generation tokens, test runs, and environment setup. The math matters when you run ten trials a week.
Split the work into two phases. Generation gets 70% of the budget. Verification gets 30%. The verification phase is non-negotiable. A patch without a control run is a rumor.
Free model access changes the calculus. MonkeyCode's free models and free server option mean the verification phase costs nothing but time. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier is a starting point, not a guarantee. Your verdict still depends on your own test suite.
The free server matters more than the free tokens. A disposable runtime means you can destroy the environment after each trial. No cleanup debt. No lingering containers. No surprise bills. The evidence file survives the destruction. That is the point.
The 70/30 split is a starting point. Some tasks need more generation budget. Some need more verification. Track your own ratios. Adjust after ten trials. The rule is simple: never spend the last token on generation. Keep some for the control run.
Who Should Not Use This
The evidence file assumes three things. You have a test suite that runs in minutes. You have a reviewer who reads the verdict. You have tasks small enough to fit one patch.
Legacy codebases fail the first test. A forty-minute build eats the whole budget. Teams without automated tests should fix that gap first. The evidence file cannot rescue a project with no oracle.
Small teams with no reviewer should also pause. The evidence file is only as good as the human who reads it. A verdict nobody checks is a checkbox. Automate the generation. Never automate the judgment.
The File Outlives the Patch
The anomaly trace pointed at that config refactor. The on-call engineer opened the evidence file. The task description explained the goal. The control run proved the patch changed behavior. The reviewer's verdict said "ship, but watch the edge case."
Three weeks later, the edge case was the bug. The evidence file did not prevent the incident. It cut the diagnosis time from hours to minutes. That is the real payoff.
The next AI patch gets an evidence file too. The script is on disk. The habit is forming. The cost is one file per patch. The value is a project that explains itself.
Try the workflow with your own agent. If you need a free runtime to test it, MonkeyCode's free server tier works. But the script, not the product, is the point.
Top comments (0)