When a task comes back as "done", the reviewer usually checks the result. Much less often, anyone checks the path it took: which brief the runner actually worked from, which commit the evidence belongs to, whether the test output was produced by the code being accepted, and who made the final call. When something breaks a month later, that path is what you need, and it's rarely written down in a form you can check.
This tutorial builds that check. There are three parts:
- A task template that the author fills in before anyone claims the work.
- An append-only event log with one line per handoff: prepared, claimed, delivered, sent back, accepted.
- A 100-line Python script that rebuilds the timeline from the log and checks it against the git history. It also re-runs each delivery's evidence on a clean checkout of the delivered commit.
Everything below comes from a toy repository I built for this post. The people are invented and the commit times were set by hand, but every commit, test run and audit output is real.
The roles
-
The task author prepares the work. Here that's Nare, a support lead. She writes
TASK.mdand commits it, so the brief has a SHA everyone can refer to. - The runner executes it with their own authorized tools. Lilit claims the task and works in her own clone, with her own git identity and push access, assisted by a coding agent. Nobody lends her a token, and the log never records one.
- The reviewer judges acceptance. Hayk runs the acceptance check on the delivered commit and either sends the work back with a reason or accepts it. He is not the runner.
Step 1: the task template
# Task: export log shows "75m 00s" for long exports
## Prepared by
Author: Nare (support lead). Reviewer: Hayk (backend).
## Outcome
Durations of one hour or more read as hours, minutes and seconds:
4500 s -> "1h 15m 00s". Under one hour stays as it is: 125 s -> "2m 05s".
## Out of scope
Changing the log format for anything other than durations.
## Acceptance check (the reviewer runs this on the delivered commit)
python3 -m unittest -v
Must include tests for 4500 s, 3600 s and 59 s, and pass.
## Evidence to attach
The delivered commit SHA and the exact output of the acceptance check.
Two lines in it matter for the audit. "Acceptance check" is a command, not a sentence, so it can be re-run. "Evidence to attach" asks for the SHA together with the output, so the output can't drift away from the code it describes.
Nare commits this as 0bc904f before the claim. If someone edits the brief later, the audit notices.
Step 2: the handoff, as it happened
The bug: the export log printed 75m 00s for a 75-minute export.
Lilit's first delivery, fc29eb4, added an hours branch and one test for 4500 s. Her own run passed 3 tests. Hayk ran the acceptance check from TASK.md, saw that the 3600 s and 59 s tests it asks for were missing, and tried 3600 s by hand. It returned 60m 00s, because the code used > where it needed >=. He sent the work back with that reason.
The second delivery, 1bbebc9, changed the comparison and added both tests. Five tests passed, and Hayk accepted it.
Each handoff is one line in events.jsonl, next to the repo:
{"at": "2026-10-15T09:20:00+04:00", "type": "prepared", "by": "Nare", "commit": "0bc904f"}
{"at": "2026-10-15T09:35:00+04:00", "type": "claimed", "by": "Lilit"}
{"at": "2026-10-15T10:12:00+04:00", "type": "delivered", "by": "Lilit", "attempt": 1, "commit": "fc29eb4", "evidence": {"cmd": "python3 -m unittest -v", "result": "Ran 3 tests: OK"}}
{"at": "2026-10-15T10:40:00+04:00", "type": "sent_back", "by": "Hayk", "attempt": 1, "reason": "TASK.md asks for 3600 s and 59 s tests. 3600 s returns '60m 00s'."}
{"at": "2026-10-15T11:15:00+04:00", "type": "delivered", "by": "Lilit", "attempt": 2, "commit": "1bbebc9", "evidence": {"cmd": "python3 -m unittest -v", "result": "Ran 5 tests: OK"}}
{"at": "2026-10-15T11:30:00+04:00", "type": "accepted", "by": "Hayk", "attempt": 2, "commit": "1bbebc9"}
The log is append-only. The rejected first attempt stays in it, which is the point. Reading it later, you can see why >= matters.
Step 3: the audit script
"""Rebuild a task's timeline from its event log and check it against git.
Usage: python3 audit.py <repo> <events.jsonl>
Exits 1 if any check fails. Stdlib only, Python 3.11+.
"""
import json
import re
import subprocess
import sys
import tempfile
from datetime import datetime
def git(repo, *args, check=True):
return subprocess.run(["git", "-C", repo, *args], check=check,
capture_output=True, text=True)
def run_evidence(repo, commit, cmd):
"""Re-run the evidence command on a clean checkout of the commit."""
with tempfile.TemporaryDirectory() as tmp:
git(repo, "worktree", "add", "--detach", "-q", tmp, commit)
try:
out = subprocess.run(cmd, shell=True, cwd=tmp,
capture_output=True, text=True)
finally:
git(repo, "worktree", "remove", "--force", tmp)
text = out.stdout + out.stderr
ran = re.search(r"Ran (\d+) tests?", text)
status = "OK" if out.returncode == 0 else "FAILED"
return f"Ran {ran.group(1) if ran else '?'} tests: {status}"
def audit(repo, events):
fails = []
when = lambda e: datetime.fromisoformat(e["at"])
times = [when(e) for e in events]
if times != sorted(times):
fails.append("events are not in time order")
prepared = next((e for e in events if e["type"] == "prepared"), None)
claim = next((e for e in events if e["type"] == "claimed"), None)
if not prepared or not claim or when(claim) < when(prepared):
fails.append("no preparation before the claim")
return fails
runner = claim["by"]
task_at_prep = git(repo, "show", f"{prepared['commit']}:TASK.md").stdout
deliveries = {}
for e in events:
if e["type"] == "delivered":
n, c = e["attempt"], e["commit"]
deliveries[n] = e
if git(repo, "cat-file", "-e", c, check=False).returncode:
fails.append(f"attempt {n}: commit {c} not in repo")
continue
if git(repo, "merge-base", "--is-ancestor", prepared["commit"],
c, check=False).returncode:
fails.append(f"attempt {n}: {c} is not built on the task")
made = git(repo, "show", "-s", "--format=%cI", c).stdout.strip()
if datetime.fromisoformat(made) > when(e):
fails.append(f"attempt {n}: delivered before {c} existed")
if git(repo, "show", f"{c}:TASK.md").stdout != task_at_prep:
fails.append(f"attempt {n}: TASK.md changed after prep")
got = run_evidence(repo, c, e["evidence"]["cmd"])
if got != e["evidence"]["result"]:
fails.append(f"attempt {n}: evidence says "
f"'{e['evidence']['result']}', re-run gives "
f"'{got}'")
elif e["type"] in ("sent_back", "accepted"):
n = e["attempt"]
if n not in deliveries:
fails.append(f"{e['type']} for attempt {n} with no delivery")
if e["by"] == runner:
fails.append(f"{e['type']} by the runner ({runner})")
if e["type"] == "accepted":
if e["commit"] != deliveries.get(n, {}).get("commit"):
fails.append(f"accepted {e['commit']}, but attempt {n}"
f" delivered {deliveries[n]['commit']}")
if e is not events[-1]:
fails.append("events after acceptance")
if not events or events[-1]["type"] != "accepted":
fails.append("no acceptance at the end of the log")
return fails
def main(repo, path):
events = [json.loads(line) for line in open(path) if line.strip()]
for e in events:
extra = e.get("commit") or e.get("reason") or ""
print(f"{e['at'][11:16]} {e['type']:<9} {e['by']:<6} {extra}")
fails = audit(repo, events)
print("\nPASS" if not fails else f"\nFAIL ({len(fails)})")
for f in fails:
print(f" - {f}")
return 1 if fails else 0
if __name__ == "__main__":
sys.exit(main(*sys.argv[1:3]))
It checks five things:
- The events are in time order, and the brief was prepared before the claim.
- Every delivered commit exists, is built on the prepared task commit, and existed before it was delivered.
-
TASK.mdis the same in every delivered commit as it was at preparation, so the scope wasn't quietly edited along the way. - Re-running the evidence command on a clean
git worktreeof the delivered commit gives the same result as the log claims. - Verdicts come from someone other than the runner, and the accepted commit is the one actually delivered in that attempt.
Running it
On the log above:
$ python3 audit.py toy logs/events.jsonl
09:20 prepared Nare 0bc904f
09:35 claimed Lilit
10:12 delivered Lilit fc29eb4
10:40 sent_back Hayk TASK.md asks for 3600 s and 59 s tests. 3600 s returns '60m 00s'.
11:15 delivered Lilit 1bbebc9
11:30 accepted Hayk 1bbebc9
PASS
To see what it catches, here is a second log for the same repo, written the way logs often get written in a hurry:
{"at": "2026-10-15T09:20:00+04:00", "type": "prepared", "by": "Nare", "commit": "0bc904f"}
{"at": "2026-10-15T09:35:00+04:00", "type": "claimed", "by": "Lilit"}
{"at": "2026-10-15T10:02:00+04:00", "type": "delivered", "by": "Lilit", "attempt": 1, "commit": "fc29eb4", "evidence": {"cmd": "python3 -m unittest -v", "result": "Ran 3 tests: OK"}}
{"at": "2026-10-15T11:15:00+04:00", "type": "delivered", "by": "Lilit", "attempt": 2, "commit": "1bbebc9", "evidence": {"cmd": "python3 -m unittest -v", "result": "Ran 3 tests: OK"}}
{"at": "2026-10-15T11:20:00+04:00", "type": "accepted", "by": "Lilit", "attempt": 2, "commit": "fc29eb4"}
$ python3 audit.py toy logs/events-sloppy.jsonl
09:20 prepared Nare 0bc904f
09:35 claimed Lilit
10:02 delivered Lilit fc29eb4
11:15 delivered Lilit 1bbebc9
11:20 accepted Lilit fc29eb4
FAIL (4)
- attempt 1: delivered before fc29eb4 existed
- attempt 2: evidence says 'Ran 3 tests: OK', re-run gives 'Ran 5 tests: OK'
- accepted by the runner (Lilit)
- accepted fc29eb4, but attempt 2 delivered 1bbebc9
Every one of these looks fine on a task board. The first delivery was logged at 10:02, three minutes before the commit it points to was made, so the evidence can't be about that commit. The second delivery's evidence was copied from the first attempt, and re-running it gives 5 tests, not 3. The runner accepted her own work. And the commit that was accepted is not the one that was delivered.
Where the log lives
The event log is only useful if it lives with the task and nobody can tidy it up later. A tracker that already keeps these as separate events saves you writing the JSONL by hand. Wagglet's request-to-delivery workflow records the claim, the delivery report with the branch, and the reviewer's verdict as separate events. Send back keeps the same runner, and independent acceptance has to come from someone other than the runner. That is the same split the audit checks. Whatever tool you use, the test is the same: could someone who wasn't there take over this task from its history alone? If so, the work is transferable AI work and not one person's memory.
Limits
-
It checks the record, not the judgment. The audit passed Lilit's first delivery. The missing 3600 s and 59 s tests were caught by Hayk reading
TASK.md, not by the script. Checking that tests exist for each case in the brief is still the reviewer's job. - It missed a gap in the sloppy log. Attempt 2 was delivered with no verdict on attempt 1, and the script didn't flag it. A rule like "every delivery after the first follows a send-back" would catch it. I left it out to keep the script short, and because some teams allow a runner to re-deliver before review.
- Re-running evidence only works for deterministic commands. Unit tests are fine. Screenshots, flaky integration tests and anything that calls a live service are not. For those, the log needs an artifact hash and the reviewer's own note.
- It trusts the log's names and times. Anyone who can write the JSONL can write "Hayk". In a real setup, the log should come from the tracker's own records, not a file the runner edits.
- It's slow on big suites. Every delivery is re-run from a clean worktree. That's fine for a five-test toy, but in CI you would cache it or re-run only the delivered attempt.
This post was drafted with AI assistance. The toy repository, commits, test runs and audit output are real and were run locally; the people and times are invented.
Top comments (0)