Your Coding Agent Says Done but Which Evidence Still Applies

A coding agent finishes a change and leaves a confident summary: the bug is fixed, the tests pass, and the work is ready for review.

A reviewer still needs answers to several ordinary questions. Which tests ran? Against which files? Did the implementation change after the run? Does the cited test actually exercise the behavior being claimed?

A chat transcript makes those questions surprisingly awkward to investigate. The evidence is spread across messages, terminal output, and a working tree that may have changed since the last test. Proofpacket is a small Python tool for keeping the command evidence and the author's claims together in a portable file.

A passing test has a scope and a time

Consider a document-sharing change with two claims:

• Guests cannot reshare a document.
• Every access path is covered.

A targeted test might provide useful evidence for the first claim. It says much less about the second. Attaching the same green checkmark to both hides a difference the reviewer needs to see.

Proofpacket stores claims, explicitly selected source files, command observations, and author-supplied links between claims and checks. Each command observation includes the arguments, working directory, exit code, output excerpts, and file fingerprints taken before and after execution.

That last detail matters. If a selected source file changes after the command ran, the old observation remains recorded but becomes stale. It cannot silently become evidence for the new version.

Run the example

Use Python 3.11 or newer. The runtime needs no third-party Python packages. Clone the repository and run these commands from its root:

git clone https://github.com/asadani/proofpacket.git
cd proofpacket
demo_dir="$(mktemp -d)"
cp -R fixtures/project "$demo_dir/project"

python -m proofpacket init \
  --task fixtures/task.md --claims fixtures/claims.json \
  --root "$demo_dir/project" \
  --path sharing.py --path test_sharing.py \
  --out "$demo_dir/packet.json"

python -m proofpacket run "$demo_dir/packet.json" \
  --root "$demo_dir/project" --id check-guests \
  -- python -m unittest test_sharing.py

python -m proofpacket link "$demo_dir/packet.json" \
  --claim guest-reshare --check check-guests

python -m proofpacket verify "$demo_dir/packet.json" \
  --root "$demo_dir/project" --out "$demo_dir/verification.json"

python -m proofpacket report "$demo_dir/packet.json" \
  --root "$demo_dir/project" --out "$demo_dir/report.md"

The targeted command passes. The guest-reshare claim has linked passing evidence; the broader all-paths claim remains unlinked. Open the Markdown report to inspect that distinction.

Now change a selected file in the temporary copy:

python -c 'from pathlib import Path; import sys; p=Path(sys.argv[1]); p.write_text(p.read_text() + "\n# Changed after the check\n")' "$demo_dir/project/sharing.py"

python -m proofpacket verify "$demo_dir/packet.json" \
  --root "$demo_dir/project" --out "$demo_dir/after-edit.json"

The second verification exits with status 1. The command's historical execution still passed, but its freshness is now stale. The linked claim needs review. Adding a comment is enough because the tool conservatively hashes whole files.

Keep the reviewer in the loop

The packet records observations; it cannot decide whether a test supports a claim. A trivial assertion can exit successfully. An author can link an irrelevant check. A selected file list can omit a dependency, configuration file, or important runtime input.

Commands also run with the current user's privileges. This is an evidence recorder, not a sandbox for unfamiliar code.

A useful review therefore has two parts: inspect the recorded execution and inspect the relationship between the test and the claim. The file hashes help answer whether the selected bytes still match. They do not establish that the behavior is correct.

Start with one disputed claim

Try this on a small change where the completion summary sounds broader than its tests. Record one concrete claim, select its relevant files, and link a targeted command. Then ask another engineer whether the packet helps them understand what still needs checking.

The interesting measure is whether reviewers catch unsupported or stale claims without spending more effort recording evidence than reviewing it. A tidy report alone would not establish that value.

The MIT-licensed repository includes the runnable fixture, packet schema, and documented limitations.

AI disclosure: This article was drafted with AI assistance from the project's source, documentation, and runnable examples. The examples illustrate the tool's behavior; they are not results from a production deployment.