An Audit Trail You Don't Have to Take on Trust

"The AI wrote it" isn't an answer. A signed record of how each change shipped is, as long as you can check it without trusting the vendor.

This is the last post in a short series. The first, Nothing Should Merge on an AI's Word, argued for a gate between "the agent finished" and "this is a pull request." The second and third covered wiring that gate into CI and keeping approvals honest. This one covers what's left after a change ships: the record.

Six months later

A bug reaches production. Someone runs git blame and finds the commit. It came from an AI session in the spring.

Now the questions start. What was the agent asked to do? Which model wrote it? Did the review flag anything, and if so, who decided it was fine? Did the tests actually run on this version? Who approved it?

For human-written code you'd ask the author. For AI-written code there's no author to ask, and the chat history is probably gone. If the answers aren't written down somewhere durable, they're lost.

What a useful record contains

A good record answers those questions without needing anyone's memory. In AI CodeOps, every change gets an evidence pack that covers:

• The task: the prompt, the model and provider, and the context files the agent was given.
• The runs: each session and follow-up run, with status, timings, tokens, cost, and which provider and model handled each agent step.
• The result: the summary, the changed files, and a SHA-256 hash of the diff.
• Approvals: who was eligible to approve, and each vote with its time and note.
• Policy decisions: every allow, deny or ask made during the change.
• Findings and tests: review findings and how each was triaged, plus test results and coverage.
• Delivery: the pull request, the deployment, and any linked incident.
• The audit trail: the related audit log entries, and the email address behind each account ID.

Notice what's missing: the code itself. The pack carries a hash of the diff, not the diff. You match the hash against the pull request, which keeps source code out of an artifact you might hand to an auditor.

A record you can edit isn't evidence

A log stored in a database the vendor controls proves very little. Anyone with write access, including the vendor, could change it after the fact. For a record to count as evidence, you need to be able to tell whether it was altered.

That's what the signature is for. Each pack is signed with Ed25519 over a canonical form of its contents: the JSON payload with object keys sorted at every level and no whitespace. Canonical form matters because the same data can be serialised many ways. Without it, re-saving a file in an editor could break a valid signature.

If anyone changes a single field, whether it's a vote, a finding's triage status or a model name, the signature no longer matches.

Don't trust the verifier either

A signature is only useful if the check doesn't depend on the party that signed it. "Our server says the pack is valid" proves nothing if the server is the thing in question.

So there are two ways to verify a pack, and the second needs no trust in the platform at all.

In the browser. Open aicodeops.dev/evidence/verify and pick the file. No account is needed. It tells you whether the signature matches and which key signed it.

Offline, on your own machine. First fetch the public keys:

curl -o keys.json "https://console.aicodeops.dev/api/evidence?action=keys"

Then run a short Node.js script. It uses only Node's built-in crypto module and needs Node 20 or later. The script:

1. Reads the pack and keys.json.
2. Finds the public key whose kid matches the pack's signature.kid.
3. Rebuilds the canonical JSON of the pack's payload.
4. Checks the Ed25519 signature against it.
5. Prints VERIFIED, or NOT VERIFIED: the pack was altered.

The full script is in the evidence pack docs, under "Verify a pack offline." It's short enough to read in a minute, which is the point: you should be able to see exactly what it checks.

node verify-evidence.mjs evidence-chg_....json keys.json

Keep the keys with the records

One step people skip: save a copy of keys.json next to your audit records. If you only ever fetch keys live, verification depends on the platform still being online, and still serving the same keys, years from now.

Keys are never deleted. A retired key stops signing new packs, but every pack it already signed keeps verifying. Still, an auditor shouldn't have to take that on faith. A local copy of the keys file turns "the vendor says so" into "the maths says so."

Who can download a pack

Evidence packs include people's email addresses and the full history of a change, so access is limited. Only owners, admins and security admins can download them, from Changes, then the change, then Download evidence pack. Each download is itself recorded in the audit log.

The checklist

1. Record the decisions, not just the code. Task, models, findings, triage, tests, approvals, policy decisions.
2. Sign the record. An unsigned log is a claim, not evidence.
3. Use canonical form. So the same data always produces the same signature.
4. Verify without the vendor. Public keys plus a script you can read.
5. Keep the keys. Store keys.json with your audit records.

Across the series, the idea has been the same: AI agents can do the work, but trust comes from evidence. Tests that ran on this version, a review that read the real diff, a second person who said yes, and a record of all of it that anyone can check.

---

AI CodeOps is free during the open beta and free forever for non-commercial open-source projects. Learn how evidence packs work in the docs.

Drafted with AI assistance and edited by the AI CodeOps team.