I tried to break my own AI audit trail 75 times. 17 attacks worked.

On September 12, Dario Amodei published an essay urging companies and governments to "pace the frontier" of AI, and Sam Altman said OpenAI would follow. One line in it is going to reach a lot of teams who never read AI safety essays: labs should embed third-party evaluators who report incidents. He pointed to a recent rogue-agent incident as the kind of thing that is coming.

If you ship an AI agent, a version of that question is headed your way, probably from a customer before a regulator: your agent did something. What did it actually do, and why should I believe your logs?

"Trust our logs" is not an answer, because the logs belong to the party with the most reason to edit them.

I build AIR Blackbox, an open-source recorder for AI agent actions. Every action is written into a hash chain, each decision carries its own Ed25519 signature, and the chain is timestamped by an outside authority whose key I don't hold. The output is a single .air-evidence file that anyone can check in a browser at airblackbox.ai/verify. Nothing is uploaded, and you don't need to trust me.

That's the claim. Here is what happened when I spent a week trying to make it lie.

The setup

I ran 75 attacks across six lenses: the ZIP container, the signed manifest, the records and chain, the receipts and counts, the external timestamp, and how findings get attributed. A "break" meant one thing: the verifier says VERIFIED about a file whose evidence has been changed, added to, removed, or misrepresented, by someone who doesn't hold the signing key.

Every claimed break then went to an independent reviewer instructed to refute it, and to default to "refuted" if it couldn't be reproduced exactly.

• 23 attacks were stopped by the existing code.
• 43 were claimed as breaks.
• 17 survived refutation, reducing to 8 distinct root causes.

All eight are addressed, one of them only partly, and the full report is public, including the one I couldn't fully fix. These are the ones worth your time.

1. One file, two stories

ZIP allows two entries with the same name. Python's zipfile reads the last one. unzip, streaming readers, and most other tools read the first.

So I put a forged record first and the genuine one second. My verifier read the genuine copy, checked every digest, and passed it. Anyone opening the same file with ordinary tools saw the forgery:

verifier:   [1/6]..[6/6] OK - VERIFIED, 0 alterations
unzip -p:   "decision_type": "advance", "human_reviewer": "compliance@corp"

One file, certified clean, showing a human review that never happened to whoever opened it. The same trick worked on the signed manifest itself.

Fix: duplicate names are rejected before any digest is computed. No legitimate bundle ever has them.

2. The number that mattered most was the one I never checked

The manifest carries summary counts: actions, blocked actions, adverse decisions, and so on. The verifier recomputed five of the eight from the raw records.

The three it skipped were adverse_decisions, adverse_decisions_missing_reviewer, and engine_outputs_without_reviewer, which are exactly the ones that exist to surface missing human review. The manifest even tells auditors that adverse_decisions_missing_reviewer is the number to read first.

So a bundle could declare zero unreviewed rejections over records that plainly showed one, sign it properly, and verify clean. The line from the report I keep coming back to:

Signing a false summary does not make it true; it makes it an attributable lie, which is only useful if something catches it.

Fix: all eight counts are recomputed, and a manifest that omits a count fails rather than skipping the comparison.

3. "All signatures valid," over nothing

Records without a receipt were skipped. Some record types legitimately carry none, so a bundle could have zero receipts, and the verifier printed:

all signatures valid

It was true in the way "all my unicorns are healthy" is true. Nothing had been checked.

Fix: it now says 0/N records carry a receipt - NOTHING was checked here, and the final verdict adds that authorship is unevidenced.

4. The hole my own fix created

To stop people slipping unsigned files into a bundle, I added a guard that rejects any member the signed manifest doesn't list, with an exemption for directory entries, meaning names ending in /.

ZIP doesn't enforce that a name ending in / is empty. attachments/reviewer_signoff.pdf/ can carry a complete file. My exemption was a door straight back through the guard it belonged to, and it arrived in the same commit as the guard.

Fix: the exemption also requires the entry to be zero bytes.

5. A public-log anchor I could forge offline

Besides the timestamp authority, bundles can be anchored in a public transparency log. The verifier checked the log receipt's signature against a public key the receipt itself carried. Offline, that proves the receipt agrees with itself and nothing else. I could generate a key, sign a receipt for any chain head, attach a plausible entry number, and get:

Public-log anchor: anchor consistent ... The entry is permanent

Fix: --rekor-verify fetches the claimed entry and confirms it exists and matches. Without that flag the verifier now says plainly that it's self-consistency only.

The one I couldn't fully fix

Any key verifies. The bundle carries its own public key. If I generate a new keypair, assemble records of my choosing and sign them, the result verifies cleanly. VERIFIED means this file is intact and hasn't changed since it was signed. It does not mean this file came from who you think it came from.

A self-describing file can't authenticate its own author. What I could do:

• --expect-key pins the signer you expect, using a fingerprint you got some other way.
• With nothing pinned, the verdict says so out loud: VERIFIED (UNATTRIBUTED, UNWITNESSED).
• --strict fails anything that isn't pinned and externally anchored.

--strict isn't the default, and the reason is worth sharing. If the default rejected most real-world files, people would learn to reach for the flag that turns checking off, which is the worst possible outcome. The default reports honestly; --strict is what belongs in a CI gate or a procurement checklist.

It kept happening after the report

I'd like to say that was the end of it. Two more turned up this month.
The browser said VERIFIED when it couldn't check anything. On browsers without Ed25519 support, the verify page correctly printed "Signature could not be checked in this browser," and directly above that, the headline read "Authentic… the contents are genuine and unaltered." It's fixed: those browsers now get "could not verify in this browser," which is neither a pass nor a fail.

A comment claimed protection that didn't exist. The policy file has a rule that forbids inferring protected attributes (race, health, religion and the like). A change that permitted a new machine-tagging action carried a comment saying that rule still applied to it. It didn't. Rules match on the action's name; the rule never looks at a tag's content. A parser tagging someone "likely 55+" under the new name was permitted. The review that caught it also found that the test written to prove the rule "still bound" passed identically before and after the change, so it proved nothing. Now machine tagging is denied unless the producer explicitly asserts a controlled vocabulary, and that assertion is itself recorded.

The pattern

Look at the list again. Almost none of these are "the cryptography was weak." Ed25519 was never the problem. Nearly every break has the same shape:

a reassuring sentence sitting on top of a check that didn't support it.

"VERIFIED" over a file other tools read differently. "0 unreviewed rejections" that nobody recomputed. "All signatures valid" over zero signatures. "Authentic" over checks that couldn't run. A comment promising a protection the engine couldn't enforce.

If you're building anything that's supposed to let someone else hold an AI system to account, that's where I'd look first. Not at your algorithms: at every place your tool tells a human that something is fine. For each one, ask what exactly was checked to earn that sentence.

Try to break it yourself

As of the latest release, it's one pip install. No server, no account:

pip install air-blackbox

import air_blackbox as air

with air.record("support-agent") as rec:
    rec.action("read_ticket", "ticket #4821: customer charged twice")
    rec.action("issue_refund", "order 991, $40",
               human_reviewer="[e-mail]", decision_type="refund")

print(rec.bundle_path)

Drop the file on airblackbox.ai/verify and watch your browser check every signature. Then change one byte and drop it again.

If you find a sixth way to make it lie, I want to know: [e-mail], subject [SECURITY]. It'll go in the report with your name on it, unless you'd rather it didn't.

The code is Apache 2.0: github.com/airblackbox/airblackbox