Two verifiers, one database, opposite answers. 1,022 green tests didn't notice.

Two ways to destroy evidence in an append-only audit chain got past 1,022 green tests. Both were caught by one sentence, not by a test: run both verifiers on the same database, and treat any disagreement as a hole. One of the two holes was created by me fixing a false positive the day before.

This is the story behind traceguard 1.6.0, shipped today. The numbers are from the repo; the reproduction at the bottom runs offline in under a second.

Background: two verifiers

traceguard is a small Python SDK for point-in-time correctness in LLM pipelines. Its opt-in audit layer chains every trace row with sha256(prev_hash || canonical(entry)) and lets you export the chain head as an anchor to somewhere the database owner can't reach. Since 1.6.0 it also exports an evidence bundle, evidence-bundle/v1: the selected traces, their chain entries, the anchors, and a verify-bundle command that checks the JSON offline, without the database.

So there are now two verifiers. verify_chain walks the database. verify_bundle walks the export. They were written days apart, by the same process, against the same threat model. That turned out to be the problem.

Hole 1: I fixed a false positive and made a false negative

The first review round found that verify_bundle compared every anchor against the bundle's own chain.head field. Both values are written by whoever produced the bundle, so the comparison constrained nothing: rewrite the trace content, re-chain the segment with the module's own hash function, leave the head and the anchor alone, and the output read VERIFIED (full) ... 1 anchor(s) match the head over an output_parsed that said "the model approved the trade". Commit ff739ec fixed that: an anchor only counts when it names an entry the bundle actually carries.

Then I hit the opposite case. A periodic anchor taken at seq 3, a chain that honestly grew to seq 6, a bundle exporting seq 4 to 6 for an investigation. The verifier reported anchor_mismatch BREAK and bundle FAILED. A verifier that fails on the normal operating case teaches people to ignore it, so commit af74f23 stopped comparing anchors outside the window at all.

That fix was one branch too wide. An anchor sitting past the head is the one case where an out-of-window anchor is proof of something: the tail was cut. At commit 6b62111:

--- tail truncation (anchor seq 6, chain now ends at 4)
verify_chain (db, from the same anchor): ok=False
verify_bundle (export)                 : ok=True  -> bundle INTERNALLY CONSISTENT (full) ...
    bundle finding: anchor_outside_window WARN

The database says the chain was truncated. The export says "internally consistent, please re-export me". Same rows, same anchor.

Hole 2: the sparse-export rule, applied to a deletion

Bundles selected by --trace-ids carry non-adjacent entries, and linkage can only be checked within an unbroken run. Commit dfda9d6 therefore downgraded the check between non-adjacent entries from link_broken (BREAK) to chain_gap (WARN). Correct for the case it was written for.

Now delete one row from the middle of the chain with raw SQL, the way a tamperer would. The tip seq doesn't move. The head row_hash doesn't change. The tip anchor still binds the head entry. Every seq-based check passes, and the gap is a WARN:

--- mid-chain deletion (anchor seq 6 still matches the head)
verify_chain (db, from the same anchor): ok=False
verify_bundle (export)                 : ok=True  -> bundle INTERNALLY CONSISTENT (full) ...
    bundle finding: chain_gap WARN

Commit b81cb4b closed it with the field nobody had been comparing: entry_count. On an append-only chain it only grows, so an anchor that counted more entries than the export carries means something was removed, wherever the anchor sits.

Both holes existed at once, in a suite that was green at 1,022 tests. The tests for each rule passed because each rule was right for the case it was tuned on. What none of them checked was the two rules against each other.

The rule

It is now appendix B3.6 of the spec, non-normative, one sentence: on the same chain data, the export's conclusion may never be stronger than the database's. Weaker is fine and expected. A bundle carrying entries 8 to 10 can't know that entry 4 was edited, and a bundle whose anchor binds nothing can only ever be "internally consistent". Stronger is a bug: the database fails and the export says verified.

The guard is tests/test_audit_differential.py: 6 mutations (delete a middle row, cut the tail, edit content without re-chaining, edit and re-chain, corrupt a prev_hash, control) × 3 export shapes × 4 anchor choices = 72 combinations, deterministic, every mutation applied through raw SQL because the ORM's append-only guard is anti-footgun, not a threat model. The 5 combinations where the damage lies wholly outside the window are enumerated and frozen in WINDOW_BLIND_SPOTS; a sixth one is a test failure, not a silence. 21 tests. Reverting b81cb4b alone fails 3 of them, reverting c655d53 alone fails 2.

Both review rounds and the implementation were run with Claude Code. The rule came out of the second round's closing note, not from a test; the tests came after, to make sure the rule stays.

What this does not prove

The matrix is six mutations I could think of. It is a regression guard for the tension between two rule sets, not a proof that the rule sets are complete. hash_only bundles still can't recompute entry hashes, so their verdict is linkage only, and the summary says so instead of borrowing the full-mode wording. A bundle is JSON; anyone can edit it. Its value comes from an anchor the bundle's author could not have moved, and traceguard still only structure-checks an RFC 3161 token. Signature verification is openssl ts with a CA you fetched yourself, on purpose.

The reason any of this matters: METR's August investigation of the OpenAI / Hugging Face incident found that roughly 7% of the transcripts it evaluated had been spoofed in places, and at least 20% of the agents expressed clear interest in tampering with their transcripts. A log that can't out-claim its own database is the minimum, not the goal.

Reproduce it

pip install "traceguard==1.6.0"

from datetime import datetime, timezone
from sqlalchemy import text
from traceguard import audit, make_engine
from traceguard.audit.bundle import anchor_record, export_bundle, verify_bundle
from traceguard.audit.verify import export_anchor, verify_chain
from traceguard.sdk.tracer import Tracer

engine = make_engine("sqlite:///:memory:")
audit.enable(engine)
tg = Tracer(engine=engine)
for i in range(6):
    with tg.span("proj", "comp", "llm_complete",
                 feature_as_of=datetime(2026, 1, 1, tzinfo=timezone.utc)) as span:
        span.record_input({"q": f"question {i}"})
        span.record_output(parsed={"answer": f"answer {i}"})

tip = export_anchor(engine)
with engine.begin() as c:                      # delete a row from the middle, as a tamperer would
    c.execute(text("delete from audit_chain_entries where seq = 3"))
    c.execute(text("delete from traces where trace_id = 3"))

print(verify_chain(engine, from_anchor=tip).ok)   # False
bundle = export_bundle(engine, trace_ids=[1, 2, 4, 5, 6], anchors=[anchor_record(tip)])
print(verify_bundle(bundle).summary())            # 1.6.0: bundle FAILED (full): 1 break(s) ...

Swap pip install for a checkout of 6b62111 and the second line prints INTERNALLY CONSISTENT. That commit is 13 commits before the release tag.

Release notes and the spec change are in the repo: https://github.com/lizhuojunx86/traceguard/releases/tag/v1.6.0. If you run two verifiers over the same evidence and have never put them on the same input at the same time, that is the cheapest audit you will do this month. Happy to compare notes if you find a third route.

Li Zhuojun