DEV Community

Cover image for The Bug That Proved My Own Point: A Post-Mortem on TSA Verification
Vesmot
Vesmot

Posted on

The Bug That Proved My Own Point: A Post-Mortem on TSA Verification

#git #rust #security #programming

In my previous post, I argued that Git history is not enough as proof because timestamps and commit data can easily be rewritten. The core thesis was simple: you cannot rely on a verification model controlled by a single interested party; you need an independent, external source of truth, such as an RFC 3161 Time Stamping Authority (TSA).

Then, I broke our own verification pipeline.

This is a technical post-mortem on how a subtle bug in our verification path nearly turned a valid cryptographic proof into a false positive "tampering" alert — and why it forced us to rethink what a trusted verification system actually means.

1. What Broke: The OpenSSL Digest Trap

During the stabilization phase of Evident Ledger v1.2.0-rc1, we audited our end-to-end evidence workflow. The write path worked flawlessly:

  1. Compute the Merkle root digest.
  2. Send the digest to an external TSA provider (freetsa.org).
  3. Store the resulting cryptographic timestamp token.

The trouble started on the read path during independent verification.

Our ledger architecture is content-free: we never store or pass raw file contents during verification, only the pre-calculated 32-byte binary digest (the Merkle root). That digest was being passed to OpenSSL like this:

openssl ts -verify -data merkle_root_digest -tsd token.tsr
Enter fullscreen mode Exit fullscreen mode

The problem: OpenSSL's -data flag expects the original raw file content, and it automatically hashes that content internally before comparing it to the timestamp token's message imprint. We were feeding it an already-hashed 32-byte digest — so OpenSSL hashed it a second time.

expected_imprint = digest
actual_imprint   = hash(digest)
Enter fullscreen mode Exit fullscreen mode

The result was a predictable, and misleading, mismatch:

Verification failed: message imprint mismatch
Expected: <digest>
Actual:   <hash of digest>
Enter fullscreen mode Exit fullscreen mode

To an automated verifier or an auditor, this technical mismatch looked identical to a severe security violation: evidence tampering. The system was flagging completely valid, unmanipulated historical records as broken — simply because the verification tool was wrapping the digest a second time.

The Fix

The correction required switching OpenSSL from data mode to direct digest mode, using the correct flag:

openssl ts -verify -digest <merkle_root_hex> -tsd token.tsr
Enter fullscreen mode Exit fullscreen mode

2. Why This Is Not Just a "Minor Bug"

In most software, a bug is an annoyance. In a cryptographic evidence system, a bug in the verification path is a different category of problem entirely.

There is a profound difference between not having a proof and producing a false tampering alert:

  • No proof means you cannot substantiate a claim.
  • A false tampering alert means a mathematically sound, honest record is falsely accused of being altered.

If an independent expert or technical consultant runs a verification command during a dispute and sees message imprint mismatch, the reputational and legal damage happens instantly — before anyone has time to check whether the tool, not the evidence, was at fault. A verification tool's failure mode must never mimic the failure mode of the data itself.

3. What Changed in the Release Process

Finding this issue right before tagging v1.2.0-rc1 prompted a deeper shift in how we handle releases. A "provable ledger" needs more than correct math and clean Rust code — it needs a strict operational boundary around that math.

  1. Fail-closed startup security. In production (APP_ENV=production), the application now enforces strict startup guards. If signing keys or TSA CA trust paths are missing, it refuses to boot (exit code != 0) instead of degrading silently.
  2. Warning classification. We audited 201 unique compiler and runtime warnings and classified every one of them, with zero hard blockers left ambiguous in the release binaries.
  3. Evidence-driven documentation. Every security guard and fix is now tracked alongside the codebase in structured audit logs (docs/audits/), instead of living as undocumented tribal knowledge.

4. Conclusion: The Chain of Trust Is Only as Strong as Its Verification

My first article argued that you cannot trust Git history, because the tools that produce it can be manipulated. This bug taught me the corollary: you cannot trust an independent timestamp if your own verification tool interprets it incorrectly.

A trust infrastructure is not a collection of isolated guarantees — Git, GPG, TSA. It is an unbroken chain where every link, from Merkle root generation down to the exact CLI verification flag, has to be deterministic and transparent.

v1.2.0-rc1 is our first release where the code, the operational guardrails, and the audit trail are locked together as a single verifiable system.

If you're interested in the architecture or the audit documentation, the Evident Ledger repository is open.

Top comments (0)