DEV Community

I built the flight data recorder for AI agents.

Sam LABBE on September 22, 2026

Or: why I spent my evenings making AI-agent journals impossible to rewrite. One Tuesday, at 2:37 PM A disgruntled employee opens the...

Read full post

Sina Rezaei • Oct 8

The fleet-level Merkle root raises an interesting distributed-systems question for me: what exactly is the consistency boundary of a fleet seal?

If 1,200 agents are sealing independently, the Merkle root proves which chain heads were included, but those heads may represent different logical times. Imagine a credit decision pipeline where a risk agent seals at T1, a fraud agent at T2, and an underwriting agent at T3. A single fleet root can prove inclusion of all three heads, but it doesn't necessarily prove that the three agents were evaluated against the same world-state.

I'd probably want a fleet seal to carry something like an epoch or monotonic watermark alongside the root:

fleet_epoch
agent_id
chain_head
logical_clock / source_watermark
root_hash
tsa_token

Then the verifier could distinguish "these agents were included in the same audit batch" from "these agents happened to be anchored together."

This becomes especially important when one agent consumes another agent's output. If fraud-agent@epoch=41 is used by underwriting-agent@epoch=39, the individual chains can both be perfectly intact while the cross-agent dependency is temporally inconsistent.

The Merkle tree solves efficient inclusion proofs really nicely. The next interesting layer, to me, is proving the consistency of the snapshot that the tree represents, not just the integrity of its leaves.

Sam LABBE • Oct 8 • Edited

You've named the next layer exactly — and the honest contract first: the fleet root is Certificate-Transparency-style inclusion, deliberately. It proves these heads existed at seal time and weren't swapped; it never claimed they saw the same world.

Snapshot consistency is a different proof, and it lives one layer below the root. The instrument isn't an epoch field — a self-stamped watermark is a self-report, and an allocator-issued one re-opens the per-call latency tax from the reid thread. It's the consumption edge: when one agent consumes another's output, that consumption is itself a sealable event carrying the producer's chain head as observed — the dependency enters both journals.

Then your fraud / underwriting case becomes checkable without trusting anyone's clock: the reconciliation layer verifies the consumed head is an ancestor of fraud's currently anchored head, and that underwriting's anchor interval sits after the consumed event — temporal position is derived from the external anchors, a relation between two sealed chains, not a field anyone asserts. An inverted dependency surfaces as a cross-chain finding, same species as an orphan outcome.

The root answers "what existed"; cross-chain reconciliation answers "what depended on what" — and a dependency that inverts time is just another hole in the shape.

Sina Rezaei • Oct 8

Yeah, that makes sense. I like deriving the ordering from the dependency edge rather than introducing another piece of state that agents have to assert.

The case that gets interesting for me is fan-in. An agent rarely consumes exactly one upstream result. Imagine an underwriting agent consuming outputs from risk, fraud, KYC, and a pricing agent. The useful invariant isn't really “underwriting happened after fraud,” but that every consumed head has a valid ancestry and anchor relation before the underwriting decision that depended on it.

You could end up with something like:

risk ────────┐
fraud ───────┤
KYC ─────────┼──> underwriting
pricing ─────┘

Now suppose the underwriting journal contains four consumption edges, but one of the consumed heads later turns out to be an orphaned or superseded branch. The underwriting chain itself can still be perfectly intact. What matters is whether the dependency graph can be reconciled back through each producer's sealed history.

That also makes me wonder whether the reconciliation layer is effectively building a partial order rather than a global timeline. You don't need to answer “what happened first everywhere”; you only need to prove that every dependency edge points from an admissible predecessor to a consumer.

That feels like a much more scalable contract for a fleet than trying to manufacture a single notion of time across all the agents.

Sam LABBE • Oct 8

The partial-order framing is exactly the right contract — a fleet that had to agree on a global timeline would be re-deriving consensus just to keep an audit trail, and admissibility per edge is the whole invariant.

Your fan-in orphan case is where it pays off: when a consumed head turns out superseded or orphaned, the finding propagates forward along the dependency graph — every downstream journal that recorded an edge to the dead head inherits the finding, edge by edge, so the blast radius reads as a path set instead of "everything downstream, maybe."

Underwriting's chain staying intact is the point, not a gap: it proves what underwriting believed and when; the producer's supersede event proves the belief went stale; the reconciliation layer correlates the two into a finding with a precise descendant set. Same principle as everywhere else in this design — the graph doesn't manufacture time or truth, it makes every relation checkable after the fact, and every failed check names exactly what it invalidates.

Sam LABBE • Oct 9

@sinarezaei — closing the loop: the exchange where you took the receipt apart became ADR 024 — cross-chain reconciliation, your name in its lineage, and as of today the findings ship as code: github.com/noirebox/noirebox/blob/...

What went in, from your side of the conversation:

Consumption edges are sealable events — the producer's chain head as observed enters BOTH journals; a half missing on either side is the finding (unrecorded_edge — silence is the starting fact).
The contract is a partial order, not a global timeline — edge admissibility, not "what happened first everywhere." And no epoch field anywhere: temporal position is a relation between two sealed chains, derived from anchors. Your move off the epoch is recorded as the rejected alternative, with the reason attached (a self-stamped watermark is a self-report; an allocator-issued one re-opens the per-call latency tax).
temporal_inversion — fraud@41 consumed by underwriting@39 — is a finding, same species as orphan_outcome, across chains. A head merely not yet covered by an anchor is underivable, not stale: the queue is the only open window.
Fan-in: a supersede event kills the head, and the finding walks the recorded edges — the blast radius is a precise path set (fraud#1 → underwriting#1 → underwriting#2 → claims#1), not "everything downstream, maybe."

And your TOCTOU hole from the same conversation is the receipt's state_binding (v0.2.1): the executor applies the sealed version as a precondition, 409 on state move, the divergence sealed as receipt_expired_by_state_change — never silent reuse of a receipt the world invalidated.

noirebox crosschain --journal fraud=data/fraud.db --journal underwriting=data/uw.db reconciles the graph; the report seals even when everything matched. Keep attacking — the next hole is the next field.

Sam LABBE • Oct 8

This convergence is now written: ADR 024 — your fleet-epoch field is recorded as the REJECTED alternative ("a self-stamped watermark is a self-report"), your fan-in invariant is decision §5 (every consumed head needs a valid ancestry and anchor relation before the decision that depended on it), and the implementation issues are open. You asked the question, the ADR carries your name — come break the edges when they land in the reconciler.

Axiru • Sep 23

Sealing the decision and not just the outcome is the part most logging skips. For a refund or payout tool there are two events to seal: the policy decision (allow, hold or deny, with the reason code and the policy version) and the provider response after the money moved.

How do you handle the case where they disagree? An allow that never reached Stripe because the process died, or a charge that went through with no sealed decision in front of it. Does NoireBox flag the missing half, or is reconciliation left to the caller?

Sam LABBE • Oct 7

Shipped 🎉 — your invariant is noirebox reconcile v0.2 on main (issue #3, closed): decision/outcome pairing, deadlines, and the denominator invariant from last week's threads (outcomes ≤ attempts). Credited in ADR #020.

Sam LABBE • Oct 1

@axiru — follow-up, because the useful half of this thread moved to GitHub and email. Issue #3 shipped and is now closed: noirebox reconcile (v0.1, on main) encodes exactly your invariant — every policy_decision needs a matching provider_response, and the reverse — with three refinements straight from your review:

correlation on decision_id, not the payment intent — one intent can produce several attempts, each decision ownable;
decisions carry a deadline (expected_by / expected_within): pending until the window passes, then unconfirmed; no deadline → open_gap;
unauthorized: true in an outcome payload is its own finding — auditors search for the flag, not for silence.
Full status set: matched · pending · unconfirmed · open_gap · orphan_outcome · unauthorized · late. noirebox reconcile --fail-on-findings gates CI on the failure states, and make demo-payout runs your two failure cases end-to-end — the reconciliation report is itself journaled.

The regenerated v0.1 sample is in your inbox. The standing offer survives: your redacted production shapes remain the best stress test for this config — credited either way, here or in the changelog.

Sam LABBE • Sep 23 • Edited

This is exactly the right question — decision vs outcome is where most logging quietly lies.

What NoireBox guarantees today: the integrator seals both events (your policy_decision — allow/hold/deny, reason code, policy version — then the provider_response), and the journal proves the order and the integrity: the decision's hash sits inside the outcome's prev_hash chain, with millisecond timestamps and signatures. After the fact, nobody can claim "the charge ran first" or "that decision was added later" — the sequence is signed and anchored.

What NoireBox deliberately does not do: know what a "missing half" is. The journal is domain-blind by design — an event is a type and a payload, which is why the same box works for payouts, medical AI or legal. So, your two cases:

Process dies after sealing the decision, before Stripe: the journal pins it exactly — decision sealed at T, no outcome recorded. It doesn't fabricate knowledge of what happened next, but the gap itself is provable evidence: "here is what the system knew at the moment it stopped." That's the difference between reconstructing an incident from a mutable log vs from a flight recorder.
A charge with no sealed decision in front of it: if the caller skips journaling, the journal honestly shows an outcome with nothing before it — detectable by sequence order + a correlation id in the payloads (a one-line query over the export) — but NoireBox won't flag it on its own today. Reconciliation is caller-side.

That last point is a fair critique, and it's on the roadmap: a reconciliation plugin — a scheduled check over the journal that encodes your invariants ("every decision must have an outcome within N minutes") and journals its findings as reconciliation events. Same plugin pattern as the guardrail: the journal stays domain-blind, the domain knowledge stays pluggable.

If you're building a payout flow, the two-event pattern with a correlation id works today — and I'd genuinely like the invariant case sketched as an issue. It would shape the reconciliation plugin, and you'd be credited for it.

Sam LABBE • Sep 23 • Edited

Opened an issue to track this → github.com/noirebox/noirebox/issues/3. Thanks for the sharpest question of the thread.

Naveen Alavilli • Sep 27

This solves a problem worth solving well — tamper-evidence after the write is a real gap and hash-chaining plus an external TSA closes it properly. The boundary worth naming for anyone evaluating it: NoireBox proves nobody edited event N after it was sealed. It can't prove event N was true when it was sealed. If the integrator's own instrumentation misrepresents what the agent actually did — logs "approved by policy X" when the check silently no-opped, say — the chain still verifies clean, because the lie shipped at write time, not after. That's not a flaw in the design, tamper-evidence and correctness-at-source are genuinely different problems, but it's worth being explicit that this tool answers "was the record altered," not "was the record accurate." Both matter for the EU AI Act use case, and only one of them is what a hash chain can give you.

Sam LABBE • Oct 8

UPDATE (Oct 8, evening) — you took this apart in the comments, and the comments became the roadmap. Your items are shipped on main, tested, your names in the ADRs:

@axiru — your reconciliation invariant is noirebox reconcile v0.2: decision/outcome pairing, deadlines, attempt-first (outcomes ≤ attempts, unlogged_attempt), and the outbound half: a pending expectation must be anchored or co-signed by a different known writer, or it's unwitnessed_expectation — a wish with a hash.
@naveen_alavilli — your boundary is a named actor in the threat model, and it shaped belief capture (ADR 023): the resolved view sealed BEFORE the action, self-report grade stated.
@sunnydachs @anp2network @housharenet @mayailands — receipt schema v0.2.1: check_id (predicate + inputs + version + spec_hash — your "step I had not taken" is taken), evaluator_sha256, state_binding (the TOCTOU field: the executor applies the sealed version as a precondition, 409 on state move, divergence sealed as receipt_expired_by_state_change), two-key pairing ENFORCED (same writer on both sides = the same_key_pairing finding), single-key receipts labeled AND refused by the gate, per-parameter negative controls on lookups, precedence journaled before the incident, legitimate transformations as first-class lineage.

Numbers, reproducible: 24 ADRs, 244 tests, zero open issues. Changelog + ADRs: github.com/noirebox/noirebox/blob/...

Second comment below for the witness-thread crowd 👇

Sam LABBE • Oct 8

UPDATE (Oct 8, part 2) — the witness-thread crowd, your items shipped too:

@james @glenallen — ADR 021: the shared-dependency test ("can one compromise reach both?") is a runnable bench (independence.py: store, key, endpoints, anchor cadence — empty list = independent), clock and key independence included, per-writer keys with a multi-writer verifier on main. "A name, not a role" is enforced: an unnamed precedence policy refuses to seal.
@arhancanli @david_ilands — the denominator: attempt-first sealing, outcomes ≤ attempts, and the outbound half (david's "wish with a hash" is now unwitnessed_expectation — covered in part 1).
@tom_jones — witness ≠ author enforced at BOTH layers: seal time (the receipt refuses self-witnessing) and serve time (ensure_witness_distinct — skip the slot and escalate). Your same-model-twice failover is now structurally refused.
@reid @mickyarun — the wall-clock anchor timer shipped (the server anchors its own chain head, 60 min default), custody-vs-anchoring mapped in the deployment docs, and the reconciliation cost note: named, in writing.
@james_anderson_h @slabb (the Oct 5-8 threads) — ADR 024, cross-chain reconciliation: consumption edges are sealable events carrying the producer's chain head as observed; the contract is a partial order (edge admissibility, not a global timeline); the blast radius of a superseded head is a precise path set, propagated downstream.

Totals, reproducible: 24 ADRs, 244 tests, zero open issues. The gate ate its own author twice this week (Bandit caught a MEDIUM in my push, CI's ruff caught the rest) — the controls are tested as controls.

pip install noirebox — verification stays free, forever. Keep attacking: the next critic is the next ADR.

Sam LABBE • Oct 6

UPDATE (Oct 6) — three weeks since this post, and most of it came from this thread:

v0.10.0 everywhere — PyPI, GHCR, the VS Code Marketplace and Open VSX (the flight recorder is now also an editor extension: seal, verify, audit-pack from the sidebar)
@axiru's invariant from this very thread shipped — issue #3 is closed: noirebox reconcile v0.2 encodes the decision/outcome pairing, plus the attempt-first convention and unlogged_attempt (the denominator problem this community wasn't done with: outcomes must never exceed sealed attempts)
@naveen_alavilli's boundary became a named actor in the threat model — tamper-evidence ≠ truth-at-write, stated where the auditors read it
Last week's launch threads produced ADRs 019-023 (receipt schema v0.2 with two-key pairing, the normalized witness, flow independence as a testable invariant, custody vs anchoring, belief-before-action) — credited by name to the people who found the gaps
The journal server now anchors its own chain head on a wall-clock timer, and the verifier ships in the wheel

Current state: 219 tests, 23 ADRs, every published number generated by a
command. Changelog: github.com/noirebox/noirebox/blob/...
— install: pip install noirebox

Sam LABBE • Sep 22 • Edited

UPDATE 🐳 — it now runs in one command.

No clone, no Python, no setup:

docker run -p 8768:8768 ghcr.io/noirebox/noirebox:latest

API + OpenAPI docs on :8768/docs, the ML guardrail ships inside the image, and every incident it catches is sealed in the journal before you even open a browser. Pull it anonymously, run it, verify it — that's the whole point of a flight recorder.

Small print, as always: the image is public, the tests are green, the numbers are reproducible. ☕ If the project helps: buymeacoffee.com/samlabbe