DEV Community

AI Systems Need Evidence, Not Just Observability

NTCTech on June 25, 2026

The gap between ai evidence observability and proof is where every AI compliance failure lives — and most infrastructure teams don't discover it un...
Collapse
 
jugeni profile image
Mike Czerwinski

The "observability is evidence for operators, evidence is proof for everyone else" cut puts a name on what observability stacks structurally cannot do, and Whatsonyourmind posted a technical version of the same primitive earlier today: three-layer decision ledger where the bijection invariant between executed spans and authorized decisions is what makes "nothing executed unauthorized, nothing authorized vanished" actually verifiable rather than asserted. Different framing, same underlying cut.

The Framework #149 four-component decomposition (execution records at authorization boundary, policy snapshots, agent action provenance, artifact portability) is the architectural specification version; Whatsonyourmind's piece is the hash-bound implementation version. Both land on the same load-bearing requirement: the artifact has to survive outside the system that generated it, or it's not evidence.

The practical test for any AI evidence layer claim: would the artifact be readable by an auditor who has no access to the live infrastructure that produced it? Observability stacks fail that test the moment the runtime stops emitting.

Collapse
 
ntctech profile image
NTCTech

That's a useful distinction.

Framework #149 is intentionally written at the architectural layer: what evidence-grade infrastructure must produce if authorization, provenance, and policy state need to survive outside the runtime that generated them.

The implementation question comes next: how do you guarantee those artifacts remain complete, attributable, and resistant to reconstruction drift? That's where ledger models, hash-chained records, signed execution artifacts, and decision-to-execution integrity checks become relevant.

The point I wanted to make in this post is that most teams are still debating observability while skipping the prior architectural decision: whether evidence is even a first-class system output.

If the platform never generates evidence artifacts at execution time, no amount of cryptographic integrity can be added later. The artifact has to exist before it can be protected.

I think both perspectives converge on the same requirement: accountability depends on preserving the relationship between authorization, execution, and evidence in a form that survives after the runtime is gone.

Collapse
 
jugeni profile image
Mike Czerwinski

Agreed on the ordering, and I'd sharpen the corollary: the thing you can't retrofit isn't just the artifact's existence, it's its provenance binding. An evidence record added after execution can prove the bytes haven't changed since you hashed them, but it can't prove which channel produced them, because that information only exists at write time. Hash-chaining a reconstructed artifact gives you tamper-evidence over a fiction.

Which is why "evidence as first-class output" is a write-path decision, not an observability layer. The artifact has to be emitted by the execution that performed the action, carrying the channel it came from, before any integrity wrapper touches it. Reconstruction drift isn't a logging gap, it's the actor's story re-derived from fields the actor could author, which is the same failure whether you're debating observability or signing records after the fact.

So the convergence is tighter than two perspectives meeting. The prior architectural decision you name (is evidence a system output at all) and the integrity decision (can it be forged) are the same constraint read at two times. Generate the non-forgeable artifact at execution, or there's nothing honest to protect.

Thread Thread
 
ntctech profile image
NTCTech

The provenance binding problem is exactly where a lot of teams accidentally substitute telemetry for evidence.

Telemetry can tell you that an event occurred. Even if it's immutable afterward, the system still has to trust that the event was faithfully represented at capture time. Once you're reconstructing from logs, traces, or downstream records, you're already depending on interpretation rather than preservation.

That's why I keep coming back to execution-time artifact generation as the architectural requirement.

The moment an action crosses an authorization boundary, the system has one opportunity to capture the authority chain, policy state, execution context, and provenance channel as a single artifact. After that moment, every reconstruction step introduces assumptions.

A useful test is: could the artifact stand on its own if every dashboard, trace store, SIEM, and runtime component disappeared tomorrow?

If the answer is no, the organization may have excellent observability, but it still doesn't have evidence.

Which gets back to your point: integrity mechanisms aren't creating trust. They're preserving trust that was established at the moment of execution. If provenance wasn't bound at write time, the cryptography is protecting a reconstruction, not the event itself.

Thread Thread
 
jugeni profile image
Mike Czerwinski

Integrity mechanisms presuppose authenticity. That's the part the hash can't give you.

The implication: the hash guarantees bytes haven't changed since capture, but it can't guarantee the captured bytes were the right bytes. A faithfully captured event and a faithfully captured reconstruction are identical to the integrity check — the distinction is established (or not) at the moment the artifact was written.

"Stand on its own if every dashboard and SIEM disappeared tomorrow" is the right operational test. If runtime context is required for interpretation, the interpretation is doing authentication work the artifact should already have done. That's exactly the boundary between preserving trust established at execution and asserting trust after the fact — and both look identical until someone tries to re-derive the claim without the runtime.

Thread Thread
 
ntctech profile image
NTCTech • Edited

I think that's the architectural consequence most teams miss.

Once authenticity is recognized as the prerequisite, evidence stops being a storage problem and becomes a system design problem.

The common assumption is that evidence quality improves as records move downstream: collect logs, aggregate them, sign them, retain them, audit them. But authenticity doesn't accumulate. It either exists at the moment the event is produced or it never exists at all.

That's why I've started thinking about evidence generation as an execution-plane responsibility rather than an observability responsibility.

The execution path is the only place where identity, authorization source, policy state, execution context, and resulting action coexist simultaneously. Every system downstream is already operating on a representation of that event rather than the event itself.

Viewed that way, reconstruction isn't merely less trustworthy. It's operating from a fundamentally different artifact category. One preserves an execution claim. The other derives an execution claim.

Which may be another way of expressing the boundary we're circling around:

Integrity answers whether an artifact changed.

Authenticity answers whether the artifact was ever authoritative.

If that second question can't be answered at write time, every downstream control is preserving uncertainty rather than preserving evidence.