The Evidence Layer Must Survive the Framework

The first design error was easy to describe after we found it: we had protected command authorization but missed tunnel authorization.

The second error was less visible. We had treated evidence as if it naturally followed enforcement.

It does not.

A control can make the right decision and still leave no durable answer to the questions an incident responder will ask later: What was requested? Which policy version decided it? What did the agent reject? Was the emergency path open? Did the evidence survive restart?

This is Part 2 of Auditability Under Pressure, a real production series about the gap between architectural intent and provable behaviour.

One system, four different responsibilities

I now model this class of system as four layers:

Layer	Question
Policy	Should this action be allowed?
Transport	How does the request reach the executor?
Enforcement	What does the executor verify before acting?
Evidence	What durable record proves the decision and outcome?

These layers can cooperate without sharing a runtime.

In our case, the control plane was TypeScript, the agent and gateway were Go, Linux used journald, and Windows used the Service Control Manager. An application-framework logger could cover only one slice of that path. It could not prove what the agent saw after the gateway transported an order, and it could not guarantee that a Windows service preserved standard output.

The evidence layer therefore needed two properties:

1. semantic stability — the same security event meant the same thing on every platform;
2. sink independence — evidence did not disappear because one framework, console, or transport was unavailable.

Evidence is not “more logs”

Turning up log volume would not have solved this problem.

Evidence needs an event contract. For the agent, we assigned stable event identifiers to lifecycle, tunnel, certificate, and update decisions. A tunnel rejection had a different identifier from an expired emergency window. An update staged event was distinct from an update applied event.

The records deliberately excluded raw commands, tokens, private keys, and passwords. Auditability cannot come at the price of building a second secret store inside log files.

The OpenTelemetry Logs Data Model makes a useful distinction between a log record's timestamp, observed timestamp, severity, body, attributes, trace context, and resource context. We did not need to adopt every OpenTelemetry component to benefit from the model. The important part was treating evidence as structured data with explicit semantics, not incidental prose.

Two evidence planes

The architecture ended up with two complementary evidence planes.

Control-plane audit

The control plane recorded decisions and results in a hash-linked audit history. Each row included the previous hash in the next row's calculation. A stored head helped detect truncation, and an optional external HMAC anchor allowed the head to be checked outside the database.

This made alteration detectable. It did not make the database immutable.

That distinction is why I avoid borrowing stronger terms from systems we did not build. RFC 9162 describes Certificate Transparency's append-only Merkle-tree log. It is a valuable reference for verifiable log design, but a linear SHA-256 chain with an HMAC anchor is not a Certificate Transparency implementation.

Agent-local security events

The agent wrote local security and lifecycle evidence. Linux retained the existing journal path and gained a rotating file. Windows gained both a rotating file and Event Log events for warnings and lifecycle changes.

The local file used bounded retention: three files of five MiB each. The design traded unlimited history for predictable disk use. Central collection remained a separate, explicitly open problem.

The rule inside the emitter was equally important: logging failure must not block the security decision. A broken Event Log provider could not turn a rejected request into an accepted one or crash the agent.

What we believed

We believed that if enforcement code logged a rejection, the rejection was observable.

That belief smuggled in several unstated assumptions:

• standard output was connected to a persistent sink;
• the service manager preserved it;
• encoding survived the collection path;
• operators knew which field contained the message;
• retention lasted long enough for an investigation.

Those assumptions held on one platform and failed on another.

What changed our mind

The decisive test was not a unit test. It was a simple inventory on a production Windows agent:

• no agent log file existed;
• no registered Event Log provider existed;
• the service was running;
• rejected requests were still rejected.

Enforcement and evidence had separated in reality even though our architecture described them together.

We added the persistent sinks, then tested the complete observation path. Distinct malformed requests produced distinct event reasons. Raw file bytes were transferred independently and decoded as valid UTF-8. Event Log insertion data was read even when one formatter exposed an empty Message property.

That last step matters. An observation tool can be wrong about the evidence. “The message is empty” and “the message data is absent” are different claims.

A practical evidence contract

For each sensitive action, I now want a compact contract:

event_id       stable semantic identifier
occurred_at    UTC time at decision source
observed_at    time the sink received it
actor          human, service, or system identity
subject        agent, tenant, or protected resource
action         requested operation
decision       accepted, rejected, expired, or failed
reason_code    machine-filterable cause
correlation    order, approval, or incident identifier
policy_version rules that produced the decision

Not every layer needs every field. But if the correlation and policy version are missing everywhere, an incident review becomes narrative reconstruction instead of verification.

This decision becomes invalid when…

Local evidence sinks can become secondary if a central collector provides all of the following:

• durable delivery during control-plane outages;
• bounded local buffering during network loss;
• stable event identifiers across platforms;
• independent retention and access control;
• a tested way to recover evidence when the collector itself is unavailable.

Even then, I would keep a small local last-resort record. The failure that removes network access is often the same failure you are trying to investigate.

The next problem: architecture records that age

Once evidence became a separate layer, our decision documents became the next weak point.

An ADR could say “rejected requests are visible” long after a platform change made that false. A test name could be renamed. A source path could move. A production limitation could remain hidden behind the word “implemented.”

Static decision documents were no longer enough. We needed ADRs that carried their own verification state and stated when the decision should be reconsidered.

That is Part 3: Living ADRs — decisions that can prove themselves wrong.

References

• OpenTelemetry Logs Data Model
• NIST SP 800-53 Rev. 5 — AU-9, Protection of Audit Information
• RFC 9162 — Certificate Transparency Version 2.0
• systemd — journald.conf

Editorial note: This article is based on a real production engineering record. AI tools assisted with structure and language review; the author verified the technical claims, source links, and final wording.