The Windows Evidence Gap

The Windows agent was online. It rejected unauthorized tunnel requests. Its binary matched the release we had tested.

And we could not show a durable record of the rejection.

This was not an enforcement failure. It was an evidence failure, and confusing the two would have sent the investigation in the wrong direction.

Part 4 of Auditability Under Pressure follows that gap from the Windows Service Control Manager to Event Log insertion strings and raw UTF-8 bytes.

The control worked only in the present tense

On Linux, the agent's existing log output flowed into journald. Operators could query a rejected tunnel after the fact.

On Windows, the same process ran as a long-lived service under the Service Control Manager. The code called the standard logger. But no rotating log file existed and no application event source had been registered.

The request was rejected in memory. Once the moment passed, the security decision had no persistent sink.

Four architecture decisions depended on evidence that did not exist on that platform:

• rejected tunnel requests should be visible;
• emergency bypass acceptance and expiry should be visible;
• certificate renewal failures should be actionable;
• update promotion and rejection should be auditable.

The mechanism was not broken. The assurance around it was.

The fix used two sinks for two audiences

We added a small event layer with stable identifiers and two local outputs.

Windows Event Log

Warnings and lifecycle events went to Windows Event Log. This was the operator and SIEM surface: lower volume, stable IDs, clear severity.

The event ranges were intentionally boring:

Range	Meaning
1000–1003	lifecycle
2001–2004	tunnel and order decisions
3001–3003	certificate lifecycle
4001–4003	update lifecycle

The values were locked by tests because changing an event ID can silently break an alert.

Rotating local file

Every platform also received a rotating file log: UTC RFC 3339 timestamps, a five-MiB limit, and three retained files.

The file carried detailed diagnosis. Event Log carried the decisions an operator was likely to monitor.

The directory and file were permission-restricted. On Windows, the installer applied an ACL for SYSTEM and Administrators. That correction mattered because a log directory beside certificate and bootstrap material can become a new read path for local users.

The Event Log message that looked empty

Our first production read produced another apparent failure: Get-WinEvent returned the event, but its formatted Message property was empty.

It would have been easy to conclude that the logger had stored no message.

That conclusion was wrong.

Windows event records can store insertion strings separately from the formatted message. Without a custom message resource DLL, one formatter may not produce friendly text. The complete event data was present in the event properties and was visible through the appropriate tooling.

The safe diagnostic sequence became:

$event = Get-WinEvent -ProviderName "agent-provider" -MaxEvents 1
$event.Id
$event.LevelDisplayName
$event.Properties[0].Value

Microsoft's EventLogEntry.Message documentation describes message formatting from replacement strings, while Get-WinEvent exposes the underlying event record. The lesson was not “always use this one property.” It was: inspect both storage and formatting before declaring data loss.

The UTF-8 corruption that was not corruption

The local file then appeared to contain broken Turkish characters when read through a remote command path.

We did not “fix encoding” immediately. We changed the observation path.

The file's raw bytes were encoded as base64, transported without text decoding, decoded locally, and validated as UTF-8. The bytes were correct. The display path had interpreted a no-BOM UTF-8 file using a legacy code page.

Windows PowerShell 5.1 is especially relevant here: Microsoft documents that Get-Content uses the default ANSI encoding for BOM-less input unless encoding is specified. Reading the file with explicit UTF-8 produced the expected text.

Get-Content .\logs\agent.log -Encoding UTF8

Again, the evidence was correct and the observer was wrong.

What we believed

We believed a cross-platform agent had cross-platform observability because the same logging calls compiled everywhere.

Compilation parity is not operational parity.

We also believed a text rendering symptom implied corrupted stored data. That assumption could have led us to alter a correct writer and create a real incompatibility.

What changed our mind

We required proof at three separate boundaries:

1. decision — did the agent reject the request?
2. storage — did a persistent sink contain the event?
3. presentation — did the chosen tool format the stored data correctly?

Four deliberately malformed tunnel requests produced four distinct rejection reasons. The same events appeared in the rotating file and Windows Event Log. Raw bytes proved UTF-8 integrity. Event properties proved the insertion string existed.

Only then did we mark platform evidence parity as restored.

The log path cannot become the decision path

The event emitter was designed never to panic. If Event Log registration failed, the file sink could continue. If both sinks were unavailable, the security decision still had to proceed.

That creates a residual risk: a rejected action may again become unobservable.

The alternative is worse. If a logging failure blocks or crashes enforcement, an attacker can turn the evidence subsystem into a denial-of-service lever.

The correct response is to monitor sink health separately, not make authorization depend on a writable log.

This decision becomes invalid when…

The Windows Event Log layer can be removed if the agent stops running as a Windows service or a central collector provides durable local buffering and equivalent event semantics.

The custom-message-resource compromise should be revisited if operator experience requires fully formatted messages in every Windows API. At that point, maintaining a message DLL may be worth the build and versioning cost.

Until then, documented insertion-string access plus stable event IDs is an honest trade-off.

The next green message

With persistent evidence in place, another production phrase became suspicious:

Update applied successfully.

The updater had downloaded a file, checked its signature and hash, and staged it beside the running binary.

It had not actually promoted the file.

Part 5 is about that gap: “Implemented” is not the same as “verified.”

References

• Microsoft — About Services
• Microsoft — Get-WinEvent
• Microsoft — EventLogEntry.Message
• Microsoft — About Character Encoding
• NIST SP 800-92 — Guide to Computer Security Log Management

Editorial note: This article is based on a real production engineering record. AI tools assisted with structure and language review; the author verified the technical claims, source links, and final wording.