Mustafa ERBAY

Posted on Sep 10 Originally published at mustafaerbay.com.tr

The Production Bugs That Weren’t Bugs

#production #debugging #architecture #devops

We fixed seven production problems in one week.

Calling all of them “bugs” would have made the fixes smaller than the lessons.

A bug suggests a local defect: a wrong condition, a missing function, an invalid state transition. Some of these cases contained exactly that. But the recurring cause was broader. A design assumption crossed a platform, process lifecycle, deployment boundary, or observation tool and stopped being true.

This is the final part of Auditability Under Pressure.

1. The Windows log gap

Symptom: Security decisions appeared in Linux journals but had no durable Windows record.

Initial diagnosis: Windows logging was broken.

Assumption failure: The same logging call implied the same persistence semantics across service managers.

The code compiled and enforcement worked. Windows Service Control Manager did not automatically turn the process's console output into the persistent evidence path our ADRs promised.

Correction: Stable event IDs, Windows Event Log for operator-facing events, rotating local files for detailed diagnosis, and explicit sink verification.

Boundary crossed: platform.

2. The non-version version

Symptom: Downgrade protection rejected updates without a useful explanation.

Initial diagnosis: The SemVer comparator had a bug.

Assumption failure: Every build identifier was an ordered version.

Tagless builds reported commit hashes. Hashes identify content; they do not define release precedence.

Correction: Require comparable versions, reject non-SemVer values with an actionable message, tag releases, and verify ordering again at promotion.

Boundary crossed: identity versus ordering.

3. The deployment without `.git`

Symptom: An ADR evidence verifier said every cited test was missing in production.

Initial diagnosis: The documentation was completely stale.

Assumption failure: The runtime source tree was a Git working tree.

The deployment deliberately excluded .git. git grep could not inspect the tree the way it did in development.

Correction: Traverse the filesystem directly. Return a different status for “cannot inspect environment” and “citation is stale.”

Boundary crossed: development versus deployment.

4. The verifier's first bug

Symptom: A tool designed to detect misleading evidence produced misleading evidence.

Initial diagnosis: One shell command needed replacement.

Assumption failure: Verification tools were outside the threat model of assumptions.

The verifier was part of the evidence chain. Its own dependencies, failure modes, and error taxonomy needed the same scrutiny as the controls it checked.

Correction: Test the verifier in repository and deployed-tree shapes; make uncertainty explicit.

Boundary crossed: observer versus observed system.

5. The UTF-8 corruption

Symptom: Turkish characters looked corrupted in a remote log read.

Initial diagnosis: The agent wrote invalid UTF-8.

Assumption failure: Rendered terminal output faithfully represented stored bytes.

Raw bytes moved through base64 and decoded locally were valid UTF-8. The legacy reader selected an ANSI code page for a BOM-less file.

Correction: Preserve raw bytes during diagnosis and specify encoding at the reading boundary.

Boundary crossed: storage versus presentation.

6. The empty Event Log message

Symptom: Get-WinEvent returned an event with an empty formatted Message.

Initial diagnosis: Event Log had lost the text.

Assumption failure: One API's formatted field was the event's complete stored representation.

The insertion string existed in the event properties. Without a custom message resource, the formatter could not produce the friendly Message value we expected.

Correction: Inspect event properties, document the formatter limitation, and decide explicitly whether a custom message DLL is worth its lifecycle cost.

Boundary crossed: stored event versus formatter.

7. The one-hour bypass

Symptom: An emergency flag described as a one-hour window could open another hour after service restart.

Initial diagnosis: The expiry timer reset incorrectly.

Assumption failure: Process lifetime and incident lifetime were equivalent.

The code correctly calculated one hour from process start. Restarting created a new process and therefore a new window.

Correction: Prefer an absolute RFC 3339 or Unix expiry; retain the process-relative form only as a documented last resort.

Boundary crossed: process lifecycle versus operational intent.

What we believed

Across all seven cases, we believed the local statement was the system statement:

“we log this” meant the platform persisted it;
“this is a version” meant it could be ordered;
“the source exists” meant Git metadata existed;
“the tool reports missing evidence” meant evidence was missing;
“the text looks corrupt” meant bytes were corrupt;
“Message is empty” meant event data was absent;
“one hour” meant one incident hour.

The code was often behaving exactly as written. The surrounding interpretation was wrong.

What changed our mind

We stopped debugging nouns and started debugging boundaries.

Instead of asking “is logging broken?” we asked:

decision → emitter → sink → storage → transport → decoder → formatter → operator

Instead of asking “did the updater succeed?”:

offered → downloaded → verified → staged → promoted → started → healthy

Instead of asking “is the architecture deployed?”:

merged → built → available → installed → active → observed

Each arrow became a claim that needed evidence.

The assumption ledger

The method I will keep from this incident is an assumption ledger beside the claim ledger.

For every high-impact decision:

Field	Example
Assumption	service output reaches a persistent sink
Boundary	Windows SCM
Evidence	named production query
Failure signal	no provider and no local file
Owner	agent platform owner
Invalid when	hosting model or sink changes

An assumption is not bad. Architecture is impossible without assumptions. The dangerous assumption is the one presented as an unconditional fact.

This decision becomes invalid when…

This investigation method can become lighter when contracts are machine-checkable end to end:

protocol negotiation proves compatibility;
release activation is transactional;
evidence schemas and sinks are continuously verified;
deployment artefacts declare their runtime shape;
observation tools preserve raw data and provenance;
emergency state has absolute expiry and central visibility.

Even then, new boundaries will appear. The ledger should shrink where automation proves a condition and expand where human inference remains.

What we proved our architecture was wrong about

The system became stronger during this work, but not because every weakness disappeared.

It became stronger because the records stopped pretending:

a tamper-evident log was no longer called immutable;
test coverage was no longer called production proof;
one upgraded agent was no longer called a protected fleet;
a staged file was no longer called an applied update;
an empty formatted field was no longer called missing data;
a necessary bypass was no longer kept outside the architecture.

The most valuable result was not a framework, an RFC, or an event sink.

It was a sentence we could attach to every important decision:

This decision becomes invalid when…

That sentence leaves room for production to be more truthful than the diagram.

References

Editorial note: This article is based on a real production engineering record. AI tools assisted with structure and language review; the author verified the technical claims, source links, and final wording.