Production Does Not Care About Beautiful Architecture

The architecture review ended with a clean statement:

The gateway cannot choose a tunnel target because the agent verifies a signed order that binds the target.

In code, that was true.

In the fleet, it was true only on agents running the new binary.

Production does not deploy a trust boundary all at once. It deploys versions, in an order, across machines that can be offline, busy, misconfigured, or rolled back.

This is Part 6 of Auditability Under Pressure.

The real unit of security was the rollout

The signed-tunnel change touched three components:

• the control plane created the order;
• the gateway transported an additive field;
• the agent required and verified it.

Compatibility was asymmetric.

An old agent ignored the new field and kept its old behaviour. A new agent receiving traffic through an old path rejected the request.

That asymmetry gave us a deployment order:

control plane
→ gateway
→ one canary agent
→ a small production wave
→ the remaining fleet

Each wave needed evidence:

• agent version from heartbeat;
• signed tunnel success;
• unsigned and malformed tunnel rejection;
• local persistent event;
• operator access still available;
• rollback procedure still viable.

Merging the agent code did not close the gateway-pivot risk. Completing the rollout did.

Rollback can reverse a security assumption

The control-plane deployment included automatic rollback when health checks failed. That was a reliability feature.

After the protocol change, it could also restore a version that did not send signed tunnel orders while new agents already required them.

Reliability automation and security rollout had become coupled.

This is where a launch checklist earns its place. Google's SRE launch guidance treats reliability reviews and launch checklists as ways to surface operational requirements before a system reaches critical scale. The value is not bureaucracy. It is forcing dependencies, failure modes, monitoring, capacity, and rollback assumptions into one review.

For this rollout, the checklist had to include protocol compatibility and recovery access, not only service health.

Break-glass is part of the architecture

Teams often draw break-glass access outside the main architecture, as if it were an operational footnote.

That is dangerous. An emergency bypass changes the authority boundary more directly than many normal features.

Our local bypass was time-bounded and fail-closed for invalid values. Yet it still carried residual risk:

• while open, the affected agent accepted the unsigned path;
• central visibility was not yet available;
• a bare one-hour flag could receive a new window after restart;
• operators needed an absolute-expiry runbook.

Those were not reasons to hide the bypass. They were reasons to include it in the threat model, evidence contract, events, tests, rollout instructions, and invalidation conditions.

An undocumented emergency path is still part of production. It is simply the least reviewed part.

Version visibility is a control

We added release tags because downgrade comparison required ordered versions. The operational benefit was broader: the control plane could now answer which version was running where.

Without fleet version visibility, a security fix has three ambiguous states:

• merged;
• available;
• active.

Only the third changes the protected system.

I now treat version inventory as a security control whenever enforcement lives in distributed agents. It tells you where a fix exists, where an exception may still be necessary, and where documentation claims overreach.

What we believed

We believed the architecture became secure when all components supported the new protocol.

“Support” was too weak. Some components generated the field, some transported it, some ignored it, and some enforced it.

We believed rollback returned the system to a known-good state.

It returned one component to an older state. In a mixed fleet, that combination was not necessarily known-good.

We believed emergency access was an availability mechanism.

It was also a temporary security policy and needed the same evidence discipline.

What changed our mind

The change came from writing a compatibility matrix instead of another component diagram:

Control plane	Gateway	Agent	Result
new	new	new	signed tunnel enforced
new	new	old	field ignored; legacy risk remains
old	new	new	tunnel rejected; recovery risk
new	old	new	order may not reach agent; tunnel rejected

The matrix made two facts unavoidable:

1. security posture existed per agent, not per repository;
2. rollback safety depended on the fleet's current version distribution.

We paired the matrix with wave evidence and stopped using “deployed” as a binary word.

A production readiness gate for trust-boundary changes

For future changes of this class, my minimum gate is:

Protocol contract locked across languages
Compatibility matrix written
Deployment and rollback order written
Canary identity chosen
Version inventory visible
Positive and negative evidence queries ready
Break-glass expiry and owner defined
Residual risk accepted by name
Invalidation condition recorded

The gate is small enough to use. A forty-page review that nobody updates would recreate the same problem as a stale ADR.

This decision becomes invalid when…

Wave deployment and local break-glass can simplify if the platform gains:

• mandatory protocol negotiation;
• transactional fleet activation;
• guaranteed out-of-band recovery;
• centrally visible emergency state;
• automatic rollback that understands compatibility, not only component health.

Until then, rollout order is part of the security model.

The last lesson was about categories

By the end, we had fixed several things that appeared to be bugs:

• missing Windows logs;
• an updater that never promoted;
• version comparison without versions;
• a verifier that needed .git;
• apparent UTF-8 corruption;
• an empty Event Log message;
• a bypass window changed by restart.

But “bug” was an incomplete diagnosis. Each one was an assumption that held in one environment and failed in another.

Part 7 closes the series with those cases: The production bugs that weren't bugs.

References

• Google SRE — Evolving SRE Engagement Model
• Google SRE — Launch Coordination Checklist
• Microsoft — Manage emergency access accounts

Editorial note: This article is based on a real production engineering record. AI tools assisted with structure and language review; the author verified the technical claims, source links, and final wording.