The updater downloaded the correct artefact.
It verified the Ed25519 signature. It verified the SHA-256 digest. It wrote the new executable beside the running one. The log said the update had been applied.
The agent restarted on the old binary.
Nothing in the cryptography was wrong. The implementation had confused staged with applied.
This is Part 5 of Auditability Under Pressure.
A green log with an old process
Replacing a running executable is platform-sensitive. The updater sensibly wrote the candidate to a .new file rather than overwriting the active binary.
That was a valid staging design. But there was no startup path that promoted .new into the active executable.
The sequence was:
download
→ verify signature
→ verify digest
→ write agent.new
→ report applied
→ restart
→ run old agent
The corrected sequence became:
download
→ verify candidate
→ write agent.new
→ report staged
→ restart
→ re-verify agent.new
→ atomically promote
→ run new agent
→ report applied
The second verification was not redundant. A staged file exists across a time boundary and can be modified after download. Promotion is the moment at which trust transfers to the executable that will run.
State names are part of the security model
“Downloaded,” “verified,” “staged,” “promoted,” “started,” and “healthy” are different states.
Collapsing them into “success” removes the evidence needed to answer where an update stopped.
We changed the event contract so Applied: false remained true until actual promotion. The agent heartbeat exposed the running version, allowing an operator to distinguish a staged candidate from the active process.
The operational trade-off remained: an agent that never restarts can keep a valid staged update waiting. That is not a hidden failure if the state and running version are visible.
The downgrade protection that compared hashes
The updater also had a downgrade guard. It parsed semantic versions and refused to install an older release.
Except our builds did not always have semantic versions.
The build pipeline used a Git-derived value. Without a release tag, that value could be a commit hash such as 9a5fd99. Two hashes have identity but no release ordering. Treating one as “greater” than another invents semantics the values do not contain.
The old guard silently returned false for incomparable values. It looked like downgrade protection while making updates mysteriously fail.
The correction was fail-closed and explicit:
- local and remote versions must both satisfy the SemVer contract;
- incomparable versions reject the update with an actionable reason;
- production releases receive SemVer tags;
- the comparison is repeated at promotion.
The SemVer specification defines numeric MAJOR.MINOR.PATCH ordering and the meaning of prerelease identifiers. A commit hash is useful provenance, but it is not a version-ordering scheme.
What we believed
We believed successful signature and digest verification meant the update mechanism worked.
It meant the candidate was authentic and intact at one point in the process.
We believed a value printed by --version could participate in downgrade comparison.
It could identify a build. It could not necessarily order that build.
We believed a green status was evidence.
It was only evidence that the code reached the line that emitted it.
What changed our mind
The proof procedure stopped reading logs and inspected state:
- What file did the updater write?
- What hash did the staged file have?
- Which executable path would the service start?
- Which version did the running process report after restart?
- Which event marked staging, promotion, and rejection?
That procedure exposed the missing promotion path immediately.
We then separated the verification status honestly:
- SemVer release and running version: observed in production;
- staging and promotion logic: covered by tests;
- first real self-update promotion: not yet production-triggered at that point.
The implementation existed. Full production verification did not.
Verification should follow the state machine
For an updater, a useful evidence matrix looks like this:
| Transition | Evidence |
|---|---|
| offered → downloaded | response metadata and candidate path |
| downloaded → verified | signature key ID and digest result |
| verified → staged | durable candidate hash and staged event |
| staged → promoted | pre-promotion re-verification and atomic rename result |
| promoted → started | process-reported version |
| started → healthy | application-specific health signal |
If the system cannot distinguish these transitions, rollback and incident diagnosis become guesswork.
This pattern applies to more than agents. Database migrations, firmware, container images, feature flags, and policy bundles all have a moment where “artefact exists” becomes “artefact governs production.”
That moment deserves its own evidence.
This decision becomes invalid when…
Startup promotion can disappear if the platform provides a trusted package manager or an atomic deployment primitive that owns staging, activation, rollback, and version reporting.
The SemVer requirement can change if releases adopt another ordering contract with explicit comparison semantics. A signed monotonically increasing build number could work. A commit hash alone still cannot.
The underlying rule survives both changes: reject updates when the system cannot prove ordering and activation state.
The architecture was still beautiful
At this point, we had the right state machine, explicit evidence, and comparable releases.
But the production fleet still contained different agent versions. Deployment order still mattered. Rollback could still strand new agents behind an old control plane. The emergency path still carried residual risk.
The architecture was correct on paper and conditional in the fleet.
Part 6 is about that difference: Production does not care about beautiful architecture.
References
- Semantic Versioning 2.0.0
- RFC 8032 — Edwards-Curve Digital Signature Algorithm (EdDSA)
- NIST SP 800-218 — Secure Software Development Framework
Editorial note: This article is based on a real production engineering record. AI tools assisted with structure and language review; the author verified the technical claims, source links, and final wording.
Top comments (0)