I have been calling ShrekOS an immutable operating system for a while now.
That sounds nice.
It also does not mean very much if the machine cannot safely replace itself.
So this week I stopped talking about the update design and made the thing actually update.
Not "downloaded some metadata."
Not "the signatures verified."
Not "systemd-sysupdate saw a newer version."
I mean:
ShrekOS discovered a new release over the public internet, verified it, wrote it into the inactive system slot, rebooted, passed Secure Boot and dm-verity, reached the login screen, and kept the previous system available for rollback.
One caveat up front, because this whole post is about not overclaiming. I proved this in a virtual machine, not on bare metal yet. The network leg is real. It pulls the signed release from the live public front over the actual internet, no mocks. But the reboot ran under virtual firmware, OVMF Secure Boot, not a physical machine's firmware and TPM. A VM lies about exactly the layer this depends on. So I am calling it proven in a VM. Bare metal is the next test, not a finished one.
And naturally, the first time I tried the complete path, I bricked the new slot.
Which turned out to be the most useful part.
The basic shape
ShrekOS is immutable.
The running base system is not something I want users, packages, agents, or random scripts mutating until it slowly becomes a unique archaeological artifact.
Instead, the machine has two system slots.
Conceptually:
running system
│
▼
┌─────────┐
│ Slot A │ ← active
└─────────┘
┌─────────┐
│ Slot B │ ← inactive
└─────────┘
▲
│
update goes here
You do not patch the system you are standing on.
You write the next system somewhere else.
Then you boot it.
If it works, great.
If it does not, the old one is still there.
That is the theory, anyway.
This week the theory became a booting machine.
I wanted the update server to be boring
One thing I did not want was to accidentally make the update infrastructure depend on the rest of my personal infrastructure.
ShrekOS already has enough weird machinery.
The update front should not be clever.
It should not depend on some development tunnel being alive.
It should not require an agent.
It should not be coupled to whichever backend I happen to be hacking on that week.
So the update endpoint is deliberately boring:
ShrekOS
│
▼
shrekos-updates.iambu.dev
│
▼
Cloudflare Worker
│
▼
public release artifacts
The Worker is independently operable and fronts the public release artifacts.
That means the machine has a stable place to ask:
Is there a newer ShrekOS?
without making my private infrastructure part of the trust model.
There was even a wonderfully mundane DNS/TLS constraint in here.
I originally wanted:
updates.shrekos.iambu.dev
But that extra subdomain level did not fit the free Universal SSL setup I was using.
So reality won.
The host became:
shrekos-updates.iambu.dev
Architecture is full of extremely sophisticated decisions like:
"The certificate does not cover that."
The signing key does not live in the repo anymore
This was another thing I wanted fixed before I called any of this real.
The private update-signing key used to be too close to the source tree for my comfort.
That is now gone.
The repository carries the public key only.
The private key lives separately under my local signing vault, and the release tooling requires the signing key explicitly.
If it cannot sign correctly, it fails closed.
That distinction matters.
GitHub is not the root of trust.
Cloudflare is not the root of trust.
The release manifest is not trusted because it came from my domain.
The host trusts artifacts because they were signed by a key it already trusts.
The rough chain is:
offline signing authority
│
▼
signed release artifacts
│
▼
GitHub / update front
│
▼
ShrekOS verifies
│
▼
inactive slot
The transport can distribute the bytes.
It does not get to decide whether those bytes are ShrekOS.
That is the important part.
I also had to answer the annoying future question
What happens when the signing key changes?
It is easy to build a system that says:
"This public key is trusted forever."
It is harder to build one that admits keys eventually need to rotate.
So the bootstrap story is now explicit.
For one transition release:
trust old key
trust new key
Then, after machines have crossed that release boundary:
drop old key
trust new key
Nothing exotic.
No magical distributed trust ceremony.
Just an intentionally boring overlap period.
I am increasingly convinced that "boring and explicit" is a security feature.
Then I tested the public edge
Before baking the update path into the OS, I tested the actual public front.
Not localhost.
Not a mock server.
Not a directory on disk pretending to be the internet.
The live endpoint.
The proof covered the normal release path plus failure cases, including a deliberately bad signature.
Fifteen checks.
Fifteen passed.
Great.
At this point I had:
- release discovery
- public distribution
- signatures
- manifests
- negative signature validation
- version selection
Everything looked good.
Which was exactly why the next failure was interesting.
The metadata all worked
I built a v2 system.
Then I published v3.
The v2 machine saw v3 on the live update front.
It downloaded it.
It verified the GPG signature.
It selected the inactive slot.
It wrote the update.
The update mechanism reported success.
Then I rebooted.
And dm-verity said:
absolutely not.
The new system would not boot.
This is where the dogfood earned its keep
The easy conclusion would have been:
The image is broken.
Except I booted the v3 image directly.
It worked.
Secure Boot worked.
dm-verity worked.
The system reached login.
So the release itself was fine.
The thing that was broken was the updated slot.
That narrowed the problem considerably.
I inspected the bytes at the start of the partition.
They were not filesystem bytes.
They were a compression header.
The updater had faithfully written the compressed release artifact directly into the system partition.
Amazing.
.zst was the trap
My release pipeline was publishing the system image compressed with Zstandard.
Something like:
system.raw.zst
I had mentally modeled the updater as:
download
↓
decompress
↓
write raw image
What actually happened was:
download
↓
write .zst bytes directly
↓
verity explodes
The version of systemd-sysupdate in the system was not decompressing that artifact the way I expected.
So every check before the physical write could pass.
The metadata was correct.
The signature was correct.
The release was correct.
The downloaded file was correct.
And the resulting operating system partition was garbage.
That is such a good failure.
The rule changed immediately
Before this, I thought proving an update meant proving things like:
release exists
signature verifies
version comparison works
updater selects correct target
Those are necessary.
They are not sufficient.
The real contract is now:
An update is not proven until the bytes written by the updater boot.
Not the source artifact.
Not a manually extracted copy.
Not the manifest.
The actual bytes written through the real updater into the real inactive slot.
Then reboot the machine.
Then let Secure Boot and dm-verity judge you.
That is the test.
I switched the release pipeline to xz
Once the problem was isolated, I changed the update artifacts from Zstandard to xz so the update machinery could handle the image correctly.
Then I rebuilt everything.
And did the whole thing again.
From the old version:
v2
│
│ discovers v3
▼
public update front
│
▼
download
│
▼
GPG verify
│
▼
decompress
│
▼
write inactive slot
│
▼
reboot
│
▼
Secure Boot
│
▼
dm-verity
│
▼
login
This time it worked.
The machine came back on v3.
No emergency shell.
No verity failure.
No manually fixing the filesystem.
And the previous system remained available as the rollback target.
That is the moment I was willing to say:
ShrekOS can update itself.
Why A/B feels so much better than "repair in place"
The more I build this thing, the less I like the traditional model of an OS slowly mutating underneath you.
An update fails halfway through?
Now go repair package state.
A dependency script modifies something unexpected?
Good luck remembering what changed.
An upgrade boots badly?
Start reconstructing the previous system from whatever state survived.
The A/B model changes the question.
Instead of:
How do I repair this machine after the update mutated it?
the question becomes:
Does the new system boot?
If yes:
A → B
If no:
A ← still here
That is a much smaller problem.
The immutable part is not "nothing changes"
This is something I think the word immutable obscures.
Of course the operating system changes.
It has to.
There are kernel updates.
Security fixes.
Driver updates.
New system components.
The important difference is how it changes.
I do not want:
current system + thousands of accumulated mutations
I want:
known system N
↓
known system N+1
with a cryptographically authenticated transition between them.
The machine should move between complete states.
Not slowly become one.
What is still not solved
Three problems are still open.
The first, and the one that decides whether any of this counts as real, is bare metal.
Everything above happened in a virtual machine. Real firmware, a real TPM, and real Secure Boot do not behave exactly like OVMF, and the entire point of this update path is that it survives the real boot chain unattended. Until I have watched a physical ShrekOS box pull a signed update over the network and come back up on its own, this is proven in a VM and nothing stronger.
The second is a broker-leg MITM issue elsewhere in the architecture.
That is a real security problem, but it is not the update trust path.
I do not want to blur unrelated boundaries just so I can claim everything is done.
The third problem is much less glamorous.
The full install image is about 5.4 GB.
GitHub's normal release asset path has a 2 GiB ceiling for what I am trying to do with it.
So I now have an operating system that can securely update itself over the internet...
and still need a better way to distribute the original giant installer image.
Perfect.
What changed for me
Before this milestone, I would have described ShrekOS as something like:
An immutable Debian-based operating system with signed system layers, transactional updates, rollback, and agent isolation.
Most of that was architecturally true.
But there is a huge difference between:
"The system is designed to do transactional updates."
and:
"I watched the old system discover a new release on the public internet, verify it, write it into the other slot, reboot through Secure Boot and dm-verity, and reach login."
The second one is a fact.
I like facts much more.
The part I am keeping
The .zst failure is probably the lesson from this milestone that sticks.
Every abstraction above the actual disk write said the update was healthy.
The machine disagreed.
And the machine was right.
So the new rule in ShrekOS is simple:
Prove the write, not just the listing.
If the updater is supposed to create the next bootable system, then the test ends when that exact system boots.
Everything before that is evidence.
The reboot is the verdict.
ShrekOS is still deeply unfinished.
There are still security boundaries I have not closed.
There are still ugly deployment problems.
There are still parts of this thing where I know just enough to discover entirely new categories of mistakes.
But the update path is not hypothetical anymore.
The machine can replace itself.
It can reject bad releases.
It can boot the new state under the same verified chain as the old one.
And if the new state is bad, the old one is still sitting there.
That feels like a real milestone.
Not because the design looks good on paper.
Because I broke it, found out why, fixed it, and watched the actual machine come back up.
This is a milestone note from my Building ShrekOS series. New to the project? Start with Why I'm Building ShrekOS When Containers Already Exist.
Top comments (0)