Originally published at tothenoc.substack.com
Why every ToTheNOC image is launched on a real instance before it is listed.
We build hardened Linux images. AlmaLinux, Rocky Linux and Debian, x86 and ARM.
In October one of our AlmaLinux 9 builds passed the hardening scan. Clean. Then we launched it on a real instance and it never came up. No SSH. The serial console said "Read-only file system", and after that cloud-init, auditd and sshd all failed.
The cause was one line. The hardening profile wants /tmp on its own mount, so the recipe added a tmpfs line in fstab. On the AlmaLinux base image, tmp.mount is masked. systemd puts that masked unit in the boot transaction, the branch falls out, the root filesystem is never remounted read-write. Everything that needs to write to disk dies, sshd included.
Nothing in the build told us. The scan checks files. It does not boot the system.
A few weeks later, same story on ARM. The image was "running" on x86 and "degraded" on Graviton, because a service waited for a hardware random source that the ARM instance does not have.
So we changed the rule. An image is ready only after:
- it boots on a real instance,
- SSH answers,
- systemctl is-system-running says "running", not "degraded",
- the same OpenSCAP / ComplianceAsCode profile runs again on the live system.
For AlmaLinux 9 today that is 265 of 265 automated checks of our Level 1 server hardening profile (OpenSCAP / ComplianceAsCode), measured on a running instance. The raw results, the tailoring file and the checksums are public for every release.
This is what "verified on boot" means for us. Not a badge. A boot.
Short video (88 s): https://www.youtube.com/watch?v=IET97Z8aREo
The images are on AWS Marketplace, 17 regions: search "ToTheNOC".
How we build and verify every release: tothenoc.com/hardened-images.
-- Alex
Top comments (0)