Your OS Abstraction Isn't Real Until the Second Family Boots

TL;DR — I have an OsProfileContract with a Debian implementation that's been happily provisioning machines for months. Today I ran the same pipeline against an enterprise-Linux family for the first time. The contract survived. Almost everything I'd assumed about it did not. Here's what a second implementation actually teaches you.

---

The abstraction looked fine

The shape is the usual driver seam. One contract, one implementation per OS family, a factory that picks by what /etc/os-release says:

interface OsProfileContract
{
    public function family(): OsFamily;

    /** @return list<string> */
    public function toolchainCommands(NativeToolchain $toolchain, ?string $version = null): array;

    /** @return list<string> */
    public function prepareWorkloadCommands(Workload $workload): array;
}

Nothing exotic. And with exactly one implementation, a driver seam is indistinguishable from a wrapper — every assumption Debian happens to satisfy is silently baked in as part of "the interface", because nothing ever contradicts it.

Then the second family boots, and you find out how much of your contract was actually a description of apt.

Lesson 1: a guard that checks less than the install provides never self-heals

This one is the most portable idea of the day, so I'll lead with it.

Every toolchain install is wrapped in a guard so it's idempotent — skip the expensive install if the tool is already there:

// The version guard for the Node toolchain
'__n=$(node -v 2>/dev/null | sed -e "s/^v//" -e "s/\..*//"); [ "${__n:-0}" -ge 20 ]'

Reasonable. On Debian-family machines the vendor package bundles node and npm together, so checking node implies npm. On the enterprise family they're separate packages, and the install only named the first one.

Result: a node with node and no npm. The guard checked node -v, saw 22, and skipped the install — forever. The build died on npm: command not found, and there was no path back. Re-running provisioning changed nothing, because the guard was permanently, confidently green.

The rule that falls out:

The guard must assert everything the install is supposed to produce. If the install brings three things, the guard checks three things — otherwise a partially-installed machine can never repair itself.

self::Node => '__n=$(node -v 2>/dev/null | sed -e "s/^v//" -e "s/\..*//"); '
    .'[ "${__n:-0}" -ge 20 ] && command -v npm >/dev/null 2>&1',

self::Php => '__p=$(php -r "echo PHP_MAJOR_VERSION*100+PHP_MINOR_VERSION;" 2>/dev/null); '
    .'[ "${__p:-0}" -ge 802 ] && command -v composer >/dev/null 2>&1',

The same bug bit twice more the same day. A PHP build failed at composer install on a node whose toolchain guard was green — because the enterprise package list was missing the zip extension, and the guard only checked for redis. Adding zip to the install fixes new machines. Adding zip to the guard is what lets every already-provisioned machine self-heal on its next run.

That's the actual payoff, and it's worth saying plainly: a guard is not just a speed optimisation, it's your repair mechanism. Widen the guard and old machines fix themselves. Widen only the install and they stay broken forever.

Lesson 2: exit code 0 does not mean "the thing you wanted happened"

Having added the missing package to the install command, I watched the very same node fail the very same way.

The install used a package-manager subcommand that syncs a package to the current stream — and that subcommand exits 0 while installing nothing when the package is merely absent. So the || fallback that was supposed to catch the never-installed case never fired. The command succeeded. Nothing was installed.

// Before: the install is behind a fallback that a zero exit will never trigger.
'sync-cmd nodejs npm || install-cmd -y nodejs npm'

// After: sync only what may be on an old stream, then install unconditionally.
'(sync-cmd nodejs || true); install-cmd -y nodejs npm'

Installing an already-present package exits 0 and does nothing, so the unconditional install is still idempotent. It just no longer depends on a non-zero exit that the tool doesn't produce.

The generalisation: || chaining encodes an assumption about a tool's exit-code semantics. Every time you write a || b in a provisioning script, you're asserting "a returns non-zero when it doesn't do its job". Verify that. A surprising number of CLI tools disagree.

And pin the shape in a test, because this is exactly the kind of fix that quietly regresses:

it('installs node and npm unconditionally on the EL family', function () {
    $commands = app(RhelOsProfile::class)
        ->toolchainCommands(NativeToolchain::Node);

    $joined = implode(' ', $commands);

    expect($joined)->toContain('install -y nodejs npm')
        ->and($joined)->not->toMatch('/\|\|\s*\S*install -y nodejs npm/');
});

Asserting a command isn't behind a fallback feels like testing an implementation detail. It isn't — it's testing the exact property that took two live iterations to establish.

Lesson 3: a permission fix can be worse than the permission failure

The enterprise family ships mandatory access control on by default; the Debian family doesn't. That's not a "few extra commands" difference — it's a whole axis the contract never had to think about.

First attempt at fixing a socket-bind denial: label the entire workload tree as web-server-writable content. It fixed the bind. It also broke something far more fundamental — the init system itself could no longer read the current symlink to resolve the service's working directory, so the unit died before spawning a single process, at a step that runs before your code exists. And because the unit restarts on failure, it crash-looped every five seconds. For hours.

Trading a scoped failure for an unscoped one is the classic shape of a permissions "fix". Two corrections came out of it:

Label surgically, and give the thing that needs a weird label its own directory. The socket moved into a dedicated run/ directory — specifically so its label can't spread to neighbours by parent-directory inheritance. Code stays a generic type that the web server may read and init may traverse. Only what the process actually writes gets the special label.

The right label is the one the vendor's own default path carries. The second iteration failed again, this time on socket creation, in a directory that had exactly the label I'd chosen. The reason is dull and useful: that policy type manages files, directories and symlinks — and simply has no rule for sockets. Sockets live under a different type, the one the vendor's own default runtime path already uses. When you're guessing at a security label, stop guessing and go read what the upstream package does by default.

That produced one new contract method:

interface OsProfileContract
{
    // …

    /**
     * Re-apply security labels to a workload tree after a deploy.
     * Creation-time labels are inherited from the parent directory —
     * never from the configured patterns — so a freshly written release
     * carries the wrong type until this runs.
     *
     * @return list<string>
     */
    public function relabelWorkloadCommands(Workload $workload): array;
}

Debian returns [].

That empty array is the interesting part. A contract method that one driver implements as a no-op is fine — much better than an if ($family === OsFamily::Rhel) sprinkled through the deploy path. The seam absorbs the difference; callers stay family-blind. This is the same reason enum-shaped capabilities beat feature flags: the caller asks the question, the driver decides the answer.

One more habit that made all three iterations survivable: the pattern-modification arm is written so it self-heals machines that already carry the previous, wrong pattern. When you're on live iteration three of a security label, "new machines are correct" isn't good enough — the fix has to reach the machines the earlier attempts damaged.

Lesson 4: defaults describe intent, not reality

A small one with a wide blast radius. The node-upgrade action picked its OS template from the fleet default (falling back to literally the first row). But the fleet default encodes the family an operator once chose for new machines. It says nothing whatsoever about this machine.

With a Debian-family default configured, every enterprise-family node's upgrade failed at the pipeline's OS-family gate. Un-upgradeable by construction — a category of bug that no test catches, because in the test environment the default happens to match.

// Read the machine, then choose a template for what it actually is.
$detected = $this->agent->execute($node, 'cat /etc/os-release');
$family = OsFamily::fromOsRelease($detected->stdout);

$template = $family !== null
    ? OsTemplate::defaultFor($family) ?? OsTemplate::firstFor($family)
    : $fleetDefault; // unreadable os-release → keep the default; the gate still protects us

Note what the fallback does: an unreadable /etc/os-release keeps the old behaviour and lets the family gate refuse. The fix narrows the failure; it doesn't paper over it.

Two honesty fixes rode along on the same path, and they're the kind I keep having to re-learn:

• The last_upgraded_at timestamp is now stamped only when the job actually reached Ready. A failed, rolled-back upgrade changed nothing on the machine — recording it would hide the node from exactly the report meant to surface it.
• The console command reports the job's failure instead of cheerfully announcing the target version over a run whose steps all rolled back.

A timestamp that means "we tried" masquerading as one that means "we succeeded" is how a fleet quietly rots.

The takeaway

If you have a driver seam with one implementation, you don't have a driver seam yet — you have a wrapper plus a strong hypothesis. The second implementation is what converts assumptions into contract:

• Guards are repair mechanisms. Assert everything the install produces, or broken machines stay broken.
• a || b encodes a claim about b's exit codes. Check the claim.
• Prefer a no-op method on one driver over a family check at the call site.
• Fixes for the last three failures have to self-heal the machines the first two attempts broke.
• Defaults describe what someone once intended, never what a given machine currently is.

None of this needed a new abstraction. It needed the existing one to meet something that disagreed with it.