TL;DR — A long day across a deployment platform and a brand-new mobile app. Two themes kept resurfacing: an abstraction is only real once something disagrees with it, and a check that runs too late is not a check. Here's the log.
1. Preflight: fail at step 1, not step 14
Most of the day went into a provisioning pipeline's ordering, not its logic.
A long pipeline — provision a node, install a toolchain, wire a database, configure a proxy, then finally clone and deploy — was asking five-second questions at step fourteen. Is the deploy key registered on the git host? Is the declared runtime version one the preset can even install? Both are answerable from the control plane before a single resource is reserved. The first live run on a new OS family paid a full node provision to learn its deploy key had never been registered.
So: a PreflightWorkloadsStep at position one, failing on everything the last step would fail on anyway. Three rules made it worth having:
- It needs nothing. Everything runs from the control plane. The moment a preflight needs infrastructure it stops being a preflight.
- It reads from the same source as the real check. The allowed runtime versions come from the same enum the deploy-time guard reads. Duplicate the list into a validator and you've built a check that will eventually pass while the real one fails.
- An empty workload set is an honest success, not a skip. A clean deployment shouldn't render as a hole in the pipeline.
And the error message is the feature. A preflight that emits Permission denied (publickey) has technically worked and practically wasted an afternoon, so an SSH refusal now says which fix to apply — generate the key, or register the existing public key on the repository.
Same instinct landed on the runtime side: a failed systemctl restart whose stderr says "see journalctl" assumes the reader has a shell on that box. They have a web UI. The failure now tails fifteen journal lines into the exception, so the operator reads the actual migration exception. One extra round trip on a path that already failed — cheapest error message you'll ever buy.
Written up properly in a separate post today.
2. Two pipeline runs is not a wasted duplicate — it's destructive
The one that genuinely surprised me. A create wizard dispatches a provisioning job; a detail page also has an Execute button. The policy refuses a second run once the status is Provisioning — but the status only flipped when a worker claimed the job, leaving the whole queued window open.
I assumed the worst case was wasted compute. The actual worst case: the run that fails first executes its rollback chain and destroys the node rows the surviving run is standing on. The survivor then fails with "No node provisioned" and rolls back a second time. Two runs, zero infrastructure.
Rollback turns a duplicate into a weapon pointed at shared state. Two locks, because there are two kinds of call site — ShouldBeUnique for the queued path, and a per-deployment cache lock inside the pipeline for the console, MCP and direct callers that never queue. Lock TTL matches the job's $timeout (a crashed holder frees it no later than the process would've been killed anyway) and the release lives in finally.
Related, one line, wide blast radius: flip the status before the dispatch, at every dispatch site. The detail page only live-polls while the status reads Provisioning, so a deployment sitting at Pending until a worker picked it up rendered as a page that looked stalled. One of the three dispatch sites had always done this correctly — which is precisely why the bug survived: the behaviour was right on the path everyone tested by hand.
3. A second OS family, and everything it disagreed with
The provisioning platform has had an OsProfileContract with a Debian-family driver for months. Today it met an enterprise-Linux family for the first time. The contract survived. My assumptions didn't. Also a post of its own, but the three transferable bits:
A guard that checks less than the install provides can never self-heal. Toolchain installs sit behind an idempotency guard. The Node guard checked node -v. On Debian the vendor package bundles node and npm; on the other family they're separate packages, and the install named only the first. Result: a machine with node, no npm, and a permanently green guard. The build died on npm: command not found with no path back. Widening the install fixes new machines. Widening the guard is what repairs the ones already broken. Same bug bit again with a missing PHP zip extension on a node whose guard only checked redis.
Exit code 0 doesn't mean the thing you wanted happened. The follow-up fix failed on the very same node, because the package manager's stream-sync subcommand exits 0 while installing nothing when the package is simply absent — so the || install fallback never fired. Every a || b in a provisioning script is an assertion about a's exit-code semantics. Verify it.
A permission fix can be worse than the permission failure. Labelling the whole workload tree as web-writable content fixed a socket bind and broke the init system's ability to read the current symlink — the unit died resolving its working directory, before any process spawned, and crash-looped every five seconds. The correction was surgical: give the socket its own directory precisely so its label can't spread by parent inheritance, and use the type the vendor's own default runtime path already carries. That added one contract method, relabelWorkloadCommands(), which the Debian driver implements as return []. A no-op in one driver beats an if ($family === …) in the deploy path every time.
4. Failure cleanup has to know what the world is pointing at
A deploy's failure path deleted its release directory — sensible, because failed builds leave a gigabyte of node_modules behind that the success-only pruner never sees, and four of them once filled a node to ENOSPC.
The comment justifying it said "the failure happened before the symlink swap, so this directory is never current." That was true when it was written. It stopped being true when activation failures joined the same catch block — the symlink swaps before the restart. So a failed activation deleted the directory current pointed at, leaving a dangling symlink under a restart-always unit. Crash loop, twice in one day.
Now the cleanup reads the symlink first: if it points at the release being removed, stop and disable the unit and take the link with it. The next deploy recreates both. Failures before the swap keep the old behaviour exactly, because a running old release must never be stopped by a failed build.
The lesson isn't about symlinks. It's that a comment asserting an invariant is load-bearing code, and the moment you widen a catch block you've potentially invalidated every one of them. If you can't express the invariant as an assertion, at least grep for the ones you're about to break.
5. Soft deletes and unique indexes, again
The workflow docs recommended a detach-then-re-attach flow. Detach soft-deletes the row. The uniqueness index over the tuple doesn't scope deleted_at. So the recommended flow died on a unique violation, rendered as a bare HTTP 500 — minutes after a user was told to use exactly that flow.
The fix is the same rule I've written before and will write again: look the key up withTrashed() and decide deliberately. An active row is a named refusal on the form ("already attached — detach it first to change how it's wired"). A trashed row is restored and refilled with fresh attributes and a reset status. Neither path is allowed to be an exception.
The existing re-attach test dodged this entirely by using a different layer, which is a good reminder that a passing test named after the flow doesn't mean the flow is tested. New tests pin the same-key path specifically.
6. On the other desk: a mobile app, an engine, and a licence audit
Completely different context — a Flutter app with a chess engine in it — but the day's most transferable decisions there were legal and architectural, not gameplay.
The licence audit came before the code grew. The initial spike used a couple of GPL-licensed packages. Swapping them for permissively-licensed equivalents on day one cost an afternoon; discovering it after six months of feature work costs a rewrite. A vendored native engine was chosen partly for its permissive licence, and there's now a scan for copyleft headers across everything vendored, plus a third-party notices page in the legal copy. Licence compatibility is an architectural constraint that happens to be written in English instead of code.
Two very different engines, one interface. A neural-network model running through an ONNX runtime and a vendored native C++ engine talking over FFI have nothing in common at the implementation level — different languages, different build systems, different failure modes. Behind one Engine contract with an evaluate() method, the app doesn't care. Same driver-seam thinking as the OS profiles above, in a completely different stack. The FFI side runs from a long-lived isolate so the UI thread never blocks on a search, which is the mobile equivalent of "a multi-minute pipeline must not run inside a request".
Naming is research, not inspiration. The product got renamed twice in one day. The first name collided with an existing well-known franchise's species family — the kind of collision that's cheap to fix at week one and ruinous at launch. Do the trademark search before the bundle identifier, not after.
What's next
The deployment platform's remaining rough edge is the honesty of its reporting — statuses and timestamps that describe intent rather than outcome. One landed today (an upgrade timestamp now stamped only when the job actually reached Ready, so a failed and rolled-back upgrade can't hide a node from the report meant to surface it), but there are more of those hiding in plain sight.
If there's a single line to take from today: most of these weren't logic bugs. They were correct logic running at the wrong time, or a comment that used to be true.
Top comments (0)