Thirty-eight commits across five repos today. Reading them back, most were the same species of bug: two different things that a system had been treating as one word. Loaded and serving. Both-S3 and copyable. Present and correct. Every one of them was cheap to fix and expensive to find.
Loaded is not serving
Found in production, and my favourite bug of the day.
A site was answering 200 to every request. The dashboard showed the workload as failed, with its last release green and its deployment active. The journal was clean — php-fpm up, migrations run, no errors anywhere. A red workload sitting next to a perfectly healthy site, which is exactly the thing that sends somebody to go and fix a deployment that is fine.
The blue-green runtime picks the "active" colour like this:
foreach ([SLOT_BLUE, SLOT_GREEN] as $slot) {
if ($this->unitLoaded($slot)) {
return $slot;
}
}
Here's the thing: systemd keeps a unit loaded after it dies. systemctl show on a stopped-but-known unit still reports LoadState=loaded. So a retired blue outranked the green that was actually serving every request, the status probe went and asked blue how it was doing, blue said ActiveState=failed, and the UI rendered red.
The quieter half of the same bug: standbySlot() is the inverse of that answer. With blue permanently "active", every release went to green. Two consecutive production releases landed on the same colour — blue-green alternating had silently stopped, which defeats the entire arrangement.
The fix reads both properties in one round trip and lets an active unit win outright, even though blue is examined first:
$state = $this->unitState($slot); // LoadState + ActiveState, one call
if ($state->isRunning()) { // active or activating
return $slot;
}
$loadedFallback ??= $slot; // only used if NEITHER is running
The fallback is deliberate — it's what keeps a genuinely stopped workload reporting Stopped rather than Pending. And the test that scripted the old single-property probe was updated rather than deleted, because it now expresses serving instead of merely loaded, which is the distinction the whole bug turned on.
Both endpoints being S3 is not "can copy server-side"
While wiring bring-your-own object storage, I hit a variant of the same mistake in my own contract design.
CopyObject in the S3 API is served by the endpoint you send it to. That endpoint cannot reach into a different service's buckets. So "source is S3 and destination is S3" does not imply a server-side copy is available — same endpoint does.
The awkward part is that the implementation is resolved per driver, not per row. An S3 destination asked "can you server-side copy from this?" genuinely cannot tell which of its own configured endpoints it's being asked about. So the method grew a parameter:
public function supportsServerSideCopyFrom(
BackupDestination $destination,
BackupArtefact $source,
): bool;
PHPStan caught the arity break across one call site and two test doubles, which is the cheapest possible way to find out you changed a contract.
The consequence is worth stating out loud: cross-endpoint S3-to-S3 has to stream, which makes the streaming path the common case rather than the rare one. That promotes the transfer size ceiling from a defensive guard to a load-bearing part of the design — and it's why the byte counter counts bytes through the control plane, so a genuine server-side copy correctly records zero.
A version that is a fact beats a version that is a guess
A one-click Drupal install on an AlmaLinux 9 node bootstrapped, built, deployed, passed the health gate — and then died at step 18 with:
The database server version 13.23 is less than the minimum required version 16.
EL9's AppStream defaults the postgresql module to 13. Drupal 11 pins a minimum of 16. Those two can never agree, which means that recipe could not have installed on any RHEL-family node, ever — and the cost of learning it was a full composer build on a 1 vCPU box, reported on a step whose name points nowhere near a package stream.
Two fixes, because either one alone leaves the failure sitting there.
Install a server modern apps accept. A per-OS hook runs before a service's packages go on. Empty on Debian, whose postgresql metapackage tracks the release's own server. On RHEL it switches the module stream, guarded three ways, each load-bearing: only when 16 actually exists in that node's AppStream, so an older minor keeps working; module reset first, because enabling a second stream over an enabled one is an error rather than a switch; and only from the "no server installed yet" branch, because a stream switch does not upgrade a cluster — it would leave the data on 13 and the packages on 16, which is worse than either.
Refuse early when it still can't be satisfied. A recipe declares its minimum engine versions and a guard checks them at the end of the provisioning step — not in preflight, and this is the part worth arguing about.
Preflight has no database yet. The only thing available there is a guess from a table of distribution defaults: a table that goes stale silently and is wrong the moment somebody installs a version by hand. Six steps later, the version is a fact — the provisioner has read SHOW server_version off the running daemon. Still ten steps earlier than the failure it replaces, and it cannot be wrong.
Two details from the same guard:
- An engine that isn't in the deployment at all is unconstrained, not forbidden. The app might use SQLite, or a database this platform never provisioned.
-
version_compare, not string comparison.'9.6'sorts after'16'as text, which is how a naive check waves through an engine seven majors too old.
A hostname nobody typed
An application with no domain isn't merely unrouted — on a plain VM it's usually undeployable. With no domain the runtime drops off fpm serving and onto the recipe's php -S 0.0.0.0:${PORT} fallback, on the port the catalogue declared for the container image, and on any node already serving a domain, nginx is holding 80.
Cloning is what makes this the common case rather than an edge case. Cloning strips every domain, correctly — a hostname is served by exactly one deployment — so the copy of a working site arrives in precisely the state that can't run, and the operator is asked to invent a hostname before they can see whether the copy even works.
So a preview hostname gets issued rather than typed:
{workload-slug}-{deployment-suffix}.{preview-base}
The deployment's own random suffix carries the uniqueness, and that's the entire reason it's in the label. A scheme that can collide needs a retry loop, a "name taken" error, and a human to resolve it — for a name nobody asked for in the first place. The workload slug is in there because one deployment may attach more than one HTTP application.
The nice part: it's almost all wiring that already existed. One new step writes the name onto the routing rule; the reverse-proxy, DNS and SSL steps then do exactly what they already do for a hostname somebody typed, and none of them needs to know where it came from.
Who owns the decision — a menu refactor that's really an access refactor
On a multi-tenant product I work on, the Administration section had quietly become where anything unclassifiable landed: 22 items, six sub-groups, three levels deep, mixing a tenant's own settings with the vendor's diagnostics. A committee secretary was being shown Telescope, Horizon, an Artisan runner and an MCP console — tools they will never touch and shouldn't see.
The dividing line I took wasn't what kind of screen is this. It was who owns the decision. Vendor-owned tooling moved into its own section, invisible to a tenant entirely. Tenant-owned administration stayed and flattened — single destinations moved to the top level instead of each sitting inside a sub-group that costs a click to reveal one item. Net effect on a tenant's sidebar: 22 items to 9, three levels to two.
One implementation note that generalises. The new section gates on a landlord-level flag rather than a permission, for the same reason the platform routes do: these screens run with no tenant current, and a permission check then has no table to consult. It throws rather than returning false.
And the test that had checked the moved item is now stronger than before — it reads the new section and short-circuits on that section's own authorisation, so the item can't leak through a per-item gate even if somebody adds one.
The rest, briefly
Re-pricing moved out of a Livewire component and into an action. Two callers now share one definition of what re-pricing means — a form and an MCP tool. If the tool had copied the "retire the active version, cut the next one" logic, there'd be two definitions free to drift, and the thing that drifts is what people get charged. The write tool also carries forward anything the caller didn't name rather than defaulting to zero, so an agent asked to raise one number can't silently wipe the parameters sitting beside it.
A statistics dashboard that couldn't reach its own rows. A mail-history package renders eight status counts and a banner reading "N messages stuck in Sending for over 1 hour" — with no way to reach those N. It can't have a list, because the table has no recipient or subject column; both live inside a JSON headers blob, which is why status is the only thing it can aggregate. Fix: add the two columns, fill them on saving in a model that extends the package's, and backfill via a deploy operation that's idempotent and deliberately does not bump updated_at — the stuck-message window measures from it, so a backfill mustn't make a stuck message look fresh.
Gateway credentials got a UI, and a rule. A stored secret is never sent back to the browser; the form shows only whether one is set, and a blank field on save means "leave it alone" — so saving the page to change a URL can't silently blank a credential. Switching to a gateway with empty credentials is refused outright, because allowing it means every checkout throws and the operator hears about it from a customer instead of from that screen.
A Livewire trap worth writing down: wire:model="settings.payment.stripe.secret" doesn't bind one value — Livewire reads each dot as nesting and builds a three-level array. It fails as "Array to string conversion", which names nothing useful. Flat field names mapped to dotted storage keys through an explicit fieldMap().
A sandbox route with no guard. A pay-by-button-press endpoint that marks an invoice paid with no money involved — registered unconditionally by a glob-based route loader, checking nothing but that the invoice exists. That's fine in a sandbox and very much not fine anywhere else.
And one genuinely small thing: a mobile app version bump for optional player accounts and Google Sign-In, plus its changelog entry and two new BSD-3 licence notices. Third-party notices are one of those chores that's trivial the day you do it and awful the day you're asked for them at once.
The takeaway
Every bug in the first half of today was a word doing two jobs. loaded meant both "systemd knows about this" and "this is serving traffic". s3 meant both "speaks the protocol" and "can copy from that". A distribution default meant both "probably installed" and "definitely installed".
They're hard to spot because the collapsed version reads fine — it's usually the shorter, more natural sentence. The tell is when a screen and reality disagree and nobody can immediately say which one is wrong. That's not a display bug. That's two concepts sharing a name, and the name picked the wrong one.
Top comments (0)