Which parts of this are real? Should be a question with an answer

TL;DR

• The most expensive bug I hit yesterday wasn't a crash. It was a stub that answered politely.
• Four shapes of the same problem: a column nobody writes to, a model nobody calls, an engine backed by a fake, and a resolver that falls back to a null implementation silently.
• The fix isn't "delete the fakes". Fakes are how you ship in slices. The fix is making which parts are real an enumerable question instead of a code-reading exercise.
• Corollary that keeps paying off: null means unknown, not zero. An unreachable provider is not a cluster that shrank to nothing.

Yesterday was a long day on a private deployment platform, and by the end of it I noticed every commit was the same commit wearing a different hat.

Nothing had thrown an exception. Nothing had failed a test. The system had simply been answering questions it had no business answering, confidently, for weeks.

Shape 1: the column with no producer

There was a ip_public column on a node record. It had a migration, it had a row in the UI panel, and something downstream preferred it over the internal IP when present.

It was always null. Not sometimes — always. No driver ever set it, and the status object the drivers return didn't even have a field for it, so the three places that re-read a node couldn't have refreshed it if a driver had.

The downstream branch that preferred it had never once executed. It looked like a feature. It was a comment.

Here's the thing about a column with no producer: it doesn't fail loudly, it fails as absence. Every reader takes the null branch, every reader is correct to, and the whole path stays green.

The interesting part came when I went to fix it, because it turned out different providers can't honestly answer the same question:

• One scheduler backend can report the cluster node's external address — or its internal one, since a single-node dev cluster has no external address and still serves traffic there.
• A hypervisor driver has to sort the guest agent's addresses by scope. Taking "the first IPv4" is how you end up publishing a Docker bridge address.
• The container drivers report null on purpose. A container's published port answers on the daemon host, and that's provider credentials, not node state. Copying it onto every node row goes stale the moment someone edits that credential.

So the fix wasn't "make every driver fill in the field". It was deciding, per driver, what the field means — and then pinning the deliberate nulls in tests so a future contributor doesn't read them as a gap and helpfully fill them in.

it('reports no public ip for container-backed nodes', function () {
    $status = app(ContainerDriver::class)->getNodeStatus('node-ref');

    // Deliberate: the published port answers on the daemon host, which is
    // provider-level config, not node state. Do not "fix" this to non-null.
    expect($status->publicIp)->toBeNull();
});

A test whose whole job is to say this null is on purpose is one of the highest-value tests you can write. It converts an absence into a decision.

Shape 2: the model nobody calls

There was a complete access-review model. Enums, migration, markInProgress(), complete(). Zero references anywhere in the application.

The compliance control it was written for — periodic access review — couldn't be evidenced at all, because there was no way to actually run one.

The load-bearing part turned out to be completion, and it's worth stating plainly:

A review that records "revoke" and leaves the access in place is a permanently green control. That's worse than an absent one, because it produces evidence certifying nothing.

So completion does the revocation and the record in one transaction — deactivate the membership, clear the roles — and writes the evidence outside it. That second half is the part people get wrong, and I've now been bitten by it twice:

DB::transaction(function () use ($session) {
    $session->applyDecisions();   // actually revokes
    $session->markCompleted();
});

// Outside. Evidence written inside the transaction it describes gets rolled
// back together with it — you lose the record of the failure precisely when
// you need it.
$this->evidence->record('access-review.completed', $session);

Two more rules fell out of it, both the "obvious in hindsight" kind:

• Completion refuses while any member is undecided. A review signed off with blanks certifies people nobody looked at.
• Opening is idempotent per scope. Two open sessions mean two reviewers working from different membership lists, and whoever finishes last silently overwrites the other.

And the membership list is snapshotted when the review opens, not recomputed on render. A review of "who has access right now" is not a review, it's a dashboard.

Shape 3: the engine backed by a fake

The rollback mechanism had a contract, a plan/step schema, and a fake implementation whose step-class strings named handlers that didn't exist.

This one has a nasty second-order effect. A previous piece of work read the whole mechanism as scaffolding and routed around it — wrote its own path. So now you have two rollback stories, one of which is decorative, and the decorative one is the one with the nice contract.

That's the real cost of a convincing fake: it doesn't just fail to work, it teaches the next engineer that this seam isn't load-bearing.

Making it real meant four idempotent step classes, resumable from the first step that isn't already successful, and one seam — a single releaser class — where a previously-built image goes back onto a workload. Both callers now go through it, which means there's exactly one file to change when this needs to become zero-downtime.

Two things the tests caught that I'd have shipped:

• Repeated rollbacks walked forward through the release history instead of back. Rolling back twice is not "go to the previous release" twice unless you're tracking where you already are.
• A workload-scoped rollback whose health check fails should escalate to a full rollback — unless the runtime refused the image outright. That's a broken artifact, not evidence the infrastructure is at fault. Escalating there just widens the blast radius of a bad build.

Same category as the promotion engine, which was also a fake: it approved everything. An approval gate that cannot say no is not a gate, it's a log line. Replacing it meant a real exception type carrying which checks failed and why, so the UI can say something an operator can act on instead of "promotion failed":

class PromotionBlockedException extends RuntimeException
{
    /** @param  array<int, PromotionCheck>  $failed */
    public function __construct(public readonly array $failed)
    {
        parent::__construct(__('Promotion blocked by :gates.', [
            'gates' => implode(', ', array_map(
                fn (PromotionCheck $c) => $c->type->label(),
                $failed,
            )),
        ]));
    }
}

A distinct exception type rather than a bare RuntimeException, because the caller has something useful to say about it. If your error only carries a message, every handler upstream is reduced to string-matching or shrugging.

Shape 4: the silent fallback

This is the worst one, and it's the one that generalises furthest beyond infrastructure.

A resolver picked a driver per provider. When it couldn't find what it needed, it fell back to a no-op implementation. Quietly. So a provider configured with credentials that could never work reported "connection fine" — because the thing answering was the fake, and the fake always says yes.

The same fallback meant one deployment reported a load balancer it had never created.

The fix that actually stuck wasn't better error messages. It was making the question enumerable:

enum DriverCapability: string
{
    case Driver = 'driver';
    case Agent = 'agent';
    case ReverseProxy = 'reverse_proxy';
    case LoadBalancer = 'load_balancer';
    case Runtime = 'runtime';

    public function description(): string
    {
        return match ($this) {
            self::Driver => __('Creates, destroys, starts and stops nodes.'),
            self::Agent => __('Runs commands and reports metrics on a node.'),
            // …
        };
    }
}

One case per resolver method. Now "which parts of this provider are real?" has an answer you can render in a table, assert in a test, and show an operator — instead of an answer you get by reading every match arm by hand.

A deliberate fake and an accidental fake look identical in the code. They only become distinguishable when something declares which is which.

And for the probes themselves: stop asking the abstraction whether it works, and ask the underlying thing. A "test connection" that goes through the resolver tests the resolver. A probe that opens a real SSH session and reads /etc/os-release proves the login and answers a question you needed anyway.

The unit problem, briefly

One more from the same day, different flavour. Every driver implements provisionNode(). On one it creates a VM. On another, a container. On another, a single-replica scheduled workload.

The interface is honest — all of them provision a node. The word isn't. Counting node rows uniformly bills fifty containers on one host exactly like fifty machines across fifty hosts.

So the difference got named:

enum NodeBillingModel: string
{
    /** We create the machine — each node row is a host. */
    case PerProvisionedNode = 'per_provisioned_node';

    /** We schedule onto machines the customer already runs. */
    case PerClusterHost = 'per_cluster_host';
}

The discriminator I landed on: does the thing have its own OS kernel? It sorts every provider type cleanly, and — the part that matters for anything commercial — it's a line a procurement officer can verify without reading the code.

Then one class owns the count. Licence checks, quotas, invoice lines, the admin UI all come through it. A second implementation anywhere is how the dashboard and the invoice come to disagree in front of a customer.

And the rule I keep coming back to:

// An outage is not a cluster that shrank to zero. Keep the last known value.
if ($count === null) {
    return;
}

Unknown and zero are different answers. Conflating them is how a monitoring system reports perfect health five minutes after the region went down.

What I'd take from this

If you're carrying a system with staged, partially-built seams — and you are, everyone is — three things:

1. Every stub declares itself. Not in a comment. In something you can enumerate, list, and assert on.
2. Deliberate nulls get a test that says so. Otherwise your next contributor "fixes" them.
3. Probe the real thing, not your own abstraction over it. Your abstraction was written by someone who assumed it worked.

None of this is exotic. It's just that "does this work?" and "does this answer?" are different questions, and only one of them has been tested.