Drift detection and autoscaling are the same shape: ask the provider, then decide

TL;DR

• Drift detection and autoscaling look like different features. They're the same loop: observe → compare to intent → decide → act.
• The single most important rule: read actual state from the provider, not from your own tables. Your table saying "running" next to a VM that no longer exists is the drift.
• Both belong behind a contract, driven by a scheduled command, so the decision logic is testable without touching real infrastructure.
• A fake context object beats mocking the world.

Yesterday I built two things on a deployment platform: a drift reconciler and a scaling engine. I expected two unrelated features. Halfway through the second one I realised I was writing the first one again with different nouns.

Sharing the shape, because it generalises well beyond infrastructure — anything with "what I intended" and "what is actually true" has this problem.

The loop

observe actual state  →  compare to desired state  →  produce a decision  →  act (or just record)

For drift: desired = the recorded blueprint snapshot, actual = what the provider reports, decision = "this layer has 2 nodes and should have 3".

For scaling: desired = the policy thresholds, actual = live utilisation from the nodes, decision = "scale up by 1".

Same skeleton. Which means the same contracts, the same scheduling, and — the part people skip — the same testing strategy.

Rule 1: actual state comes from the provider

This is the one that decides whether the feature is worth building at all.

The lazy implementation counts your own rows:

// Wrong. This can never detect the failure you care about.
$actual = $deployment->nodes()->where('status', 'running')->count();

A row saying running beside a VM that somebody deleted by hand at 2am is exactly the drift you're hunting. Counting rows reports a perfectly healthy deployment right up until someone tries to use it.

So the reconciler asks the provider:

interface DriftReconcilerContract
{
    public function recordDesiredState(Deployment $deployment): void;

    public function detectDrift(Deployment $deployment): DriftReport;

    public function reconcile(Deployment $deployment): DriftReport;
}

and detectDrift() goes out to the provider driver for each node's real status. Slower, chattier, occasionally times out. Also the only version that's true.

Generalise the lesson: if your "verification" reads the same table your "write" wrote, you haven't verified anything. You've asserted your own database is consistent with itself.

Rule 2: no baseline means no drift

The first bug I wrote: every layer of every pre-existing deployment reported as drifted, because nothing had ever recorded a desired state for them.

$desired = $this->latestDesiredState($deployment, $layer);

// Nothing was ever recorded for this layer, so there is no baseline to
// diff against. Reporting drift here would flag every layer of every
// deployment provisioned before this ran.
if ($desired === null) {
    continue;
}

A comparison with a missing side isn't a difference — it's an unknown. Alerting on unknowns is how you train a team to ignore alerts in week two.

Related: write the baseline to a table you actually read. Mine was being snapshotted into a metadata JSON column that nothing queried, so the reconciler had to record it properly itself. Worth checking whether your "we already store that" is stored somewhere anyone reads.

Rule 3: pick a unit and make everything agree

The scaling engine works in utilisation percentages, not absolute cores and gigabytes. Not because percentages are nicer — because the node agent already reports fractions, and converting to absolutes needs per-node capacity that nothing reliably records yet.

Same for replicas. A layer's replica count is its node count — the same definition the provisioning pipeline uses when it honours min_replicas. If scaling and bootstrap disagreed about what a replica is, they'd fight each other forever, quietly, on a schedule.

Pick the definition your cheapest reliable source can produce, and make every component use that one. A more "correct" unit that only two of five components can compute is worse than a rough unit everybody shares.

Absolute consumption is still needed — for cost and quotas — so it lives in a separate resource tracker. Two concerns, two units, no fudging one into the other.

Rule 4: the schedule is the trigger, not the logic

Both features ship as an invokable service behind a contract, plus a thin console command:

// routes/console.php
Schedule::command('drift:detect')->everyFifteenMinutes()->withoutOverlapping();
Schedule::command('scale:evaluate')->everyFiveMinutes()->withoutOverlapping();

The command does argument parsing, iteration and output. The engine does the deciding. That split is what lets you run the whole decision path in a test in milliseconds, and it's why --dry-run is three lines rather than a parallel code path.

withoutOverlapping() is not optional here. A scaling evaluation that starts before the previous one's provisioning finishes will see the old node count and scale up again. Twice.

Rule 5: fake the context, don't mock the world

The genuinely useful trick. Both engines take a context object — the thing that knows how to reach this deployment's provider, node agent and DNS. In tests, hand them a fake:

$context = new FakeScalingContext(
    metrics: ['web' => ['cpu' => 0.91]],
    nodes: ['web' => 2],
);

$decisions = app(ScalingEngineContract::class)->evaluate($deployment);

expect($decisions)->toHaveCount(1)
    ->and($decisions[0]->action)->toBe(ScalingAction::Up)
    ->and($decisions[0]->delta)->toBe(1);

Not Http::fake(). Not five chained mock expectations. One object that answers the questions the engine asks, constructed with the scenario you're testing.

The difference matters when you're covering the cases that actually bite:

• utilisation sitting exactly on the threshold
• already at max_replicas and still hot
• provider throws mid-evaluation
• a node the provider has never heard of
• two layers, one scaling up, one scaling down, same pass

Every one of those is a constructor argument away with a fake context. With mocks, each is a small research project, so in practice they don't get written.

it('does not scale beyond max_replicas', function () {
    $policy = ScalingPolicy::factory()->create(['max_replicas' => 2, 'scale_up_threshold' => 70]);

    $decision = evaluateWith(cpu: 0.95, nodes: 2);

    expect($decision->action)->toBe(ScalingAction::None)
        ->and($decision->reason)->toBe('at_max_replicas');
});

Note the decision carries a reason. "Did nothing" and "did nothing because it's capped" are very different answers at 3am, and a nullable reason string costs you nothing.

What I'd watch out for

• Reconciling automatically is a big step. Detect-only first, log the reports, look at a week of them. Auto-correcting drift you don't understand yet means your platform fights your ops team.
• Cooldowns before cleverness. A crude "don't act again within N minutes" prevents more incidents than any sophisticated prediction.
• Provider calls fail. A failed status probe is unknown, not missing. Treating a timeout as "node is gone" is how a scheduled job deletes production.
• Every decision should be recorded, including the no-ops. The scaling event log is what makes the thresholds tunable later.

The general pattern, stripped of infrastructure: record what you intended, observe reality from the authoritative source, diff on a schedule, and record every decision — including the decision to do nothing. That's an accounting reconciliation, a config sync, a cache invalidator, an inventory audit. Same shape every time.