Fail at Step 1, Not Step 14: Preflight Checks for Long Pipelines

TL;DR — I spent today running a provisioning pipeline against a fresh OS family for the first time, and most of what broke wasn't the logic — it was when the logic ran. A check that only runs at step 14 is a check no failure meets in time. Here's the preflight step, the single-flight lock, and the status flip that came out of it.

---

The shape of the problem

A deployment pipeline with fourteen-ish steps. Provision a node. Install a toolchain. Set up a database. Configure a reverse proxy. Somewhere near the end: clone the repository and deploy the workloads.

That last step is where the boring failures live. The deploy key was never registered on the git host. The declared runtime version isn't one the preset can install. Both are five-second questions — and both were being asked after ten steps of real, billable, side-effecting work had already happened.

The first live run on a new OS family paid a full node provision to learn that a deploy key had never been registered. That's the whole lesson in one sentence.

Step 1 asks everything step 14 will ask

The fix isn't clever. It's a step that runs first and fails on anything the last step would fail on anyway:

final class PreflightWorkloadsStep implements DeploymentStepContract
{
    public function name(): string
    {
        return 'Preflight workloads';
    }

    public function handle(DeploymentContext $context): void
    {
        $failures = [];

        foreach ($context->deployment->workloads() as $workload) {
            $failures = [...$failures, ...$this->checkOne($workload)];
        }

        if ($failures !== []) {
            throw new RuntimeException(implode("\n", $failures));
        }
    }

    public function rollback(DeploymentContext $context): void
    {
        // Checks only. Nothing changed, so there is nothing to undo.
    }
}

Three design rules did the real work here.

1. The preflight must need nothing. Everything in it runs from the control plane. No node, no SSH to a machine that doesn't exist yet. The moment a preflight needs infrastructure, it stops being a preflight and becomes step 3.

2. It must read from the same source as the real check. The runtime-version validation reads its allowed list from the same enum the deploy-time guard reads:

$recipe = app(BuildRecipes::class)->for($preset);

if ($recipe?->versionedToolchain === null) {
    $failures[] = "'{$preset}' does not take a runtime_version — remove it.";
} elseif (! in_array($declared, $recipe->versionedToolchain->allowedVersions(), true)) {
    $failures[] = "runtime_version '{$declared}' is not available for "
        .$recipe->versionedToolchain->label().'. Available: '
        .implode(', ', $recipe->versionedToolchain->allowedVersions()).'.';
}

Duplicating the allowed list into a validator is how you get a preflight that passes and a deploy that fails on the same input six months later. One enum, two readers. The checks cannot drift because there's nothing to drift from.

3. An empty set is an honest success, not a skip. No workloads to check means the preflight passed. Marking the step "skipped" would make a genuinely clean deployment look like a hole in the pipeline.

Point at the fix, not just the failure

A preflight that says Permission denied (publickey) has technically done its job and practically wasted everyone's afternoon. The error a preflight produces is the entire user interface of the feature:

private function gitHint(string $url, bool $hasDeployKey): string
{
    if (GitRemote::parse($url)?->isSsh() !== true) {
        return '(Check the repository URL, and the credential for this host if the repo is private.)';
    }

    return $hasDeployKey
        ? '(The key exists but the host refused it — register the public key on the repository as a read-only deploy key.)'
        : '(Generate a deploy key on the application page and register it, or use an HTTPS URL with a credential.)';
}

Same thinking landed elsewhere in the day: when a systemctl restart fails, its stderr is famously just "see journalctl" — which assumes the reader has a terminal on that box. They don't; they have a web UI. So the failure now tails the last fifteen journal lines into the exception message. The operator reads the actual migration exception instead of being sent somewhere they can't go.

if (! $result->isSuccessful()) {
    $journal = $this->agent->execute($node, sprintf(
        'journalctl -u %s -n 15 --no-pager 2>/dev/null | tail -15',
        $unit,
    ));

    throw new RuntimeException(trim(
        "restart failed on {$node}: {$result->stderr}"
        .(trim($journal->stdout) !== '' ? "\nLast log lines:\n".trim($journal->stdout) : '')
    ));
}

Fetching the journal costs one extra round trip on a path that has already failed. That's the cheapest error message you'll ever buy.

Two runs of the same pipeline is not a wasted duplicate

Here's the one that genuinely surprised me. The create wizard dispatches a provisioning job. The detail page also has an Execute button. The authorization policy refuses a second run once the status is Provisioning — but the status only flipped when a worker claimed the job. So the queued window was wide open, and clicking Execute during it dispatched a second run.

I'd assumed the worst case was wasted compute. The actual worst case: the first run to fail executes its rollback chain and destroys the node rows the surviving run is standing on. The survivor then fails with "No node provisioned" — and rolls everything back a second time. Two runs, zero infrastructure, and a rollback that ate a healthy machine.

Two locks, because there are two kinds of call site:

class ProvisionDeploymentJob implements ShouldBeUnique, ShouldQueue
{
    /** Held while queued AND running; released on completion. */
    public int $uniqueFor = 1800;

    public function uniqueId(): string
    {
        return (string) $this->deploymentId;
    }
}

ShouldBeUnique covers the queue. But console commands, MCP tools and direct calls never touch the queue, so the pipeline holds its own lock:

public function execute(): Deployment
{
    $lock = Cache::lock('deployment-pipeline:'.$this->deployment->id, self::LOCK_SECONDS);

    if (! $lock->get()) {
        throw new RuntimeException(
            'A provisioning run is already in progress — wait for it to finish.',
        );
    }

    try {
        return $this->runSteps();
    } finally {
        $lock->release();
    }
}

Two details worth stealing: the lock TTL matches the job's $timeout, so a crashed holder frees the lock no later than the process would have been killed anyway; and the release lives in finally, not at the end of the happy path.

Pest makes this cheap to pin:

it('refuses a second concurrent run for the same deployment', function () {
    $deployment = Deployment::factory()->create();

    Cache::lock('deployment-pipeline:'.$deployment->id, 1800)->get();

    expect(fn () => app(DeploymentPipeline::class)->for($deployment)->execute())
        ->toThrow(RuntimeException::class, 'already in progress');
});

it('releases the lock when a step throws', function () {
    // …execute with a step that throws, then assert a second execute proceeds.
});

The second test matters more than the first. A lock you never release is a nastier outage than the race it was preventing.

Status is part of the contract with the UI

Related, and easy to miss: the detail page only live-polls while the status reads Provisioning. So a deployment sitting at Pending until a worker picks the job up rendered as a page that looked stalled and never updated without a manual refresh.

The fix is one line, but the rule behind it generalises: flip the status before the dispatch, at every dispatch site.

$deployment->forceFill(['status' => DeploymentStatus::Provisioning])->save();

ProvisionDeploymentJob::dispatch($deployment->id);

"Every dispatch site" is the load-bearing part. There were three — the wizard's save-and-deploy path, the deploy-pending path, and an MCP tool. One of them had always done it correctly, which is exactly why the bug survived so long: the behaviour was right on the path everyone tested by hand.

If you find yourself repeating a two-line invariant at three call sites, that's a nudge toward an invokable action — DispatchProvisioningAction — so a fourth call site can't be added without it.

The takeaway

Ordering is a design decision, not an implementation detail:

• A validation that runs after side effects isn't a validation, it's a post-mortem.
• Preflights must read from the same source of truth as the real check, or they'll drift into lying.
• An error message on a long-running pipeline is a UI. Include the fix, not just the symptom.
• Concurrency bugs in pipelines with rollback aren't wasteful — they're destructive, because rollback is a loaded weapon pointed at shared state.
• If a UI polls on a status, then setting that status is part of the API, not cosmetics.

Next up: the failure paths themselves. A cleanup routine that deletes a release directory needs to know whether it's deleting something the world is currently pointing at — which is a whole post of its own.