Three Clicks, Three Concurrent Builds: Move Long Work Out of the HTTP Request

TL;DR — A Redeploy button in a Livewire component ran the full clone-and-build inline in the HTTP request. Impatient clicking meant three concurrent multi-minute builds in three FPM workers on the same 512MB box. It fell over, all three releases failed, and nothing showed up in Horizon because there was no job. The fix is boring and that's the point: dispatch the same queued job the webhook already used, pin the queue in the job's constructor, and refuse a second dispatch while one is in flight.

The bug that looks like a UX problem

The symptom: a deploy that works fine when you press the button once, and dies when you press it three times. Easy to misread as "user clicked too fast, add a spinner".

It isn't a UX problem. Look at the shape of the old handler:

public function redeploy(string $uuid): void
{
    // ...

    try {
        $context = app(DeploymentContextFactory::class)->make($this->deployment);

        if ($dw->status === WorkloadStatus::Running) {
            $context->runtime->stop($dw);
        }

        // Clone. Install. Build. Restart. All of it. Right here.
        (new DeployWorkloadsStep)->deployOne($context, $dw->refresh());

        $this->dispatch('toast', type: 'success', message: __('Redeployed.'));
    } catch (Throwable $e) {
        report($e);
        $this->dispatch('toast', type: 'error', message: __('Redeploy failed.'));
    }
}

A Livewire action is an HTTP request. So this is a multi-minute git clone and npm build running inside PHP-FPM, holding a worker the entire time. Every extra click is a new request, in a new worker, starting a new concurrent build against the same target machine. Nothing in the code says "one at a time", because HTTP requests are concurrent by nature and nobody told this one otherwise.

The failure mode isn't a timeout, either — it's resource exhaustion on the target. Three simultaneous npm installs on a small node, tarball fetches crawling, everything failing together.

The fix, in three parts

1. Dispatch the job that already existed

The push-webhook path had done this correctly the whole time: receive the event, dispatch a job, return. The UI button just... didn't use it. So the change is mostly deletion:

// Queued, never inline: a git build runs for minutes, and running it
// inside the HTTP request meant every extra click started another
// concurrent build.
AutoDeployWorkloadJob::dispatch($dw->id, [
    'source' => 'ui-redeploy',
    'requested_by' => auth()->id(),
    'requested_at' => now()->toIso8601String(),
]);

$this->dispatch('toast', type: 'info', message: __(
    'Redeploy of ":name" queued — the build runs in the background and this page updates as it progresses.',
    ['name' => $dw->workload->name],
));

A nice side effect of converging on one path: the UI deploy picked up behaviour it had been quietly missing (deploy scripts, in this case) because the queued path went through the proper action class and the inline path didn't. Two code paths for the same operation always drift. The one you exercise less is the one that's wrong.

That trigger payload — source, requested_by, requested_at — costs nothing and pays for itself the first time someone asks "who redeployed prod at 21:44?".

2. Pin the queue in the constructor, not at the call site

A build takes minutes. It must not sit on default in front of a batch of ten-millisecond jobs. That means a dedicated queue — and the queue choice belongs to the job, not to whoever dispatches it:

public function __construct(
    public int $deploymentWorkloadId,
    public array $trigger = [],
) {
    $this->onQueue('deployments');
}

Before this, the webhook did ->onQueue('deployments') at the dispatch site. That works right up until a second dispatch site appears and forgets — and then you have a job sitting on a queue no worker consumes, which is a silent failure. Enqueued successfully, never runs, nothing in the logs.

Same rule as $tries, $backoff, $timeout: the job knows how it needs to run. Encode it once, in the job.

3. One in flight at a time

Dispatching still has to be idempotent-ish from the user's point of view. The cheapest correct guard is to ask the domain, not to add a lock:

/**
 * One deploy per workload at a time. A release still pending or in progress
 * means a build is already running (or queued) — dispatching another would
 * run concurrent builds on the same node.
 */
private function deployInFlight(DeploymentWorkload $dw): bool
{
    $inFlight = $dw->releases()
        ->whereIn('status', [ReleaseStatus::Pending, ReleaseStatus::InProgress])
        ->exists();

    if ($inFlight) {
        $this->dispatch('toast', type: 'info', message: __(
            'A deploy of ":name" is already in progress — wait for it to finish.',
            ['name' => $dw->workload->name],
        ));
    }

    return $inFlight;
}

Worth being honest about the trade-off: this is a check-then-act, so two requests landing in the same millisecond can both pass it. For a human clicking a button that's fine — and if it stops being fine, ShouldBeUnique or a Cache::lock() keyed on the workload is the next step up. I'd rather ship the readable guard that also gives the user an explanation than a lock that silently swallows the second click.

Note the toast is info, not error. "Already running, hang on" isn't a failure. Telling the user what the system is doing is most of what makes a queued action feel as good as an inline one.

Testing it

Queue::fake() plus Livewire's test helper makes the important assertions cheap — and note that the first one asserts both halves: that it was queued at all, and that it landed on the right queue.

it('queues a redeploy on the deployments queue instead of building inline', function () {
    [$dw, $s] = GitScenario::workload('https://github.com/acme/api.git');
    $this->actingAs($s->user);

    Queue::fake();

    Livewire::test(Workloads::class, ['deployment' => $dw->deployment])
        ->call('redeploy', $dw->uuid);

    Queue::assertPushedOn('deployments', AutoDeployWorkloadJob::class,
        fn (AutoDeployWorkloadJob $job) => $job->deploymentWorkloadId === $dw->id
            && $job->trigger['source'] === 'ui-redeploy');
});

it('refuses a second deploy while a release is still in flight', function () {
    [$dw, $s] = GitScenario::workload('https://github.com/acme/api.git');
    $this->actingAs($s->user);

    $dw->releases()->create([
        'image_tag' => 'build-pending',
        'strategy' => 'rolling',
        'status' => ReleaseStatus::InProgress,
    ]);

    Queue::fake();

    Livewire::test(Workloads::class, ['deployment' => $dw->deployment])
        ->call('redeploy', $dw->uuid);

    Queue::assertNothingPushed();
});

assertPushedOn is the one people skip. assertPushed alone passes for a job queued onto a queue nothing consumes.

Authorization moves too

One thing to think through before you convert an inline action to a queued one: the job runs without a session. A webhook has no authenticated user at all — its HMAC signature is the authentication — so a user-centric policy can't be satisfied inside the job.

The resolution is to be explicit about where the gate lives: authorize at every dispatch site (policy check in the Livewire action for the UI, signature plus the per-workload auto-deploy toggle for the webhook), and document in the job's docblock that it deliberately runs in system context. What you must not do is let "the job can't call the policy" quietly become "nothing checks anything".

The rule

If it can take longer than a request should, it doesn't belong in a request. Not because of PHP-FPM timeouts — those are the polite failure. The impolite one is that HTTP gives you unbounded concurrency for free, and heavy work plus unbounded concurrency is a small server falling over on the third click.

Queue it, pin the queue in the job, guard the second dispatch, and tell the user what's happening.