DEV Community: laravel-o11y

Laravel Middleware: Requests, Queued Jobs and the HTTP Client

laravel-o11y — Mon, 12 Oct 2026 09:55:50 +0000

Laravel has middleware in three places that most apps use every day. HTTP middleware wraps each incoming request, job middleware wraps each queued job, and Guzzle middleware wraps each outgoing call made through the HTTP client. The three signatures differ, but each one is a layer that receives the input and a callable for everything below it.

Key takeaways

All three kinds are a function wrapped around "the rest". Each layer gets the thing being processed and a $next. It can run code around the call to $next, or return without making it.
Request and job middleware run on the same class. Both go through Illuminate\Pipeline\Pipeline. HTTP client middleware is Guzzle's own handler stack, which is why its signature looks nothing like the other two.
Job middleware also covers queued listeners, mailables, notifications and broadcast events. They all run through the same pipeline, so RateLimited works on a queued notification exactly as it does on a job.
A job middleware that doesn't call $next counts as a success. Skip, a dontRelease() limiter or your own early return deletes the job, and if it was part of a chain the next link still runs.
Http::globalMiddleware() only sees Laravel's client. SDKs that build their own Guzzle client need the middleware pushed onto their stack too. Faked requests do pass through it, so it can be tested with Http::fake().
On a Livewire page, route middleware mostly runs once. Component actions only re-run middleware on Livewire's persistent list, which includes auth and can but not verified, throttle or your own classes. Filament's isPersistent flag adds to the same list.

What middleware is

Middleware is code that runs between a caller and a handler, and can change the input or stop it before the handler sees it. The handler might be a controller, a job's handle() method or the cURL call at the bottom of Guzzle. The middleware is given only the input and a callable that runs everything below it, so the same pattern works for all of them.

Each middleware is a layer. The input passes inwards through every layer to the handler, and the result passes back out through the same layers in reverse, so a layer can run code before the handler and after it:

public function handle($input, Closure $next)
{
    // on the way in: inspect or change $input, or return early

    $result = $next($input);

    // on the way out: inspect or change $result

    return $result;
}

If a layer returns without calling $next, nothing inside it runs. An authentication middleware uses this to turn away a guest before the controller is constructed, and a rate limiter uses it to put a job back on the queue before the job calls the API.

Why use it

Some code has to run for many handlers but isn't part of what any of them does. Every admin route needs an authenticated admin, and every job that calls a vendor API has to stay under the vendor's rate limit. Written inline, that check is copied into dozens of controllers and jobs, and sooner or later one of them is missed.

As middleware, the check is written once and attached by name. The controller keeps what the endpoint does, with nothing about who may call it, and the job keeps its work, with nothing about pacing. The order is written down too: the session starts before authentication, and authentication runs before authorisation, in a list you can read in one place.

The cost is that you can't see middleware from the handler. If a middleware rewrites input or drops a job, the cause is in a file the reader of the controller never opens. Most of the gotchas later in this post are of that kind.

How Laravel runs it

Request and job middleware both run on Illuminate\Pipeline\Pipeline, which you can also use directly. It takes an object, a list of stages, and a final callback, and folds the stages into nested closures so the first stage in the list is the outermost layer:

use Illuminate\Support\Facades\Pipeline;

$order = Pipeline::send($order)
    ->through([
        ApplyCoupon::class,
        CalculateTax::class,
        ReserveStock::class,
    ])
    ->then(fn (Order $order) => $order->save());

Each stage is a class with a handle($passable, Closure $next) method, resolved from the container, or a closure with the same signature. The HTTP kernel builds exactly this with the request as the passable and the router as the final callback. The queue worker builds it with the job as the passable and the job's handle() as the final callback.

Guzzle middleware is built differently. It is a function that takes the next handler and returns a new handler, and a handler takes a PSR-7 request and options and returns a promise. The layering is the same, but code that runs after the handler goes in a then() on the promise:

Kind	Signature	Attached with
Request	`handle(Request $request, Closure $next)`, on `Pipeline`	`bootstrap/app.php`, `->middleware()` on routes, controllers
Job	`handle(object $job, Closure $next)`, on `Pipeline`	The job's `middleware()` method, or `->through()`
HTTP client	`fn (callable $handler): callable`, on Guzzle's `HandlerStack`	`Http::withMiddleware()`, `Http::globalMiddleware()`

Request middleware

A request middleware receives the Illuminate\Http\Request and a $next, and must return a response, either its own or the one $next produced. Generate one with php artisan make:middleware:

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Context;
use Illuminate\Support\Str;
use Symfony\Component\HttpFoundation\Response;

class AssignRequestId
{
    public function handle(Request $request, Closure $next): Response
    {
        $id = $request->header('X-Request-Id') ?: (string) Str::uuid();

        Context::add('request_id', $id);

        $response = $next($request);

        $response->headers->set('X-Request-Id', $id);

        return $response;
    }
}

Context::add() puts the id on every log line written during the request, and on every job dispatched from it, since Laravel carries context into the job payload. The response header lets a client quote the id back to you when something goes wrong.

Where it gets attached

Request middleware can be declared in four places: on every request, in a group, on a route, or on a controller. The place decides which requests the middleware sees, and together they decide the order it runs in. Since Laravel 11 the first two are configured in bootstrap/app.php.

Global middleware runs before the router has matched a route, so it also sees 404s and requests with no matching route. That also means it runs too early to read the session, the logged-in user or route parameters.

->withMiddleware(function (Middleware $middleware) {
    $middleware->append(AssignRequestId::class);  // after Laravel's defaults
    $middleware->prepend(BlockBadBots::class);    // before them
    $middleware->remove(ConvertEmptyStringsToNull::class);
    $middleware->replace(TrustProxies::class, CloudflareTrustProxies::class);
})

$middleware->use([...]) replaces Laravel's global list with your own.

A group is a named list of middleware. Laravel defines two: web (cookies, session, CSRF and route model binding) and api (binding, plus throttling if you turn it on). withRouting() wraps routes/web.php in the first and routes/api.php in the second, which is why you never see them in your route files. You can change either group or define your own:

->withMiddleware(function (Middleware $middleware) {
    $middleware->web(append: [SetLocale::class]);
    $middleware->api(prepend: [EnsureApiVersion::class]);

    $middleware->group('partner', [
        AuthenticatePartner::class,
        'throttle:partner',
    ]);
})

appendToGroup(), prependToGroup(), removeFromGroup() and replaceInGroup() do the same for any group by name.

Routes take middleware by class name or by an alias registered in bootstrap/app.php. Groups of routes pass theirs down to every route inside, and a route adds its own on top:

// bootstrap/app.php
$middleware->alias(['tenant' => IdentifyTenant::class]);

// routes/web.php
Route::middleware(['auth', 'tenant'])->group(function () {
    Route::get('/invoices', [InvoiceController::class, 'index']);

    Route::post('/invoices/{invoice}/send', SendInvoice::class)
        ->middleware('role:billing')
        ->withoutMiddleware('tenant');
});

Parameters go after a colon, so role:editor,admin calls handle($request, $next, 'editor', 'admin'). withoutMiddleware() takes a middleware back off a route or a group of routes. It works for route and group middleware, including subclasses of the class you name, but not for the global stack.

Laravel 13 added the #[Middleware] attribute, which goes on a controller class or on one of its methods. only and except limit a class-level attribute to some methods, and #[WithoutMiddleware] removes one:

use Illuminate\Routing\Attributes\Controllers\Middleware;
use Illuminate\Routing\Attributes\Controllers\WithoutMiddleware;

#[Middleware('auth')]
#[Middleware('verified', except: ['show'])]
class InvoiceController extends Controller
{
    #[Middleware('throttle:exports')]
    public function export(Invoice $invoice)
    {
        // ...
    }

    #[WithoutMiddleware('auth')]
    public function show(Invoice $invoice)
    {
        // ...
    }
}

The older way, which still works, is to implement HasMiddleware and return the list from a static method:

use Illuminate\Routing\Controllers\HasMiddleware;
use Illuminate\Routing\Controllers\Middleware;

class InvoiceController extends Controller implements HasMiddleware
{
    public static function middleware(): array
    {
        return [
            'auth',
            new Middleware('verified', except: ['show']),
        ];
    }
}

The order it runs in

A request goes through two pipelines. The kernel runs the global middleware, then hands the request to the router, which matches a route and runs that route's middleware around the controller. The route's list is put together in four steps:

Collect. Group middleware comes first, from the outermost group inward, so web leads for anything in routes/web.php. Then the route's own ->middleware(), then the controller's: HasMiddleware entries, then attributes, with a parent class's before a child's and the class's before the method's.
Expand. Aliases become class names and group names become their contents.
Remove. Anything named in withoutMiddleware() or #[WithoutMiddleware] is dropped, and so is any repeat of a middleware already in the list.
Sort. The list is reordered against the priority list, described below.

The global list is never sorted. It runs exactly as written: prepends, then Laravel's defaults, then appends. On the way back the response passes through every layer in reverse, so the last middleware to run before the controller is the first to see the response.

The priority list holds the middleware that depend on each other, in the order they have to run: HandlePrecognitiveRequests, EncryptCookies, AddQueuedCookiesToResponse, StartSession, ShareErrorsFromSession, anything implementing AuthenticatesRequests, ThrottleRequests, ThrottleRequestsWithRedis, AuthenticatesSessions, SubstituteBindings and Authorize. When two of these appear out of order, the later one is moved up in front of the earlier one. Middleware that isn't in the list keeps the position you wrote it in.

For the POST /invoices/{invoice}/send route above, after the global stack, that gives:

EncryptCookies               web group
AddQueuedCookiesToResponse   web group
StartSession                 web group
ShareErrorsFromSession       web group
PreventRequestForgery        web group
Authenticate                 'auth', moved up by the priority list
SubstituteBindings           web group
role:billing                 the route's own
                             ('tenant' removed by withoutMiddleware)

auth was written after the whole web group, but it runs before SubstituteBindings. A guest gets the login redirect and doesn't learn whether invoice 42 exists.

Your own middleware goes into the list when it has to run before or after one of these. A tenant middleware usually has to run before route model binding, so that {invoice} is looked up in the tenant's data:

$middleware->prependToPriorityList(
    before: SubstituteBindings::class,
    prepend: IdentifyTenant::class,
);

appendToPriorityList(after: ..., append: ...) does the opposite, and priority([...]) replaces the whole list.

A middleware with a terminate($request, $response) method gets called again after the response has been sent to the browser. Route middleware is terminated before global middleware, each in the order it ran rather than in reverse. Each call gets a fresh instance unless you bind the class as a singleton, so state set in handle() is gone by then.

What it is used for

Most request middleware decides who may make the request: authentication, roles and permissions, signed URLs, verified email, API token abilities and IP allow-lists. The next biggest group sets something up for the rest of the request, such as the current tenant, the locale or a request id. The rest either cleans up the request (trusting proxy headers, trimming strings, rejecting oversized bodies) or changes the response, by adding security headers, cache headers and ETags, or by serving a cached copy of the whole page. Throttling, honeypots and idempotency keys for payment endpoints are middleware too.

Laravel's built-in request middleware

A new Laravel 13 app runs these on every request, in this order:

Global middleware	What it does
`ValidatePathEncoding`	Rejects a path that isn't valid UTF-8 with a `MalformedUrlException`.
`InvokeDeferredCallbacks`	Runs the callbacks registered with `defer()` after the response is sent.
`TrustHosts`	Only when enabled with `$middleware->trustHosts()`. Rejects requests for a `Host` you don't serve.
`TrustProxies`	Reads the client IP, scheme and host from `X-Forwarded-*` headers set by proxies you trust.
`HandleCors`	Answers preflight requests and adds CORS headers from `config/cors.php`.
`PreventRequestsDuringMaintenance`	Serves the 503 page while `php artisan down` is in effect.
`ValidatePostSize`	Rejects a body larger than PHP's `post_max_size`.
`TrimStrings`	Trims whitespace from input, except password fields.
`ConvertEmptyStringsToNull`	Turns empty input strings into `null`.

The web group adds EncryptCookies, AddQueuedCookiesToResponse, StartSession, ShareErrorsFromSession, PreventRequestForgery and SubstituteBindings. The api group has only SubstituteBindings, plus Sanctum's EnsureFrontendRequestsAreStateful after $middleware->statefulApi() and throttle:api after $middleware->throttleApi().

PreventRequestForgery is new in Laravel 13 and replaces ValidateCsrfToken, which remains as a deprecated subclass. It lets a state-changing request through when the browser's Sec-Fetch-Site header says same-origin, and falls back to the CSRF token otherwise. allowSameSite extends that to sibling subdomains, and originOnly drops the token fallback altogether.

These are registered as aliases for use on routes:

Alias	Class	What it does
`auth`	`Authenticate`	Requires a logged-in user, optionally for a given guard: `auth:sanctum`.
`auth.basic`	`AuthenticateWithBasicAuth`	HTTP Basic authentication against your user provider.
`auth.session`	`AuthenticateSession`	Logs out other sessions when the password changes.
`cache.headers`	`SetCacheHeaders`	Sets `Cache-Control` and optionally an ETag: `cache.headers:public;max_age=3600;etag`.
`can`	`Authorize`	Runs a gate or policy ability: `can:update,post`.
`guest`	`RedirectIfAuthenticated`	Sends logged-in users away from login and register pages.
`password.confirm`	`RequirePassword`	Asks for the password again before a sensitive action.
`precognitive`	`HandlePrecognitiveRequests`	Runs validation only, for live form validation with Precognition.
`signed`	`ValidateSignature`	Rejects a URL whose signature is missing, wrong or expired.
`throttle`	`ThrottleRequests`	Rate limits by a named limiter or inline numbers: `throttle:uploads`, `throttle:60,1`. Becomes `ThrottleRequestsWithRedis` after `throttleWithRedis()`.
`verified`	`EnsureEmailIsVerified`	Requires a verified email address.

A few more ship in the framework but aren't registered anywhere, so you attach them by class name. FrameGuard sets X-Frame-Options: SAMEORIGIN. CheckResponseForModifications answers a matching If-None-Match with a 304. AddLinkHeadersForPreloadedAssets turns the assets Vite preloads into Link headers, so the browser can start fetching them before it has parsed the HTML. Laravel 13 also has PrefersJsonResponses, enabled with ->prefersJsonResponses() on the application builder, which treats a request with no Accept header, or only wildcards, as asking for JSON. That suits an API whose clients forget the header and then get an HTML error page back.

Popular request middleware packages

Package	Middleware it adds
spatie/laravel-csp	`AddCspHeaders`, a Content Security Policy built from preset classes, with nonces.
spatie/laravel-responsecache	`CacheResponse`, `DoNotCacheResponse` and the newer `FlexibleCacheResponse`, which store whole responses.
spatie/laravel-honeypot	`ProtectAgainstSpam`, which rejects forms filled in by bots.
bepsvpt/secure-headers	`SecureHeadersMiddleware`, for HSTS, CSP, Permissions-Policy and the rest from one config file.
spatie/laravel-http-logger	`HttpLogger`, which logs incoming requests, by default only the ones that change data.
monicahq/laravel-cloudflare	A `TrustProxies` that keeps Cloudflare's IP ranges up to date, so `$request->ip()` is the visitor's.
akaunting/laravel-firewall	`firewall.ip`, `firewall.sqli`, `firewall.xss`, `firewall.all` and a dozen more, as a basic application firewall.
square1/laravel-idempotency	`IdempotencyMiddleware`, which replays the stored response for a repeated `Idempotency-Key`.

Two of the most installed middleware packages are now in core and shouldn't be added to a new app. fruitcake/laravel-cors is marked abandoned and was replaced by HandleCors in Laravel 9.2, and fideloper/proxy by the framework's TrustProxies in Laravel 9. If an upgrade still has either in composer.json, remove it. We wrote up what responsecache's cache key does with tracking parameters in why our response cache kept missing.

Job middleware

Job middleware wraps one attempt at a queued job. The worker builds a pipeline from the job's middleware with the job's handle() at the centre, so the middleware runs on the worker, every time the job is picked up, retries included. Return the list from a middleware() method, or attach it at dispatch time with through():

use Illuminate\Queue\Middleware\RateLimited;
use Illuminate\Queue\Middleware\WithoutOverlapping;

class SyncInvoiceToXero implements ShouldQueue
{
    public function __construct(public Invoice $invoice) {}

    public function middleware(): array
    {
        return [
            new RateLimited('xero'),
            (new WithoutOverlapping($this->invoice->account_id))->expireAfter(120),
        ];
    }
}

// or for one dispatch only
SyncInvoiceToXero::dispatch($invoice)->through([new RateLimited('xero')]);

Instead of calling $next($job), a job middleware can call $job->release($seconds) to put the job back on the queue for later, $job->fail($e) to send it to the failed jobs table, or $job->delete() to drop it. Writing one takes a class with a handle() method. This one stops calling a vendor for a minute after a call fails:

namespace App\Jobs\Middleware;

use Closure;
use Illuminate\Support\Facades\Cache;
use Throwable;

class PauseWhenVendorIsDown
{
    public function __construct(private string $vendor) {}

    public function handle(object $job, Closure $next): void
    {
        if (Cache::has("vendor-down:{$this->vendor}")) {
            $job->release(60);

            return;
        }

        try {
            $next($job);
        } catch (Throwable $e) {
            Cache::put("vendor-down:{$this->vendor}", true, 60);

            throw $e;
        }
    }
}

It also runs on listeners, mail and notifications

Anything Laravel queues goes through the same pipeline. A queued listener can define middleware(OrderShipped $event), a queued mailable middleware(), and a queued notification middleware(object $notifiable, string $channel), which is called once per channel. That last one is useful for limiting only the channel that has a limit:

public function middleware(object $notifiable, string $channel): array
{
    return match ($channel) {
        'vonage' => [new RateLimited('sms')],
        default => [],
    };
}

What it is used for

The common case is someone else's limit. A vendor allows 60 calls a minute, so every job that calls it shares a limiter. Close behind is mutual exclusion: two jobs updating the same account must not interleave, so they take a lock keyed by the account. Job middleware also backs off from a failing dependency, for example pausing for ten minutes after ten exceptions in a row rather than spending every retry. It drops work that no longer matters, such as a job whose order or batch was cancelled. And it sets up context before the job runs, such as the tenant database or the locale.

Laravel's built-in job middleware

All of these live in Illuminate\Queue\Middleware.

Middleware	What it does	Options
`RateLimited`	Applies a limiter defined with `RateLimiter::for()` and releases the job when the limit is reached.	`releaseAfter()`, `dontRelease()`
`RateLimitedWithRedis`	The same, using Redis directly rather than the cache store.	As above, plus `connection()`
`WithoutOverlapping`	Takes a cache lock on a key so only one job with that key runs at a time.	`releaseAfter()`, `dontRelease()`, `expireAfter()`, `shared()`
`ThrottlesExceptions`	After a number of exceptions, releases the job for a cool-down instead of running it.	`backoff()`, `by()`, `byJob()`, `when()`, `deleteWhen()`, `failWhen()`, `report()`
`ThrottlesExceptionsWithRedis`	The same, counting in Redis.	As above, plus `connection()`
`Skip`	Deletes the job without running it when a condition holds.	`Skip::when()`, `Skip::unless()`
`Release`	New in Laravel 13. Puts the job back on the queue without running it when a condition holds.	`Release::when($condition, $seconds)`, `Release::unless()`
`FailOnException`	Fails the job straight away, without retries, on the exceptions you list.	An array of classes, or a closure
`SkipIfBatchCancelled`	Returns without running the job when its batch has been cancelled.	None

Skipped jobs count as done, and releases use up tries

Not calling $next is a success. When the pipeline returns without the job being released or failed, the worker deletes it as completed. That is what Skip and SkipIfBatchCancelled do, and also what RateLimited and WithoutOverlapping do after dontRelease(). In a chain, a skipped link, or one whose middleware simply returns, is treated as done and the next link is dispatched. If the rest of the chain depends on the skipped work, fail the job instead. A ShouldBeUnique link works the other way round, since a held lock skips it and silently ends the rest of the chain.

Every release spends an attempt. The worker counts the attempt when it picks the job up, before any middleware runs. A job that keeps meeting a held WithoutOverlapping lock or an exhausted limiter is released again and again, and can reach its $tries and fail without handle() having run once. Give limited jobs a retryUntil() deadline rather than a small number of tries. We cover the limiter options in depth in rate-limited APIs and Laravel queues, the locking ones in ShouldBeUnique, WithoutOverlapping and their gotchas, and how releases and retryUntil() bend a job's backoff in Laravel job retries and backoff with unique locks.

Popular job middleware packages

The core classes cover most needs, so there are few job middleware packages with real usage.

Package	What it adds
spatie/laravel-rate-limited-job-middleware	A Redis-backed `RateLimited` with a fluent `allow(30)->everySeconds(60)` API. It predates core's `RateLimited`, which now overlaps with it.
harris21/laravel-fuse	`CircuitBreakerMiddleware`, a circuit breaker with closed, open and half-open states. While open it releases jobs rather than failing them, and its failure classifier keeps 429s and auth errors from tripping it. Still pre-1.0.

Multi-tenancy packages make jobs tenant-aware too, but not through job middleware. spatie/laravel-multitenancy uses a TenantAware interface and stancl/tenancy a queue bootstrapper, so the tenant is restored before your middleware runs.

We maintain a small one ourselves. Unique Job Middleware checks a ShouldBeUnique job's lock again when a worker picks it up, because Laravel only checks it at dispatch and chains, batches and retries go around that check.

HTTP client middleware

Laravel's Http facade builds a Guzzle client for each request, and lets you add Guzzle middleware to it. There are three ways in, each available per request or for the whole application:

use Illuminate\Support\Facades\Context;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Log;
use Psr\Http\Message\RequestInterface;
use Psr\Http\Message\ResponseInterface;

// change the outgoing request
Http::withRequestMiddleware(
    fn (RequestInterface $request) => $request->withHeader('X-Trace-Id', Context::get('request_id'))
)->get('https://api.example.com/orders');

// look at or change the response
Http::globalResponseMiddleware(function (ResponseInterface $response) {
    Log::debug('vendor responded', ['status' => $response->getStatusCode()]);

    return $response;
});

// a full Guzzle middleware, which sees the request, the response and the failure
Http::globalMiddleware(new LogOutgoingRequests);

The first two are shortcuts for Guzzle's Middleware::mapRequest() and mapResponse(). The third takes any Guzzle middleware, which is what you need for anything that has to see a request that never got a response, such as a timeout. A full one looks like this:

class SignWithHmac
{
    public function __construct(private string $secret) {}

    public function __invoke(callable $handler): callable
    {
        return function (RequestInterface $request, array $options) use ($handler) {
            $signature = hash_hmac('sha256', (string) $request->getBody(), $this->secret);

            return $handler($request->withHeader('X-Signature', $signature), $options)
                ->then(function (ResponseInterface $response) {
                    // runs once the response has arrived
                    return $response;
                });
        };
    }
}

What it is used for

Authentication is the most common job: adding a bearer token and refreshing it when it expires, or signing the request with OAuth 1, HMAC or AWS SigV4. Logging and timing each call is next, often with a trace id passed on to the vendor. Then there is retrying on a 503 while honouring Retry-After, limiting how fast you call an API, and stopping calls to a vendor that keeps failing. Caching middleware follows the vendor's Cache-Control so an unchanged resource isn't fetched twice, and in tests a history middleware records each request to assert on.

Built-in HTTP client middleware

Laravel doesn't ship Guzzle middleware classes of its own. What it ships is the registration API above, plus a retry() method that is implemented outside the stack: Http::retry(3, 100) sends the request through the whole stack again for each attempt, so your middleware sees every one. Guzzle itself has these, on GuzzleHttp\Middleware:

Middleware	What it does
`httpErrors()`	Throws on 4xx and 5xx responses. In the default stack, but Laravel's client turns it off with `http_errors => false` and leaves throwing to `->throw()`.
`redirect()`	Follows redirects according to `allow_redirects`. In the default stack.
`cookies()`	Sends and stores cookies through a cookie jar. In the default stack.
`prepareBody()`	Sets `Content-Length` and `Content-Type` for the body. In the default stack.
`auth()`	Guzzle 8 only. Handles authentication challenges, and sits in the default stack after `redirect()`.
`retry($decider, $delay)`	Retries when your decider callback says so, with your delay.
`log($logger, $formatter)`	Logs requests and responses through a PSR-3 logger and a `MessageFormatter`.
`history($container)`	Appends each request, response and error to an array.
`tap($before, $after)`	Calls a function before and after each request without changing it.
`mapRequest()`, `mapResponse()`	Transform the request or the response with a function.

Laravel pushes its own handlers after yours, and the one that serves Http::fake() responses is the innermost. So your middleware runs for faked requests too, and a test can assert that the signature header was added without a real server.

Http::globalMiddleware() only reaches requests made through Laravel's client. SDKs that create their own GuzzleHttp\Client, the AWS SDK among them and with it every S3 call, never see it. For those, push the same middleware onto a HandlerStack and give the SDK a client built on it. Instrumenting Guzzle in Laravel walks through that, along with why a logging middleware should record from on_stats rather than from the response.

Popular HTTP client middleware packages

Most of these are plain Guzzle middleware, written for any PHP project, and plug into Laravel through Http::withMiddleware() or globalMiddleware(). Laravel 13 accepts Guzzle 7 or 8 and Laravel 12 only Guzzle 7. Some packages' latest major needs Guzzle 8, so on Laravel 12 you stay on their previous one.

Package	What it adds
kevinrob/guzzle-cache-middleware	`CacheMiddleware`, HTTP caching that follows `Cache-Control`, with a `LaravelCacheStorage` adapter for your cache store.
guzzlehttp/oauth-subscriber	`Oauth1`, OAuth 1.0 request signing. The current release needs Guzzle 8.1.
caseyamcl/guzzle_retry_middleware	`GuzzleRetryMiddleware`, retries on 429 and 503 that honour `Retry-After`. Version 3 needs Guzzle 8, so use 2.x on Laravel 12.
kamermans/guzzle-oauth2-subscriber	`OAuth2Middleware`, fetching and refreshing OAuth 2 tokens for client-credentials and other grants.
spatie/guzzle-rate-limiter-middleware	`RateLimiterMiddleware::perSecond(3)` or `perMinute()`, which delays calls to stay under a limit.
ackintosh/ganesha	`GuzzleMiddleware`, a circuit breaker with Redis, Memcached and APCu storage.
bilfeldt/laravel-http-client-logger	An `Http::log()` macro and a `LoggingMiddleware` for Laravel's client.

Two logging middlewares that still appear in older answers, gmponos/guzzle_logger and rtheunissen/guzzle-log-middleware, haven't had a release since 2022 and 2023. For metrics rather than log lines, our free httptheus is a Guzzle middleware that exports request durations and failures by host to Prometheus, and Requizon uses the same hook to keep a browsable record of every call.

Livewire and Filament middleware

Livewire doesn't add a new kind of middleware. It uses ordinary request middleware, but changes where it runs. A Livewire page is rendered by a normal route, so that route's middleware runs on the first load as usual. After that, every click, form submit and wire:model update is a POST to one shared endpoint, /livewire-{hash}/update in Livewire 4, which has only the web group on it. The page's own route middleware doesn't run on those requests unless it is persistent.

What persistent middleware is

Persistent middleware is route middleware that Livewire runs again on each component update, as if the update were a request to the page's original route. Livewire saves the page's path and method in every component's snapshot. On an update it builds a copy of the request with that path, looks up the middleware on the matching route, and runs the ones that are also on its persistent list.

The default list holds Authenticate (the auth alias), Authorize (can), SubstituteBindings, AuthenticateWithBasicAuth, Sanctum's EnsureFrontendRequestsAreStateful, Jetstream's AuthenticateSession, and the App\Http\Middleware versions of Authenticate and RedirectIfAuthenticated. So a user who is logged out, or whose can: ability is revoked, while a page is open is stopped at their next click. Anything else on the route, such as verified, password.confirm, throttle or your own middleware, protects the page load and none of the actions taken on the page. Add a class to the list in a service provider:

use App\Http\Middleware\EnsureTeamIsActive;
use Illuminate\Auth\Middleware\EnsureEmailIsVerified;
use Livewire\Livewire;

public function boot(): void
{
    Livewire::addPersistentMiddleware([
        EnsureEmailIsVerified::class,
        EnsureTeamIsActive::class,
    ]);
}

For middleware that should run on every update whatever page it came from, Livewire lets you replace the update route with Livewire::setUpdateRoute() and put middleware on it. That middleware runs before any component is restored, which is why tenancy packages use it: the tenant database has to be connected before Livewire loads the component's models.

Livewire::setUpdateRoute(function ($handle, $path) {
    return Route::post($path, $handle)
        ->middleware(['web', InitializeTenancyByDomain::class]);
});

Where persistent middleware behaves differently

None of these is a bug, but each one can leave a check out without any error.

It has to be on the route and on the list. Adding a class to the list does nothing for pages whose route doesn't have it, and a class on the route does nothing on updates until it is on the list.
The list matches class names. Livewire compares it with the route's middleware after aliases and groups are expanded, so a closure or a group name never matches. Parameters come from the route, so can:update,post works once Authorize is listed.
Only the "before" half counts. The middleware runs on a copy of the request, after the real web group, and the pipeline ends in an empty response that is thrown away. Headers or cookies set after $next never reach the browser. A redirect it returns is turned into an abort() with that response.
Livewire::test() skips it. Persistent middleware only runs on requests to the real update endpoint, so a component test passes whether or not a class is on the list. A browser test or a feature test that posts to the endpoint will catch a missing entry.
The endpoint moved in Livewire 4. It changed from /livewire/update to /livewire-{hash}/update, with the hash taken from your APP_KEY. CSRF exceptions, firewall rules and maintenance-mode exceptions written for livewire/* stop matching after the upgrade.
Locale middleware doesn't need to be listed. Livewire saves the app locale in the snapshot and restores it on each update.

Filament panels

Filament panels are Livewire pages, and the panel provider sets their middleware in three lists. Each takes an isPersistent argument, which passes the classes on to Livewire's list:

public function panel(Panel $panel): Panel
{
    return $panel
        ->middleware([/* the panel's own stack, in place of the web group */])
        ->authMiddleware([Authenticate::class])
        ->tenantMiddleware([ApplyTenantScopes::class], isPersistent: true);
}

Filament's docs say it directly: by default, panel middleware runs when the page is first loaded and not on later Livewire requests. Their tenancy guide uses a middleware that adds global scopes for the current tenant, and that one has to be persistent. Without the flag the scopes are missing on every action taken after the page loads, and a table filter or a bulk action can query other tenants' rows.

Two other details catch people. A panel builds its own stack rather than using your app's web group, so middleware you append to web skips panel page loads, but it does run on the panel's Livewire updates, which go through web. And Filament doesn't rely on middleware to keep users out of pages and resources. It runs canAccess() again from a Livewire hook on every update, so removing a user's access to a page takes effect on their next click.

Livewire and Filament's built-in middleware

Livewire's own middleware is small. RequireLivewireHeaders sits on the update route and returns a 404 to any request without the X-Livewire header. DisableBackButtonCacheMiddleware is pushed onto the global stack and adds no-store headers when Livewire's back-button cache feature is active for the page, so pressing Back reloads it instead of showing a stale copy. Livewire also turns off TrimStrings and ConvertEmptyStringsToNull for its own requests, and its file upload route is throttled with throttle:60,1 by default.

Filament middleware	What it does	Persistent
`Authenticate`	Requires a logged-in user who passes `canAccessPanel()`.	Yes
`AuthenticateSession`	Filament's version of Laravel's `AuthenticateSession`.	Yes
`SetUpPanel`	Makes the panel the current one. Always first in the panel stack.	Yes
`IdentifyTenant`	Resolves the tenant from the URL and returns a 404 unless the user passes `canAccessTenant()`.	Yes
`DispatchServingFilamentEvent`	Fires the `ServingFilament` event, which plugins listen to.	Yes
`DisableBladeIconComponents`	Turns off Blade Icons' components inside the panel.	Yes
Email verification	Added per page when the panel has `->emailVerification()`.	No
`EnsureMultiFactorAuthenticationIsEnabled`	Sends users without MFA to set it up, when the panel requires it.	No

Popular Livewire and Filament packages with middleware

There are no widely used Livewire middleware packages. The most installed rate-limiting package, danharrin/livewire-rate-limiting, is a trait you call from a component action, because a throttle route middleware can't tell one Livewire action from another. Filament's own login and password-reset pages use it. Filament plugins do ship middleware, and whether they register it as persistent varies:

Package	Middleware it adds	Persistent
bezhansalleh/filament-shield	`SyncShieldTenant`, which sets spatie/laravel-permission's team id to the current tenant.	Yes
jeffgreco13/filament-breezy	`MustTwoFactor`, which sends users to the 2FA challenge, added through `authMiddleware()`.	No
bezhansalleh/filament-language-switch	`SwitchLanguageLocale`, which applies the language the user picked.	No, which is fine because Livewire keeps the locale
stephenjude/filament-two-factor-authentication	`TwoFactorChallenge` and `ForceTwoFactorSetup`.	No

The two 2FA plugins check for a completed challenge on page loads only. From reading their source, a page that is already open keeps accepting actions after the 2FA session expires, until the user navigates. If 2FA guards something sensitive, add the plugin's middleware with Livewire::addPersistentMiddleware(). Inertia has none of this to think about: every Inertia visit is a normal request to its route, so the route's middleware always runs.

Laravel AI SDK agent middleware

An agent in Laravel's AI SDK (laravel/ai) runs in steps. Each step sends the conversation to the model, and if the model asks for tools, the SDK runs them and starts another step with the results. Agent middleware wraps each step. It receives a PendingStep, which holds the model, instructions, messages and tools about to be sent, and a $next that calls the model.

This changed in v1.0.0, released on 23 September 2026. Before that, agent middleware wrapped the whole prompt once and received an AgentPrompt, so tutorials and packages written earlier in 2026 use a signature that no longer works. An agent declares its middleware by implementing HasMiddleware:

use App\Ai\Middleware\SendOnlyAfterDraft;
use Laravel\Ai\Contracts\Agent;
use Laravel\Ai\Contracts\HasMiddleware;
use Laravel\Ai\Promptable;

class SupportAgent implements Agent, HasMiddleware
{
    use Promptable;

    public function middleware(): array
    {
        return [new SendOnlyAfterDraft];
    }
}

Generate the middleware with php artisan make:agent-middleware. This one keeps the agent from emailing a customer until it has drafted the email in an earlier step:

namespace App\Ai\Middleware;

use Closure;
use Laravel\Ai\PendingStep;

class SendOnlyAfterDraft
{
    public function handle(PendingStep $step, Closure $next)
    {
        $drafted = collect($step->steps)
            ->flatMap(fn ($done) => $done->toolCalls)
            ->contains(fn ($call) => $call->name === 'DraftEmail');

        if (! $drafted) {
            $step = $step->withoutTools('SendEmail');
        }

        return $next($step);
    }
}

PendingStep is immutable. withModel(), withInstructions(), withMessages(), withTools(), onlyTools(), withoutTools(), withToolChoice(), withMaxTokens() and withProviderOptions() each return a changed copy, and the change applies to this step only. The step also carries the steps already completed and their combined token usage. Code that should run after the model answers goes in $next($step)->then(), which receives the StepResponse before any tools run, for streamed responses too. A middleware can also return a StepResponse of its own instead of calling $next, and the model isn't called for that step.

What it is used for

Logging is the usual first one: the model, the prompt and the tokens each step used. The Laravel docs show two more. One summarises the middle of a conversation once it passes 40 messages, so a long tool-calling run stays inside the context window; the shortened history is sent for that step only and the stored conversation keeps everything. The other drops an expensive search tool after the first step. Beyond the docs, middleware can send later steps to a cheaper model with withModel(), answer repeated prompts from a cache, and act as a guardrail that rejects prompt injection or removes personal data before the messages leave your server.

Built-in agent middleware

The SDK doesn't ship agent middleware for you to attach. The one class it has, RememberConversation, stores the conversation for agents that use RemembersConversations, and the SDK adds it itself in a separate pipeline around the whole prompt.

Agent middleware packages

The SDK is new and so is its package ecosystem. The one package written for the per-step API is promptphp/intercept (v1.0.0 from 27 September 2026). It ships PromptInjectionGuard, PIIRedactor and ToolApprovalGuard. padosoft/laravel-ai-guardrails has input and output guardrail middleware but only installs with the SDK's 0.x releases. Outside the SDK, Prism has no middleware, and NeuronAI has its own agent and workflow middleware built from before() and after() hooks rather than $next.

Other places Laravel uses middleware

Outside the kinds above, middleware turns up in a handful of smaller places:

Where	What it looks like
The command bus	`Bus::pipeThrough([...])` registers pipes that wrap every command dispatched with `dispatchSync()` and every queued job when it runs, inside the job's own middleware. It isn't in the docs, but it's the one place to add something to every job at once. Each call replaces the list, so two packages that both use it overwrite each other.
`Pipeline` itself	The facade is available for your own multi-step processes, such as checkout or import steps, with `send()`, `through()`, `then()` and `withinTransaction()`.
Folio	Page middleware, declared in the page with `middleware(['auth'])` or for a path with `Folio::path()->middleware()`.
Laravel MCP	Ordinary route middleware on the server route, for example `Mcp::web('/mcp', WeatherServer::class)->middleware('throttle:mcp')`. There is no per-tool middleware; a tool's `shouldRegister()` method decides whether a request can see it at all.
Kafka consumers	mateusjunges/laravel-kafka takes `->withMiddleware(fn ($message, $next) => $next($message))` on a consumer, wrapping each message before your handler.
Broadcasting	Route middleware on the channel authorisation endpoint, set in `withBroadcasting()`, for example `auth:sanctum` for an API.

Some places have hooks that look like middleware but can't wrap or stop anything. Queue::before() and Queue::after() are event listeners, scheduled tasks have before(), after() and when(), and Eloquent has observers and scopes. If you need to stop something from running, use one of the pipelines above.

A job dropped by middleware leaves no failed job and no exception, so it is hard to notice. In Skyline, the Locks & Limits screen counts how often each limiter released or dropped a job. Its log also records jobs dropped by dontRelease() and chain links skipped by a unique lock.

Common questions

What kinds of middleware does Laravel have?

Three main kinds. HTTP middleware wraps incoming requests and is configured in bootstrap/app.php, on routes or on controllers. Job middleware wraps each attempt at a queued job, and also runs for queued listeners, mailables, notifications and broadcast events. HTTP client middleware is Guzzle middleware added with Http::withMiddleware() or Http::globalMiddleware(). There are smaller ones too: Bus::pipeThrough() pipes for the command bus, page middleware in Folio, persistent middleware in Livewire and Filament, and agent middleware in the Laravel AI SDK, which wraps each step an agent sends to the model.

What is the difference between global, group and route middleware in Laravel?

Global middleware runs on every request, including ones that match no route, and is added with $middleware->append() or prepend(). Group middleware runs on every route in a group such as web or api, and is added with $middleware->web() or api(). Route middleware runs only where you attach it with ->middleware(), usually by an alias. withoutMiddleware() can remove route and group middleware but not global middleware.

What job middleware does Laravel include?

Nine classes in Illuminate\Queue\Middleware: RateLimited and RateLimitedWithRedis, WithoutOverlapping, ThrottlesExceptions and ThrottlesExceptionsWithRedis, Skip, Release (new in Laravel 13), FailOnException and SkipIfBatchCancelled. Return them from a job's middleware() method or attach them at dispatch time with ->through().

What happens when a Laravel job middleware does not call $next?

If the middleware does not release or fail the job, the worker deletes it as completed. Skip, SkipIfBatchCancelled and RateLimited or WithoutOverlapping with dontRelease() all work this way. In a chain, the next link is still dispatched, so a skipped job does not stop the rest of the chain.

Does Http::globalMiddleware() apply to every outgoing request?

Only to requests made through Laravel's Http facade, including Http::pool() and faked requests under Http::fake(). SDKs that build their own GuzzleHttp\Client, such as the AWS SDK behind the S3 filesystem driver, never pass through it. Push the same middleware onto a Guzzle HandlerStack and give the SDK a client built on that stack.

What is persistent middleware in Livewire?

Route middleware that Livewire runs again on component update requests. After a Livewire page loads, every action posts to one shared update endpoint, so the page route's middleware does not run on it. Livewire re-runs the middleware that is both on the original route and on its persistent list, which by default includes auth, can and SubstituteBindings but not verified, password.confirm, throttle or custom middleware. Add classes with Livewire::addPersistentMiddleware(), or in Filament pass isPersistent: true to middleware(), authMiddleware() or tenantMiddleware().

How does middleware work in the Laravel AI SDK?

An agent that implements HasMiddleware returns middleware classes with a handle(PendingStep $step, Closure $next) method. Since laravel/ai 1.0 the middleware runs once for each generation step, not once per prompt, and can change the model, messages or tools for that step, run code after the model answers with ->then(), or return its own StepResponse to skip the model call. Generate one with php artisan make:agent-middleware.

Laravel Horizon Alerting: What Horizon Sends on Its Own and What Prometheus Adds

laravel-o11y — Wed, 30 Sep 2026 20:06:32 +0000

Horizon will tell you about one thing: a queue whose estimated wait has crossed a threshold. It will not tell you that the wait ended, that jobs are failing, that a supervisor was killed for memory, or that Horizon itself stopped an hour ago. Most teams find this out the usual way, from a customer asking where their invoice is.

This post walks up the layers you can add. It starts with what ships in Horizon, moves to the gauges spatie/laravel-prometheus exports and the alert rules those support, then to a counter-based exporter and the rules that only counters make possible. The last part covers the conditions no scrape carries, which is where Skyline's alerts come in. The rules are the ones we run in production, and for each we say which exporter can and cannot evaluate it.

Key takeaways

Horizon has one notification, LongWaitDetected. It is built on a forecast of time to clear, repeats every five minutes while the condition lasts, never reports recovery, and is evaluated by a supervisor, so it is silent when Horizon is down.
Spatie's Horizon collectors are gauges read at scrape time. They are enough for "Horizon is not running", "Horizon is paused" and "this backlog has not moved". They cannot express a failure rate, and they have nothing per job class.
Counters unlock the rules worth paging on. With processed, failed and retried counters plus the age of the oldest pending job, you can alert on failure ratio, on a backlog nobody is draining, and on latency instead of length.
In Grafana, keep the comparison out of the PromQL. A query like x > 900 returns no series when it is false, and Grafana reads that as No Data. Put the threshold in a threshold expression instead.
Some failures never become a time series. A stranded unique lock, a job dropped by a rate limiter, a reservation that expired mid-run and a timeout above retry_after produce no failures and no backlog. Those need checks that read queue state directly.

What Horizon sends on its own

Horizon's notification is configured in two places. The thresholds live in config/horizon.php, keyed by connection and queue, and the destinations are set in your HorizonServiceProvider:

// config/horizon.php
'waits' => [
    'redis:default' => 60,
    'redis:reports' => 1800,
    'redis:housekeeping' => 0,   // 0 turns the check off for that queue
],

// app/Providers/HorizonServiceProvider.php
Horizon::routeSlackNotificationsTo('https://hooks.slack.com/services/...', '#ops');
Horizon::routeMailNotificationsTo('ops@example.com');
Horizon::routeSmsNotificationsTo('+15555550100');

A queue you do not list gets 60 seconds. Once a minute, one supervisor in the fleet takes a lock, calculates the wait for every queue it knows about, and raises LongWaitDetected for each one over its threshold. That event sends the notification. It is a reasonable first alert, and it is worth knowing exactly how it behaves before relying on it.

The wait is a forecast. It comes from WaitTimeCalculator: ready jobs multiplied by the queue's average runtime, divided by the processes serving it. Ten thousand fast jobs produce a large number and a single job stuck at the head of a queue produces a small one.
It repeats every five minutes. The listener takes a 300-second lock per connection and queue before sending, so a backlog that lasts two hours sends twenty-four messages.
It never says the wait ended. There is no recovery message, so a channel that went quiet might mean the queue drained or might mean someone muted it.
Queues balanced together are one entry. A supervisor working high,default is keyed redis:high,default, and the threshold applies to the pool.
It needs a running supervisor. The check hangs off the supervisor's loop. When Horizon is down there is no loop, and the one alert you have cannot fire during the outage it was meant for.

Events Horizon fires and tells nobody about

Horizon raises several other events at the moments you would want to hear about, with no notification attached. SupervisorOutOfMemory fires when a supervisor exceeds its memory limit and is about to be terminated, MasterSupervisorOutOfMemory does the same for the master, and UnableToLaunchProcess fires when a worker process will not start. You can listen to them yourself:

use Illuminate\Support\Facades\Event;
use Illuminate\Support\Facades\Log;
use Laravel\Horizon\Events\SupervisorOutOfMemory;
use Laravel\Horizon\Events\UnableToLaunchProcess;

Event::listen(SupervisorOutOfMemory::class, function (SupervisorOutOfMemory $event) {
    Log::channel('slack')->critical('Horizon supervisor out of memory', [
        'supervisor' => $event->supervisor->name,
        'memory_mb' => $event->getMemoryUsage(),
    ]);
});

Event::listen(UnableToLaunchProcess::class, function (UnableToLaunchProcess $event) {
    Log::channel('slack')->critical('Horizon could not launch a worker', [
        'command' => $event->process->process->getCommandLine(),
    ]);
});

The same pattern with Laravel's JobFailed event is the alert most teams write first, and the one that gets muted first. One malformed record on a job dispatched four hundred times an hour fills the channel before anyone is awake. A failure alert has to be about a rate, and a rate needs something that counts.

For Horizon being down, the built-in tool is php artisan horizon:status. It exits 0 when Horizon is running, 1 when a master is paused and 2 when no master is reporting, so a cron job or an uptime monitor that runs commands can watch it from outside the fleet.

Gauges from spatie/laravel-prometheus

spatie/laravel-prometheus is a general Prometheus library for Laravel with a set of Horizon collectors. Uncomment $this->registerHorizonCollectors() in the published PrometheusServiceProvider and the /prometheus route serves seven gauges. The default namespace is app, so the series arrive with that prefix.

Series	What it reads
`app_horizon_status`	1 running, 0 if any master is paused, -1 if no master is reporting
`app_horizon_master_supervisors`	How many master supervisors are reporting
`app_horizon_current_workload{queue}`	Ready jobs per supervisor pool
`app_horizon_current_processes{queue}`	Worker processes per supervisor pool
`app_horizon_failed_recent_jobs`	Failures inside the `trim.recent_failed` window, a week by default
`app_horizon_jobs_per_minute`	Horizon's own jobs-per-minute figure for the current snapshot window
`app_horizon_recent_jobs`	Jobs inside the `trim.recent` window

Our production rule file has five Horizon alerts. Four of them only ask a current-value question, which is what a gauge is for, and they port to these series directly:

groups:
  - name: horizon-spatie
    rules:
      - alert: HorizonNotRunning
        expr: app_horizon_status == -1 or absent(app_horizon_status)
        for: 5m
        labels:
          severity: critical

      - alert: HorizonPaused
        expr: app_horizon_status == 0
        for: 15m
        labels:
          severity: critical

      # The backlog is the same size it was 15 minutes ago, and not empty.
      - alert: QueueProcessingStalled
        expr: |
          (app_horizon_current_workload offset 15m == app_horizon_current_workload)
          and (app_horizon_current_workload > 0)
        for: 15m
        labels:
          severity: critical

      # The queue has not reached zero once in 24 hours. The inner max_over_time
      # bridges short gaps in the series left by restarts and deploys.
      - alert: QueueBacklogNotZeroFor24Hours
        expr: |
          min_over_time(
            max_over_time(app_horizon_current_workload[10m])[24h:1m]
          ) > 0
        labels:
          severity: critical

absent() matters in the first rule. If the application is unreachable or the scrape target was removed, the series is gone, and a comparison against a series that does not exist matches nothing.

Two details change compared with the originals. app_horizon_status folds paused and running into one series with no label, so HorizonPaused cannot name which master is paused. And the queue label is the supervisor's queue string: a pool working high,default arrives as one series with queue="high,default", and the workload is the pool's total.

app_horizon_master_supervisors gives you one more rule for free. Set it against the number of servers that should be running Horizon, and it catches both a server that dropped out and the duplicate master tree from our supervisor upgrade postmortem:

- alert: HorizonMasterCountWrong
  expr: app_horizon_master_supervisors != 2   # your server count
  for: 10m

The rule that does not port

Our fifth rule is "more than 20 failed jobs in an hour, sustained for two hours". It is written with increase() over a failure counter, and the Spatie collectors have no counter. app_horizon_failed_recent_jobs is a gauge over a sliding window, so it goes down as old failures leave the window. increase() treats every drop as a counter reset and the result means nothing.

There is a workaround if you can live with its side effect. The gauge counts failures newer than trim.recent_failed minutes, so setting that to 60 turns it into "failures in the past hour":

// config/horizon.php
'trim' => [
    'recent_failed' => 60,   // was 10080
    'failed' => 10080,       // the failed jobs list keeps its week
],

- alert: FailedJobsIncreasing
  expr: app_horizon_failed_recent_jobs > 20
  for: 2h

The side effect is that the dashboard's failed-jobs figure becomes the past hour as well, since it reads the same setting. The figure is also fleet-wide. There is no queue or job class on it, so the alert tells you that something is failing and you go to the dashboard to find out what.

What is missing from the Horizon collectors

There is no age of the oldest pending job, no wait time, no retries and nothing per job class. Spatie's separate queue collectors do export queue_oldest_pending_job_age, but their documentation says not to run them alongside the Horizon collectors. The jobs-per-minute gauge has its own problems as an alert input, covered in the Horizon in Prometheus post. Also set allowed_ips before deploying the route, because an empty list allows everyone.

Counters change which rules you can write

Our Horizon Prometheus Exporter is open source and runs on stock Horizon. Workers count each outcome into Redis, the counts only go up, and /horizon/prometheus serves them. It exports processed, failed and retried counters per queue and per job class, a wait-time summary, and gauges for queue length, oldest pending age, time to clear, processes and paused state. Every queue label names one queue, with the pool in a group label.

composer require boring-o11y/horizon-prometheus-exporter

On this exporter our five production rules run as they are written. We scrape Skyline's endpoint rather than this package, and the two use the same metric names, so the file is the same either way:

groups:
  - name: HorizonAlerts
    rules:
      - alert: HorizonNotRunning
        expr: horizon_up != 1 or absent(horizon_up)
        for: 5m

      - alert: HorizonPaused
        expr: horizon_master_supervisor_paused == 1
        for: 15m

      - alert: QueueProcessingStalled
        expr: |
          (horizon_queue_length{queue!=""} offset 15m == horizon_queue_length{queue!=""})
          and (horizon_queue_length{queue!=""} > 0)
        for: 15m

      - alert: QueueBacklogNotZeroFor24Hours
        expr: |
          min_over_time(
            max_over_time(horizon_queue_length{queue!=""}[10m])[24h:1m]
          ) > 0

      - alert: FailedJobsIncreasing
        expr: sum(increase(horizon_queue_failed_total[1h])) > 20
        for: 2h

HorizonPaused now carries a master label, so the alert names the machine. FailedJobsIncreasing is a real increase() over a counter and needs no change to Horizon's trimming.

Those five were written when gauges were all we had. With counters and the oldest-pending gauge there are better ones, and these are the rules we would add.

Latency instead of length

- alert: QueueBacklogAgeing
  expr: max by (queue) (horizon_queue_oldest_pending_seconds) > 900
  for: 5m

- alert: QueueWaitHigh
  expr: |
    sum by (queue) (rate(horizon_queue_wait_seconds_sum[10m]))
      / sum by (queue) (rate(horizon_queue_wait_seconds_count[10m])) > 120
  for: 10m

The first is how long the job at the head of the queue has been ready to run. It is 0 on an empty queue, small on a fast one however deep it is, and climbs without limit when nothing is consuming. The second is the average wait of the jobs that were picked up, which catches a queue that is moving but slower than it promised. Set both per queue from what the queue is for. A password reset is late at a minute and a nightly export is fine at half an hour.

A backlog nobody is draining

- alert: QueueNotMoving
  expr: |
    max by (queue) (horizon_queue_length) > 0
    unless sum by (queue) (rate(horizon_queue_processed_total[10m])) > 0
    unless max by (queue) (horizon_queue_paused) == 1
  for: 10m

This replaces QueueProcessingStalled. Comparing a length with itself fifteen minutes ago has two failure cases: a busy queue that happens to land on the same number, and a stalled queue that keeps receiving dispatches, whose length changes every minute. Asking whether anything finished has neither. The last clause leaves out queues somebody paused on purpose, which get their own low-severity rule:

- alert: QueueLeftPaused
  expr: max by (queue) (horizon_queue_paused) == 1
  for: 1h
  labels:
    severity: ticket

Failure ratio, per queue and per job class

- alert: QueueFailureRatio
  expr: |
    sum by (queue) (rate(horizon_queue_failed_total[5m]))
      / (
        sum by (queue) (rate(horizon_queue_processed_total[5m]))
        + sum by (queue) (rate(horizon_queue_failed_total[5m]))
      ) > 0.05
    and sum by (queue) (rate(horizon_queue_processed_total[5m])) > 0.1
  for: 10m

- alert: JobClassFailing
  expr: sum by (job_class) (increase(horizon_job_failed_total[15m])) > 10

The and clause is a throughput floor of about six jobs a minute. Without it a queue that ran one job in five minutes and failed it reads as 100 per cent. The per-class rule is there because a ratio per queue dilutes: one class failing every run on a queue shared with twenty others can stay under five per cent indefinitely.

Falling behind, and retry churn

- alert: QueueFallingBehind
  expr: |
    max by (queue) (deriv(horizon_queue_length[10m])) > 0
    and max by (queue) (horizon_queue_time_to_clear_seconds) > 900
  for: 10m
  labels:
    severity: ticket

- alert: JobRetryChurn
  expr: |
    sum by (job_class) (rate(horizon_job_retried_total[15m]))
      / sum by (job_class) (rate(horizon_job_processed_total[15m])) > 1
  for: 30m
  labels:
    severity: ticket

Time to clear is a forecast, and a forecast over a line is a bad page on its own. Paired with a backlog that is still growing it says arrivals are outpacing workers while the fix is still cheap. The retry rule counts jobs released after an exception, so a class retrying more often than it completes is leaning on its tries to get through.

The same rules as Grafana alerts

Everything above is a Prometheus rule file, evaluated by Prometheus and routed by Alertmanager. Grafana lists those rules read-only under its alerting pages. If you would rather have Grafana evaluate them, each one becomes a Grafana-managed rule with the PromQL as the query and for as the pending period.

One thing needs to change on the way over. A PromQL comparison filters: horizon_queue_oldest_pending_seconds > 900 returns no series at all when every queue is healthy, and Grafana reports that as No Data. Query the bare metric and put the 900 in a threshold expression, so the rule always has a value to evaluate. Then set "Alert state if no data" to Alerting on the horizon_up rule, which does the job absent() does in the rule file.

Whichever evaluates them, scrape one web server. Every instance reads the same Redis, so three targets return the same series three times and every sum() triples.

Where a scrape runs out

A scrape carries numbers that exist for every queue or job class at every moment. That shape fits throughput, backlog and latency well. It fits badly when the fault is a fact about one lock, one job or one line of config.

Take a job class that has silently stopped running because a unique lock outlived the worker that held it. Every dispatch is skipped. Nothing fails, so the failure ratio is flat. Nothing is queued, so length and age are zero. Throughput for that class drops to nothing, which looks the same as a quiet afternoon. You could write an "expected throughput" rule per class, and you would be maintaining a list of what each job ought to be doing.

The other gap is detail. A rule can say the failure ratio on default is eight per cent. It cannot say which lock, which limiter or which supervisor setting is responsible, because those were never labels.

The alerts Skyline adds

Skyline is our fork of Horizon, and its alert pipeline runs inside the application against the same Redis. It covers the ground above without a metrics stack: Horizon down, oldest job age per queue, a queue with workers and no completions, a backlog that keeps growing, a pause nobody resumed and a failure rate per queue or per job class. If you have built the Prometheus rules you already have those, and Skyline's endpoint exports the same metric names, so the rule file keeps working.

The checks below are the ones the rule file has no equivalent for. They read lock owners, limiter buckets, worker exit reasons and supervisor config, none of which either exporter publishes.

Check	Fires when	Why the metrics miss it
`stranded_lock`	A unique or overlap lock is still held and the job that took it is gone	No failures and no backlog; the dispatches are skipped
`limiter_drop`	Rate-limit, overlap or exception-throttle middleware dropped or deleted jobs	A dropped job is recorded as completed
`reservation_expired`	A job was still reserved when `retry_after` ran out, so it will run twice	Both runs succeed and count as processed
`misconfiguration`	A supervisor's `timeout` is not below its connection's `retry_after`	It is config, and nothing has gone wrong yet
`job_timeout`	One job class keeps timing out	A timed-out attempt is retried, and only counts as failed when attempts run out
`worker_crash_loop`	Workers die without reporting a reason, faster than a threshold	The supervisor replaces them, so the process count never drops
`memory_restarts`	Workers keep being recycled for their memory limit	Nothing fails; throughput sags while workers boot
`process_launch_failed`	A worker process would not start	A single event between two scrapes
`supervisor_out_of_memory`, `master_out_of_memory`	A supervisor or the master exceeded its memory limit and was restarted	A single event between two scrapes

Work that vanishes without failing

stranded_lock is the case from the previous section. Skyline records which job holds each ShouldBeUnique and WithoutOverlapping lock for its Locks & Limits screen, so the check can tell a lock whose job is still queued or running from one whose job no longer exists. After five minutes in that state it alerts with the job class, the lock key, how long it has been held, when it expires and how many dispatches it has skipped. How locks get stranded in the first place is in the ShouldBeUnique post.

limiter_drop watches the middleware that discards jobs by design: RateLimited with dontRelease(), WithoutOverlapping without releaseAfter(), and ThrottlesExceptions with deleteWhen(). Each leaves the job marked completed, so the processed counter goes up for work that did not happen. The check fires at ten drops in an hour for one limiter or job class. Some applications drop on purpose, a "latest wins" sync for example, so each limiter takes its own threshold or a zero to leave it out:

'limiter_drop' => [
    'threshold' => 10,
    'circuit_open' => 50,   // also alert when an exception throttle holds back 50 jobs an hour
    'groups' => [
        'App\\Jobs\\SyncLatestPrices' => 0,
        'rate_limiter:uploads' => 200,
    ],
],

Jobs that run twice

When a job is still running at the moment its reservation expires, the queue hands it to a second worker. Both runs complete, both count as processed, and the customer is charged twice. reservation_expired reports it once per queue with a count and the job classes involved, at critical severity, because it is a correctness problem and not a slow one.

misconfiguration is the same fault before it happens. A supervisor whose timeout is at or above its connection's retry_after will produce a double run as soon as one job runs long enough. Stock Horizon accepts that config without comment. Skyline prints a warning when it starts, where nobody is looking after a deploy that went fine, so the check also alerts with the supervisor, the connection and both values, repeats once a day, and resolves when a deploy picks up the corrected config. The timeout vs retry_after post explains why the two have to be ordered that way.

Workers dying, by cause

Stock Horizon shows the configured number of processes whether workers are healthy or being replaced every thirty seconds. Skyline records why each worker stopped, and three checks read that record. worker_crash_loop counts workers that died without giving a reason, which means a PHP fatal or a kill from outside, and fires at five in five minutes per supervisor. memory_restarts counts workers recycled for reaching their memory limit and fires at ten in fifteen minutes. One memory restart is the supervisor doing its job, so the crash check leaves those out and this one looks at the rate.

The memory alert also names the job classes that retained the most heap over the window, since Skyline measures heap growth per class. That turns "workers on default keep running out of memory" into a class name to open. Both checks need Insights enabled, which is what records the restarts.

job_timeout fires when one class times out five times in ten minutes. A timed-out job is killed, retried, and often times out again, so the failure rate only sees it once the attempts are spent. By then a slow dependency has been holding workers for an hour. Timeouts are counted per class and per queue, and Skyline exports them as horizon_job_timed_out_total and horizon_queue_timed_out_total if you would rather write that rule in Prometheus.

The last three are the events from the top of this post: process_launch_failed, supervisor_out_of_memory and master_out_of_memory. They report once when the event happens, through the same channels as everything else, in place of the listeners you would otherwise write.

How they are delivered

Every check holds for a for window before notifying, repeats at most every fifteen minutes while it lasts, and sends a recovery message saying how long the condition held. Notifications are withheld for five minutes after a deploy while the timers keep counting. Each alert has a severity, and routes decides which channels a severity goes to, across mail, Slack, SMS and a JSON webhook for PagerDuty or Opsgenie.

HORIZON_ALERTS=true
HORIZON_ALERT_SLACK="https://hooks.slack.com/services/..."
HORIZON_ALERT_WEBHOOK="https://events.example.com/queue-alerts"

Anything specific to your application goes in a class that extends Check and is registered with Horizon::alertCheck(). "No invoices processed since midnight on a weekday" is a rule like that, and it gets the same hold window, cooldown, recovery message and routing as the built-in ones.

Which layer to stop at

If you run no metrics stack, configure LongWaitDetected with a threshold per queue and put horizon:status behind a cron or an uptime check. That covers a growing backlog and Horizon being down, which are the two outages that cost the most.

If you already run Prometheus and Spatie's package, the four gauge rules are worth the ten minutes they take. Once you want a failure rate, or an alert that separates a busy queue from a stuck one, you need counters and the oldest-pending age, and that is the point to add the exporter.

The Skyline checks are for the failures that leave the graphs looking normal. A rule file and those checks do not compete: the rules stay in Prometheus next to your database and host alerts, and the checks cover the conditions that never reach it.

Common questions

Does Laravel Horizon have built-in alerts?

One. Horizon sends a LongWaitDetected notification by mail, Slack or SMS when a queue's estimated wait passes its threshold in horizon.waits, 60 seconds by default. It repeats every five minutes while the wait lasts and sends nothing when it clears. There is no built-in notification for failed jobs, for a supervisor running out of memory, or for Horizon not running.

How do I get alerted when Horizon stops running?

From outside Horizon, because its own notification is evaluated by a running supervisor. Run php artisan horizon:status from cron or an uptime monitor: it exits 0 when running, 1 when paused and 2 when inactive. With Prometheus, alert on app_horizon_status == -1 from spatie/laravel-prometheus or horizon_up == 0 from a counter exporter, and add absent() so a missing series also fires.

Can I alert on failed jobs with spatie/laravel-prometheus?

Only roughly. Its horizon_failed_recent_jobs gauge counts failures inside Horizon's trim.recent_failed window, a week by default, and has no queue or job class label. It is a sliding window and not a counter, so increase() and rate() do not work on it. Setting trim.recent_failed to 60 makes it failures in the past hour, which also changes the figure on Horizon's dashboard. A failure ratio per queue needs an exporter that keeps counters.

Why does my Grafana alert on a Horizon metric show No Data?

Usually because the comparison is inside the PromQL. A query such as horizon_queue_oldest_pending_seconds > 900 returns no series when every queue is under the threshold, and Grafana treats an empty result as No Data. Query the metric without the comparison and add a threshold expression for the 900, so the rule always has a value to evaluate.

Which Horizon problems can Prometheus not alert on?

The ones that produce no failures and no backlog. A unique lock left behind by a dead worker makes every dispatch of that job get skipped. A job dropped by RateLimited with dontRelease() is recorded as completed. A job that outlives retry_after runs twice and both runs count as processed. These need a check that reads lock owners, limiter counts or supervisor config directly, which is what Skyline's alert checks do.

Rate-Limited APIs and Laravel Queues: One Request at a Time, Without Starving the Rest

laravel-o11y — Tue, 29 Sep 2026 14:42:29 +0000

You are integrating with an API that allows sixty calls a minute and no parallel requests at all. Send two at once and it returns a 429, or worse, corrupts the record you were updating. Your queue workers, meanwhile, are built to run as many jobs at once as they can.

Each of Laravel's answers to this problem addresses one of two different limits, and most teams reach for the wrong one. This article goes through the options the framework and Horizon give you (the middleware, the limiters, the process counts) and what each one costs. It ends with a single serial slot, shared by a fast lane and a slow lane, where urgent work can still go first.

Key takeaways

"Sixty a minute" and "one at a time" are different limits. A rate limit bounds how often a job may start; a concurrency limit bounds how many may be in flight. RateLimited handles the first and does nothing about the second: sixty workers can fire simultaneously and stay within sixty a minute.
Laravel has no "don't pull" primitive. Every built-in throttle pops the job, finds it can't run it, and pushes it back. BullMQ leaves rate-limited jobs waiting in the queue; Laravel turns them into churn.
That churn has three costs: a released job goes to the back of the queue, it consumes an attempt, and it lands in the delayed set, where Horizon mostly stops showing it to you.
The cheapest answer is not in the docs: a dedicated supervisor with maxProcesses => 1. Concurrency of one comes from the process count, with no locks, no releases and no burned attempts.
Priority on a single worker needs weights. Put api-high and api-default on that one worker and they share the limiter with no extra code. With strict priority order, though, a flood on api-high starves api-default to zero. Weighting the queues gives api-default a fixed share instead.

Two limits that look like one

The vendor's docs state two limits. "60 requests per minute" and "no concurrent requests" are different constraints, and no single setting satisfies both:

A rate limit bounds how often a call may start: 60 per minute, 5 per second. It says nothing about how many are in flight at once.
A concurrency limit bounds how many calls may run simultaneously, in our case exactly one. It says nothing about how often they may start.

They are independent. Sixty calls per minute, each taking ten seconds, is ten requests in flight at any moment: within the rate limit and well over the concurrency limit. One call at a time, each taking ten milliseconds, is a hundred requests a second: serial, and far over the rate limit. You need both limits, and Laravel provides them in different places.

The distinction is visible in Laravel's API. Redis::throttle() is a rate limiter and Redis::funnel() is a concurrency limiter; the RateLimited middleware is the former and WithoutOverlapping is the latter. Most "how do I rate-limit a Laravel job" answers online cover the rate half and leave the concurrency half unsolved. The options below cover both, with what each one costs.

Option 1: the `RateLimited` middleware

The documented answer, and the right tool for the "60 per minute" half. Define a named limiter, then attach the middleware to the job:

use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Support\Facades\RateLimiter;

// In AppServiceProvider::boot()
RateLimiter::for('vendor-api', function (object $job) {
    return Limit::perMinute(60);
});

use Illuminate\Queue\Middleware\RateLimitedWithRedis;

public function middleware(): array
{
    return [new RateLimitedWithRedis('vendor-api')];
}

When the limit is hit, the middleware calls $job->release() with a delay of "time until the window resets, plus three seconds." You can override that with releaseAfter(), or call dontRelease(), which is easy to misread. dontRelease() does not fail the job and does not defer it. It deletes it: no exception, no failed_jobs row, no failed() hook.

Prefer RateLimitedWithRedis over plain RateLimited whenever the limit is an external contract. The generic version checks the counter and increments it as two separate cache operations. That is a time-of-check-to-time-of-use race: several workers can pass the check before any of them increments, and together they exceed your vendor's limit. The Redis variant does both in a single atomic Lua script. The docs describe it only as "more efficient."

Pros: documented, declarative, per-job, and the limiter key is shared across every job that names it, so two different job classes can draw on one budget. Cons: it limits rate and nothing else. Sixty workers can still hit the API in the same instant and remain within sixty per minute. And every throttled job pays the release costs described below.

Option 2: `WithoutOverlapping`, a mutex

This covers the concurrency half, and it gives you the number you want: one.

use Illuminate\Queue\Middleware\WithoutOverlapping;

public function middleware(): array
{
    return [
        (new WithoutOverlapping('vendor-api'))
            ->shared()
            ->releaseAfter(5)
            ->expireAfter(120),
    ];
}

All three settings in that snippet matter, and all three defaults are wrong for this use case:

shared() is required here. By default the lock key includes a hash of the job's class name, so SyncContact and PushInvoice get separate mutexes and call your no-parallel-requests API in parallel. shared() drops the class from the key, which is what makes both jobs contend for the same lock.
releaseAfter() defaults to 0. A blocked job is re-queued with no delay: pop, fail the lock, re-enqueue, pop, fail the lock. That loop keeps a worker core busy and burns an attempt on every pass.
expireAfter() defaults to never. The lock is released in a finally block, so an exception is fine, but a worker killed by --timeout or SIGKILL never runs it. The lock is then orphaned permanently, and that key stays locked until someone force-releases it by hand. Always set it above your job timeout.

Pros: mutual exclusion, works on any lock-capable cache driver, and it composes with RateLimited: stack both and you have covered both limits. Cons: the concurrency is fixed at one; there is no ->limit(n). It is release-based, so it pays the full churn cost. And a lock is a coordination mechanism you have to operate, with expiries to tune and orphans to clean up.

Option 3: funnels and throttles inside the job

Below the middleware sit the primitives they are built on. Redis::funnel() is the one that matters here, because it is the only thing in Laravel that expresses "at most N of these at once" for N greater than one:

use Illuminate\Support\Facades\Redis;

public function handle(): void
{
    Redis::funnel('vendor-api')
        ->limit(1)
        ->releaseAfter(120)
        ->block(0)
        ->then(
            fn () => $this->callTheVendor(),
            fn () => $this->release(5),
        );
}

funnel caps how many run at once; its sibling throttle()->allow(60)->every(60) caps how often they start. Both are Lua-backed and atomic, and both are undocumented. They were in the queue docs up to Laravel 8 and were dropped when RateLimited landed. The classes still ship and still work, but tutorials that teach them cite documentation that no longer exists.

There are two traps. First, block() defaults to three seconds, and that is three seconds of the worker sleeping inside your job's runtime. It pushes the job toward retry_after, at which point a second worker picks the job up while the first is still holding it, and you make the duplicate API call you were trying to prevent. Pass block(0) to fail fast. Second, omitting the failure closure throws a LimiterTimeoutException instead of releasing, which counts as a job failure.

The releaseAfter(120) above is the slot's safety TTL. If a worker is hard-killed mid-job, the slot is not returned until it expires, so it must exceed your job timeout. Otherwise a second worker is handed the slot while the first is still running, and the limit(1) guarantee no longer holds. Set it too long, though, and one stuck job stalls your serial pipeline until the TTL runs out.

Laravel 12 added the same concept to the cache layer as Cache::funnel(), which works on any lock-capable driver and is documented. The semantics are the same, without the Redis dependency.

Pros: atomic, and the only way to express a concurrency limit above one. Cons: the Redis variants are undocumented, they block by default, there is no middleware wrapper, and the code lives inside handle(), which mixes coordination logic into your business logic.

Option 4: one queue, one worker

Every option so far gets concurrency of one by having many workers compete for a lock, with the releasing, re-queueing and attempt-burning that implies. The queue already has a concurrency control, and it is the most reliable one available: the number of worker processes.

Give the API its own queue and give that queue exactly one worker, and concurrency of one follows from the setup instead of being enforced. There is nothing to contend for. No lock, no release, no attempt burned, and FIFO order is preserved because no job ever loses its place.

'supervisor-vendor-api' => [
    'connection'   => 'redis',
    'queue'        => ['api-high', 'api-default'],
    'balance'      => false,
    'minProcesses' => 1,
    'maxProcesses' => 1,
    'rest'         => 1,
    'timeout'      => 30,
],

maxProcesses => 1 is the setting that matters, and it is easy to get wrong. It is tempting to assume balance => false already pins the process count, but it does not: even unbalanced, Horizon scales the pool toward the number of ready jobs, clamped between minProcesses and maxProcesses. Pinning both ends to 1 is what guarantees a single serial slot.

rest, one of Horizon's more obscure options, sleeps the worker for N seconds after each job. On a single-process supervisor that works as a rate limit: one job per (duration + 1s) stays comfortably under sixty a minute. There is no middleware, no limiter key, no released job, no consumed attempt, and nothing is pushed to the back of the queue. The option appears in no version of the Laravel documentation.

Pros: the cheapest correct answer. No release churn, no lock administration, FIFO preserved, and it covers both limits at once. Cons: two. It is coarse: rest paces every job on that worker identically and cannot burst up to a limit the way a token bucket can. And maxProcesses is a per-host cap, not a cluster-wide one.

The per-host cap breaks the guarantee without any error. Every machine running php artisan horizon builds its own provisioning plan and deploys every supervisor matching its environment. Nothing in Horizon coordinates process counts across hosts: there is no cluster-wide cap, no lock and no leader election. Three app servers running Horizon against the same Redis give you three of these "single" workers, and three parallel calls to an API that permits none.

The fix is to make exactly one host provision this supervisor by giving it an environment of its own. Horizon resolves its environment from --environment, then config('horizon.env'), then APP_ENV, so the simplest lever is the flag, set in that host's Supervisor or systemd unit:

# On the one host that owns the vendor API. Everywhere else: php artisan horizon
php artisan horizon --environment=serial

'environments' => [
    'production' => [
        // ...your normal supervisors, on every app server
    ],

    'serial' => [
        'supervisor-vendor-api' => [
            'connection'   => 'redis',
            'queue'        => ['api-high', 'api-default'],
            'balance'      => false,
            'minProcesses' => 1,
            'maxProcesses' => 1,
            'rest'         => 1,
        ],
    ],
],

Environment matching is a first-match wildcard lookup, and a host whose environment matches nothing starts zero supervisors, silently. If you cannot guarantee single-host ownership, or would rather not stake your vendor contract on a deployment detail, keep a funnel(1) or a WithoutOverlapping as a backstop. It should almost never fire.

Option 5: chains and drip-feeds

Job chains (Bus::chain()) serialize by construction: job N+1 is not enqueued until job N succeeds. For a known, finite, ordered batch, such as "sync these 200 records, in order", this works well and needs no locks. But a chain is a closed list. Producers elsewhere in your app cannot append to it, two chains run in parallel with no mutual exclusion between them, and one failure abandons the rest of the chain. It serializes one batch, not access to the API.

Drip-feeding means a scheduled command that dispatches N jobs a minute, or a self-redispatching "pacemaker" job that pulls work off a list on an interval. Teams build it when the built-ins fall short. It works, and it has the same advantage as the single worker: nothing is popped that cannot be run. But you are now hand-rolling a scheduler on top of a queue, the scheduler's one-minute floor sets your resolution, and you own every edge case yourself. Use it only if the single-worker supervisor cannot express your pacing.

Option 6: the packages

The ecosystem has filled part of the gap, and the two mature packages solve different problems:

spatie/laravel-rate-limited-job-middleware wraps Redis::throttle in a fluent middleware and adds what core lacks: exponential backoff on release, time-based attempts, and events when the limit is hit. It is maintained and widely used. It is still a rate limiter, and still release-based.
mxl/laravel-queue-rate-limit throttles at the worker level, which core cannot do. Configure 'allows' => 1, 'every' => 5 for a queue and it makes the worker sleep instead of releasing the job. No attempts consumed, no re-queueing, no reordering. The cost is coarseness, since it throttles the whole queue regardless of job class. On a dedicated API queue that is what you want anyway.

Concurrency limiting above one is still missing. Core gives you exactly one (WithoutOverlapping) or an undocumented funnel, and no maintained package covers the range in between well.

The options, side by side

Approach	Limits rate?	Limits concurrency?	Release churn?	Main cost
`RateLimited`	Yes	No	Yes	Non-atomic; can overshoot a hard limit
`RateLimitedWithRedis`	Yes	No	Yes	Solves only half the problem
`WithoutOverlapping`	No	Yes, exactly 1	Yes	Lock orphans; three wrong defaults
`Redis::funnel()`	No	Yes, N	Yes	Undocumented; blocks the worker by default
`Redis::throttle()`	Yes	No	Yes	Undocumented; blocks the worker by default
Chains	No	Within a chain	No	Closed set; can't serialize the API globally
One worker + `rest`	Yes	Yes, exactly 1	No	Coarse pacing; per-host, not cluster-wide

The cost of nearly every option: Laravel pops before it checks

Every lock- and middleware-based option in the table has release churn. It is why teams who add the RateLimited middleware often end up with a queue that is slower, drops more jobs and is harder to see than before.

When a job is rate-limited, Laravel has already popped it: reserved it, unserialized it and re-fetched its Eloquent models from the database. Only then does the middleware run, find the budget spent, and push the job back. In BullMQ, rate-limited jobs stay in the waiting state: the worker declines to pull, and nothing else happens. Laravel has no "don't pull" primitive, so it pulls the job and puts it back. Rate limiting in Laravel therefore does not reduce work. It converts work into churn, and the churn grows with how far over the limit you are.

That design has three consequences, and the documentation mentions only one of them:

Released jobs go to the back of the queue. On Redis, release() puts the job in the delayed set; when it matures it is RPUSHed onto the tail. On the database driver it is a new row with a new auto-increment id, sorted last. The job that was next in line becomes last in line every time it is throttled. Under sustained load, a job can be starved indefinitely while newer work overtakes it. For the jobs you are throttling, FIFO order is inverted.
Released jobs consume attempts. This one is documented, and it is still the most common way this setup fails: your retry budget, which exists to protect you from failures, is spent on successful backpressure. With the default tries of 1, a job fails permanently the first time it is throttled. The fix is to count time instead of attempts, with retryUntil().
Released jobs collide again as a group. Every job blocked on the same limiter gets the same release delay; middleware releases have no jitter and no exponential backoff. They all mature at the same instant, are migrated back in one batch, and are popped and throttled again together on the next tick. The group stays synchronized indefinitely.

use DateTime;

// Retry until a deadline instead of for a fixed number of attempts.
public function retryUntil(): DateTime
{
    return now()->plus(hours: 6);
}

Teams have followed this path to its end (custom throttling middleware, then dynamic per-user queues, then a forked Horizon autoscaler) and written post-mortems about it. That is a lot of engineering for calling an API slowly, and it is the strongest argument for an approach that never releases a job in the first place.

The shape that works: one slot, one limiter

The API needs serial access and a paced rate, you want urgent work to overtake routine work, and you want as little churn as possible. That gives you one supervisor, two queues and one worker:

'supervisor-vendor-api' => [
    'connection'   => 'redis',
    'queue'        => ['api-high', 'api-default'],
    'balance'      => false,
    'minProcesses' => 1,
    'maxProcesses' => 1,   // the serial slot
    'rest'         => 1,   // the rate limit
    'timeout'      => 30,
],

Because both queues are drained by the same single process, they cannot produce a parallel request. No lock forbids it; there is no second worker to make one. The two queues share the concurrency budget by sharing the worker. And if you add a RateLimited('vendor-api') middleware for a limit rest can't express, both queues share that budget too, because the limiter is keyed by its name, not by the queue.

Keep the rest of your queues, and the other twenty supervisors doing your main work, out of this. This supervisor is a deliberate bottleneck.

Letting urgent jobs jump the line

A customer clicks "sync now" and expects their record pushed within seconds, but there are 4,000 nightly-reconciliation jobs on the same queue and the vendor allows one call a second. That is an hour of waiting behind work nobody is watching.

With balance => false, the queue list is a strict left-to-right priority order: a worker only looks at api-default when api-high is empty. Dispatching to api-high therefore does what you want. The urgent job is picked up next, and it uses the same single slot and the same limiter, so jumping the line cannot breach the API contract.

SyncContact::dispatch($contact)->onQueue('api-high');

Strict priority is absolute. Push a few thousand jobs onto api-high and api-default gets nothing for as long as api-high stays non-empty. With one worker and a rate limit throttling you on purpose, that can be hours. This is the same problem covered in Horizon's balancing trade-offs, and a rate-limited single worker is its worst case, because your throughput is capped by contract.

Turning balancing on does not help. With balance => 'auto', Horizon ignores queue order and gives each queue its own process pool, which means two workers and two parallel API calls, the thing the vendor forbids. Auto-balancing and serial access are mutually exclusive.

Skyline's weighted queues keep the single shared worker of balance => false, so the serial slot survives, and have it check the queues proportionally instead of in strict order:

'supervisor-vendor-api' => [
    'connection'   => 'redis',
    'queue'        => ['api-high', 'api-default'],
    'balance'      => false,
    'minProcesses' => 1,
    'maxProcesses' => 1,
    'rest'         => 1,
    'queueWeights' => [
        'api-high' => 5,
        // 'api-default' omitted => default weight of 1
    ],
],

At five to one, roughly five in every six pickups favour api-high, and the sixth still reaches api-default. Urgent work gets most of a scarce, contractually capped resource, and the nightly reconciliation keeps draining instead of stalling until morning. There is still one worker and one API call at a time.

For one-off urgency (a single job, now, without a second queue), front-of-queue dispatching pushes a single job onto the head of the Redis list rather than the tail:

use Laravel\Horizon\InteractsWithFrontOfQueue;

class SyncContact implements ShouldQueue
{
    use Dispatchable, InteractsWithFrontOfQueue, Queueable;
}

SyncContact::dispatch($contact)->onFront();

onFront() acts on the ready list. A job sitting in the delayed set because a rate limiter released it is not on that list yet, so it cannot be moved to the front until it matures. This is the ordering problem from the release section again: once a job has been released, its position is decided by when it matures, not by you.

What you can no longer see

This architecture has one more cost: it makes your queue harder to see.

Every job that a rate limiter releases lands in the delayed set. It is not running, not failed and not in the ready list; it is in a fourth state that Horizon barely shows. A backlog of throttled work can be almost absent from the dashboard you are checking to find out why nothing is happening. Jobs delayed beyond your retention window can drop off the dashboard altogether while still pending, a long-standing Horizon issue that was closed without a fix.

The single-slot pattern adds a second visibility cost. With balance => false, Horizon registers the pool under the comma-joined name of its queues, so your two lanes stop being two rows and become one pseudo-queue called redis:api-high,api-default. You cannot set a per-queue wait threshold for them, and the metrics dashboard shows them as a single entry. The pattern that gave you a serial slot costs you per-queue visibility, on the queue you most need to watch because you throttle it on purpose.

In Skyline, delayed and retrying jobs are first-class: you can see what is waiting and when each job is due, and run any of them immediately instead of waiting out a backoff window. You can open api-default and read the jobs inside it, in order. And when the vendor goes down, you can pause that one queue and let the work accumulate, instead of a rate limiter running 4,000 jobs through their retry budgets against an API that is returning 503s. None of that needs a migration or a code change: see Skyline vs Horizon for what swapping the package does and does not touch.

One change is worth making by hand. Import RateLimitedWithRedis from Laravel\Horizon\Middleware instead of Illuminate\Queue\Middleware, and Skyline's Locks & Limits screen counts how often each limiter released or dropped a job, while a job dropped by dontRelease() is logged instead of vanishing.

If you are deciding which queue a job belongs on in the first place, that is practice #2 in 12 best practices for Laravel background jobs. And if the jobs piling up behind your rate limiter are duplicates of each other rather than distinct work, read the ShouldBeUnique gotchas next.

Common questions

How do I limit a Laravel job to one at a time?

Two ways. The lock-based way is the WithoutOverlapping middleware with ->shared(). Without shared(), the lock key includes the job class, so two different job classes still run in parallel. The cheaper way is structural: give the job its own queue and a Horizon supervisor with maxProcesses => 1, so a single worker process drains it serially. That needs no locks, releases no jobs and consumes no attempts.

Does the RateLimited middleware prevent parallel requests?

No. A rate limit bounds how often a job may start; it says nothing about how many run at once. Sixty workers can call an API simultaneously and still be within a limit of sixty per minute. To bound concurrency you need WithoutOverlapping, Redis::funnel(), or a supervisor pinned to one worker process.

What is the difference between Redis::throttle() and Redis::funnel()?

throttle() is a rate limiter: at most N starts per time window. funnel() is a concurrency limiter: at most N running at once, with each slot held for as long as the job runs and returned when it finishes. Both are atomic Lua scripts. Both were dropped from the Laravel documentation after version 8, and both still ship and work.

Why do my rate-limited jobs fail after a few minutes?

Because releasing a rate-limited job back onto the queue still increments its attempt count. With the default tries of 1, a job fails permanently the first time it is throttled, having used its only attempt on waiting for the limiter. Define retryUntil() on the job so its retry budget is a deadline instead of an attempt count.

How do I let an urgent job jump ahead of a rate-limited queue?

Run an api-high and an api-default queue on the same single-worker supervisor: they share the worker, so they share the concurrency budget, and they share the rate limiter because it is keyed by name rather than by queue. With balance => false the queue list is a strict priority order, so a flood on api-high starves api-default entirely. Skyline's queueWeights check the two in proportion instead (5:1 here), so urgent jobs mostly go first and api-default is still checked on every poll.

Full-Page Caching in Laravel: Large Redis Entries, predis vs phpredis, and Compression

laravel-o11y — Tue, 29 Sep 2026 11:22:31 +0000

A full-page cache stores the whole rendered response, so an entry is as large as the page it holds. Our Redis monitor showed response-cache entries of about 25MB, and ordinary listing pages were taking 1.6 to 2.8MB each. The hit rate looked fine. Redis memory and the size of every read did not.

Most of that size goes away with compression, and where it happens depends on the Redis client. phpredis can compress every value on a connection with one config option. predis can't, so on predis the compression has to happen in the cache serializer. The examples use spatie/laravel-responsecache, the open-source full-page cache for Laravel that Spatie maintains, but the Redis side applies to any cache that stores rendered HTML.

Key takeaways

Rendered HTML compresses well. Our listing pages went from 2.71MB to 0.08MB with zlib, and a 1.47MB article page went to 169KB with zstd.
predis has no compression option. phpredis compresses values with lzf, lz4 or zstd when you set compression on the connection, provided the extension was built with that algorithm.
Give the response cache its own connection. On a compressed connection, Cache::increment() silently returns false unless pack_ignore_numbers is set.
Enabling compression needs no flush, but disabling it does. A compressed connection reads old plain entries, and an uncompressed client can't read compressed ones.
On predis, compress in the serializer. gzcompress() reaches the same size as zstd but costs about seven times as much per write.

Why a full-page cache entry gets big

responsecache serializes the response to JSON with its status, headers and body, and stores that string. The body is everything the browser receives, including inline SVG icons, JSON-LD blocks, inline scripts and a Livewire snapshot for every component on the page. A listing page with a card per result, each carrying its images, prices and links, reaches megabytes quickly. JSON then escapes every quote and slash, so the stored string is larger than the HTML.

The number of entries multiplies that. Our hasher keys each page by locale, currency and user, so one URL can hold dozens of guest entries, and every query-string variant gets its own set. The 25MB entries came from a URL that no page links to, which was still rendered and cached for every visitor who reached it.

redis-cli --memkeys samples the keyspace and reports the largest key of each type by memory, and MEMORY USAGE gives the size of a single key:

redis-cli -n 1 --memkeys
redis-cli -n 1 MEMORY USAGE "laravel_cache:9f2c...-eur-en-guest"

Redis keeps every entry in memory, so a few hundred page variants can push the instance into maxmemory eviction. If the cache store is shared with the rest of the app, the keys it evicts can be sessions, locks or anything else. Each cache hit also moves the full value from Redis to PHP and decodes it there, which on a busy page is most of the traffic between the two.

predis and phpredis

Laravel talks to Redis through one of two clients, chosen by database.redis.client (the REDIS_CLIENT variable). The framework defaults to phpredis, but many apps run predis because it installs with Composer and needs no extension, which is the only option on hosts that don't allow extensions.

	predis	phpredis
What it is	Pure PHP library, `composer require predis/predis`	C extension, `pecl install redis` or a distro package
Protocol parsing	In PHP	In C
Value compression	None	lzf, lz4, zstd, set per connection
Value serializers	None, Laravel's cache calls `serialize()`	php, igbinary, msgpack, JSON
Deploy cost	Nothing	The extension in every image, CI runner and dev machine

For small values the two perform close enough that the choice rarely matters. For values the size of a page, phpredis's built-in compression is the main reason to prefer it.

What compression costs

These are the stored size and the time per put and get for a 1.47MB response-cache payload written through Laravel's RedisStore:

Client and compression	Stored	Put	Get
predis	1,500KB	1.03ms	0.55ms
predis + `gzcompress()` level 6	172KB	17.1ms	2.15ms
phpredis	1,500KB	0.40ms	0.51ms
phpredis lzf	334KB	2.29ms	2.46ms
phpredis lz4	277KB	1.52ms	0.52ms
phpredis zstd level 3	169KB	2.42ms	0.81ms
phpredis zstd level 9	143KB	11.6ms	0.70ms

zstd at its default level stores the page in about a ninth of the space. It adds 2ms to a write and about 0.3ms to a read. Writes happen only on a cache miss, which already pays for a full render, so the write cost barely registers. Reads happen on every hit, and zstd and lz4 both decompress quickly enough that a hit stays under a millisecond. lz4 suits you better if write time matters more than memory. Level 9 saves another 15% for five times the write cost, which we didn't think was worth it.

The timings assume Redis close to PHP. With a managed Redis in another zone, or any link where bandwidth is the limit, the uncompressed reads slow down in proportion to their size, and the compressed ones barely change. The ratio depends on the markup too. Our own listing pages repeat the same card markup many times, and they compressed about 33 times with zlib, where the article page above compressed about 9 times.

Enabling phpredis with compression

The extension compresses only with algorithms it was compiled with. The PECL build asks about lzf, zstd and lz4 at configure time and skips any whose library headers are missing. A Dockerfile that gets all three:

RUN apt-get update && apt-get install -y liblz4-dev libzstd-dev \
    && pecl install --configureoptions \
        'enable-redis-igbinary="no" enable-redis-msgpack="no" enable-redis-lzf="yes" enable-redis-zstd="yes" enable-redis-lz4="yes"' \
        redis \
    && docker-php-ext-enable redis

A distro package or a hosting panel's build may leave some of them out, so check before relying on it:

$ php --ri redis | grep compression
Available compression => lzf, zstd, lz4

If the line is missing or doesn't list the algorithm you configure, rebuild the extension first. Then switch the client. database.redis.client applies to every Redis connection in the app, so this moves the default connection, the cache, the queue and Horizon to phpredis together:

REDIS_CLIENT=phpredis

This step needs no flush. With no serializer or compression options set, both clients store and read the same bytes.

Compression is set per connection, and we keep it on the response cache only. Add a connection and a cache store for it:

// config/database.php, under 'redis'
'responses' => [
    'url' => env('REDIS_URL'),
    'host' => env('REDIS_HOST', '127.0.0.1'),
    'username' => env('REDIS_USERNAME'),
    'password' => env('REDIS_PASSWORD'),
    'port' => env('REDIS_PORT', '6379'),
    'database' => env('REDIS_RESPONSE_CACHE_DB', '2'),
    'compression' => Redis::COMPRESSION_ZSTD,
    'compression_level' => 3,
    'pack_ignore_numbers' => true,
],

// config/cache.php, under 'stores'
'responses' => [
    'driver' => 'redis',
    'connection' => 'responses',
    'lock_connection' => 'responses',
],

RESPONSE_CACHE_DRIVER=responses

Redis::COMPRESSION_ZSTD is a constant of the extension's class, so every environment that loads the config needs phpredis, including CI and config:cache in a build step.

A separate store also limits what a clear removes. With no responsecache.cache.tag set, ResponseCache::clear() flushes the whole store it's pointed at, and on a store of its own that means only responses.

Counters on a compressed connection

phpredis compresses every value it writes through that connection, integers included. Redis's INCRBY then finds a compressed blob where it expects a number and refuses it, and phpredis reports that as false with no exception:

Cache::store('responses')->put('n', 5, 60);
Cache::store('responses')->increment('n');   // false, and 'n' is still 5

Laravel's RateLimiter works around this by turning compression off around its own writes, so throttle keeps counting. Your own Cache::add() and Cache::increment() pairs don't get that treatment. The pack_ignore_numbers option in the config above tells phpredis to store numbers as plain values, and with it the same two lines return 6. A connection used only by the response cache has no counters on it in the first place.

Deploying and rolling back

phpredis with compression on still reads values written without it, with lzf, lz4 and zstd alike. The deploy that turns compression on therefore needs no flush. Old entries are served until they expire, and new ones are written compressed.

Going back is different. predis, or phpredis without the compression option, gets the compressed bytes back as they are, and RedisStore's unserialize() fails on them. If you roll back the config, clear the response cache store as part of the rollback:

php artisan responsecache:clear

The same applies to anything else reading that Redis database directly, such as a script or a second app on predis.

If you have to stay on predis

Without the extension, compress before the value reaches the client. responsecache lets you replace its serializer, and a subclass of its JsonSerializer can zlib the JSON using only ext-zlib, which nearly every PHP build has:

namespace App\ResponseCache;

use Spatie\ResponseCache\Exceptions\CouldNotUnserialize;
use Spatie\ResponseCache\Serializers\JsonSerializer;
use Symfony\Component\HttpFoundation\Response;

class CompressedJsonSerializer extends JsonSerializer
{
    public function serialize(Response $response): string
    {
        return gzcompress(parent::serialize($response), 6);
    }

    public function unserialize(string $serializedResponse): Response
    {
        // Entries cached before compression are plain JSON, and a zlib stream never starts with '{'
        if (str_starts_with($serializedResponse, '{')) {
            return parent::unserialize($serializedResponse);
        }

        $json = @gzuncompress($serializedResponse);
        if ($json === false) {
            throw new CouldNotUnserialize('Could not gzuncompress cached response');
        }

        return parent::unserialize($json);
    }
}

// config/responsecache.php
'serializer' => App\ResponseCache\CompressedJsonSerializer::class,

This took our listing pages from 2.71MB to 0.08MB. The value is already a binary string when Laravel receives it, so it works with predis and phpredis alike, and the fallback for JSON entries means the deploy needs no flush. A corrupt entry throws CouldNotUnserialize, which the middleware treats as a miss, and the next render overwrites it.

zlib is slower than phpredis's zstd. In the table above, gzcompress() at level 6 stores the page at about the same size as zstd but takes 17ms per write against 2.4ms, and 2.2ms per read against 0.8ms. The write cost falls only on misses, which makes it acceptable when you can't install an extension. If you can install ext-zstd to use in the serializer, installing phpredis takes the same effort and needs no custom class.

Compression makes each entry smaller but doesn't change how many of them get rendered. If the cache is missing more than it should, the responsecache hit-rate post goes through the click ids and key mismatches that caused most of our misses.

Common questions

Does predis support compression?

No. predis sends values to Redis as it gets them. To compress on predis, compress before the value reaches the client, for example in a custom responsecache serializer that calls gzcompress(). phpredis has compression built in: set compression to Redis::COMPRESSION_ZSTD, COMPRESSION_LZ4 or COMPRESSION_LZF on the connection in config/database.php.

How do I check which compression algorithms phpredis supports?

Run php --ri redis and look for the "Available compression" line. The extension only compresses with algorithms it was compiled with, so a PECL build needs the liblz4 and libzstd headers installed and the enable-redis-lz4 and enable-redis-zstd configure options set.

Why does Cache::increment() return false with phpredis compression?

phpredis compresses every value it writes through the connection, integers included, so INCRBY finds a compressed blob instead of a number and fails, which phpredis returns as false. Set pack_ignore_numbers to true on the connection so numbers are stored plain, or keep counters off the compressed connection. Laravel's RateLimiter turns compression off around its own writes and is not affected.

Do I need to flush Redis when enabling phpredis compression?

No. A compressed phpredis connection still reads values written without compression, so old entries are served until they expire. Disabling compression is different: predis or an uncompressed phpredis connection cannot read compressed values, so clear the cache store when rolling back.

Instrumenting Guzzle in Laravel: Logging and Metrics for Every Outgoing Request

laravel-o11y — Tue, 29 Sep 2026 11:21:48 +0000

Most Laravel apps spend a good share of every request waiting on someone else's server: the payment provider, the shipping API, the CRM, S3. When one of them gets slow, you see your own response times rise and have to work out whose fault it was. Instrumenting the outgoing side answers that directly, and in Laravel nearly all of that traffic goes through Guzzle, which gives you a very good place to hook in.

This post builds that hook. We start with a Guzzle middleware registered once on Laravel's HTTP client, look at what the client's own events can and can't tell you, cover the SDKs that build their own Guzzle client, and then deal with the case that most logging setups miss: the request that never got a response at all. At the end, we compare logs with metrics and point at the two packages we maintain for this.

Key takeaways

Register a Guzzle middleware with Http::globalMiddleware(). It runs for every Http:: call, including each request in Http::pool(), and it is the one hook that sees the request, the response and the failure.
Record from Guzzle's on_stats option, not from the response. on_stats fires for every transfer, including a timeout or a refused connection, with the transfer time and cURL's error number. A handler that only looks at responses never sees those.
Laravel's ConnectionFailed event carries no timing. It holds the request and the exception, and the only way to tell a timeout from a DNS failure is to read the exception's message.
SDKs with their own Guzzle client are invisible to Http::globalMiddleware(). The AWS SDK (and with it the S3 filesystem driver) is one. Push the same middleware onto their handler stack.
You're counting transfers, not calls. A redirect produces one transfer per hop and Http::retry(3) up to three, and each one passes through the middleware.

Where your outgoing requests go

Laravel's Http facade is a wrapper around Guzzle. Every call through it builds a Guzzle handler stack, and Laravel lets you add to that stack for every request the application makes. That covers your own integration code, Http::pool(), and any package that uses Laravel's client internally.

It doesn't cover code that creates a GuzzleHttp\Client of its own. Many SDKs do, because they are written to work outside Laravel too. Those requests go through a separate stack that Laravel never sees, so they need instrumenting separately. The mechanism is the same in both cases, which is why it's worth writing the middleware as a plain Guzzle middleware rather than something Laravel-specific.

A Guzzle middleware for every `Http::` call

A Guzzle middleware is a function that takes the next handler and returns a new one. The handler receives the request and the request options and returns a promise. Here is one that logs every transfer:

namespace App\Http;

use GuzzleHttp\TransferStats;
use Illuminate\Support\Facades\Log;
use Psr\Http\Message\RequestInterface;
use Throwable;

class LogOutgoingRequests
{
    public function __invoke(callable $handler): callable
    {
        return function (RequestInterface $request, array $options) use ($handler) {
            $previous = $options['on_stats'] ?? null;

            $options['on_stats'] = function (TransferStats $stats) use ($previous) {
                try {
                    $response = $stats->getResponse();
                    $errno = $stats->getHandlerErrorData();

                    Log::channel('outbound')->info('http.out', [
                        'method' => $stats->getRequest()->getMethod(),
                        'host' => $stats->getEffectiveUri()->getHost(),
                        'path' => $stats->getEffectiveUri()->getPath(),
                        'status' => $response?->getStatusCode(),
                        'ms' => (int) round(($stats->getTransferTime() ?? 0) * 1000),
                        'curl_error' => $response === null && is_int($errno) ? $errno : null,
                    ]);
                } catch (Throwable $e) {
                    report($e);
                }

                if ($previous) {
                    $previous($stats);
                }
            };

            return $handler($request, $options);
        };
    }
}

use App\Http\LogOutgoingRequests;
use Illuminate\Support\Facades\Http;

public function boot(): void
{
    Http::globalMiddleware(new LogOutgoingRequests);
}

The middleware doesn't touch the promise. All the recording happens in on_stats, an option that Guzzle's cURL handler calls once per transfer, after the transfer has finished, whatever the outcome. We keep any on_stats callback the caller already set and call it afterwards, so a request that passes its own on_stats still gets it.

The try block matters more than it looks. Guzzle treats an exception thrown from the stats callback as a failure of the request itself, so a full disk or a misconfigured log channel would turn every outgoing call in the app into an exception. Instrumentation should never be able to break the thing it observes.

At this layer a 500 is still a response

Laravel's client doesn't throw on a 4xx or 5xx unless you call throw() or exhaust a retry(). A plain Guzzle client does, through its http_errors middleware, but that sits outside the middleware you pushed. Your middleware sees the 500 as an ordinary response before it becomes a ServerException, so recording status codes there works the same whichever client made the call.

What Laravel's HTTP client events give you

Laravel also dispatches three events from its client, and a listener is less code than a middleware. They are worth knowing, because they are where most "log every outgoing request" answers start:

Event	Fires when	Carries
`RequestSending`	Before every attempt	The request
`ResponseReceived`	A response arrived, of any status	The request and the response. `$response->transferStats` holds Guzzle's `TransferStats`, so the timing is there.
`ConnectionFailed`	No response arrived	The request and a `ConnectionException`. Nothing else.

The successful path is well covered. The failed path isn't. ConnectionFailed has no transfer time, so you can't tell a request that failed instantly from one that held a worker for 30 seconds. And the cause is only in the exception message, as a string like cURL error 28: Connection timed out after 1001 milliseconds, which you would have to parse to count timeouts separately from DNS failures.

The events also exist only on Laravel's client. The SDK traffic in the next section never dispatches them, so an events-based setup can't be extended to cover it. The middleware can.

SDKs that bring their own Guzzle client

We checked which common PHP SDKs depend on Guzzle directly. The AWS SDK, Google's API client and Resend's SDK all do, and each builds its own GuzzleHttp\Client. That includes every S3 call made through Laravel's filesystem, because the S3 driver uses the AWS SDK. A test with a plain new Client() confirms that Http::globalMiddleware() never sees these requests.

The fix is to push the same middleware onto a handler stack and give the SDK a client built on it:

use App\Http\LogOutgoingRequests;
use GuzzleHttp\Client;
use GuzzleHttp\HandlerStack;

$stack = HandlerStack::create();
$stack->push(new LogOutgoingRequests);

$client = new Client(['handler' => $stack]);

How you hand that client over depends on the SDK. Most accept one through a constructor option or a factory method. openai-php/client, for example, takes it through OpenAI::factory()->withHttpClient($client). SDKs built on PSR-18 discovery, like openai-php/client and Mailgun's, will pick Guzzle up if it's installed but still construct their own instance, so the global middleware doesn't reach them either.

Not every SDK uses Guzzle. Stripe's and Twilio's PHP SDKs ship their own cURL client and don't require Guzzle at all, so a Guzzle middleware won't see their traffic. Check the SDK's composer.json before assuming your middleware covers it.

The requests that never got a response

The outages that hurt most often don't produce a 500 at all. The vendor's load balancer stops answering and your requests hang until the timeout, or a DNS change goes wrong and the hostname stops resolving. There is no response to log in either case, and an instrumentation layer that hangs off the response, such as a then() on the promise or a response middleware, has nothing to record.

on_stats still fires. We tested it against Laravel 13.24 and Guzzle 7.15 with a closed port, a hostname under .invalid, an unroutable address and a server that answered too slowly. In each case the callback ran once, with hasResponse() returning false and cURL's error number available from getHandlerErrorData():

What happened	cURL error	Transfer time	Laravel throws
Connection refused	7	~0 s	`ConnectionException`
DNS lookup failed	6	The lookup time	`ConnectionException`
Connect timeout (`connectTimeout(1)`)	28	1.00 s	`ConnectionException`
Response timeout (`timeout(1)`)	28	1.00 s	`ConnectionException`

The transfer time on a timeout is how long your code waited, roughly the timeout you configured. Keep those rows in your latency data rather than filtering them out. A p95 that ignores timed-out calls will look healthy during exactly the incident you wanted it for. TLS failures arrive the same way under their own error numbers: 35 for a failed handshake, 60 for a certificate that won't verify.

Redirects and retries each count as a transfer

The middleware sits inside Guzzle's redirect handling, so a request that is redirected once produces two transfers, and on_stats fires for both. Laravel's retry() goes further and sends the request through the whole stack again for each attempt. Http::retry(3) against a host refusing connections logged three transfers and dispatched three RequestSending and ConnectionFailed pairs. Against an endpoint returning 500 it logged three transfers too.

That is the right thing to record, since each attempt had its own latency and its own way of failing. It does mean your transfer count will be higher than the number of calls your code made. When a vendor bills per request, or a rate limit counts attempts, the transfer count is the number they see.

Logs or metrics?

Everything so far writes one log line per transfer, which answers questions about a particular call: what happened to the refund request for order 8134, and what did the vendor say? It's poor at trends. Working out the p95 latency to one host over the last week from log lines means a log pipeline that can aggregate, and most can't do it cheaply.

Metrics are the other way round. A duration histogram labelled by host and outcome gives you latency percentiles and error rates per vendor at almost no storage cost, and it can drive alerts. It can't tell you about any single call. Most teams end up wanting both, because you use them at different moments: the chart shows you something is wrong, and the individual records show you what.

If you build the metrics yourself from the middleware above, the label to watch is the path. Every distinct value becomes its own series, and /v1/orders/8134 and /v1/orders/8135 are different values. Replace numeric and UUID segments with a placeholder before using the path as a label, or leave the path out and label by host only.

Prometheus metrics with httptheus

httptheus is the metrics half of this post, packaged. It's free and MIT-licensed, and it's built the way this post describes: a Guzzle middleware registered with Http::globalMiddleware(), recording from on_stats. Install it and there is nothing else to wire up:

composer require boring-o11y/httptheus

It exports two metrics. httptheus_client_request_duration_seconds is a histogram labelled by host, method, endpoint and status class, where a transfer with no response gets status_class="error", so timeouts stay in your latency percentiles. httptheus_client_request_errors_total counts only transfers that produced no response, with a reason label (timeout, dns, connection_refused, tls, network or other) read from the cURL error number:

# p95 latency to one vendor
histogram_quantile(0.95, sum by (le) (
  rate(httptheus_client_request_duration_seconds_bucket{host="api.stripe.com"}[5m])
))

# timeouts per second, by host
sum by (host) (rate(httptheus_client_request_errors_total{reason="timeout"}[5m]))

Paths are normalised into the endpoint label before they are recorded, with an allow-list if you want a hard cap on cardinality. If you already run spatie/laravel-prometheus, the metrics appear on its /prometheus route with no configuration. Otherwise they're served at /httptheus/metrics. It also ships a Grafana dashboard, and SDK clients are covered by pushing its RecordHttpMetrics middleware onto their stack, as above.

A record of every call, with Requizon

The log-line half is where we think a dashboard does better than a log channel. When a vendor integration starts failing, the questions are which calls failed, how, since when, and what the vendor sent back. A grep through JSON logs answers them slowly, and only if you logged the response body, which you probably shouldn't do for every call.

Requizon is the dashboard we built for that. It uses the same on_stats hook, stores each transfer in your own MySQL database and serves a Horizon-style UI at /requizon. Each call is classified as a connection error, an HTTP error or an application error, and a failed call keeps its response body so you can read what the vendor actually said.

Application errors are the case metrics can't catch: an API that answers 200 OK and puts the failure in the body. You register one detector that knows what failure looks like for each API, and those calls stop counting as successes. Request parameters are stored with anything that looks like a credential redacted, and headers are never stored. What Requizon records lists exactly what goes into a row, and how it compares with Nightwatch and Telescope covers where it overlaps with the tools you may already run.

SDK traffic is handled the way it is everywhere else in this post:

use BoringO11y\Requizon\Guzzle\RecordHttpRequests;

$stack = HandlerStack::create();
$stack->push(app(RecordHttpRequests::class));

$client = new Client(['handler' => $stack]);

httptheus and Requizon read the same hook and can run side by side. If your integrations are the kind that go wrong in ways a status code doesn't show, the rate limits and backoff in rate-limited APIs and Laravel queues are the natural next read.

Common questions

How do I log every outgoing HTTP request in Laravel?

Register a Guzzle middleware with Http::globalMiddleware() in a service provider. It runs for every call made through Laravel's HTTP client, including Http::pool(). Record from Guzzle's on_stats request option rather than from the response, because on_stats also fires for timeouts, DNS failures and refused connections, where there is no response to log.

Why doesn't Http::globalMiddleware() see my SDK's requests?

Because the SDK builds its own GuzzleHttp\Client, with its own handler stack, and never goes through Laravel's HTTP client. The AWS SDK, and with it the S3 filesystem driver, Google's API client and Resend's SDK all do this. Push the same middleware onto a HandlerStack and give the SDK a client built on it. Stripe's and Twilio's SDKs use their own cURL client, so a Guzzle middleware can't see them at all.

How can I tell a timeout from a DNS failure in Laravel's HTTP client?

Both throw a ConnectionException, and the ConnectionFailed event carries only the request and that exception. The cause is cURL's error number: 28 for a timeout, 6 for a DNS failure, 7 for a refused connection. Guzzle's TransferStats exposes it through getHandlerErrorData() in an on_stats callback, along with the transfer time. Otherwise it only appears inside the exception message.

Does Http::retry() count as one request or several?

Several. Each attempt goes through the whole Guzzle handler stack again, so Http::retry(3) against a failing host produces three transfers, three on_stats callbacks and three RequestSending events. Redirects are counted the same way, one transfer per hop, because middleware you add sits inside Guzzle's redirect handling.

Laravel Job timeout vs retry_after: The Ordering Rule Nothing Enforces

laravel-o11y — Fri, 18 Sep 2026 07:20:24 +0000

Two numbers, in two different configuration files, decide whether a queued job runs once or twice. One is retry_after, in config/queue.php. The other is your worker's timeout, in config/horizon.php or on the job class itself. Laravel ships them 30 seconds apart and mentions the relationship twice in prose. No code checks it at boot, at dispatch, or when a supervisor starts a worker.

Get the order wrong and a job that is still running is handed to a second worker. What you see next depends on a third setting, tries. Either two workers execute the same job concurrently and both report success, or a job you never saw run twice lands in your failed jobs list with MaxAttemptsExceededException, for work that completed. The rest of this article covers the mechanism behind both symptoms, the exact defaults, and the ordering rule you have to enforce yourself.

Key takeaways

retry_after is a lease, not a delay. Popping a job reserves it for retry_after seconds. When the lease expires the job goes back on the ready queue whether or not it is still running.
The defaults are 60 and 90, if you kept the shipped config. Worker timeout defaults to 60s, the shipped retry_after to 90s. Omit retry_after from a connection and the framework falls back to 60, making the two equal.
Equal values are not safe. The lease starts at pop() and the timeout alarm is armed a few steps later, so with matching values the reservation always expires first.
The symptom depends on tries. With tries => 1 (the value in Horizon's published config) you get a failure recorded against a job that succeeded. With tries => 2 or more you get concurrent double execution, and nothing fails.
The rule: job timeout < supervisor timeout < retry_after, with margin at each step. Nothing in Laravel tells you when it stops holding.

The two numbers and their defaults

Both settings are measured in seconds and are easy to confuse. retry_after belongs to the queue connection and sets how long a reservation is honoured. timeout belongs to the worker and sets how long a single job may occupy a process. They have different owners, run on different clocks, and nothing enforces a relationship between them.

	`timeout`	`retry_after`
Owned by	The worker process (or the job class)	The queue connection
Configured in	`config/horizon.php`, `--timeout`, `$timeout`, `#[Timeout]`	`config/queue.php`
Default	60 (`queue:work --timeout=60`, and the Horizon supervisor default)	90 as shipped in `config/queue.php`; 60 if the key is absent
Enforced by	`pcntl_alarm()` in the worker	A score on the reserved set, checked by whichever worker pops next
On expiry	The worker kills itself	The job becomes available to everyone again

A stock Laravel skeleton pairs a 60-second timeout with a 90-second retry_after, which is a correct configuration with 30 seconds of slack. But the 90 is a value in your configuration file, not a framework default. The framework's own fallback, used whenever a connection array has no retry_after key, is 60:

// Illuminate\Queue\Connectors\RedisConnector
return new RedisQueue(
    $this->redis, $config['queue'],
    Arr::get($config, 'connection', $this->connection),
    Arr::get($config, 'retry_after', 60),   // not 90
    // ...
);

A connection written by hand (a second Redis connection for a dedicated queue, a connection built in a test, anything copied from a blog post rather than from config/queue.php) gets 60 with no warning, equal to the default worker timeout. That configuration is broken from the first job that runs long, and it looks reasonable in a pull request.

What `retry_after` does

The name suggests a delay before a retry. It is a lease: popping the job sets a deadline on its reservation, and once that deadline passes any worker can take the job, including while the first worker is still running it.

On Redis, popping a job is a single Lua script. It lpops the payload off the ready list, increments the payload's attempts counter, and adds the result to a sorted set, scored with the moment the reservation expires:

-- Illuminate\Queue\LuaScripts::pop()
local job = redis.call('lpop', KEYS[1])
local reserved = false

if(job ~= false) then
    -- Increment the attempt count and place job on the reserved queue...
    reserved = cjson.decode(job)
    reserved['attempts'] = reserved['attempts'] + 1
    reserved = cjson.encode(reserved)
    redis.call('zadd', KEYS[2], ARGV[1], reserved)
    redis.call('lpop', KEYS[3])
end

ARGV[1] is now + retry_after. No timer watches that score. Instead, every pop() on that queue begins by sweeping it:

// Illuminate\Queue\RedisQueue
public function pop($queue = null, $index = 0)
{
    $this->migrate($prefixed = $this->getQueueRedisKey($queue));
    // ...
}

migrate() moves two things back onto the ready list: delayed jobs whose time has come, and reserved jobs whose lease has expired. It cannot tell the difference between a job whose worker was killed by a deploy and a job whose worker is still running it. Both look like an expired score.

The database driver reaches the same outcome through SQL rather than a sorted set. isReservedButExpired() widens the "next available job" query to include any row whose reserved_at is older than now - retry_after, so it also has no way to know whether the original worker is alive.

Two drivers behave differently. Beanstalkd has no retry_after default of its own and falls back to Pheanstalk's TTR. SQS has no retry_after at all: the connector never reads the key, because the lease lives on Amazon's side as the queue's Default Visibility Timeout. Its default is 30 seconds, half of Laravel's default worker timeout. An SQS queue created with the console defaults and consumed by a stock worker is misconfigured from the start, and no edit to config/queue.php will fix it.

What `timeout` does

The worker's timeout is independent of the queue. It is a POSIX alarm, set per job, inside the worker process:

// Illuminate\Queue\Worker::registerTimeoutHandler()
pcntl_signal(SIGALRM, function () use ($job, $options) {
    // ...record the failure, dispatch JobTimedOut...
    $this->kill(static::$timedOutExitCode ?? static::EXIT_ERROR, $options, WorkerStopReason::TimedOut);
}, true);

pcntl_alarm(
    max($this->timeoutForJob($job, $options), 0)
);

Three consequences follow from this code, and each can break the ordering rule without anyone touching config/queue.php:

The job's own timeout takes precedence. timeoutForJob() returns the job's $timeout property or #[Timeout] attribute if it has one, and only falls back to the worker's option. A single #[Timeout(300)] on a slow report job overrides a correctly configured 60-second supervisor and outlasts a 90-second lease.
--timeout=0 disables it. pcntl_alarm(0) cancels the alarm rather than firing immediately, so a zero timeout means no timeout. Nothing bounds the job's runtime, while the lease keeps expiring on schedule.
Without pcntl there is no timeout. The handler sits behind a supportsAsyncSignals() check. On a build without the extension no alarm is set and nothing reports that, so long jobs are duplicated with no local symptom to reproduce.

Why the two collide, and why equal values fail

Take a 90-second job with retry_after and timeout both set to 90, a configuration that looks tidy in a config file:

t=0.00   worker A pops the job; reservation scored to expire at t=90.00
t=0.02   JobReserved fires, the worker enters process()
t=0.03   pcntl_alarm(90) armed, will fire at t=90.03
t=90.00  worker B pops the same queue; migrate() sees an expired score
         and moves the still-running job back onto the ready list
t=90.00  worker B pops it, attempts becomes 2
t=90.03  worker A's alarm fires, 30ms after worker B took the job

The lease starts at the pop. The alarm is armed several steps later: after the reservation is written, after the JobReserved event and its listeners run, and after the worker enters process(). With identical values the reservation expires first every time, by the length of that setup. Even when the alarm does fire first, the handler still has to record the failure and exit before the process lets go of anything.

This is also why the bug is hard to reproduce. Migration is lazy: it only happens when somebody pops that queue. A supervisor with a single worker, occupied by the long job, never sweeps its own expired reservation, so the misconfiguration stays invisible until a second worker exists. Autoscale from one process to two, or move from a laptop to production, and the same code starts running jobs twice.

The two symptoms, and which one you get

Once the job has been migrated back and popped by a second worker, tries decides what happens next. The Lua script incremented attempts on that second pop, and Worker::process() checks the attempt count before it fires the job:

// Illuminate\Queue\Worker::process()
$this->raiseBeforeJobEvent($connectionName, $job);

$this->markJobAsFailedIfAlreadyExceedsMaxAttempts(
    $connectionName, $job, (int) $options->maxTries
);

// ...

$job->fire();

With `tries => 1`: a failure on work that succeeded

One attempt is what most Horizon installs run with: the config/horizon.php that Horizon publishes sets 'tries' => 1 on its supervisor, and a job class with no $tries of its own inherits it. (Drop the key entirely and horizon:supervisor falls back to --tries=0, which means unlimited.) The migrated copy arrives at worker B carrying attempts = 2, exceeds the limit before fire() is reached, and is failed immediately with MaxAttemptsExceededException. Its body never runs.

There is no duplicate execution, but you now have a failed job in the dashboard for work that is still running and completes successfully a few seconds later. Worker A finishes, tries to delete its reservation, and removes nothing, because the entry was migrated away while it worked. The same job id ends up with one failure and one success, and the stack trace points at a timeout that never happened.

If you have chased "MaxAttemptsExceededException on a job that clearly worked", this is one of two ways to get there. The other is lock-based middleware spending attempts on releases, which has its own set of failure modes.

With `tries => 2` or more: concurrent execution with no error

Most applications raise tries eventually, because retries are the point of a queue and because middleware like RateLimited and WithoutOverlapping consume attempts. With two or more attempts allowed, the check passes, and worker B calls fire() while worker A is inside the same job's handle().

Both copies run to completion, so the card is charged twice or the email goes out twice. Neither copy throws or is marked failed, and the dashboard shows one completed job. Laravel, Horizon and your error tracker record nothing unusual. The evidence is in your data, as duplicate rows, doubled counters or a customer with two receipts.

Four ways a correct config becomes an incorrect one

timeout usually ends up above retry_after as a side effect of a change that looked local:

A job gained a timeout of its own. A nightly export starts taking four minutes, someone adds #[Timeout(300)] to the job class, and the supervisor's 60 no longer applies to that job while the connection's 90-second lease is unchanged. The job class is the highest-precedence setting and the one furthest from the config file where the constraint lives.
A supervisor timeout was raised to fix timeouts. Jobs are being killed at 60 seconds, so the supervisor timeout goes to 120. The kills stop and the lease problem above starts, because retry_after is in a different file and was not part of the change.
A new connection was added without the key. It gets the 60-second framework fallback described above. The values are equal from the first deploy, and the diff contains no line that would catch it.
The queue moved to SQS. The lease is now a Default Visibility Timeout of 30 seconds set on the AWS side, and config/queue.php has no say in it.

The opposite mistake is setting 'retry_after' => null on a Redis connection. That skips reserved-job migration altogether, so a job whose worker dies is never recovered. You avoid duplicates and lose jobs instead, with no error to show for it.

The ordering rule, with numbers

The full chain has three links, because Horizon's supervisor timeout is a separate value from the job's, and with balance => 'auto' Horizon force-kills workers it considers hung during scale-down:

job timeout < supervisor timeout < retry_after

Each step needs more than a second of headroom. A worked example for a job whose realistic worst case is three minutes:

Setting	Value	Why
Realistic worst-case runtime	180s	Measured: the p99 from your own metrics
Job `#[Timeout]`	240s	Above the worst case, so normal runs are never killed
Supervisor `timeout`	300s	Above every job timeout in the supervisor, so scale-down does not kill live work
Connection `retry_after`	390s	Above the supervisor timeout, plus room for the alarm handler to finish and exit

Over-shooting retry_after has a small, bounded cost: a job whose worker did die waits longer before it is recovered. Under-shooting it causes duplicate execution, so when in doubt, make the lease longer.

A supervisor serves one connection, so you can check the constraint in a test:

// tests/Feature/QueueTimeoutOrderingTest.php
public function test_supervisor_timeouts_stay_below_their_connection_lease(): void
{
    foreach (config('horizon.defaults') as $name => $supervisor) {
        $connection = $supervisor['connection'];

        // The 60 mirrors the framework's own fallback when the key is absent.
        $retryAfter = config("queue.connections.{$connection}.retry_after", 60);

        $this->assertLessThan(
            $retryAfter,
            $supervisor['timeout'] ?? 60,
            "Supervisor [{$name}] can hand a running job to a second worker.",
        );
    }
}

The test covers supervisors. It cannot see a $timeout property or a #[Timeout] attribute on an individual job class, which is the most common way the chain breaks. Those you can only catch at runtime, when the worker resolves the effective timeout for the job in front of it.

Skyline runs the supervisor half of that check for you: php artisan horizon warns at startup about any supervisor whose timeout is not below its connection's retry_after.

Catching it in the logs

This misconfiguration can survive in production for months because the queue's own instrumentation reports it as ordinary activity. An expired reservation looks like a recovered job, and a second worker picking the job up looks like any other pickup.

Skyline's job lifecycle logging writes one line per transition, tagged with the job id, including the transitions Laravel does not log: reserved, migrated, released and timed out. When a job leaves the reserved set because its lease ran out, the line is a warning that names the cause, and the pattern to look for is a single job id reserved twice with that warning between the two reservations:

[11:04:12] queue.DEBUG: [job:91827364] reserved from [exports] and started processing.
[11:05:42] queue.WARNING: [job:91827364] was still reserved on [exports] when its reservation expired (retry_after=90s) and has been put back on the queue. Either the worker running attempt 1 died, or the job runs longer than retry_after and this copy will run while the first is still going; keep every job's timeout below retry_after (reason=reservation_expired).
[11:05:45] queue.DEBUG: [job:91827364] reserved from [exports] and started processing.
[11:06:20] queue.DEBUG: [job:91827364] completed.

The job was picked up at 11:04:12. Ninety seconds later, still running because no completion line had appeared, it was put back on the ready queue and picked up again three seconds after that. A grep for the job id is enough to tell a retry from a double reservation, which the dashboard alone cannot do. The reserved lines are debug, so enable that level on the channel for the investigation and turn it off afterwards; the expired-reservation line is a warning and can stay on permanently.

Common questions

What is retry_after in Laravel queues?

It is a lease on a reserved job, not a delay before a retry. Popping a job writes it to the connection's reserved set with an expiry of now plus retry_after seconds. Every later pop on that queue first sweeps that set and moves anything expired back onto the ready queue, whether or not the worker that reserved it is still running it. Nothing checks the original worker is alive, so an expired lease on a live job looks the same as one on a worker killed by a deploy.

What are the default values of timeout and retry_after?

The worker timeout defaults to 60 seconds. That is the default of queue:work --timeout, queue:listen --timeout and the Horizon supervisor timeout option. retry_after is 90 seconds in the config/queue.php that ships with the Laravel skeleton, but the framework's own fallback, used whenever a connection array omits the key, is 60. SQS has no retry_after at all; its lease is the queue's Default Visibility Timeout on the AWS side, which defaults to 30 seconds.

Why is my Laravel job running twice?

Almost always because its effective timeout is not smaller than the connection's retry_after, so the reservation expires while the job is still running and a second worker picks the same payload up. The effective timeout may not be the one in your supervisor config: a $timeout property or #[Timeout] attribute on the job class takes precedence, and --timeout=0 disables the alarm entirely rather than firing it immediately.

Is it safe to set timeout equal to retry_after?

No. The reservation clock starts at pop(), while the timeout alarm is armed several steps later: after the reservation is written, after JobReserved and its listeners run, and after the worker enters process(). With identical values the reservation expires first every time, and the alarm handler still has to record the failure and exit after that.

Why did my job fail with MaxAttemptsExceededException when it actually succeeded?

Because the reserved copy was migrated back and popped by a second worker, and the pop incremented the attempt count. With tries set to 1, as in the config/horizon.php that Horizon publishes, the second copy exceeds its limit before fire() is reached and is failed immediately, while the first copy is still running and goes on to complete. With tries above 1 the attempt check passes and both copies execute, with no failure recorded to tell you.

Laravel Horizon Queue Balancing: Idle Workers vs. Starved Queues

laravel-o11y — Sun, 13 Sep 2026 16:51:50 +0000

Every Horizon supervisor makes you choose. Turn balancing on and you pay for a floor of idle workers. Turn it off and a busy queue starves the queues listed after it. The more queues you run, the more each option costs, and many teams find this out only when a flood on one queue stalls all the others.

This article explains where the trade-off comes from, why running many small queues makes it worse, and how Skyline's weighted queues give you a shared pool of workers without starvation.

Key takeaways

Balancing on ('auto' or 'simple') gives every queue its own worker pool. Nothing starves, but you pay a permanent floor of resident workers that scales with your queue count.
Balancing off (false) runs one shared pool with no idle floor, but the queue list becomes a strict priority order, so a flood on one queue starves every queue below it.
Both costs scale with the number of queues, which penalises the teams who split work into many queues to get per-queue pausing, draining and visibility.
Skyline's queueWeights keeps the shared pool of unbalanced mode but checks queues in proportion to their weights (3 : 2 : 1) rather than in strict order. Important work gets most of the pickups and no queue drops to zero.

Why you end up with thirty queues in the first place

A queue is the smallest unit Horizon lets you observe and operate on, and Laravel's queue:pause and queue:clear commands both act on one queue at a time. So when you want operational control ("show me how exports is doing", "pause webhooks while the downstream API is down", "empty the reindex backlog without touching payments"), you split work into more, narrower queues. (Skyline adds pausing and resuming a single queue from the dashboard rather than the whole supervisor.)

That is how a healthy app ends up with thirty queues: payments, emails, webhooks-stripe, webhooks-github, exports, imports, reindex, thumbnails, and on down the list. Each one gets its own dashboard row and its own pause and drain switch.

Both balancing modes have a cost that grows with the number of queues, so the teams that split work most finely pay the most.

Option A: `balance => 'auto'` and the idle-worker floor

With balance => 'auto' (or 'simple'), Horizon allocates worker processes per queue. The auto balancer scales each queue's worker count up and down with its workload, but never below minProcesses, which defaults to 1:

'supervisor-1' => [
    'connection'   => 'redis',
    'queue'        => ['payments', 'emails', 'exports', /* ...27 more */],
    'balance'      => 'auto',
    'minProcesses' => 1,
    'maxProcesses' => 40,
],

minProcesses guarantees that every queue always has at least one worker watching it, so a flood on exports can't prevent payments from being serviced. Each queue is isolated, and for many setups that makes this the right default.

The cost is that one minimum worker per queue across thirty queues is a permanent floor of thirty worker processes. They are booted, resident, and idle most of the time, because most queues are empty most of the time. Each PHP worker is a full framework boot holding tens of megabytes of RAM whether or not it ever picks up a job.

The floor also counts against your ceiling. If minProcesses across thirty queues already pins thirty workers and maxProcesses is 40, the auto balancer has only ten processes left for the queue that is busy. Raising maxProcesses gives that queue more room, but the thirty idle workers are still there.

Option B: `balance => false` and starvation by list order

With balance => false, Horizon runs a single shared pool of workers. Every worker listens to every queue and checks them in the order you listed them. There is no per-queue floor and no idle minimum, and workers go wherever the work is. On paper this is the efficient option:

'supervisor-1' => [
    'connection'   => 'redis',
    'queue'        => ['high', 'default', 'low'],
    'balance'      => false,
    'maxProcesses' => 10,
],

The problem is the list order. Without balancing, the queue list is a strict left-to-right priority. A worker only looks at default when high is empty, and only looks at low when both are empty. Under light load you won't notice. Under sustained load, the order decides who gets served at all:

A flood on a higher-priority queue starves every queue below it until the flood drains.

Push fifty thousand jobs onto high and every worker in the pool works on it. default and low get no workers for as long as high stays non-empty. Dropping the idle-worker floor cost you the isolation that balancing guaranteed.

What is the difference between `balance` auto, simple and false?

Horizon ships two balancing strategies and an off switch. The two strategies are closer than most people assume: both 'auto' and 'simple' give every queue its own pool of worker processes, and they differ only in whether that allocation can move.

'simple' splits the supervisor's processes evenly across the queues in the list and leaves them there. Ten processes across two queues means five each, even if one of them has been empty all day.
'auto' starts from the same per-queue pools but continuously reallocates them to match each queue's workload, scaling a busy queue up toward maxProcesses and a quiet one down to minProcesses. (Whether "workload" means queue size or estimated time-to-clear is configurable via autoScalingStrategy.)
false is not a balancing strategy. There are no per-queue pools: one shared pool of workers listens to every queue and checks them in the order you listed them.

Side by side, with Skyline's weighted mode in the last column:

	`'simple'`	`'auto'`	`false`	`false` + weights
Worker pools	One per queue, fixed even split	One per queue, resized to load	One shared pool	One shared pool
Idle-worker floor	`maxProcesses` ÷ queues, always resident	`minProcesses` per queue (defaults to 1)	None	None
Can a queue starve?	No	No	Yes, strict list order	No, every queue keeps a share
Priority model	None (even split)	None (load-driven)	Strict left-to-right	Proportional (3 : 2 : 1)
Adds capacity to a spike	No	Yes, scales processes up	Shared pool absorbs it	Shared pool absorbs it

The upstream reference for the two strategies is the Horizon balancing documentation.

Why the trade-off gets worse with every queue you add

The two options are mirror images:

Balancing on gives you isolation (no queue starves) and charges you a per-queue process floor (idle workers, and less of the ceiling left for busy queues).
Balancing off gives you a shared pool (no idle floor) and charges you starvation (strict priority order means a flood on one queue stops every queue below it).

Both costs scale with queue count, which is why teams with many queues feel them most. More queues means a taller idle floor under Option A, and under Option B a longer priority list with more queues below any flood. The granularity that made thirty queues attractive makes both options expensive.

Skyline's third option: weight the queues

The dilemma assumes that "balancing off" must mean strict priority: all of high before any of default. Skyline keeps the single shared pool of balance => false, with no per-queue minimum and no idle floor, and replaces strict list order with a proportional one via a queueWeights map:

'supervisor-1' => [
    'connection'   => 'redis',
    'queue'        => ['high', 'default', 'low'],
    'balance'      => false,
    'maxProcesses' => 10,
    'queueWeights' => [
        'high'    => 3,
        'default' => 2,
        // 'low' is omitted, so it gets the default weight of 1
    ],
],

Instead of always checking high first, workers check the queues in proportion to their weights, roughly 3 : 2 : 1. About 50% of pickups favour high, 33% default and 17% low. Queues you leave out of the map get a weight of 1, so a queue you forget to list still gets picked up.

Take the same flood of fifty thousand jobs on high. Under strict priority, low would get no workers until high drained. Under weights, most pickups still go to high, which is weighted highest, but one in six reaches low, so low keeps draining instead of stalling.

How weighted queues compare with both balancing modes

No idle-worker floor. Weights live under balance => false, so there's no per-queue minProcesses. Thirty queues do not mean thirty resident workers. They share one pool, the same as plain unbalanced mode.
No starvation. Every queue has a weight of at least 1, so every queue is checked on a regular cadence. A flood can reduce a sibling queue's share, but not its throughput to zero.
Explicit priority. A 3 : 2 : 1 map says how much each queue matters, instead of encoding priority in list position or paying for it with dedicated worker pools.
Per-queue operations stay. You keep per-queue stats, pause, drain and inspect for all thirty queues without paying the cost of balancing.

When should you still use `balance => 'auto'`?

Weighted queues don't replace balance => 'auto' everywhere. Auto-balancing's strength is elastic capacity: it starts more workers for a queue that is spiking, stops them afterwards, and scales the total process count to demand. Weights don't add processes. They redistribute a fixed pool across queues. If your bottleneck is throughput on one queue and you have CPU headroom, auto-scaling is still the better fit.

Weighting fits a specific, common setup best: many queues kept for operational granularity, most of them idle most of the time, with occasional floods on one queue that should not hold up the others. That is also where the balance on/off choice costs the most.

Extending the Horizon Dashboard: Adding a UI to a Compiled Vue Bundle

laravel-o11y — Sun, 13 Sep 2026 16:39:28 +0000

Horizon's dashboard is a compiled Vue application with no plugin API. If you want to add a panel to the job details page, the usual options are to fork the package or to publish its assets and patch the bundle, and both mean redoing the work on every release. knobik/laravel-horizon-job-output takes a third approach that needs neither.

The package gives a queued job the same output API an Artisan command has ($this->info(), table(), withProgressBar()) and streams that output onto Horizon's job details page while the job runs. It also adds a Reserved Jobs page to the sidebar and a Cancel Batch card to the batch screen. None of this forks Horizon, publishes its assets or modifies Horizon's code.

This article walks through how it does that. The same techniques apply to any package that extends a Laravel package whose frontend ships pre-compiled.

Credit where it is due

Every technique below comes from knobik/laravel-horizon-job-output, by knobik, MIT licensed. The code quoted in this article is theirs, abridged for length. We are writing it up because it solves cleanly a problem most people solve by forking. If any of it is useful to you, star the repo.

Key takeaways

Get your data in through the repository, not a new endpoint. RedisJobRepository::$keys is a public whitelist read with HMGET. Append a field to it and that field flows through Horizon's existing /api/jobs/{id} response with no route or controller override.
Override the layout view, then render the original from a second namespace. addNamespace('horizon-original', …) plus prependNamespace('horizon', …) lets you patch Horizon's real rendered HTML instead of shipping a copy of it that drifts.
Mount inside Vue's in-DOM template, not into Vue-owned DOM. A <div> spliced in after <router-view></router-view> compiles to a static node: Vue renders it once and never patches it again, so your plain JavaScript owns it safely.
Horizon's catch-all route gives you SPA routes for free. It serves the layout for any path under the dashboard prefix, so a URL the compiled router has no route for renders an empty router view, which leaves your mount as the only thing on the page.
Register routes in register(), not boot(). Every provider's register() runs before any provider's boot(), which is the only placement that beats Horizon's catch-all regardless of package discovery order.
Assume every anchor will move. Each patch is independently optional, logs what the dashboard will be missing, and a scheduled CI job runs the suite against laravel/horizon:dev-master so drift arrives as a warning rather than a bug report.

The problem: a dashboard with no seams

Horizon ships its frontend as a compiled bundle in public/vendor/horizon, mounted by a Blade layout that is little more than one <div id="horizon"> containing a <router-view>, a sidebar and a script tag. There is no Horizon::registerPanel(), no view slot and no JavaScript event bus.

That leaves three usual options:

Approach	Cost
Fork Horizon	You now maintain a queue dashboard. Every upstream release is a merge.
Publish and patch the assets	Your patch is a build artifact in `public/`. `horizon:publish` overwrites it on the next deploy without warning.
Ship a copy of the layout view	Works until Horizon changes its layout, at which point your users get the old dashboard with none of the new features and no error to explain it.

The package does none of these. It uses Horizon's rendered HTML as the extension point and treats every assumption about that HTML as provisional: it patches the render output, not the source. The rest of the design follows from that decision.

Getting your data into Horizon's API

The output has to reach the browser first. The obvious way is a new endpoint, GET /horizon/api/job-output/{id}, with your own controller and your own Redis read. The package avoids adding one.

Horizon reads each job out of Redis in RedisJobRepository, and it does not use HGETALL. It reads a fixed whitelist of hash fields with HMGET, and that whitelist is a public property:

// Knobik\HorizonJobOutput\HorizonJobOutputServiceProvider
protected function exposeOutputOnJobRepository(object $repository): void
{
    if (! property_exists($repository, 'keys')) {
        return;
    }

    if (! in_array(JobOutputStore::FIELD, $repository->keys, true)) {
        $repository->keys[] = JobOutputStore::FIELD;
    }
}

The repository is a singleton, so appending output to it once at boot applies process-wide. From then on, Horizon's own /api/jobs/{id} endpoint returns the field alongside status, payload and the rest. The dashboard makes no second request, and there is no new route or controller whose authorization could be wrong: the data comes back from an endpoint Horizon already gates.

Storage follows the same approach. The output is written as a field on Horizon's own job hash rather than under a key of its own:

// Knobik\HorizonJobOutput\RedisJobOutputStore
public function put(string $jobId, string $output): bool
{
    $connection = $this->connection();

    if (! $connection->exists($jobId)) {
        return false;
    }

    $connection->hset($jobId, self::FIELD, $output);

    return true;
}

Sharing the key means sharing its TTL. Horizon already trims completed, failed and recent jobs on the schedule set by horizon.trim.*, and because the output is a field on that same hash, the same policy trims it. There is no cleanup command to run and no way for the two lifetimes to drift apart; a separate key would have needed a retention policy of its own. The exists() guard is what keeps this safe: HSET on a missing key would create a new hash with no expiry, a leak that grows by one key per job and never shrinks.

Owning the layout without forking it

Laravel's view finder resolves horizon::layout through a namespace hint. A package that registers its own view directory under the horizon namespace with prependNamespace() wins that lookup. That part is well known. The problem is that you then have to supply a layout, which usually means copying Horizon's and keeping the copy in sync with every release.

The package avoids that by keeping Horizon's own view path reachable under a second name before taking over the first:

// Knobik\HorizonJobOutput\HorizonJobOutputServiceProvider
protected function registerViewOverride($view): void
{
    $finder = $view->getFinder();
    $hints = $finder->getHints();

    if (! isset($hints['horizon'])) {
        return;
    }

    $finder->addNamespace('horizon-original', $hints['horizon']);
    $finder->prependNamespace('horizon', __DIR__.'/../resources/views');
}

The override view is then three lines, none of them Horizon's markup:

{!! app(\Knobik\HorizonJobOutput\LayoutDecorator::class)->decorate(
    view('horizon-original::layout', ['isDownForMaintenance' => $isDownForMaintenance])->render()
) !!}

Horizon renders its real layout, and the decorator receives the resulting HTML string and splices things into it. A Horizon release that changes the sidebar, the theme switcher or the asset URLs changes them here too, because this is Horizon's layout with a few extra nodes added.

Two more details are worth copying. The registration runs from $this->app->booted() and through callAfterResolving('view', …), so a request that never renders a view never pays to construct Blade's finder. And every splice goes through one method, which logs a missing anchor and returns the HTML unchanged:

// Knobik\HorizonJobOutput\LayoutDecorator
protected function patch(string $html, string $anchor, string $insert, string $missing, bool $before = false, int $from = 0): string
{
    $position = strpos($html, $anchor, $from);

    if ($position === false) {
        $this->warn($anchor, $missing);   // logs what the dashboard will be missing

        return $html;
    }

    return substr_replace($html, $insert, $before ? $position : $position + strlen($anchor), 0);
}

Each caller passes its own $missing string, such as "the output panel will not be shown", "the Reserved Jobs link will be missing" or "nothing this package adds will load". If a Horizon release moves an anchor, the result is a log line naming the feature that disappeared, rather than a 500 on the dashboard or a blank panel with nothing in the logs.

Where to put a mount point in a Vue app you don't control

The usual approach is to wait for the SPA to render and then appendChild into it. That puts your node inside DOM that Vue's virtual DOM believes it owns, and the next patch (a poll updating the job status, or a route change) either removes it or leaves Vue diffing against a tree that no longer matches.

The package inserts its mounts server-side, into the HTML string, immediately after Horizon's router view:

protected const ROUTER_VIEW_ANCHOR = '<router-view></router-view>';

// ...

return $this->patch($html, self::ROUTER_VIEW_ANCHOR, $mounts, 'the output panel will not be shown');

That position puts the <div id="hjo-root"> inside #horizon, which Vue uses as its in-DOM template. Vue compiles the element's existing markup into its render function, and a node with no directives, bindings or interpolation compiles to a static node: rendered once, then skipped by every later patch. So the mount sits inside the app, at the right place in the layout, and the diff never touches it. Plain JavaScript can own it.

The scripts and styles go just before </body>, outside #horizon, so Vue never tries to compile them.

The general rule

If you need to add DOM to a compiled SPA you don't control, add it to the template the app compiles from, before the app boots, and not to the DOM the app has already rendered. Markup in the template becomes a static node the framework leaves alone. A node appended to rendered DOM can be removed by the next re-render.

Adding a whole page the compiled router has never heard of

The package also adds a new page, Reserved Jobs at /horizon/reserved, to a compiled router that has no route for it.

This works because Horizon's dashboard routes end in a catch-all GET route that matches every path under the prefix and returns the same layout. So /horizon/reserved already serves the app. The compiled Vue router finds no route matching that path and renders an empty <router-view>, which leaves the package's own mount, placed immediately after it, as the only content in the column.

The sidebar link differs from Horizon's own nav markup in one way:

<li class="nav-item">
    <a href="{$href}" class="nav-link d-flex align-items-center" data-hjo-nav>
        <svg …></svg>
        <span>Reserved Jobs</span>
    </a>
</li>

It is a plain <a href>, not a <router-link>. The nav is inside #horizon, so Vue compiles whatever is put there, and a router-link pointing at a route the bundle does not know about resolves to nothing. A plain href performs a full navigation, and Vue leaves it alone. The href is built from horizon.proxy_path and horizon.path, mirroring how Horizon's own bundle computes its base path, so the link still works with a custom dashboard path or behind a reverse proxy.

Registering routes Horizon's catch-all won't swallow

The Reserved Jobs page needs an API endpoint, and that endpoint lives under the same prefix as the catch-all. When two routes match, the one registered first wins.

Horizon adds its catch-all from boot(), so the package registers its routes from register():

public function register(): void
{
    $this->mergeConfigFrom(__DIR__.'/../config/horizon-job-output.php', 'horizon-job-output');

    // ... bindings ...

    // Registered here rather than in boot(). Horizon's dashboard ends in a
    // catch-all route matching everything under its prefix, added from its
    // own boot(), and whichever route is registered first wins. Every
    // provider's register() runs before any provider's boot(), so this is
    // the only placement that beats the catch-all no matter what order the
    // packages were discovered in.
    $this->registerRoutes();
}

You don't control package discovery order, but the container guarantees that every register() runs before any boot(), so a route registered in register() always comes first.

The cost is that Horizon has not booted yet, so its route group cannot be reused and has to be rebuilt, middleware stack included:

protected function middleware(): array
{
    $middleware = (array) config('horizon.middleware', ['web']);

    if (class_exists(SentinelMiddleware::class)) {
        array_unshift($middleware, SentinelMiddleware::class.':horizon');
    }

    return $middleware;
}

Newer Horizon versions collect this into a named horizon middleware group, but that group does not exist across the whole ^5.0 range the package supports, and naming a group that was never registered makes the router try to resolve a class by that name. Rebuilding the list works on every version in the range.

Authorization needs more care. Horizon attaches its Authenticate middleware in its base controller, not on the route group, so a controller that does not extend that base class gets no authorization check. The package applies the middleware explicitly, around the whole route file rather than per route:

// routes/reserved-jobs.php
Route::middleware(Authenticate::class)->group(function () {
    Route::get('/api/reserved-jobs', [ReservedJobsController::class, 'index']);
    Route::post('/api/reserved-jobs/release', [ReservedJobsController::class, 'release']);
});

If you are writing your own Horizon extension, check where the package you are extending applies its gate. A route group under horizon.middleware alone gets you the web stack and nothing else, so your endpoint would be open to anyone who can reach the dashboard's URL, whether or not they pass the viewHorizon gate.

The feature toggles are enforced in the controllers, not around route registration:

public function index(): array
{
    abort_unless(config('horizon-job-output.reserved_page', true), 404);

    return ['jobs' => $this->reserved->all()->all()];
}

Gating the registration would bake the setting into a cached route table, so changing the config would have no effect until someone ran route:clear. Checking in the controller avoids that.

Reacting to navigation without access to the router

The panel has to know when the user navigates to a job details page. It cannot ask the router, because the router is inside a bundle the package has no reference to. Vue Router in history mode pushes state rather than reloading, so the package wraps the two history methods it calls and re-announces them as an ordinary DOM event:

['pushState', 'replaceState'].forEach((method) => {
    const original = history[method];

    history[method] = function () {
        const result = original.apply(this, arguments);
        window.dispatchEvent(new Event('hjo:navigated'));
        return result;
    };
});

function onNavigation(sync) {
    window.addEventListener('hjo:navigated', sync);
    window.addEventListener('popstate', sync);

    if (document.readyState === 'loading') {
        document.addEventListener('DOMContentLoaded', sync);
    } else {
        sync();
    }
}

The browser fires popstate for the back and forward buttons; the wrapper covers navigation the app triggers itself, which does not fire it. This code lives in a shared support module concatenated ahead of the feature scripts, so the history methods are wrapped once however many features are enabled, instead of by whichever feature script happened to load first.

The current screen is then read from the URL, since every Horizon screen has its own path:

function currentJobId() {
    const path = support.dashboardPath();

    const preview = path.match(/^\/jobs\/[^/]+\/([^/]+)\/?$/);
    if (preview) {
        return preview[1];
    }

    const failed = path.match(/^\/failed\/([^/]+)\/?$/);
    if (failed) {
        return failed[1];
    }

    return null;
}

Because Vue creates the mount point when it compiles the layout template, on a cold load this code can run before the element exists. whenElementExists retries up to 50 times at 100ms intervals instead of assuming a boot order.

Shipping frontend assets with no build step

A package extending a compiled dashboard cannot add itself to that dashboard's build. Publishing files into public/ works, but it needs a publish step in every deploy, and Horizon's own horizon:publish writes to the same directory.

So everything is inlined into the layout at render time: one <style>, one <script type="module">, and a settings object serialised with Js::from(). That is the same helper Horizon uses for its own settings, and it applies the escaping needed to embed data inside a script tag.

The terminal renderer takes more work. The package vendors an xterm.js build so a progress bar redraws in place on the dashboard as it would in a shell. An ESM build ends in an export statement, and nothing can import from an inline module, so the export does nothing there. The package rewrites it into a global assignment as it inlines the file:

protected const EXPORT_PATTERN = '/export\s*\{\s*(\w+)\s+as\s+Terminal\s*\}\s*;?/';

// The export is the last statement in the bundle, so only the tail is
// searched. Running the pattern over the whole 345KB build would repeat
// that scan on every dashboard request for no added certainty.
$tail = substr($js, -self::TAIL_BYTES);

if (! preg_match(self::EXPORT_PATTERN, $tail, $matches, PREG_OFFSET_CAPTURE)) {
    Log::warning('[horizon-job-output] Could not rewrite the xterm export, so the terminal renderer was skipped. …');

    return ['css' => '', 'js' => ''];
}

$js = substr_replace($js, 'globalThis.HorizonJobOutputTerminal = '.$matches[1][0].';', /* … */);

Rewriting a vendored bundle with a regular expression is brittle, and the package handles it accordingly. The pattern runs over the last 512 bytes only, a miss is logged, and the panel falls back to an HTML renderer that collapses the control sequences and needs no extra payload. A failed rewrite costs you the xterm renderer, not the output panel.

Capturing output from a job that is already running

The dashboard side needs output to show, and getting an output object into a running job has problems of its own.

It cannot happen at dispatch. A queued job is serialised, and an unserialised object never runs its constructor, so anything the constructor attached is gone by the time the worker has it. The attachment has to happen at execution time, inside the worker, which is what a global bus pipe allows:

protected function registerBusPipe(): void
{
    $dispatcher = $this->app->make(BusDispatcherContract::class);

    if (! $dispatcher instanceof BusDispatcher) {
        return;
    }

    try {
        $property = new ReflectionProperty($dispatcher, 'pipes');
        $pipes = (array) $property->getValue($dispatcher);
    } catch (Throwable) {
        $pipes = [];
    }

    if (in_array(CaptureJobOutput::class, $pipes, true)) {
        return;
    }

    $pipes[] = CaptureJobOutput::class;

    $dispatcher->pipeThrough($pipes);
}

pipeThrough() replaces the pipe list outright and there is no getter, so the package reads the existing pipes by reflection and appends to them. Replacing the list instead would drop the pipes other packages have registered, with no error.

Two cases need their own handling:

Artisan commands run inside a job. The console kernel writes a command's output to whatever buffer it is handed and discards it when handed nothing. So the kernel is decorated for the length of the job (its call() supplies the job's output as the default buffer) and restored in a finally, because a worker handles one job after another in the same process and a stale decorator would feed a finished job's output. The facade's resolved instance is cleared alongside the binding, since Artisan::call() is how a job runs a command in practice and a facade holds on to whatever it resolved first.
Queued Artisan commands. Artisan::queue() dispatches a QueuedCommand, which does not use InteractsWithQueue, so nothing ever sets a job on it and the bus pipe has no way to reach the Horizon id its output belongs on. The package listens to Queue::before() / after(), keeps the job the worker has in hand in a small singleton, and only hands out its id when the payload's commandName matches the command being piped. That check stops a command dispatched inside another job from writing over the outer job's output.

The write path is buffered with a flush interval, capped at max_bytes, and flushed with force: true from a finally. A job that throws keeps whatever it wrote before the exception, which is usually the output you most want to read.

Designing for the day Horizon changes

Everything above depends on internals that carry no compatibility guarantee: a public property on a repository, a private property read by reflection, two string anchors in rendered markup, and the shape of a trailing export in a vendored bundle. Any Horizon release could break one of them, so the package is built to make that breakage cheap and visible.

Every patch is independently optional: a missing anchor costs you that one feature and logs which one. Every failure is a Log::warning naming the anchor and the consequence, so the first person to hit it can diagnose it without reading the package source. And a scheduled CI job tests against Horizon's development branch:

# .github/workflows/horizon-canary.yml
on:
  schedule:
    - cron: '41 6 * * 1'

# ...

      - name: Install Horizon from its development branch
        run: |
          composer require --no-update "laravel/horizon:dev-master"
          composer update --no-interaction --prefer-dist

It runs the full suite against laravel/horizon:dev-master once a week and, on failure, opens a labelled issue listing which four internals might have moved. Upstream drift shows up as an issue on a Monday morning, not as a user's bug report after a release.

The pattern, generalised

The same mechanisms work for extending any Laravel package that ships a compiled frontend:

You want to…	Mechanism	What it depends on
Add a field to an existing API response	Mutate the repository's field whitelist at boot	The property staying public
Add markup to a view you don't own	Re-register the original under a second namespace, prepend your own, render and patch	String anchors in the rendered HTML
Own DOM inside a compiled SPA	Splice a bare element into the in-DOM template server-side; it compiles to a static node	The framework not patching static nodes
Add a page to a compiled router	Use the host's catch-all route; render into your own mount when the router matches nothing	A catch-all existing at all
Beat a catch-all route	Register from `register()`, not `boot()`	Nothing; the container guarantees the order
Observe SPA navigation	Wrap `history.pushState`/`replaceState`, re-dispatch as an event, plus `popstate`	History-mode routing
Attach state to a running job	A global bus pipe, appended to the existing pipes reflectively	`Dispatcher::$pipes` staying where it is

Apart from the route ordering, none of these dependencies is guaranteed. The approach holds up because each assumption is isolated to one feature, degrades to that feature missing plus a log line saying so, and is checked by CI against the upstream development branch.

The source is around 1,500 lines of PHP and JavaScript, and close to half of it is comments explaining why the code is written the way it is.

Laravel Background Jobs: 12 Best Practices for Production Queues

laravel-o11y — Wed, 26 Aug 2026 16:18:00 +0000

Queued jobs are the part of a Laravel app that runs when you're not looking — after the request is gone, during a deploy, against data that may have changed or vanished, on code one version newer than the code that dispatched them. Most job bugs aren't logic errors; they're assumptions about when and how many times a job runs that quietly hold in development and fall apart under production load.

Here are twelve practices that keep a queue healthy under real traffic. They build on each other: small arguments make idempotency easier, idempotency makes retries safe, safe retries make backoff useful, and so on. The last one — backwards compatibility across deploys — is the one that bites hardest and no linter catches.

Key takeaways

Assume every job runs at least twice. Laravel's queues are at-least-once, and a worker killed mid-job re-runs it from scratch — with the side effects of the first attempt still committed.
Keep jobs small, atomic and explicitly queued. One job per record beats one job that loops over 10,000: a failure then retries one unit of work instead of everything.
Let the framework handle failure. Declarative $tries and exponential backoff() rather than manual re-runs, and afterCommit() so a worker can't outrun your transaction.
Read the whole contract before using ShouldBeUnique or batches. Both fail silently — a missing uniqueId() drops jobs with no error anywhere.
The queue outlives your deploy. Jobs serialized against old code are deserialized by new workers, so renaming a job class or adding a constructor argument without a class-level default destroys work that is already in flight.

1. Keep arguments small

Every byte of constructor arguments is a byte in Redis, multiplied by every queued instance. Don't pass big arrays, blobs, or rich DTOs "to save a query." Pass the minimal identifiers and re-read what you need inside handle(). A queue backlog of fat jobs is how you OOM Redis.

If you pass Eloquent models, Laravel can serialize just the class and the ID and automatically re-reload the record on execution — as long as you include the SerializesModels trait.

2. Every job should declare its queue

class GenerateImageThumbnails implements ShouldQueue
{
    public $queue = 'media';
}

Horizon routes work by queue name, and each supervisor is tuned for a workload — concurrency, timeout, balance strategy. A job with no $queue lands on default, where a slow image resize sits behind — or ahead of — a latency-sensitive job.

Mixing fast and slow, or urgent and bulk, on one queue defeats the entire point of having supervisors.

Pick the queue that matches the work's shape and SLA, and make sure a matching Horizon supervisor processes it.

3. Make jobs idempotent

Horizon delivers at-least-once. A worker that's killed mid-job — deploy, OOM, timeout — leaves the job to be retried, so the same job body can run twice. Design every job so that running it twice produces the same end state as running it once:

Check before you create (firstOrCreate, updateOrCreate, "already sent?" guards).
Use unique constraints as a backstop, not wishful thinking.
Never assume "this ran, therefore it ran exactly once."

The realistic failure mode isn't a clean re-run from the top — it's a job that did partial work and then got killed before completing. Redis only removes a job from its reserved set on success, so a hard kill (SIGKILL from a per-job timeout, or the supervisor force-killing a worker that outlived the shutdown grace window during a deploy) leaves the job to be migrated back and re-run from scratch — with whatever side effects the first attempt already committed still in place.

Because attempts is incremented when the job is reserved, not when it fails, each kill-and-recover cycle burns one try; after maxTries such cycles the job is marked failed having run its side effects several times with no clean completion. So idempotency has to cover resuming after a partial run, not just "don't duplicate a fully-successful run."

4. Keep jobs atomic and short

A deploy restarts workers; a job that exceeds its $timeout is killed mid-flight. If your job does five writes and dies after three, the retry redoes all five — and the first three had better be idempotent (see above).

Prefer one job per record over one job that loops 10,000 records. A per-record job that dies retries one record; a mega-job retries everything and may never finish inside the timeout window. Fan out:

Item::where(...)->eachById(fn (Item $i) => SyncItemJob::dispatch($i));

5. Retries and backoff — let the framework do it

Don't manually re-run failed jobs and don't catch-and-swallow. Configure retries declaratively:

public $tries = 5;

// exponential backoff — required for anything hitting an external API
public function backoff(): array
{
    return [10, 30, 60, 300];
}

Hammering an API with immediate retries turns a transient blip into an outage. Back off. Let jobs that genuinely can't succeed land in failed_jobs — that's a signal to fix the root cause, not a thing to mass-retry. Use $this->fail($e) to bail early when you know a retry won't help.

6. Dispatch after the transaction commits

DB::transaction(function () {
    $booking = Booking::create([...]);
    SendBookingConfirmation::dispatch($booking->id)->afterCommit();
});

Without afterCommit() (or the connection-level 'after_commit' => true), a fast worker can pick up the job and find() a booking that hasn't been committed yet — ModelNotFoundException, a spurious failure that only reproduces under load. Dispatch the side effect only once the data it depends on is durable.

7. The `ShouldBeUnique` contract — three ways to get it wrong

Unique jobs are where the most expensive, silent bugs live. This section is the short version; the long one — including how uniqueness interacts with WithoutOverlapping, retries and orphaned locks — is in Laravel job uniqueness controls.

Set uniqueFor. A plain ShouldBeUnique lock is acquired at dispatch and released only when processing reaches a terminal state (success, or final failure after maxTries) — it is held across the whole processing window and across backoff retries. Without uniqueFor the lock is created with no TTL (forever()), so it is reclaimed only by that eventual terminal run. Normally that's fine: a worker killed mid-job leaves the job to be retried, and the retry's terminal state clears the lock. The exposure is twofold:

Until that retry completes, every new dispatch of the same unique job is silently dropped — not errored, just gone. uniqueFor bounds this window to a known TTL instead of "however long the kill→retry→finish cycle takes."
If the job never reaches a terminal run — a uniqueId() that reads external mutable state (so the release computes a different key than acquire), a lost reserved entry, or maxTries: 0 with a job that's killed every attempt — the forever() lock is never reclaimed and the job is silently undispatchable until the key is cleared by hand. uniqueFor is the only thing that auto-recovers it.

Caveat: uniqueFor must exceed your worst-case total processing-plus-retry time, or the lock expires mid-legitimate-run and a duplicate can be dispatched. It's a ceiling on the deadlock risk, not a free safety net — keep uniqueId() a pure function of the job's own data.

public int $uniqueFor = 3600; // lock self-heals after an hour

uniqueId is mandatory for parameterized jobs. The lock key is laravel_unique_job:<class>:<uniqueId>. Omit uniqueId on a job that takes arguments and the key collapses to the class name alone — so SyncCompany(1) and SyncCompany(2) share one lock and one of them is silently dropped at dispatch. Lost work, no error.

public function uniqueId(): string
{
    return (string) $this->companyId;
}

A genuinely class-wide singleton is fine — make the intent explicit by returning a constant from uniqueId().

8. Never bulk or batch a unique job

Queue::bulk() / Bus::bulk() push raw payloads straight to Redis, skipping the dispatcher that acquires the lock — uniqueness silently does nothing. And batching a unique job means a dropped duplicate desyncs the batch's up-front job count, so its then/finally callbacks never fire and the batch hangs as "pending" forever. Dispatch unique jobs individually.

Periodic / scheduled jobs should implement ShouldBeUnique so a slow run doesn't overlap the next tick.

9. A batchable job must honour cancellation

Cancelling a batch only stops future dispatches — jobs already on the queue still wake up and run their full body unless they check. For anything that mutates state (writes files, calls APIs, charges cards), that's wasted work against a batch the caller abandoned. Guard it:

public function handle(): void
{
    if ($this->batch()?->cancelled()) {
        return;
    }
    // ... heavy work
}

…or centralise it with the Illuminate\Queue\Middleware\SkipIfBatchCancelled middleware.

10. Don't `sleep()` in a job

A sleeping job pins a worker doing nothing, starving every other job behind it. If you need to wait — rate limits, a not-yet-ready upstream — release the job back with a delay instead:

$this->release(60); // re-queue, free the worker, try again in a minute

11. The queue outlives your deploy — backwards compatibility

This is the one that bites hardest and isn't covered by any linter, so read it twice.

When you deploy, there are already jobs sitting in the queue, serialized against the old code. A worker on the new code has to deserialize and run them. Two ways this goes wrong.

Renaming or moving a job class

The serialized payload stores the fully-qualified class name — App\Products\Jobs\SyncStockJob. Rename the class, move it to another namespace, or delete it, and every already-queued instance becomes unresolvable: deserialization throws, the job lands in failed_jobs, the work is lost.

So:

Two-phase it. Keep the old class (even as a thin subclass of the new one) for one deploy cycle, let the queue drain, then remove it in a follow-up deploy.
Or drain the queue of that job type before shipping the rename.
Don't rename a hot job class in the same PR that renames its behaviour.

Changing constructor arguments

The subtle one. PHP's unserialize() does not call the constructor — it restores the saved properties directly. So your constructor's default values do nothing for jobs already in the queue.

class SyncStockJob implements ShouldQueue
{
    // ⚠️ promoted, no class-level default.
    // An old payload serialized before $force existed restores WITHOUT it →
    // accessing $this->force throws "must not be accessed before initialization".
    public function __construct(
        public int $productId,
        public bool $force = false,
    ) {}
}

The constructor default = false only helps new dispatches. An old queued job never goes through the constructor, so its $force stays uninitialized, and the new handle() that reads it crashes. Fix: give the property a class-level default so deserialized old jobs fall back gracefully:

class SyncStockJob implements ShouldQueue
{
    public bool $force = false; // ← real default, survives unserialize

    public function __construct(public int $productId, bool $force = false)
    {
        $this->force = $force;
    }
}

Which job changes are safe to deploy?

The question to ask of any change is: what happens to a payload that was serialized an hour ago, by the code you are about to replace?

Change	Safe in one deploy?	What happens / what to do
Edit `handle()` body	Yes	Behaviour isn't serialized — only properties are. Just make sure it tolerates the old payload shape.
Add a property with a class-level default	Yes	Old payloads restore without the key and fall back to the default.
Add a promoted constructor argument	No	`unserialize()` never calls the constructor, so the property stays uninitialized and reading it throws. Give it a class-level default instead.
Remove or rename a property	No	In-flight payloads still carry the old key. Deprecate over one deploy cycle, then remove.
Rename / move / delete the job class	No	The FQCN is stored in the payload; queued jobs become unresolvable and land in `failed_jobs`. Two-phase it, or drain first.
Change the queue name	Yes, with care	Jobs already queued stay on the old queue — keep a supervisor processing it until it drains.

The general rules that fall out of that:

New properties get a class-level default value (or are nullable), never a bare typed property.
Don't remove or rename a property a queued payload still carries without a deprecation window.
handle() must tolerate both the old and the new payload shape for the length of one deploy cycle.
When in doubt, two-phase: ship the backwards-compatible change, let the old jobs drain, clean up in a later deploy.

12. Watch it run

Use Horizon. Watch queue wait times, failure rates, and throughput per queue — a backlog on one supervisor is invisible from the others. A job that's "working" in the sense of not erroring can still be silently 20 minutes behind. Instrumentation is how you find out before a customer does.

The core takeaway

Write every job assuming it will run twice, run late, run against a deleted record, and run on code that's one version newer than the code that dispatched it. Idempotency, atomicity, backoff, afterCommit, an explicit queue, the full ShouldBeUnique / batch contracts, and — above all — backwards compatibility across deploys.

DEV Community: laravel-o11y

Laravel Middleware: Requests, Queued Jobs and the HTTP Client

Key takeaways

What middleware is

Why use it

How Laravel runs it

Request middleware

Where it gets attached

The order it runs in

What it is used for

Laravel's built-in request middleware

Popular request middleware packages

Job middleware

It also runs on listeners, mail and notifications

What it is used for

Laravel's built-in job middleware

Skipped jobs count as done, and releases use up tries

Popular job middleware packages

HTTP client middleware

What it is used for

Built-in HTTP client middleware

Popular HTTP client middleware packages

Livewire and Filament middleware

What persistent middleware is

Where persistent middleware behaves differently

Filament panels

Livewire and Filament's built-in middleware

Popular Livewire and Filament packages with middleware

Laravel AI SDK agent middleware

What it is used for

Built-in agent middleware

Agent middleware packages

Other places Laravel uses middleware

Common questions

What kinds of middleware does Laravel have?

What is the difference between global, group and route middleware in Laravel?

What job middleware does Laravel include?

What happens when a Laravel job middleware does not call $next?

Does Http::globalMiddleware() apply to every outgoing request?

What is persistent middleware in Livewire?

How does middleware work in the Laravel AI SDK?

Laravel Horizon Alerting: What Horizon Sends on Its Own and What Prometheus Adds

Key takeaways

What Horizon sends on its own

Events Horizon fires and tells nobody about

Gauges from spatie/laravel-prometheus

The rule that does not port

Counters change which rules you can write

Latency instead of length

A backlog nobody is draining

Failure ratio, per queue and per job class

Falling behind, and retry churn

The same rules as Grafana alerts

Where a scrape runs out

The alerts Skyline adds

Work that vanishes without failing

Jobs that run twice

Workers dying, by cause

How they are delivered

Which layer to stop at

Common questions

Does Laravel Horizon have built-in alerts?

How do I get alerted when Horizon stops running?

Can I alert on failed jobs with spatie/laravel-prometheus?

Why does my Grafana alert on a Horizon metric show No Data?

Which Horizon problems can Prometheus not alert on?

Rate-Limited APIs and Laravel Queues: One Request at a Time, Without Starving the Rest

Key takeaways

Two limits that look like one

Option 1: the RateLimited middleware

Option 2: WithoutOverlapping, a mutex

Option 3: funnels and throttles inside the job

Option 4: one queue, one worker

Option 5: chains and drip-feeds

Option 6: the packages

The options, side by side

The cost of nearly every option: Laravel pops before it checks

The shape that works: one slot, one limiter

Letting urgent jobs jump the line

What you can no longer see

Option 1: the `RateLimited` middleware

Option 2: `WithoutOverlapping`, a mutex

A Guzzle middleware for every `Http::` call

What `retry_after` does

What `timeout` does

With `tries => 1`: a failure on work that succeeded

With `tries => 2` or more: concurrent execution with no error

Option A: `balance => 'auto'` and the idle-worker floor

Option B: `balance => false` and starvation by list order

What is the difference between `balance` auto, simple and false?

When should you still use `balance => 'auto'`?