DEV Community: Dewald Hugo

Laracon EU 2026: What Amsterdam told us about the future of Laravel

Dewald Hugo — Fri, 15 May 2026 06:22:54 +0000

Laracon EU 2026 was the largest European Laravel conference on record. Roughly 1,000 developers descended on the Passenger Terminal Amsterdam on March 2 and 3, a glass-and-steel venue perched directly on the River IJ. What they witnessed across those two days was not a routine framework update. It was a restatement of intent.

Taylor Otwell didn’t just announce Laravel 13 at Laracon EU 2026. He released a new vision: “The clean stack for Artisans and agents.” That framing matters. “Agents” is not marketing copy here. It is a concrete architectural direction. If you work on AI-adjacent Laravel applications (and increasingly, that means most of us) what happened in Amsterdam directly shapes the decisions you make this quarter. This piece covers every headline announcement, the talks worth watching, and the practical implications for production teams.

This article is also part of our broader AI Architecture coverage. If you’re thinking through how AI fits your Laravel stack at a structural level, that is where to start.

The Venue, the Scale, and the Energy

The Passenger Terminal is not a conventional conference hall. It is a working cruise terminal with a main hall large enough to seat a thousand people without feeling cramped. The energy was palpable from the moment Nuno Maduro opened the conference. This year’s Laracon EU, with around 1,000 attendees, was the largest European Laravel conference ever.

That attendance figure matters beyond the headline number. The community that showed up was not uniform. Teams from agencies, product companies, and freelancers. Developers who had never attended a Laracon before alongside people who have been at every edition. As always with Laracons, Laracon EU 2026 was more than a conference. It’s not just about the talks. It’s about spending time with like-minded people and sharing an experience as well as celebrating the Laravel community. The pre-party the evening before was where half the real conversations happened. The dinner conversations after Taylor’s keynote ran until midnight.

The practical takeaway here: if you are weighing whether to attend in 2027, the value is not just the stage content. It is the corridor conversations with people debugging the same problems you are.

Taylor Otwell’s Keynote: The “Merge It” Moment

By far the most discussed moment of the conference was Taylor Otwell’s keynote on the evening of the first day. It was scheduled to close Day 1, and by 9pm the main hall was still packed. What followed was genuinely hard to describe without sounding like you are exaggerating.

If there was one single moment that made the vision behind the entire Laravel ecosystem tangible, it was Taylor’s live demo on stage: The complete Software Development Lifecycle, from error detection to deployment, in under two minutes, live in front of 1,000 attendees.

The sequence was deliberate. A bug occurs in the application. Nightwatch detects the error automatically. An AI agent with Nightwatch MCP integration analyses and fixes the error autonomously. A pull request is automatically opened. Laravel Cloud creates a preview environment. Taylor reviews, says “Merge it”. The change goes live.

Voice-driven. No manually written code. Two minutes.

The jawdropping moment was when the AI agent called Taylor and told him “There’s a PR in the repo, do you want me to merge it?” And then it happened: bug fixed and auto-deployed, all with just voice, without writing any code.

The room reaction was not the usual polite applause of a conference keynote. It was the particular silence that precedes a standing ovation, because people needed a few seconds to process what they had just seen. We’ve all talked about AI-assisted development in the abstract. This was a functioning, live demonstration of the entire loop closing: monitoring, diagnosis, remediation, review, and deployment, as a single integrated workflow.

Taylor himself put it succinctly: Laravel, with its clear conventions, comprehensive documentation, and 15 years of example code, is ideally positioned for an era in which AI agents are increasingly becoming part of the development process.

That is not a boast. It is a defensible structural argument. The reason LLM agents work well with Laravel is precisely because the framework has consistent, well-documented conventions that agents can reason about reliably. Convention over configuration turns out to be excellent training data.

Laravel 13: What Actually Shipped

Laravel 13 was released on March 17, 2026, announced by Taylor Otwell at Laracon EU 2026, with zero breaking changes, making it the smoothest upgrade in Laravel’s history. The only requirement change is PHP 8.3 minimum.

The headline features are worth taking individually.

PHP Attributes across 15+ locations. Laravel 13 will include PHP attributes across 15+ locations. These are optional, so nothing breaks if you don’t adopt them immediately. Middleware registration, authorisation policies, and job configuration can all now be expressed declaratively on the class itself rather than in separate registration files. Whether you adopt them on Day 1 or wait for the ecosystem to settle is a judgement call, but the direction is clear: Laravel is moving toward the attribute syntax as its idiomatic configuration style. If you are still registering middleware in long arrays, your code is already going to read as dated within a release cycle.

Reverb database driver. A Reverb database driver arrives with Laravel 13, eliminating the need for Redis in smaller broadcasting setups. This is a significant quality-of-life improvement for teams that run Reverb but do not need Redis for anything else. It reduces infrastructure surface area. Whether it holds up under real concurrency pressure is something we will find out over the next few months as teams hit it in production.

Passkey authentication. Passkey authentication is coming to all starter kits and Laravel Fortify, complemented by a client-side companion package. This is the right move, and it is overdue. Password-based authentication has a long tail of support burden attached to it. Passkeys remove that tail.

Cache::touch(). A small addition, but one that solves a genuine annoyance: extending a cache entry’s TTL without re-fetching or re-computing the value. We have implemented this manually more than once using custom cache wrappers. Having it built in is welcome.

For the complete feature analysis from an AI development perspective, see what Laravel 13 actually changes for AI development.

The Laravel AI SDK Goes Stable

This is the announcement with the longest tail. On the same date as Laravel 13’s release, the Laravel AI SDK was tagged as officially stable and out of beta.

The SDK ships with several capabilities that matter in production:

Provider-agnostic design. The SDK abstracts the interface, allowing you to switch between different LLM providers without rewriting application logic. Model failover: if one model goes down, a fallback automatically kicks in, critical for production applications. Agent Classes with schema-defined output: structured AI responses that integrate in a type-safe manner into existing application logic. Vector embedding and similarity search: integrated directly into Eloquent, significantly simplifying semantic search in Laravel applications. Queuing of AI tasks: AI tasks can be processed through Laravel’s queue system as usual.

The provider-agnostic design is the decision that matters most here. Any team currently hard-coding an OpenAI or Anthropic SDK directly into their service layer is building technical debt. The Laravel AI SDK gives you the abstraction layer that most teams were building manually anyway. Our production guide to provider-agnostic AI integration in Laravel covers the architectural patterns that the SDK now formalises at the framework level. If you are migrating existing AI integration code, that is the reference.

[Architect’s Note] The SDK’s Eloquent-integrated vector search is the quiet headline most teams will underestimate. Semantic search without a separate vector database layer is not a complete RAG replacement for every use case, but for the majority of content-search scenarios most Laravel apps actually need, it removes an entire infrastructure dependency. Assess whether you still need pgvector or Pinecone before reaching for them by default.

NativePHP Super Native: Blade Goes Truly Native

If one talk generated the most feverish corridor conversations after it ended, it was the NativePHP thread across both conference days.

Simon Hamp presented the current state of NativePHP for Mobile: the modular plugin architecture, the free open-source core “NativePHP Air,” and the tooling around Jump and Bifrost for App Store deployments. That was already interesting. Then Shane Rosenthal took the stage on Day 2 and went further.

“Super Native” is an entirely new approach where Blade components and Tailwind CSS classes are translated directly into native UI elements through a custom parser. No WebView, no bridge, no overhead. The benchmarks showed UI interactions in the microsecond range, in some tests faster than React Native and Flutter. The live demo showed a functional to-do app with Eloquent, various native UI components, and app clones generated by an LLM in a single pass that rendered as 100% native UI.

We were sceptical on first pass. Then the demo ran. The implications are real: PHP teams could build native mobile and desktop applications without leaving the Laravel ecosystem. Whether Super Native holds up under complex UI requirements in production apps is an open question. The honest answer is that nobody knows yet, because the benchmarks shown were controlled environments. But the proof of concept is convincing enough that it warrants serious evaluation for internal tools and MVPs.

The Talks That Will Shape Your Next Sprint

Beyond the keynote, Laracon EU 2026 had one of the strongest community talk lineups in recent memory. A few that are worth watching back:

Yannick Kupferschmidt: AI Won’t Fail Loudly, It’ll Fail Quietly. The danger of AI agents doesn’t lie in obvious errors, but in subtle bugs and gradual architecture drift. The antidote is “Context Engineering”: not just better prompts, but structured guidelines files, reusable skills, and deliberate control over what an agent receives as context. Larger context windows don’t automatically mean better results. Targeted context does. This is the talk most teams building AI features need to watch first.

Peter Suhm: Unblocking Your Users with AI. A pragmatic framework for integrating AI into existing Laravel applications rather than greenfield projects. Three practical entry points depending on where users get stuck, with real-world examples. No vague “AI-first” positioning. Concrete integration points.

Marcel Pociot: Refactoring to Parallel. Marcel’s talk on asynchronous refactoring patterns paired well with the AI SDK’s queue integration story. If you are building AI pipelines and not yet thinking about concurrency, this one is relevant immediately.

Tobias Petry: One Billion Records with Laravel. One of the more technically dense talks of the conference. Query optimisation, indexing strategy, and Eloquent behaviour at scale. If your application is approaching data volumes where default Eloquent patterns start to break, this is required watching.

Inertia v3. The beta for Inertia v3 was due to be tagged at any time following the conference. Simpler, slimmer, and with reduced client-side overhead. If your team runs Inertia-based frontends, the upgrade path is worth evaluating now rather than waiting for it to become urgent.

Dan Harrin: Abstractions through description, not instruction. One of the underrated talks of the event. The core argument: better abstractions emerge from describing the problem well, not from instructing your way to a solution. Directly applicable to AI prompt design and to API contract design in equal measure.

For teams looking to build agentic Laravel applications that hold up in production, Kupferschmidt’s talk on quiet AI failure is the one to pair with your architecture decisions.

AI as Infrastructure, Not a Feature

The most important thing about Laracon EU 2026 was not any single announcement. It was the tone.

AI was not presented as an add-on or a differentiator you bolt onto an existing application. It was treated as infrastructure, the same category as queues, caching, and broadcasting. The Laravel AI SDK ships with queue integration. Vector search ships inside Eloquent. Model failover ships as a first-class SDK concern.

That framing carries a practical implication: teams that are building AI features as isolated packages or service classes bolted onto otherwise conventional Laravel applications are going to find themselves refactoring again in twelve months. The framework is standardising around conventions for AI integration. Teams that adopt those conventions early will carry less technical debt.

If you are currently evaluating the current AI development stack for Laravel, the SDK stable release changes several of the decisions that were previously in play.

The production deployment implications deserve equal attention. The “Merge It” demo was not just about AI. It was about what a closed-loop deployment pipeline looks like when every component in the ecosystem integrates at the API level: Nightwatch, Cloud, the AI agent, the queue system. If you are still running manual deployment workflows, the production deployment guide is the right place to close that gap before the tooling gets further ahead of your process.

The Honest Verdict

Laracon EU 2026 was not a conference about what might be possible. What unfolded in real time wasn’t some future scenario. It’s the concrete direction the Laravel ecosystem is heading: A fully integrated workflow where monitoring, AI-powered debugging, code review, and deployment seamlessly interlock.

The announcements are real. The SDK is stable. Laravel 13 is live. The “Merge It” demo ran without edits in front of a thousand people.

What Laracon EU 2026 made clear is that the question for production Laravel teams is no longer “should we integrate AI?” The question is “how close to the framework’s conventions are we building when we do it?” The teams that answer that question well in the next six months will be significantly further ahead by the time Laracon EU 2027 arrives.

Laravel Horizon in Production: Configuring AI Queue Workloads That Actually Hold

Dewald Hugo — Fri, 15 May 2026 00:02:44 +0000

Laravel Horizon in production looks deceptively simple until your first LLM inference job times out silently and your users start receiving empty responses. Standard queue jobs (sending emails, processing images, syncing records), complete in milliseconds. AI inference jobs do not. A cold claude-sonnet-4-6 call with a dense system prompt can run for 45 seconds. A gemini-2.5-pro batch summarisation job can breach two minutes under load. Horizon’s defaults were not built for this, and the failure modes are nasty: jobs that disappear without landing in failed_jobs, rate limit retries that exhaust the tries budget in under 30 seconds, and expensive inference work discarded mid-completion.

This guide is part of the AI Deployment & Production Operations module, which covers the full surface area of running Laravel AI applications in production. If you are still wiring up the surrounding deployment infrastructure, the complete production deployment guide is the right starting point.

What follows covers the three layers where AI queue workloads require deliberate configuration: supervisor setup, job class design, and operational monitoring.

Why AI Jobs Break Standard Horizon Assumptions

Standard Horizon configuration assumes workers cycle through jobs in seconds. The defaults reflect that: 60-second timeouts, three retries with no backoff configuration, and supervisor settings tuned for throughput. Those assumptions collapse the moment you start queuing LLM inference.

Three failure modes come up repeatedly.

Silent timeout kills. Horizon’s default timeout of 60 seconds is aggressive for AI inference. A gpt-4o call with a large context window can sit at 50 seconds before returning its first token. Add network variance and the worker process receives a SIGKILL mid-call. No exception is logged. The job does not land in failed_jobs. It just vanishes. This is the most common support ticket pattern we see from teams that have not tuned Horizon for AI: “jobs are disappearing.”

Rate limit mishandling. Provider 429 responses from OpenAI, Anthropic, and Google are not errors in the traditional sense. They are expected, temporary, and recoverable. Retrying immediately burns through the tries budget in seconds. Without a backoff array defined on the job, Laravel uses zero delay between retries by default. A job hitting a rate limit five times in 15 seconds has failed just as permanently as one that hit a genuine error.

Partial output loss. AI jobs often do useful work before failing. A document summarisation job might process 80% of its input before hitting a context limit. Standard job failure handling discards that state entirely. For expensive inference workloads on long documents, that is a measurable cost.

The fix requires changes at three levels: supervisor configuration, job class design, and monitoring.

Configuring Laravel Horizon in Production for AI Workloads

Install Horizon if you have not already:

composer require laravel/horizon
php artisan horizon:install

The critical configuration lives in config/horizon.php. The default supervisor configuration is intentionally generic. For AI workloads, you need a dedicated supervisor pool with materially different settings.

// config/horizon.php

'environments' => [
    'production' => [

        'supervisor-ai-inference' => [
            'connection'          => 'redis',
            'queue'               => ['ai-high', 'ai-default', 'ai-low'],
            'balance'             => 'auto',
            'autoScalingStrategy' => 'time',
            'minProcesses'        => 2,
            'maxProcesses'        => 12,
            'balanceMaxShift'     => 2,
            'balanceCooldown'     => 5,
            'timeout'             => 300,  // 5 minutes — covers streaming completions
            'sleep'               => 3,
            'tries'               => 5,
            'nice'                => 0,
        ],

        'supervisor-default' => [
            'connection'  => 'redis',
            'queue'       => ['default', 'notifications', 'mail'],
            'balance'     => 'simple',
            'minProcesses'=> 1,
            'maxProcesses'=> 8,
            'timeout'     => 60,
            'sleep'       => 3,
            'tries'       => 3,
        ],
    ],
],

A few decisions here that are worth explaining.

autoScalingStrategy: time scales workers based on queue wait time rather than queue size. For AI workloads, queue size is a poor signal: three jobs waiting sounds manageable, but if each takes 90 seconds, you are looking at a 4-minute tail for the last user. Time-based scaling catches this earlier.

timeout: 300 gives generous headroom for streaming completions and large context calls. This is not a ceiling you should approach routinely; it is a safety net. If jobs are regularly running past 120 seconds, that is a prompt engineering problem, not a timeout problem.

balanceCooldown: 5 prevents the auto-balancer from thrashing worker counts during a burst of short AI calls followed by a trough. Default is 3 seconds, which is too reactive for inference workloads.

Supervisord server configuration is equally important. The stopwaitsecs value must exceed Horizon’s timeout value, or the process manager will kill a running Horizon worker before it finishes a long inference job during deployments:

[program:laravel-horizon]
process_name=%(program_name)s
command=php /var/www/html/artisan horizon
autostart=true
autorestart=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/horizon.log
stopwaitsecs=360

Set stopwaitsecs to at least timeout + 60. We have seen rolling deployments silently truncate in-flight inference calls because this was left at the default 10 seconds.

Designing the Job Class: Timeout, Retry, and Rate Limit Handling

The supervisor configuration sets the outer boundary. The job class defines behaviour within it. For AI inference jobs, three properties are non-negotiable: $timeout, $tries, and $backoff.

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\Middleware\RateLimited;
use Illuminate\Support\Facades\Log;

class GenerateAIInsightJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    /**
     * Hard kill threshold. Horizon's supervisor timeout is the outer wall;
     * this property is the job's own declaration to the queue system.
     * Set it below the supervisor timeout to allow graceful error handling.
     */
    public int $timeout = 240;

    /**
     * Total attempts before the job is moved to failed_jobs.
     * 5 attempts with exponential backoff covers transient provider outages.
     */
    public int $tries = 5;

    /**
     * Seconds to wait before each retry attempt.
     * Indices correspond to attempt number: attempt 1 waits 30s, attempt 2 waits 60s, etc.
     */
    public array $backoff = [30, 60, 120, 180, 240];

    public function __construct(
        private readonly int    $documentId,
        private readonly string $prompt,
        private readonly string $model = 'claude-sonnet-4-6',
    ) {}

    public function middleware(): array
    {
        return [new RateLimited('ai-inference')];
    }

    public function handle(): void
    {
        $document = Document::findOrFail($this->documentId);

        try {
            $response = \Anthropic::messages()->create([
                'model'      => $this->model,
                'max_tokens' => 2048,
                'messages'   => [
                    ['role' => 'user', 'content' => $this->prompt],
                ],
            ]);

            $document->update([
                'ai_insight'       => $response->content[0]->text,
                'insight_model'    => $this->model,
                'insight_token_count' => $response->usage->inputTokens + $response->usage->outputTokens,
            ]);

        } catch (\Throwable $e) {
            if ($this->isRateLimitException($e)) {
                // Release back to queue with the appropriate backoff delay.
                // Do NOT throw — throwing counts as a failed attempt.
                $this->release($this->backoff[$this->attempts() - 1] ?? 240);
                return;
            }

            Log::error('AI insight generation failed', [
                'document_id' => $this->documentId,
                'attempt'     => $this->attempts(),
                'error'       => $e->getMessage(),
            ]);

            throw $e;
        }
    }

    public function failed(\Throwable $exception): void
    {
        // Preserve whatever partial work exists rather than nulling it.
        Document::where('id', $this->documentId)->update([
            'ai_insight_status' => 'failed',
            'ai_insight_error'  => $exception->getMessage(),
        ]);

        Log::critical('AI insight job exhausted all retries', [
            'document_id' => $this->documentId,
            'model'       => $this->model,
        ]);
    }

    public function retryUntil(): \DateTime
    {
        // Absolute deadline. Even with $tries remaining, the job will not
        // retry after this point. Critical for time-sensitive inference pipelines.
        return now()->addHours(3);
    }

    private function isRateLimitException(\Throwable $e): bool
    {
        return str_contains($e->getMessage(), '429')
            || str_contains($e->getMessage(), 'rate_limit')
            || str_contains($e->getMessage(), 'Too Many Requests');
    }
}

The $this->release() pattern inside isRateLimitException is the correct approach for provider rate limits. Throwing the exception counts as a failed attempt and triggers a retry cycle. Calling release() puts the job back on the queue with a delay and does not decrement the tries counter. Rate limits are not job failures; they are scheduling signals.

retryUntil() is the safety valve. Without it, a job sitting in exponential backoff across five attempts could theoretically retry for hours after the result is no longer needed. Set this to match the actual business requirement.

Registering the Rate Limiter

The RateLimited middleware on the job references a named rate limiter. Register it in your AppServiceProvider:

// app/Providers/AppServiceProvider.php

use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Support\Facades\RateLimiter;

public function boot(): void
{
    RateLimiter::for('ai-inference', function (object $job) {
        // Adjust this to match your provider tier.
        // Anthropic Tier 2: ~1,000 RPM. OpenAI Tier 3: ~5,000 RPM.
        // Start conservative and increase as you verify throughput.
        return Limit::perMinute(60)->by('global');
    });
}

For multi-tenant applications with per-tenant provider keys, scope the limiter by tenant:

RateLimiter::for('ai-inference', function (object $job) {
    $tenantId = $job->tenantId ?? 'global';
    return Limit::perMinute(30)->by("tenant:{$tenantId}");
});

If you are building the broader governance layer around token budgets and per-tenant limits, the AI middleware article covering token tracking covers the HTTP request layer equivalent of this pattern.

Job Duration: Why This Visualisation Matters

The chart below illustrates why AI inference jobs cannot share a supervisor pool with standard queue jobs. Standard jobs cluster almost entirely below 200ms. AI inference jobs distribute across a 5-to-90-second range, with a meaningful tail. A shared 60-second timeout kills the tail of the AI distribution silently.

Bar chart comparing job duration distribution between standard queue jobs and AI inference jobs, illustrating why a shared 60-second timeout is insufficient for AI workloads.

Monitoring What Actually Matters for AI Queues

The Horizon dashboard gives you throughput, wait time, and recent job runtime out of the box. For standard workloads, those three numbers tell you most of what you need. For AI inference workloads, the signal you need most is not surfaced by default: the ratio of jobs that exit via SIGKILL versus jobs that complete normally.

The table below outlines which Horizon metrics carry real weight for AI workloads, and the thresholds worth building alerts around.

For the silent timeout kill problem, there is no native Horizon alert. The symptom is a job that leaves no trace in failed_jobs despite not completing. You can detect this indirectly by tracking job dispatch counts against completion counts in your application layer:

// Dispatch side — record intent
Cache::increment("jobs:dispatched:{$this->documentId}");
GenerateAIInsightJob::dispatch($document->id, $prompt)->onQueue('ai-default');

// Handle side — record completion
Cache::increment("jobs:completed:{$this->documentId}");

A growing gap between those two counters, without a corresponding growth in failed_jobs, is the fingerprint of silent SIGKILL kills. Add a scheduled command to routes/console.php that alerts when the gap exceeds a threshold:

// routes/console.php

use Illuminate\Support\Facades\Schedule;

Schedule::call(function () {
    $dispatched  = Cache::get('jobs:dispatched:total', 0);
    $completed   = Cache::get('jobs:completed:total', 0);
    $failed      = DB::table('failed_jobs')
                     ->where('queue', 'like', 'ai-%')
                     ->where('failed_at', '>=', now()->subHour())
                     ->count();

    $missing = $dispatched - $completed - $failed;

    if ($missing > 5) {
        Log::critical('AI jobs disappearing without trace', [
            'dispatched' => $dispatched,
            'completed'  => $completed,
            'failed'     => $failed,
            'missing'    => $missing,
        ]);
    }
})->everyFiveMinutes();

[Production Pitfall] The silent SIGKILL kill is by far the most dangerous failure mode in AI queue workloads, because it produces no actionable output. Teams routinely run in this state for weeks without realising it, attributing the missing outputs to “the AI being slow.” Check your stopwaitsecs and timeout alignment before anything else. If stopwaitsecs in your supervisord config is lower than Horizon’s timeout, every deployment is silently truncating in-flight inference calls.

If you are building the broader observability layer around your AI architecture, the governance and telemetry patterns in the production AI architecture guide cover how to centralise this kind of cross-cutting operational signal across providers.

Failed Job Strategy for LLM Inference

When a job does land in failed_jobs, the default response is to re-dispatch it manually via php artisan queue:retry and hope the error was transient. For AI inference, that is rarely sufficient. Inference failures tend to cluster around specific causes (provider outages, malformed prompts, context window overflows, or inference parameters that produce invalid output), and each warrants a different response.

Structure your failed job handling to capture enough context to triage correctly:

public function failed(\Throwable $exception): void
{
    $reason = match (true) {
        str_contains($exception->getMessage(), 'context_length_exceeded') => 'context_overflow',
        str_contains($exception->getMessage(), '429')                     => 'rate_limit_exhausted',
        str_contains($exception->getMessage(), 'invalid_request_error')   => 'bad_prompt',
        default                                                           => 'unknown',
    };

    Document::where('id', $this->documentId)->update([
        'ai_insight_status' => 'failed',
        'ai_failure_reason' => $reason,
        'ai_insight_error'  => $exception->getMessage(),
    ]);

    Log::error('AI inference job failed permanently', [
        'document_id' => $this->documentId,
        'model'       => $this->model,
        'reason'      => $reason,
        'attempts'    => $this->attempts(),
    ]);
}

The reason field is the important addition. context_overflow failures need prompt truncation logic, not a retry. bad_prompt failures need a developer looking at the prompt template, not an automated re-queue. Retrying them blindly burns provider quota for no benefit.

For agentic pipelines where the inference job is one step in a multi-step chain, the failure handling becomes more complex. The question of what to do with downstream jobs that depend on the failed step, and how to validate the partial output that does exist, is covered in depth in the agentic workflow schema validation guide. The core principle applies here too: validate what you have before deciding whether a retry is warranted.

One final note on the Horizon dashboard and Laravel Telescope together: Telescope’s job watcher captures the full job payload, exception stack trace, and timing for every failed job. For AI workloads, the job payload includes the prompt, which makes post-mortem analysis significantly faster. Enable the job watcher in non-production environments at minimum, and consider enabling it in production with payload scrubbing for anything containing PII. See the Laravel Horizon documentation for tag-based filtering, which lets you isolate AI job failures without noise from other queue traffic. Anthropic’s rate limit documentation is the authoritative reference for per-tier RPM and token-per-minute limits if you are calibrating your RateLimiter::for('ai-inference') ceiling.

Building Fail-Safes for Incomplete LLM Responses in Laravel Echo

Dewald Hugo — Thu, 14 May 2026 18:35:48 +0000

Broadcasting LLM token streams through Laravel Echo feels elegant right up until the moment it silently dies halfway through a response. No error. No terminal event. Just a client sitting there, waiting for tokens that will never arrive.

We hit this in production during a multi-user document generation feature. The Pusher connection degraded during a particularly long Anthropic response. The queue job had no idea the client had disconnected, and the user stared at a spinner for three minutes before refreshing. The partial response was gone. No retry surface. No recovery path. The incident report had three action items and none of them were obvious beforehand.

That experience shaped the *Laravel LLM streaming fail-safe* architecture this article covers. Every pattern below is oriented toward the specific failure modes that broadcasting-based LLM streams introduce, and in several cases those failure modes are different from what you encounter with SSE.

One framing note before we start. Laravel Echo names a client-side JavaScript library for subscribing to broadcast channels. It is not an SSE wrapper. The architecture here is: a queued job calls the LLM API, iterates the stream, and broadcasts each token as a private channel event. Echo subscribes on the client. This pattern earns its weight when you need multiple subscribers on the same stream (team collaboration, agent monitoring, admin oversight), or when you are already running Reverb or Pusher in your stack. The tradeoffs between Livewire, SSE, and WebSockets for AI streaming are worth understanding before committing to this path. If you only have one client per stream, SSE is simpler and has materially lower infrastructure overhead.

Why Incomplete Streams Are More Dangerous in Echo

SSE gives you a persistent, unidirectional HTTP connection with browser-native reconnect semantics. When the connection drops, the browser reconnects and sends Last-Event-ID. You have backpressure. You know whether the connection is alive.

Echo over WebSockets gives you none of that for free. The WebSocket channel stays open even when the underlying LLM stream on the server has failed. The client shows “connected” while the queue job is dead or retrying. Pusher and Reverb will accept broadcast events and deliver them on arrival, but they will not tell your client that no more events are coming.

That gap is exactly where incomplete responses live. Four categories to prepare for.

Token limit truncation. The LLM hits max_tokens and stops mid-sentence. The finish_reason comes back as max_tokens, not end_turn. If your client only listens for a generic “done” signal without inspecting the finish reason, a truncated response renders as complete. Users won’t know. You won’t know until they complain.

Queue job failure. The job throws after token 47. Laravel retries it. The client has already rendered tokens 1 through 47 and will now receive them again from the retry run, duplicated, in the same Echo channel.

Silent connection drop. The WebSocket drops between the server and Pusher, or between Pusher and the client. Events broadcast during the gap are gone. The job may complete successfully on the server and broadcast a terminal event that the client never receives.

Orphaned streams. The queue worker is killed mid-flight by an OOM event or a forced deployment restart. The stream record in your database stays in streaming status indefinitely, because nothing transitions it out.

Each one needs a different fix. Build the layers in order.

The Architecture Baseline

The core pattern is a queued job that streams from the LLM API and broadcasts each token as a private channel event scoped to a unique stream_id. Every event carries a monotonically increasing sequence number. That sequence number is the foundation of every fail-safe that follows.

Start with the broadcast event. The Laravel Broadcasting documentation covers private channel authorization in full, but the event structure below is what drives the client-side recovery logic.

<?php

namespace App\Events;

use Illuminate\Broadcasting\PrivateChannel;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;

class LlmTokenReceived implements ShouldBroadcast
{
    public function __construct(
        public readonly string  $streamId,
        public readonly string  $token,
        public readonly int     $sequence,
        public readonly string  $status,       // 'streaming' | 'complete' | 'truncated' | 'error' | 'dead'
        public readonly ?string $finishReason = null,
    ) {}

    public function broadcastOn(): PrivateChannel
    {
        return new PrivateChannel("stream.{$this->streamId}");
    }

    public function broadcastAs(): string
    {
        return 'token.received';
    }

    public function broadcastWith(): array
    {
        return [
            'stream_id'     => $this->streamId,
            'token'         => $this->token,
            'sequence'      => $this->sequence,
            'status'        => $this->status,
            'finish_reason' => $this->finishReason,
        ];
    }
}

The status field carries the terminal state explicitly. complete means the LLM finished naturally (end_turn). truncated means it hit a token limit. error and dead mean infrastructure or retry failure. The client does not guess. The server tells it.

The database layer makes recovery possible at all:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::create('llm_streams', function (Blueprint $table) {
            $table->id();
            $table->string('stream_id')->unique();
            $table->foreignId('user_id')->constrained()->cascadeOnDelete();
            $table->text('prompt');
            $table->enum('status', ['pending', 'streaming', 'complete', 'truncated', 'error', 'dead'])
                  ->default('pending');
            $table->longText('partial_content')->nullable();
            $table->longText('final_content')->nullable();
            $table->unsignedInteger('last_sequence')->default(0);
            $table->string('finish_reason')->nullable();
            $table->text('error_message')->nullable();
            $table->timestamp('started_at')->nullable();
            $table->timestamp('last_checkpoint_at')->nullable();
            $table->timestamp('completed_at')->nullable();
            $table->timestamp('failed_at')->nullable();
            $table->timestamps();

            $table->index(['status', 'last_checkpoint_at']); // orphan detection
            $table->index(['user_id', 'status']);
        });
    }
};

Server-Side Fail-Safes

The queue job carries most of the server-side protection. Three responsibilities: stream tokens and broadcast them with sequence numbers, write periodic checkpoints so recovery is possible, and guarantee a terminal broadcast event regardless of how the job ends.

The Anthropic Messages Streaming documentation defines the event types used in the iterator below. The message_stop event is the authoritative signal that the LLM has finished, and its stop_reason field is what you map to your own status enum.

<?php

namespace App\Jobs;

use App\Events\LlmTokenReceived;
use App\Models\LlmStream;
use Anthropic\Laravel\Facades\Anthropic;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Throwable;

class StreamLlmResponseJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public int $timeout = 120;
    public int $tries   = 3;
    public int $backoff = 10;

    public function __construct(
        private readonly string $streamId,
        private readonly string $prompt,
        private readonly int    $userId,
    ) {}

    public function handle(): void
    {
        $sequence  = 0;
        $buffer    = '';
        $terminated = false;

        LlmStream::where('stream_id', $this->streamId)
            ->update(['status' => 'streaming', 'started_at' => now()]);

        try {
            $stream = Anthropic::messages()->stream([
                'model'      => 'claude-sonnet-4-5',
                'max_tokens' => 4096,
                'messages'   => [
                    ['role' => 'user', 'content' => $this->prompt],
                ],
            ]);

            foreach ($stream as $event) {
                if ($event->type === 'content_block_delta' && isset($event->delta->text)) {
                    $token   = $event->delta->text;
                    $buffer .= $token;
                    $sequence++;

                    broadcast(new LlmTokenReceived(
                        streamId: $this->streamId,
                        token:    $token,
                        sequence: $sequence,
                        status:   'streaming',
                    ));

                    if ($sequence % 50 === 0) {
                        $this->checkpoint($buffer, $sequence);
                    }
                }

                if ($event->type === 'message_stop') {
                    $finishReason = $event->message?->stop_reason ?? 'unknown';
                    $status       = $finishReason === 'end_turn' ? 'complete' : 'truncated';
                    $terminated   = true;
                    $sequence++;

                    broadcast(new LlmTokenReceived(
                        streamId:     $this->streamId,
                        token:        '',
                        sequence:     $sequence,
                        status:       $status,
                        finishReason: $finishReason,
                    ));

                    LlmStream::where('stream_id', $this->streamId)->update([
                        'status'        => $status,
                        'finish_reason' => $finishReason,
                        'final_content' => $buffer,
                        'last_sequence' => $sequence,
                        'completed_at'  => now(),
                    ]);
                }
            }
        } catch (Throwable $e) {
            if (! $terminated) {
                $sequence++;

                broadcast(new LlmTokenReceived(
                    streamId:     $this->streamId,
                    token:        '',
                    sequence:     $sequence,
                    status:       'error',
                    finishReason: 'exception',
                ));

                LlmStream::where('stream_id', $this->streamId)->update([
                    'status'          => 'error',
                    'error_message'   => $e->getMessage(),
                    'last_sequence'   => $sequence,
                    'partial_content' => $buffer,
                    'failed_at'       => now(),
                ]);
            }

            throw $e; // preserve queue retry behaviour
        }
    }

    private function checkpoint(string $buffer, int $sequence): void
    {
        LlmStream::where('stream_id', $this->streamId)->update([
            'partial_content'    => $buffer,
            'last_sequence'      => $sequence,
            'last_checkpoint_at' => now(),
        ]);
    }

    public function failed(Throwable $exception): void
    {
        LlmStream::where('stream_id', $this->streamId)->update([
            'status'        => 'dead',
            'error_message' => $exception->getMessage(),
            'failed_at'     => now(),
        ]);

        // Last-resort broadcast after all retries are exhausted.
        // PHP_INT_MAX sequence guarantees the client accepts this regardless
        // of how many gaps have accumulated.
        broadcast(new LlmTokenReceived(
            streamId:     $this->streamId,
            token:        '',
            sequence:     PHP_INT_MAX,
            status:       'dead',
            finishReason: 'retries_exhausted',
        ));
    }
}

Three decisions in this code are worth calling out explicitly.

The $terminated flag prevents double-broadcasting. If message_stop fires and then the foreach cleanup throws, you do not want a second terminal event with status: 'error' landing after you have already broadcast status: 'complete'. Clients will not handle that gracefully.

The throw $e at the end of the catch block is intentional. Removing it tells Laravel the job succeeded, killing retry behaviour. The partial broadcast for error is informational to the client. The throw is what keeps the queue system honest.

The failed() hook runs after all retry attempts are exhausted and is your last-resort client notification. This method is called by Laravel’s Queue Worker, not by your own code.

[Edge Case Alert] If you are running Laravel Reverb rather than Pusher, broadcasts inside failed() may not deliver if Reverb is also under load or restarting at the same time as your queue worker. failed() is best-effort in that scenario. The orphan detection command below is your backstop.

Client-Side Fail-Safes

Most teams spend three lines on the Echo subscription and zero lines on what happens when it goes quiet. The watchdog and sequence tracking below are the part that actually makes this production-safe.

class LlmStreamClient {
    constructor(streamId, options = {}) {
        this.streamId  = streamId;
        this.tokens    = [];
        this.lastSeq   = 0;
        this.gaps      = [];
        this.watchdog  = null;
        this.silenceMs = options.silenceMs  ?? 10000;
        this.onToken   = options.onToken    ?? (() => {});
        this.onComplete = options.onComplete ?? (() => {});
        this.onError   = options.onError    ?? (() => {});
        this.channel   = null;
    }

    subscribe() {
        this.channel = window.Echo
            .private(`stream.${this.streamId}`)
            .listen('.token.received', (e) => {
                this.resetWatchdog();
                this.handleEvent(e);
            });

        this.startWatchdog();
        return this;
    }

    handleEvent({ sequence, token, status, finish_reason: finishReason }) {
        if (sequence > this.lastSeq + 1) {
            this.gaps.push({ expected: this.lastSeq + 1, received: sequence });
        }
        this.lastSeq = Math.max(this.lastSeq, sequence);

        if (status === 'streaming') {
            this.tokens.push(token);
            this.onToken(token, this.tokens.join(''));
            return;
        }

        this.clearWatchdog();
        const partial = this.tokens.join('');

        if (status === 'complete') {
            this.onComplete({ content: partial, gaps: this.gaps, finishReason });
        } else {
            this.onError({ type: status, partial, gaps: this.gaps, finishReason });
        }

        this.teardown();
    }

    startWatchdog() {
        this.watchdog = setTimeout(() => {
            this.onError({
                type:    'client_timeout',
                partial: this.tokens.join(''),
                message: `No event received for ${this.silenceMs}ms`,
                gaps:    this.gaps,
            });
            this.teardown();
        }, this.silenceMs);
    }

    resetWatchdog() { this.clearWatchdog(); this.startWatchdog(); }

    clearWatchdog() {
        if (this.watchdog) { clearTimeout(this.watchdog); this.watchdog = null; }
    }

    teardown() {
        this.clearWatchdog();
        if (this.channel) {
            window.Echo.leave(`stream.${this.streamId}`);
            this.channel = null;
        }
    }
}

Usage with the full error surface handled:

const client = new LlmStreamClient('abc-123', {
    silenceMs: 10000,

    onToken: (token, full) => {
        document.getElementById('output').textContent = full;
    },

    onComplete: ({ content, gaps, finishReason }) => {
        if (gaps.length > 0) {
            console.warn('Stream gaps detected — some tokens may be missing.', gaps);
        }
        saveResponse(content);
    },

    onError: ({ type, partial, finishReason }) => {
        savePartial(partial); // always persist what arrived

        if (type === 'truncated') {
            showRetryButton('Response was cut off. Retry to continue.');
        } else if (type === 'client_timeout') {
            showRetryButton('Connection timed out. Retry when ready.');
        } else {
            showError('Something went wrong. Your partial response has been saved.');
        }
    },
});

client.subscribe();

The watchdog fires after silenceMs of silence. Ten seconds without a token means something is wrong: the job died, the Pusher connection degraded, or the LLM API paused for an unusually long time between tokens. All three warrant user feedback.

[Production Pitfall] Do not set silenceMs below 8000. Claude’s API pauses several seconds between tokens during complex structured outputs before streaming resumes. We set ours to 5000 initially and got false timeout fires on longer document generation tasks. 10000 is the minimum safe floor for production. If you are generating very long documents, consider 15000.

The gaps array is diagnostic, not restorative. Pusher and Reverb do not offer event replay for private channels. Gap detection tells you whether a broadcast delivery problem contributed to an incomplete response, which helps you distinguish API truncation from infrastructure failure. Log it to your analytics or error tracker on every onComplete call.

Detecting and Recovering Orphaned Streams

The orphan case does not surface at runtime. The queue worker is killed. The stream stays in streaming status. The client timed out via the watchdog and showed the retry UI. But nothing tells the database the stream is dead.

Checkpointing every 50 tokens gives you a heartbeat. An active stream always updates last_checkpoint_at within a predictable window. Silence beyond that window is evidence of death.

<?php

use Illuminate\Support\Facades\Schedule;

Schedule::command('streams:reap-orphans')->everyFiveMinutes();

The command itself:

<?php

namespace App\Console\Commands;

use App\Events\LlmTokenReceived;
use App\Models\LlmStream;
use Illuminate\Console\Command;

class ReapOrphanedStreams extends Command
{
    protected $signature   = 'streams:reap-orphans';
    protected $description = 'Mark streams with no checkpoint activity as orphaned.';

    public function handle(): void
    {
        $orphans = LlmStream::query()
            ->where('status', 'streaming')
            ->where(function ($q) {
                $q->where('last_checkpoint_at', '<', now()->subMinutes(2))
                  ->orWhere(function ($q2) {
                      $q2->whereNull('last_checkpoint_at')
                         ->where('started_at', '<', now()->subMinutes(2));
                  });
            })
            ->get();

        foreach ($orphans as $stream) {
            $stream->update([
                'status'        => 'error',
                'error_message' => 'Orphaned: no checkpoint activity for 2+ minutes.',
                'failed_at'     => now(),
            ]);

            broadcast(new LlmTokenReceived(
                streamId:     $stream->stream_id,
                token:        '',
                sequence:     $stream->last_sequence + 1,
                status:       'error',
                finishReason: 'orphaned',
            ));

            $this->line("Reaped: {$stream->stream_id}");
        }

        $this->info("Reaped {$orphans->count()} orphaned stream(s).");
    }
}

Two minutes is conservative. On a fast model with short prompts, 50 tokens arrive in seconds. If you are streaming long documents, increase the reap window to match, or reduce the checkpoint interval from 50 to 20 tokens.

[Architect’s Note] The last_checkpoint_at column is a poor man’s distributed heartbeat, and that is not a criticism. This is the same liveness-detection pattern event-driven systems have used for decades. If you later move to Laravel Horizon with job event hooks, you can correlate orphan detection directly with actual queue worker failures and remove the scheduler command. For most setups, the scheduled Artisan command approach costs nothing and requires no additional infrastructure.

Retry Strategy

When a stream ends in error, truncated, or dead, your user needs a retry button. Always generate a new stream_id for each retry. Reusing the original stream_id means the retry job broadcasts into the same channel the client already abandoned, and the partial content from attempt one gets mixed with attempt two in your database record. Neither is recoverable cleanly.

<?php

namespace App\Http\Controllers;

use App\Jobs\StreamLlmResponseJob;
use App\Models\LlmStream;
use Illuminate\Http\JsonResponse;
use Illuminate\Http\Request;
use Illuminate\Support\Str;

class StreamRetryController extends Controller
{
    public function __invoke(Request $request, string $streamId): JsonResponse
    {
        $original = LlmStream::query()
            ->where('stream_id', $streamId)
            ->where('user_id', $request->user()->id)
            ->whereIn('status', ['error', 'truncated', 'dead'])
            ->firstOrFail();

        $newStreamId = (string) Str::uuid();

        LlmStream::create([
            'stream_id' => $newStreamId,
            'user_id'   => $request->user()->id,
            'prompt'    => $original->prompt,
            'status'    => 'pending',
        ]);

        StreamLlmResponseJob::dispatch($newStreamId, $original->prompt, $request->user()->id);

        return response()->json(['stream_id' => $newStreamId]);
    }
}

Route registration:

<?php

use App\Http\Controllers\StreamRetryController;
use Illuminate\Support\Facades\Route;

Route::middleware('auth:sanctum')->group(function () {
    Route::post('/streams/{streamId}/retry', StreamRetryController::class);
});

The original stream record stays in the database with its status and partial_content intact for audit purposes. The retry creates a clean record. The client receives the new stream_id in the JSON response and calls client.subscribe() fresh.

If your streaming endpoints are not locked down with proper token-based authentication, the retry endpoint becomes an unauthenticated job dispatch surface. Sanctum’s API token middleware is the right guard here, not session auth, because the retry call typically originates from JavaScript, not a form submission.

The Service Layer Is Not This Job’s Problem

The job above handles the transport layer. It does not handle token budget management, provider fallback, cost attribution, or prompt construction. Those belong one layer up.

If you are building this into a larger system, the production AI architecture for Laravel covers the contracts and driver abstraction layer that should wrap around the streaming job described here. The job should be thin: stream, checkpoint, broadcast. Everything else belongs in the service it calls.

That separation also matters when your streamed output is structured. Schema validation against LLM hallucinations belongs at the service layer. A truncated JSON response is harder to recover from than truncated prose. Catching a max_tokens truncation in a partial JSON structure before it reaches your client is far cheaper than handling the parse error downstream.

Monitoring What Actually Matters

Before going live, instrument three metrics from your llm_streams table:

Stream completion rate. status = 'complete' divided by total streams initiated, measured hourly. Below 95% in production warrants investigation. Below 90% is an incident.

Orphan rate. Streams reaped by the Artisan command as a percentage of total streams. A spike here usually correlates with deployment restarts or queue worker OOM events. Check your Horizon metrics or system logs alongside it.

Gap frequency. Log the gaps array from client-side onComplete callbacks via a lightweight analytics endpoint. Persistent gaps from specific geographic regions point to Pusher or Reverb delivery problems, not LLM API issues. The distinction matters: one is your infrastructure, the other is your vendor’s.

Connecting these to Laravel’s AI middleware for token tracking gives you cost visibility alongside reliability metrics. That combination is what you need to have an honest conversation about whether Echo-based streaming is worth its operational complexity for your specific use case.

Zero-Downtime Laravel Deployments with GitHub Actions: A Complete CI/CD Pipeline for Production

Dewald Hugo — Sat, 02 May 2026 17:30:48 +0000

Most Laravel teams have a deployment ritual that sounds reasonable until you think about it carefully. Push to main. SSH into the server. Run git pull && composer install. Fire off php artisan migrate. Restart the queue. Hope the app comes back cleanly. It often does. When it does not, you are debugging in production, in front of users, mid-request.

Zero-downtime Laravel deployments with GitHub Actions break that cycle. The strategy borrows Envoyer’s core mechanic (timestamped release directories and an atomic symlink swap) and pairs it with a CI/CD pipeline that runs your full test suite before anything reaches production, verifies the live deployment against a real health endpoint, and rolls back automatically when that check fails. No Envoyer license. No Laravel Forge dependency. Full control over every step.

One scoping note before we get into it: this article assumes a server that is already provisioned, Nginx-configured, and accessible over SSH with a deploy user. If you are building that foundation, start with the complete Laravel production deployment guide first. This article picks up where that one leaves off.

Why the “Pull and Pray” Approach Eventually Breaks

The problem with a naive git pull + composer install deploy is not that it is lazy. The problem is that it is a long, destructive operation performed against a live directory. Composer can take 30 to 90 seconds to resolve and install packages. During that window, your application is in an inconsistent state. Old PHP files from the previous release sit alongside whatever files git has already updated. If a new class is referenced by code that has already landed on disk but Composer has not yet installed its package, you get a fatal. Users get a 500.

Migrations compound the problem. Running php artisan migrate against a live database means the schema is changing while requests are in flight. An ALTER TABLE that adds a NOT NULL column can acquire a table lock on MySQL. Even on PostgreSQL, you are relying on the migration being fast enough that no concurrent request trips over the constraint change mid-transaction.

Queue workers are the third failure mode, and the one teams notice last. Workers running old code against a migrated schema produce type errors, silent write failures, or jobs that retry indefinitely. We have seen this cause lost inference jobs (queued API calls to Anthropic or OpenAI) that were mid-retry when the schema changed underneath them. By the time someone noticed, the queue had thousands of failed jobs and no useful error message beyond a column-not-found exception.

The fix is conceptually simple: prepare the new release while the old one is still serving traffic, then swap in a single atomic operation.

How Zero-Downtime Deployments Actually Work

Everything depends on the directory structure on the server. Instead of a single /var/www/myapp directory modified in place, you maintain three:

/var/www/myapp/
├── current -> releases/20260502143012_a1b2c3d   (symlink)
├── releases/
│   ├── 20260502143012_a1b2c3d/
│   ├── 20260430091201_f4e5d6a/
│   └── 20260428181345_9c8b7a6/
└── shared/
    ├── .env
    └── storage/

current is a symlink. Nginx’s document root points at /var/www/myapp/current/public. Each deployment creates a new timestamped directory in releases/, prepares it fully (dependencies installed, assets built, framework caches generated), and then swaps current to point to the new directory. The swap itself is a single filesystem rename call. From Nginx’s perspective, it is instantaneous.

Shared state lives outside all release directories in shared/. Your .env file and the storage/ directory are symlinked into each release. Uploads, runtime cache files, and logs persist across every deployment.

The Nginx configuration for this pattern uses $realpath_root rather than $document_root. This is not optional:

server {
    listen 443 ssl http2;
    server_name myapp.com;

    root /var/www/myapp/current/public;
    index index.php;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php8.3-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
    }
}

$realpath_root resolves the symlink before handing the path to PHP-FPM. Without it, OPcache may hold inode references to files in the old release directory after the swap, serving stale bytecode until the cache expires or the process restarts. Reloading PHP-FPM (not restarting it) after the symlink swap flushes OPcache’s resolved paths gracefully, with no dropped connections.

Setting Up the GitHub Actions Pipeline

The pipeline has two jobs: test and deploy. The deploy job does not run unless tests pass. GitHub’s environment feature gives you an optional manual approval gate for production, useful for regulated contexts. The GitHub Actions documentation on deployment environments covers protection rules in detail.

A reliable test suite is the actual quality gate between your code and production. Building robust test factories gives you the factory-backed integration test coverage that makes a CI gate meaningful rather than ceremonial.

Here is the complete workflow:

# .github/workflows/deploy.yml
name: Test and Deploy

on:
  push:
    branches: [main]

env:
  PHP_VERSION: '8.3'

jobs:
  test:
    name: Run Test Suite
    runs-on: ubuntu-latest

    services:
      mysql:
        image: mysql:8.0
        env:
          MYSQL_ROOT_PASSWORD: root
          MYSQL_DATABASE: testing
        options: >-
          --health-cmd="mysqladmin ping"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=3
        ports:
          - 3306:3306

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup PHP ${{ env.PHP_VERSION }}
        uses: shivammathur/setup-php@v2
        with:
          php-version: ${{ env.PHP_VERSION }}
          extensions: mbstring, pdo, pdo_mysql, redis, pcov
          coverage: pcov

      - name: Cache Composer dependencies
        uses: actions/cache@v4
        with:
          path: vendor
          key: composer-${{ hashFiles('**/composer.lock') }}
          restore-keys: composer-

      - name: Install Composer dependencies
        run: composer install --no-progress --prefer-dist --no-interaction

      - name: Prepare test environment
        run: |
          cp .env.testing .env
          php artisan key:generate

      - name: Run migrations against test database
        run: php artisan migrate --force
        env:
          DB_CONNECTION: mysql
          DB_HOST: 127.0.0.1
          DB_PORT: 3306
          DB_DATABASE: testing
          DB_USERNAME: root
          DB_PASSWORD: root

      - name: Execute test suite
        run: php artisan test --parallel
        env:
          DB_CONNECTION: mysql
          DB_HOST: 127.0.0.1
          DB_PORT: 3306
          DB_DATABASE: testing
          DB_USERNAME: root
          DB_PASSWORD: root

  deploy:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: test
    environment: production
    concurrency:
      group: production-deploy
      cancel-in-progress: false

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Run deployment script via SSH
        uses: appleboy/ssh-action@v1.0.3
        with:
          host: ${{ secrets.DEPLOY_HOST }}
          username: ${{ secrets.DEPLOY_USER }}
          key: ${{ secrets.DEPLOY_KEY }}
          port: ${{ secrets.DEPLOY_PORT }}
          script: /home/deploy/scripts/deploy.sh ${{ github.sha }}

      - name: Verify health check
        run: |
          sleep 15
          HTTP_STATUS=$(curl --silent --output /dev/null \
            --write-out "%{http_code}" \
            --max-time 10 \
            https://${{ secrets.APP_DOMAIN }}/up)
          if [ "$HTTP_STATUS" -ne 200 ]; then
            echo "Health check returned HTTP $HTTP_STATUS — triggering rollback"
            exit 1
          fi
          echo "Deployment verified: HTTP $HTTP_STATUS"

      - name: Rollback on failure
        if: failure()
        uses: appleboy/ssh-action@v1.0.3
        with:
          host: ${{ secrets.DEPLOY_HOST }}
          username: ${{ secrets.DEPLOY_USER }}
          key: ${{ secrets.DEPLOY_KEY }}
          port: ${{ secrets.DEPLOY_PORT }}
          script: /home/deploy/scripts/rollback.sh

Two things to highlight. The concurrency block with cancel-in-progress: false prevents a rapid second push from interrupting a deployment that is already in flight. Two concurrent deploys racing to swap the same symlink is a scenario you do not want to debug at 2am. The needs: test key creates a hard dependency: no green tests, no deploy job runs, full stop.

The Deployment Script: Symlink-Based Releases

The deploy.sh script does the actual work on the production server. It accepts the commit SHA from GitHub Actions so the release directory is traceable back to a specific commit.

#!/usr/bin/env bash
set -euo pipefail

COMMIT_SHA="${1:-HEAD}"
APP_DIR="/var/www/myapp"
RELEASES_DIR="$APP_DIR/releases"
SHARED_DIR="$APP_DIR/shared"
CURRENT_LINK="$APP_DIR/current"
RELEASE_PATH="$RELEASES_DIR/$(date +%Y%m%d%H%M%S)_${COMMIT_SHA:0:7}"
KEEP_RELEASES=5

echo "==> Creating release directory"
mkdir -p "$RELEASES_DIR"

echo "==> Cloning at $COMMIT_SHA"
git clone --depth 1 git@github.com:yourorg/yourapp.git "$RELEASE_PATH"
cd "$RELEASE_PATH"
git checkout "$COMMIT_SHA"

echo "==> Linking shared resources"
rm -rf "$RELEASE_PATH/storage"
ln -sfn "$SHARED_DIR/storage" "$RELEASE_PATH/storage"
ln -sfn "$SHARED_DIR/.env" "$RELEASE_PATH/.env"

echo "==> Installing Composer dependencies"
composer install \
  --no-dev \
  --no-interaction \
  --prefer-dist \
  --optimize-autoloader \
  --quiet

echo "==> Building frontend assets"
npm ci --silent
npm run build

echo "==> Running database migrations"
php artisan migrate --force

echo "==> Warming framework caches"
php artisan optimize

echo "==> Swapping symlink atomically"
ln -s "$RELEASE_PATH" "${CURRENT_LINK}.next"
mv --no-target-directory "${CURRENT_LINK}.next" "$CURRENT_LINK"

echo "==> Reloading PHP-FPM"
sudo systemctl reload php8.3-fpm

echo "==> Restarting long-running services"
php "$CURRENT_LINK/artisan" reload

echo "==> Pruning old releases (keeping $KEEP_RELEASES)"
cd "$RELEASES_DIR" && ls -t | tail -n "+$((KEEP_RELEASES + 1))" | xargs --no-run-if-empty rm -rf

echo "==> Release complete: $RELEASE_PATH"

The symlink swap line deserves its own explanation because the naive version contains a subtle bug. ln -sfn "$RELEASE_PATH" "$CURRENT_LINK" looks correct, but when $CURRENT_LINK already resolves to a directory (which it does after the first deploy), some ln implementations create the new symlink inside the target directory rather than replacing the link itself. The ln followed by mv --no-target-directory pattern sidesteps this entirely. On Linux, mv on the same filesystem is a kernel-level atomic rename. There is no window between the unlink and the new symlink creation where a request could resolve to a missing path.

php artisan optimize is the Laravel 12 shorthand for running config:cache, route:cache, view:cache, and event:cache in sequence. Run it inside the new release directory, before the symlink swap, so the first request to the new release hits a fully warm cache rather than triggering cold compilation on a live request.

Migrations, Cache Warming, and Service Restarts

Notice that php artisan migrate --force runs before the symlink swap. This is deliberate, and it introduces a constraint that every team using this pattern must internalise.

[Production Pitfall] When migrations run before the symlink swap, both old and new application code handle requests simultaneously during the migration window. The old release is still current while the database schema changes. This means every migration you write must be strictly backward-compatible with the previous release. Adding a nullable column: safe. Dropping a column the old code still reads: immediate 500s in production on every affected query during the migration window. Renaming a column that old code references: same result, every time.

The pattern that works is two-phase migration: deploy one adds the new column (old code ignores it, new code writes to it), deploy two removes the old column after the transition is complete and no live code references it. This discipline is enforced by services like PlanetScale’s branching workflow and documented explicitly in the Laravel deployment documentation. Treat it as a hard rule, not a guideline.

For service restarts, php artisan reload is the correct command in Laravel 12. It sends a graceful termination signal to queue workers, Laravel Reverb servers, and Octane processes in a single call. Workers finish their current job before exiting. Supervisor (or your process monitor of choice) detects the exit and starts a fresh worker process, which picks up /var/www/myapp/current/artisan — now pointing at the new release.

Supervisor configuration for your worker pool should use the current symlink path, not a hardcoded release directory:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/myapp/current/artisan queue:work \
  --sleep=3 \
  --tries=3 \
  --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/log/supervisor/laravel-worker.log
stopwaitsecs=3620

stopwaitsecs must exceed --max-time. If Supervisor sends SIGTERM and the worker has not exited within stopwaitsecs seconds, it escalates to SIGKILL. A SIGKILL mid-job is a lost job. Set stopwaitsecs to --max-time + 20 at minimum.

If you are running Laravel Horizon for queue management, the reload command handles it. Horizon monitors its own worker pool and restarts processes automatically after they exit gracefully. No Supervisor configuration changes are needed for Horizon workers specifically, though you will still want Supervisor managing the horizon process itself.

Health Checks and Automatic Rollback

By default, the health check route is served at /up and returns HTTP 200 when the application has booted without exceptions, and HTTP 500 otherwise. It is registered in bootstrap/app.php via the withRouting() method:

// bootstrap/app.php
return Application::configure(basePath: dirname(__DIR__))
    ->withRouting(
        web: __DIR__.'/../routes/web.php',
        commands: __DIR__.'/../routes/console.php',
        health: '/up',
    )
    ->withMiddleware(function (Middleware $middleware) {
        //
    })
    ->withExceptions(function (Exceptions $handler) {
        //
    })->create();

When HTTP requests are made to this route, Laravel dispatches a DiagnosingHealth event, allowing you to perform additional health checks. Within a listener for this event, you can check your application’s database or cache status, and throwing an exception will cause the endpoint to return 500.

For production, register a listener that verifies database connectivity and your primary cache store. A bootstrap-only health check catches a broken Service Container but misses a misconfigured database driver, which is a common failure mode after an environment variable change.

The pipeline waits 15 seconds after the SSH deploy step before hitting /up. That window absorbs PHP-FPM’s reload time and the brief period before Nginx resolves the new symlink on the next request cycle. Adjust it upward if your server is under high load during deployments.

The rollback script is intentionally simple. It does not run migrations, does not install dependencies, and does not build assets. It only swaps the symlink back and restarts services:

#!/usr/bin/env bash
set -euo pipefail

APP_DIR="/var/www/myapp"
RELEASES_DIR="$APP_DIR/releases"
CURRENT_LINK="$APP_DIR/current"

echo "==> Identifying previous release"
PREVIOUS=$(ls -t "$RELEASES_DIR" | sed -n '2p')

if [ -z "$PREVIOUS" ]; then
  echo "ERROR: No previous release found. Cannot roll back."
  exit 1
fi

echo "==> Rolling back to: $PREVIOUS"
ln -s "$RELEASES_DIR/$PREVIOUS" "${CURRENT_LINK}.rollback"
mv --no-target-directory "${CURRENT_LINK}.rollback" "$CURRENT_LINK"

echo "==> Reloading PHP-FPM"
sudo systemctl reload php8.3-fpm

echo "==> Restarting services on previous release"
php "$APP_DIR/current/artisan" reload

echo "==> Rollback complete"

After rollback, current points to the second-most-recent release. Workers restart and load that release’s code. If the health check failure was caused by a broken migration (not just a code error), the rollback restores application behaviour but the database schema remains in its post-migration state. Plan for this. Automated rollback is a traffic restoration tool, not a database recovery mechanism. If you need genuine point-in-time database state restored, you need a pre-migration snapshot.

Secrets, Environments, and the .env Problem

Never commit .env to your repository. The shared .env file on the server should be provisioned once, manually or through a secrets management tool, and then left in place across all deployments. This is one area where a well-defined infrastructure provisioning strategy pays off directly. The Infrastructure as Code guide for Laravel teams covers managing server state deterministically, including initial environment file provisioning and ongoing drift prevention.

Configure these secrets in GitHub under Settings > Secrets and variables > Actions, scoped to the production environment:

Secret	Purpose
DEPLOY_HOST	Production server IP or hostname
DEPLOY_USER	SSH user with write access to releases dir
DEPLOY_KEY	Private SSH key (Ed25519 recommended)
DEPLOY_PORT	SSH port (default: 22)
APP_DOMAIN	Application domain for the health check curl

The deploy user should own /var/www/myapp/releases/ and have a narrow sudoers entry permitting only the systemctl reload php8.3-fpm command. Do not deploy as root. This is non-negotiable.

For teams running multiple environments (staging and production), create separate GitHub environments with separate secret sets. Keeping your local development stack as close to production as possible reduces the chance that an environment-specific failure makes it to the pipeline at all. The Laravel AI development stack setup guide covers environment parity for teams running AI-integrated Laravel applications, where model API keys, inference configuration, and queue drivers all need consistent configuration across environments.

Running the Full Pipeline

With .github/workflows/deploy.yml committed, deploy.sh and rollback.sh on the server at /home/deploy/scripts/, and GitHub secrets configured, the deployment cycle is:

Developer pushes to main
GitHub Actions starts the test job with a MySQL service container
Pest/PHPUnit runs against a fresh migrated test database in CI
On green: the deploy job starts (pending any environment approval rules you have set)
SSH executes deploy.sh with the commit SHA on the production server
A new timestamped release directory is created, the repository cloned at that exact commit
Composer installs production dependencies, assets build
Migrations run against the production database (old release still serving)
php artisan optimize warms the framework cache inside the new release directory
Symlink atomically swaps to the new release
PHP-FPM reloads, services restart gracefully via php artisan reload
GitHub Actions waits 15 seconds, then hits https://yourdomain.com/up
HTTP 200: deployment succeeds, pipeline passes
Non-200: rollback.sh executes, symlink reverts, services restart on the previous release

The cycle from push to verified production deployment runs in roughly 3 to 5 minutes, depending on Composer install time and asset build complexity. That is fast enough to ship multiple times per day without ceremony or coordination overhead.

The git pull approach worked for years. But it is a manual process dressed up as a system, and manual processes fail the same way every time: at the worst possible moment, on the highest-traffic day of the quarter. This pipeline is not perfect either. It is a single-server pattern that needs adaptation when you scale horizontally, and at that point, purpose-built tools like Deployer PHP or a blue-green deployment model via your load balancer justify the added complexity. For teams running one to three application servers, this handles the vast majority of production deployments cleanly, automatically, and with a recoverable failure path. That is the bar to clear.

If your deployment pipeline grows to include pre-deploy validation steps such as AI inference middleware health checks or token budget verification, the job structure here accommodates additional pipeline steps naturally. The Laravel AI middleware guide covering token tracking and rate limiting describes middleware patterns that translate directly into pre-deployment validation hooks for applications that depend on AI provider availability.

OpenAI Structured Outputs in Laravel: Enforcing JSON Schema for Production AI Pipelines

Dewald Hugo — Sat, 02 May 2026 16:13:19 +0000

If you've been using OpenAI's JSON mode and calling it "structured output," you're building on a guarantee that's weaker than it looks. JSON mode promises syntactically valid JSON. It does not promise that status will be a string, that confidence will be a float, or that items won't quietly disappear from the response entirely. In production, that distinction costs you. We've watched pipelines collapse at 3am because a field the downstream queue worker expected simply wasn't there, and JSON mode had no complaint.

Using OpenAI structured outputs in Laravel, specifically response_format with type: json_schema and strict: true, closes that gap. Schema compliance is enforced at the generation level using constrained decoding. The model literally cannot emit a token that would violate your schema. This article covers how to wire that up correctly in Laravel, what the real failure modes are (they're not what you'd expect), and how structured outputs slot into an agentic pipeline where downstream steps depend on typed, predictable payloads.

This article focuses specifically on OpenAI's implementation. Gemini exposes the same concept via generationConfig.responseSchema, and Claude approaches it through tool definitions. The JSON Schema structure is largely portable across all three. What differs is the implementation layer: how each provider receives the schema, the refusal mechanism, strict mode constraints, and the error surface. Those differences warrant their own focused articles. If you want the cross-provider picture first, the Laravel AI integration architecture guide covers how they compare at the architectural level.

Why JSON Mode Is No Longer Good Enough

The table below is worth keeping. The differences are subtle on paper and catastrophic under load.

Feature	`json_object` mode	`json_schema` strict mode
Guarantees valid JSON	✅	✅
Guarantees all required keys present	❌	✅
Guarantees correct types	❌	✅
Enforces enum values	❌	✅
Enforces `additionalProperties: false`	❌	✅
Exposes `refusal` field	❌	✅
Compliant with JSON Schema spec subset	❌	✅

JSON mode guarantees syntactic correctness but does not guarantee schema adherence, even fields you mark as "required" in your prompt can go missing from the response. OpenAI's own documentation now treats json_object mode as legacy. Strict mode with json_schema is the production default for data extraction and agentic workflows.

If your Laravel application is making decisions (routing jobs, updating records, triggering downstream agents) based on AI-generated JSON, you need the stronger guarantee.

How OpenAI Structured Outputs Work Under the Hood

Understanding the mechanism matters because it explains both the capability and the constraints.

OpenAI uses a Context-Free Grammar engine to mask invalid tokens before generation, the model literally cannot produce a non-conforming response. This is not prompt-level enforcement. No amount of instruction-following or fine-tuning does what constrained decoding does. The schema is compiled into the generation process itself.

Safety policies still apply. The model will abide by its existing safety rules and may refuse an unsafe request. When it does, it returns a refusal string value on the response, allowing you to programmatically detect that the model generated a refusal instead of schema-conforming output.

Two things to internalise before writing a line of code:

If the model can answer, it will conform to your schema. That's the guarantee.
If the model won't answer (safety refusal), you get a refusal field instead of content. You must handle both branches explicitly.

Defining Your JSON Schema in Laravel

Schema definitions scattered across service classes are a maintenance problem. Keep them in a dedicated config file: the Laravel-native home for static, environment-aware definitions that require no class instantiation to access.

<?php

// config/ai-schemas.php

return [

    'review_analysis' => [
        'name'   => 'review_analysis',
        'strict' => true,
        'schema' => [
            'type'                 => 'object',
            'additionalProperties' => false,
            'required'             => ['sentiment', 'confidence', 'summary', 'flags', 'escalate'],
            'properties'           => [
                'sentiment'  => [
                    'type' => 'string',
                    'enum' => ['positive', 'negative', 'neutral', 'mixed'],
                ],
                'confidence' => [
                    'type' => 'number',
                ],
                'summary'    => [
                    'type' => 'string',
                ],
                'flags'      => [
                    'type'  => 'array',
                    'items' => ['type' => 'string'],
                ],
                'escalate'   => [
                    'type' => ['boolean', 'null'],
                ],
            ],
        ],
    ],

];

Pull it into your service with a single config call:

'response_format' => [
    'type'        => 'json_schema',
    'json_schema' => config('ai-schemas.review_analysis'),
],

Config is publishable, overridable per environment, and version-controlled alongside your codebase. When your schema changes, you change one file. No class to instantiate, no namespace to import.

Strict Mode Constraints You Must Know

OpenAI's strict mode is enforced at the generation level using constrained decoding, not at the prompt level. That distinction matters because it means the constraints are hard, not advisory. The API will reject your request outright if your schema violates them, so know the rules before you deploy.

additionalProperties must be set to false for every object in the schema, and all fields in properties must appear in required. If your schema violates either rule, the API rejects the request immediately. You won't get a gracefully degraded response, you'll get an HTTP error.

The following JSON Schema keywords are not supported and will cause the API to reject your request:

default values
minLength / maxLength on strings
minimum / maximum on numbers
pattern for regex constraints
if / then / else conditional schemas

Work within these constraints, not around them. Post-API validation in Laravel (covered below) handles the domain rules that OpenAI won't enforce.

[Edge Case Alert] JSON Schema $defs are supported by strict mode for reusable sub-schemas. If you reference a $def that isn't declared in the same schema object, the API will reject the request silently in some SDK versions rather than returning a useful error. Always validate complex schemas locally before deploying. A unit test that calls json_validate() on your schema definition is cheap insurance.

Handling Optional Fields

This is where developers get tripped up. You cannot simply omit a field from required, strict mode requires every property to be required. The idiomatic solution is to make the type nullable:

'escalate' => [
    'type' => ['boolean', 'null'],
],

The model can now return null for escalate, which your PHP code handles cleanly. Do not try to work around this by removing optional fields from your schema entirely; you lose the benefit of a complete, predictable contract.

Making the API Call with Strict Schema Enforcement

Wrap the API call in a dedicated service. This is not optional in production. You need a single place to apply retry logic, log token usage, and handle the response contract.

<?php

namespace App\AI\Services;

use App\Exceptions\AI\ModelRefusalException;
use App\Exceptions\AI\TruncatedResponseException;
use Illuminate\Support\Facades\Log;
use OpenAI\Laravel\Facades\OpenAI;
use OpenAI\Exceptions\TransporterException;
use OpenAI\Exceptions\ErrorException;

class ReviewAnalysisService
{
    private const MODEL = 'gpt-4o';
    private const MAX_RETRIES = 3;
    private const RETRY_DELAY_MS = 500;

    public function analyse(string $reviewText): array
    {
        $attempt = 0;

        while ($attempt < self::MAX_RETRIES) {
            $attempt++;

            try {
                $response = OpenAI::chat()->create([
                    'model'           => self::MODEL,
                    'messages'        => [
                        [
                            'role'    => 'system',
                            'content' => 'You are a review analysis assistant. Analyse the provided customer review and respond according to the schema.',
                        ],
                        [
                            'role'    => 'user',
                            'content' => $reviewText,
                        ],
                    ],
                    'response_format' => [
                        'type'        => 'json_schema',
                        'json_schema' => config('ai-schemas.review_analysis'),
                    ],
                    'max_tokens' => 1024,
                ]);

                return $this->parseResponse($response);

            } catch (ErrorException $e) {
                // Rate limit: 429, server error: 5xx
                if ($e->getCode() === 429 || $e->getCode() >= 500) {
                    if ($attempt < self::MAX_RETRIES) {
                        usleep(self::RETRY_DELAY_MS * 1000 * $attempt);
                        continue;
                    }
                }

                Log::error('OpenAI structured output error', [
                    'message' => $e->getMessage(),
                    'code'    => $e->getCode(),
                ]);

                throw $e;

            } catch (TransporterException $e) {
                // Network-level failure, retry
                if ($attempt < self::MAX_RETRIES) {
                    usleep(self::RETRY_DELAY_MS * 1000 * $attempt);
                    continue;
                }
                throw $e;
            }
        }

        throw new \RuntimeException('Max retries exceeded for structured output request.');
    }

    private function parseResponse(mixed $response): array
    {
        $choice = $response->choices[0];

        // Handle truncation before checking refusal
        if ($choice->finishReason === 'length') {
            throw new TruncatedResponseException(
                'Structured output response was truncated. Increase max_tokens or simplify the schema.'
            );
        }

        $message = $choice->message;

        if (!empty($message->refusal)) {
            throw new ModelRefusalException($message->refusal);
        }

        return json_decode($message->content, true, 512, JSON_THROW_ON_ERROR);
    }
}

Register this service in the Service Container via a provider, and inject it where needed. Never reach for the facade directly in controllers if you can avoid it.

The Two Failure Modes You Must Handle

Schema enforcement eliminates the malformed JSON failure class entirely. But two failure modes survive strict mode, and both require explicit handling.

Refusals

When the model declines to answer due to a safety policy, message.content is null and message.refusal contains the refusal explanation. Do not treat this as a generic exception, it's a first-class response state. Log it separately, because a spike in refusals often signals a prompt injection attempt or a shift in input data quality.

<?php

namespace App\Exceptions\AI;

class ModelRefusalException extends \RuntimeException
{
    public function __construct(string $refusalReason)
    {
        parent::__construct("Model refused the request: {$refusalReason}");
    }
}

In production, we've routed these to a separate monitoring channel. A refusal on a data-extraction job is almost always either a malformed upstream payload or a policy edge case worth reviewing manually.

Truncated Responses

finish_reason: 'length' means the model ran out of output tokens before completing the schema. In strict mode, a truncated response is always invalid JSON. The constrained decoder cannot produce partial schema-conforming output, so the token budget cuts the response mid-stream. The fix is almost always to increase max_tokens or to reduce schema complexity.

[Production Pitfall] Truncation in structured outputs is not recoverable at the application level. You cannot repair a half-written JSON object. Always set max_tokens generously and monitor finish reasons in your telemetry. A length finish reason on a structured output call is a configuration bug, not an edge case.

Laravel Validation Beyond the Schema

Strict mode guarantees structural compliance. It does not guarantee semantic correctness. A confidence field of 999.5 is perfectly valid JSON, schema-compliant, and completely wrong for your business logic.

A dedicated Laravel Data Transfer Object with validation handles the gap cleanly:

<?php

namespace App\AI\DTOs;

use Illuminate\Support\Facades\Validator;
use Illuminate\Validation\ValidationException;

readonly class ReviewAnalysisResult
{
    public function __construct(
        public string $sentiment,
        public float  $confidence,
        public string $summary,
        public array  $flags,
        public ?bool  $escalate,
    ) {}

    public static function fromArray(array $data): self
    {
        $validator = Validator::make($data, [
            'sentiment'  => ['required', 'string', 'in:positive,negative,neutral,mixed'],
            'confidence' => ['required', 'numeric', 'between:0,1'],
            'summary'    => ['required', 'string', 'min:10', 'max:1000'],
            'flags'      => ['required', 'array'],
            'flags.*'    => ['string'],
            'escalate'   => ['nullable', 'boolean'],
        ]);

        if ($validator->fails()) {
            throw new ValidationException($validator);
        }

        return new self(
            sentiment:  $data['sentiment'],
            confidence: (float) $data['confidence'],
            summary:    $data['summary'],
            flags:      $data['flags'],
            escalate:   $data['escalate'] ?? null,
        );
    }
}

This follows the same validation patterns documented in our complete guide to Laravel OpenAI integration. The schema enforces structure; the DTO enforces domain rules. Both layers earn their place.

[Architect's Note] The combination of strict schema + DTO validation gives you something genuinely new: an AI response you can type-hint. Downstream code that receives a ReviewAnalysisResult can trust its shape as reliably as any other value object in your application. This is how you stop writing defensive null-checks six layers deep.

Integrating Structured Outputs Into an Agentic Pipeline

This is where the real value materialises. In an agentic workflow, one AI call produces the input that triggers the next step. Without schema guarantees, you're writing defensive code at every handoff. With strict mode, the handoff is a typed contract.

<?php

namespace App\Jobs;

use App\AI\DTOs\ReviewAnalysisResult;
use App\AI\Services\ReviewAnalysisService;
use App\Exceptions\AI\ModelRefusalException;
use App\Exceptions\AI\TruncatedResponseException;
use App\Models\Review;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Support\Facades\Log;

class AnalyseReviewJob implements ShouldQueue
{
    use InteractsWithQueue, Queueable;

    public int $tries = 1; // Retries are handled inside the service

    public function __construct(
        private readonly int $reviewId
    ) {}

    public function handle(ReviewAnalysisService $service): void
    {
        $review = Review::findOrFail($this->reviewId);

        try {
            $raw    = $service->analyse($review->body);
            $result = ReviewAnalysisResult::fromArray($raw);

            $review->update([
                'sentiment'     => $result->sentiment,
                'ai_confidence' => $result->confidence,
                'ai_summary'    => $result->summary,
                'flags'         => $result->flags,
            ]);

            // Gate the next pipeline step on a semantic condition
            if ($result->escalate === true) {
                dispatch(new EscalateReviewJob($this->reviewId));
            }

        } catch (ModelRefusalException $e) {
            Log::warning('Review analysis refused by model', [
                'review_id' => $this->reviewId,
                'reason'    => $e->getMessage(),
            ]);

            $review->update(['ai_status' => 'refused']);

        } catch (TruncatedResponseException $e) {
            Log::error('Review analysis truncated', [
                'review_id' => $this->reviewId,
            ]);

            $this->fail($e);
        }
    }
}

Notice what we're not doing: we're not writing isset($result['escalate']) before every downstream action. The DTO guarantees the key exists and has the correct type. The agentic branching logic is clean because the data contract is clean. This is what our production AI architecture contracts and governance guide refers to as treating AI outputs as typed system interfaces rather than raw text.

For heavier pipelines where multiple agents hand off between each other, pair this approach with schema validation against LLM hallucinations. Structured outputs remove structural hallucinations, but semantic drift (a confidence value that's technically valid but meaninglessly high) still needs an application-level assertion layer.

On the infrastructure side, every structured output job should run through Laravel AI middleware for token tracking so you capture token consumption per job class. Structured outputs tend to have stable, predictable token counts once your schema is stable, which makes anomaly detection on token spend genuinely useful. And if you want to understand how the temperature and max_tokens parameters interact with constrained decoding, the Laravel LLM inference control guide covers the mechanics.

Using a Laravel Abstraction Layer

Everything covered in this article is the raw implementation. For teams who don't want to manage schema definitions, response parsing, and retry logic manually, Prism PHP handles it through a unified SDK that wraps OpenAI, Anthropic, and Gemini behind a consistent interface.

The low-level approach earns its place for two reasons. First, Prism's structured output support is built on exactly these primitives, so understanding what the abstraction does underneath makes you a better consumer of it. Second, when the abstraction breaks (and it will, at a provider API boundary, under load, with an edge-case schema), you need to know what to reach for.

The decision point is straightforward. Greenfield project with multiple AI providers, or a team that shouldn't be managing provider-specific quirks: use Prism. Single-provider integration where you want explicit control over the request lifecycle, retry behaviour, and token telemetry: go direct.

Laravel AI Integration: A Production-Ready Architecture Guide (OpenAI vs Gemini vs Claude)

Dewald Hugo — Mon, 27 Apr 2026 00:47:06 +0000

Every Laravel AI integration starts the same way. You install a package, copy an API key into .env, write a controller method that fires a prompt, and it works. That approach is fine for a proof of concept. It is not fine for production.

The first incident makes that clear, a 429 rate limit silences a customer-facing feature at 2AM, a model deprecation breaks a workflow three sprints before anyone noticed the announcement, or a runaway token loop burns through your monthly budget in four hours. Unlike a payment or CRM API where breaking changes arrive with a migration guide and six months notice, LLM providers will deprecate model strings mid-sprint, shift safety filter behaviour between deployments, and bury the changelog entry in a developer newsletter. The maintenance surface is larger and less predictable than any other API category most Laravel developers have worked with. That changes what the architecture needs to do.

This guide assumes you have already decided to integrate AI into your Laravel application. The question it answers is how to do that without it becoming a production liability. No authentication walkthroughs, no step-by-step setup. What you get is the system-level map: the problem space, the architectural patterns that address it, a clear-eyed comparison of OpenAI, Gemini, and Claude, and a decision framework you can put in front of your team before the next kickoff.

The Laravel AI Integration Problem Space

Before picking a provider or drawing a service diagram, you need to be honest about what you are actually building. The problems that make AI difficult in production are not unique to any one provider. They are structural.

Provider Abstraction

You will almost certainly swap or combine providers over the lifecycle of a product. The model that makes sense at launch is rarely the model that makes sense at scale, or after a pricing change, or when your use case evolves. If your application code couples directly to OpenAI’s SDK or Anthropic’s client, a provider change means rewriting feature logic, not just swapping a configuration value.

The correct answer is a provider interface your application depends on, with concrete adapters behind it. We cover the interface contract below.

Token Usage and Cost Control

Language models charge by the token. That sounds simple. In practice it means a single misconfigured prompt, multiplied by your queue throughput, can produce a five-figure bill in an afternoon. We have seen this happen.

Token management is not optional in production. You need budget enforcement before the API call, usage logging at the response level, and alerting when spend crosses threshold. This belongs in the service layer – not in a controller, not in a scheduled job that runs after the damage is done.

Latency and Streaming

A synchronous AI call in an HTTP handler is almost always wrong. Inference latency runs from 800ms on a lightweight model to 30 seconds on a long-form reasoning task. PHP-FPM holds a connection open for that entire window. Under any meaningful traffic, that collapses your worker pool.

Streaming helps on the UX side, but it does not fix the connection pressure problem. Background jobs fix that. The streaming question is a separate decision, covered in its own section below.

Error Handling and Retries

AI APIs fail. They return 429s during traffic spikes, 500s during model instability, and occasionally time out with no response body at all. None of these are exceptional conditions. They are routine.

Your retry logic must distinguish between transient failures (worth retrying with backoff) and structural failures (wrong API key, malformed payload, context length exceeded). Treating all failures the same is one of the most common production bugs we see in AI-integrated Laravel applications.

Vendor Lock-in

Every provider has proprietary extensions – function calling formats, system prompt conventions, streaming event shapes, vision attachment schemas. The moment you hardcode those shapes into your application logic, you own a migration project the next time you need to switch.

Abstraction is the mitigation. But abstraction has a cost. Build it at the right layer, and keep provider-specific behaviour in the adapter, not in your domain services.

Provider Comparison: OpenAI vs Gemini vs Claude

No provider is the right answer for every use case. Here is an honest comparison of the three dominant options for Laravel AI integration.

Capability	OpenAI (GPT-4o)	Google (Gemini 2.5 Pro)	Anthropic (Claude Sonnet 4.5)
Context window	128K tokens	1M+ tokens	200K tokens
Streaming support	✓	✓	✓
Function / tool calling	✓	✓	✓
Vision / multimodal	✓	✓	✓
Native embeddings	✓ (text-embedding-3)	✓	✗ (use OpenAI or Gemini)
Laravel SDK support	First-party	Via laravel/ai	Via laravel/ai
Reasoning quality	Strong, broad	Strong, long-context	Exceptional, safety-aware
Rate limit tolerance	Good	Variable	Conservative
Pricing tier	Mid	Mid	Mid-High

Honest notes on each:

OpenAI has the most mature Laravel ecosystem. The first-party package is stable, the API surface is well-documented, and function calling is production-tested at scale. It is the right default for teams building general-purpose AI features with limited architectural budget.

Gemini’s one-million-token context window is not a marketing number, it genuinely changes what is possible for document processing, long-session memory, and RAG-less retrieval patterns. The laravel/ai SDK wraps it cleanly. The downside is that rate limits are inconsistently enforced and the model behaviour at the boundary of that context window is less predictable than Anthropic’s or OpenAI’s.

Claude is the most consistent model for reasoning-heavy tasks, structured output extraction, code review pipelines, and anything that requires following complex multi-step instructions reliably. The safety constraints are real and occasionally frustrating for edge cases, but the instruction-following quality justifies the slightly higher cost for high-stakes features.

Recommended Architecture for Laravel AI Systems

The pattern that holds up across every provider, every scale, and every use case is the same: isolate AI behind a typed contract, implement concrete adapters per provider, and register the active provider via the Service Container.

namespace App\Contracts\AI;

interface AIProviderInterface
{
    public function generate(string $prompt, array $options = []): string;

    public function stream(string $prompt, array $options = []): \Generator;

    public function embed(string $text): array;
}

Each provider - OpenAI, Gemini, Claude - gets its own adapter class that implements this interface. Your application services, queued jobs, and controllers depend on AIProviderInterface. They never know which provider sits behind it.

// AppServiceProvider or bootstrap/app.php
$this->app->bind(
    \App\Contracts\AI\AIProviderInterface::class,
    \App\AI\Adapters\ClaudeAdapter::class,
);

Swapping providers is now a one-line configuration change. No domain logic moves.

On top of this interface sits your governance layer – budget enforcement, telemetry, prompt versioning. Our guide on production-grade AI architecture in Laravel covers that layer in full, including typed AiResult data objects, decorator chains, and audit trail patterns. Read that guide before you build the adapter layer, the contract it defines will inform how you structure the interface above.

[Architect’s Note] The three-method interface above is intentionally minimal. Resist the temptation to add provider-specific capabilities - vision input, citation formats, tool schemas - to the shared interface. Those belong in extended interfaces or provider-specific services consumed explicitly. The shared interface exists to make the 80% case portable. The 20% case is allowed to know which provider it is talking to.

Configuration Management

Provider credentials live in config/ai.php, not scattered across service provider constructors. Each adapter reads from config('ai.providers.claude.api_key') and so on. Environment variables map cleanly through the config layer, and you can define per-provider defaults - temperature, max tokens, retry budget - without touching application code.

Token Management and Cost Control

This is the section most architectural guides skip. It is also the section that determines whether your AI feature is financially viable.

Token costs compound. A feature that runs 500 times per day at 2,000 tokens per call is one million tokens daily. Multiply by your model’s per-token rate, multiply by 30, and you have a monthly infrastructure line item that belongs in your budget conversations from day one.

The enforcement point must be pre-dispatch. That means checking the estimated token count against a configured budget before the API call fires, not after the response arrives. You cannot claw back tokens you have already consumed.

For systems that incorporate retrieval, the token problem gets harder. Injecting retrieved documents into a prompt is the right architectural move for accuracy, but each document chunk contributes to input token count. RAG systems need token-aware chunking at retrieval time, not just at prompt assembly time. If you are building retrieval into your Laravel application, the RAG and vector retrieval implementation guide covers the chunking and embedding strategy in detail.

Telemetry is the other half of cost control. Log prompt tokens, completion tokens, model, latency, and user or tenant identity on every response. Without that data, you cannot attribute cost, identify expensive call patterns, or make informed decisions about model selection.

Streaming vs Standard Responses

The choice between streaming and synchronous responses is a UX decision that has infrastructure consequences.

Synchronous calls are appropriate for background jobs, scheduled tasks, and any context where the result is written to storage before the user sees it. They are simpler to implement, easier to test, and do not require any special transport infrastructure. If your AI call feeds a Queue Worker writing to Eloquent, synchronous is the right default.

Streaming is appropriate when a human is waiting and the output is long enough that a full-response delay degrades the experience. The threshold in practice is around 1.5–2 seconds. Below that, streaming adds complexity for no perceivable gain.

The transport mechanism behind streaming is a separate decision. Server-Sent Events are the right default for Laravel, they require no additional infrastructure, work through standard HTTP, and integrate cleanly with Livewire. WebSockets are justified when you need bidirectional communication or multi-client broadcast. Livewire polling is a reasonable fallback for teams that want to avoid streaming infrastructure entirely, at the cost of perceived responsiveness.

The tradeoffs between these three approaches are significant enough to warrant their own analysis. Our streaming transport decision guide covers all three with production code and explicit recommendations for each scenario.

Choosing the Right Provider: A Decision Framework

Stop treating provider selection as a preference. Treat it as an engineering decision with clear inputs.

Use OpenAI when:

You need native embeddings alongside generation (text-embedding-3-small covers most retrieval use cases)
Your team needs the most widely documented integration path
You are building function-calling pipelines where tool reliability matters more than reasoning depth
You want the lowest overhead Laravel AI integration for an MVP that may evolve later

The complete OpenAI integration guide covers transport, function calling, and retry patterns against the GPT-4o and GPT-4o-mini endpoints.

Use Gemini when:

Your use case involves documents or sessions that exceed 128K tokens
You are processing multimodal inputs — documents, images, mixed media — at scale
You want to experiment with RAG-less patterns using Gemini’s long context as a retrieval proxy
Cost at volume is a primary concern (Gemini 2.0 Flash is competitive at high call rates)

For Gemini setup via the Laravel AI SDK, including the GeminiManager failover pattern and model string verification, that guide covers everything from provider registration to streaming response handling.

Use Claude when:

The task requires following multi-step instructions precisely and reliably
You are extracting structured output from unstructured text (JSON extraction, data normalisation)
Safety and refusal behaviour matter — regulated industries, content moderation pipelines
Code generation or review is a significant part of the workload

The Laravel Claude API integration guide covers the full Anthropic client setup, system prompt patterns, and streaming implementation with Laravel’s HTTP client.

Mixed-provider architectures

The interface pattern described earlier makes multi-provider setups practical. Routing by task type, OpenAI for embeddings, Claude for structured extraction, Gemini for long-document summarisation, is a legitimate production architecture, not over-engineering. The Service Container resolves the correct adapter per task type, and your domain code stays clean.

[Production Pitfall] Mixed-provider setups introduce a subtle budget tracking problem. If you log token costs per provider separately, you lose the aggregate view. Make sure your telemetry table records the provider name alongside every call, and your cost dashboards aggregate across providers by feature or tenant, not just by provider account.

Conclusion

If you take one thing from this guide, make it this: the providers are interchangeable. The architecture is not.

OpenAI, Gemini, and Claude will all improve, deprecate models, change pricing, and introduce capabilities over the next twelve months. Any of the three can be the right choice today and the wrong choice in two quarters. The teams that weather those changes without incident are the teams that built the abstraction layer before they needed it, not the teams that coupled their application to a specific SDK and are now rewriting feature logic to swap providers.

Production Laravel AI integration is a system design problem. The interface contract, the adapter layer, the token budget enforcement, the streaming transport decision, the telemetry pipeline — those are the decisions that determine whether your AI feature is operationally sustainable, or just a demo that scaled badly.

Build the layer. Then pick the provider.

Laravel OpenAI Integration: The Complete Production Guide

Dewald Hugo — Sun, 26 Apr 2026 20:59:38 +0000

Laravel OpenAI integration is not simpler than Claude integration. The APIs differ, the model families differ — but the architectural requirements are nearly identical once you care about production stability.

Laravel works well here because it already solves the problems AI integrations create: dependency boundaries, environment isolation, background work, failure handling, and observability. You’re not fighting the framework. You’re leaning on it.

On model names: This guide uses gpt-4o and gpt-4o-mini as examples. These are current as of early 2026. OpenAI’s model catalogue evolves quickly — always verify against OpenAI’s official models reference before deploying. gpt-3.5-turbo is deprecated; gpt-4o-mini is its cost-effective replacement.

Before committing to OpenAI as your sole provider, it is worth understanding where it sits in the broader Laravel AI provider landscape. Our provider comparison and architecture decision guide covers how OpenAI stacks up against Gemini and Claude for different production use cases — useful context before you build the service layer around any single provider.

Understanding the OpenAI API Surface

OpenAI’s platform has evolved faster than its documentation. Many tutorials conflate multiple API generations or mix incompatible approaches.

For Laravel applications, think in terms of three distinct capabilities: the Chat Completions API for text generation and reasoning, the Embeddings API for semantic search and retrieval, and the Audio/Vision APIs for transcription and multimodal inputs.

For image generation specifically – gpt-image-1, queued jobs, S3 storage, and per-request cost tracking — see the Laravel OpenAI image generation guide.

OpenAI is not “one API”. Your Laravel architecture should reflect that separation early. Controllers that call the Completions and Embeddings endpoints directly will become unmaintainable fast.

Foundations

Authentication and Environment Configuration

OpenAI uses API keys issued at the account or project level. Treat them as production secrets — same as database credentials or payment tokens.

OPENAI_API_KEY=sk-...
OPENAI_ORG_ID=optional

Never read these values directly from env() outside of configuration files. This is one of the most common structural mistakes we see in Laravel AI codebases. Expose them through a dedicated config file:

// config/openai.php
return [
    'api_key'      => env('OPENAI_API_KEY'),
    'organization' => env('OPENAI_ORG_ID'),
    'base_url'     => env('OPENAI_BASE_URL', 'https://api.openai.com/v1'),
];

This centralizes OpenAI configuration and allows overrides for testing, proxying, or self-hosted gateways. Skipping this step is how vendor logic leaks across the codebase and makes rotation painful.

Security: Never commit API keys to git. Use Laravel’s encrypted environment handling in production, and rotate keys periodically. Treat a leaked OpenAI key with the same urgency as a leaked database password — it has a direct billing impact.

Why You Should Not Use the OpenAI SDK Directly

OpenAI provides official SDKs, including PHP community implementations. Injecting these directly into controllers or jobs is a mistake.

SDKs change. Model names change. Response formats evolve. Your application should not absorb that churn.

Access OpenAI through a service boundary you control:

class OpenAiClient
{
    public function __construct(
        protected HttpClient $http
    ) {}
}

And then:

class OpenAiService
{
    public function __construct(
        protected OpenAiClient $client
    ) {}
}

This prevents your controllers from becoming vendor adapters. It also makes the Laravel Service Container work for you — you can bind a mock in tests without touching a single controller.

HTTP Client Configuration and Timeouts

AI requests can be long-running, bursty, expensive, and sensitive to retries. A default Http::post() is insufficient. Configure a dedicated client with explicit behaviour:

Http::withHeaders([
    'Authorization' => 'Bearer ' . config('openai.api_key'),
    'Content-Type'  => 'application/json',
])
->timeout(60)
->retry(0, 0);

The timeout(60) matters because OpenAI requests can legitimately take tens of seconds with large prompts or slow models. The retry(0, 0) matters even more — retrying AI requests blindly can double your cost and produce inconsistent outputs. If you need retries, implement them at the application level with full context awareness, not at the transport layer.

Error Handling as a First-Class Concern

OpenAI’s Chat Completions API fails in ways standard REST APIs rarely do: partial responses, rate limit throttling, model overload, malformed outputs despite 200 responses.

Your service layer must treat OpenAI responses as untrusted input, even when HTTP status codes indicate success.

$response = $this->client->post(...);

if (! $response->successful()) {
    throw new OpenAiRequestFailed(
        $response->status(),
        $response->body()
    );
}

But that’s only half the problem. You must also validate semantic correctness. Did the model return text when you expected JSON? Did it ignore system instructions? Did it hallucinate fields? That validation logic does not belong in a controller.

Service Architecture: Designing for Change

Your OpenAI service should not expose low-level API primitives unless absolutely necessary.

Instead of:

$openai->chatCompletion([...]);

Prefer intent-based methods:

$openai->generateText(...)
$openai->summarize(...)
$openai->extractStructuredData(...)

Internally, these methods may all use the same API endpoint. Externally, they express business intent. This gives you three architectural wins: prompts become versionable assets, you can A/B test models without touching callers, and you can migrate providers with minimal surface-area changes.

If you later introduce Claude alongside OpenAI — a decision many teams make as they want provider redundancy — this design pays for itself immediately. The service boundary is what makes multi-model routing tractable.

Dependency Injection and Testability

Every OpenAI-facing class should be injectable and mockable. No static helpers, no facades hiding HTTP calls, no env() reads at runtime.

In tests, you should be able to swap the OpenAI client with a fake implementation that returns deterministic responses. If you cannot do that without modifying production code, you are not integrating OpenAI — you are coupling your application to it.

Core Integration Patterns

Once your foundation is in place, OpenAI becomes a capability your application depends on. These patterns are not OpenAI-specific — they’re standard Laravel patterns applied to probabilistic systems.

Pattern 1: Simple Request–Response (One-Shot Tasks)

The simplest OpenAI interaction is still the most common: you send a prompt, you receive a response, the request lifecycle ends. Classification, extraction, summarization, content drafting — all one-shot.

Most tutorials implement this directly in controllers. Don’t.

Designing the Service Method

class OpenAiService
{
    public function generateText(string $prompt): string
    {
        // implementation
    }
}

Implementing the API Call

$response = $this->client->post('/chat/completions', [
    'model'    => 'gpt-4o',
    'messages' => [
        [
            'role'    => 'user',
            'content' => $prompt,
        ],
    ],
]);

Model selection is explicit. Never rely on defaults — model choice is an architectural decision with cost and quality implications.

Extracting the Result Safely

$data    = $response->json();
$content = data_get($data, 'choices.0.message.content');

if (! $content) {
    throw new UnexpectedOpenAiResponse($data);
}

return $content;

Use data_get() here. Future models may return multiple content blocks, safety filters may suppress output, and partial responses can occur under load. Your service method should either return valid output or fail loudly — silent nulls are debugging nightmares.

Pattern 2: Streaming Responses (Long-Running Output)

Streaming is not about novelty. It’s about user experience under latency. Large outputs can take seconds. Streaming turns that wait into visible progress.

When Streaming Makes Sense

Use streaming when output length is unbounded and perceived latency matters. Avoid it when you need structured JSON output, must validate the full response before use, or results are consumed only by background jobs.

Server-Side Streaming with OpenAI

OpenAI supports streaming via Server-Sent Events (SSE). On the Laravel side:

return response()->stream(function () use ($prompt) {
    $this->openAi->streamText($prompt, function ($chunk) {
        echo "data: " . json_encode($chunk) . "\n\n";
        ob_flush();
        flush();
    });
}, 200, [
    'Content-Type'     => 'text/event-stream',
    'Cache-Control'    => 'no-cache',
    'X-Accel-Buffering' => 'no',
]);

Implementing the Stream Method

⚠️ The sink Option Defeats Streaming Entirely

A common implementation mistake is using ->withOptions(['stream' => true, 'sink' => fopen('php://temp', 'w+')]) in Laravel’s HTTP client. The sink option tells Guzzle to buffer the entire response to that destination before returning. Your while (!feof($handle)) loop runs after the connection is already closed. You are not streaming — you are reading a completed buffer with extra steps. The UX benefits you expect don’t exist.

For true streaming, use Guzzle directly with stream => true and read the PSR-7 body incrementally:

public function streamText(string $prompt, callable $onChunk): void
{
    $client   = new \GuzzleHttp\Client();
    $response = $client->post('https://api.openai.com/v1/chat/completions', [
        'headers' => [
            'Authorization' => 'Bearer ' . config('openai.api_key'),
            'Content-Type'  => 'application/json',
        ],
        'json'   => [
            'model'    => 'gpt-4o',
            'messages' => [['role' => 'user', 'content' => $prompt]],
            'stream'   => true,
        ],
        'stream' => true,
    ]);

    $body = $response->getBody();
    $resource = $body->detach();
    while (! feof($resource)) {
        $line = rtrim(stream_get_line($resource, 4096, "\n"));

        if (empty($line) || $line === 'data: [DONE]') {
            continue;
        }

        if (str_starts_with($line, 'data: ')) {
            $data = json_decode(substr($line, 6), true);

            if (isset($data['choices'][0]['delta']['content'])) {
                $onChunk($data['choices'][0]['delta']['content']);
            }
        }
    }
}

Senior Dev Tip: If you need to decide between Livewire polling, SSE, and WebSockets for your AI UI, that architectural choice has more impact than the streaming implementation itself. We compared all three approaches in Livewire vs SSE vs WebSockets for AI UIs — worth reading before you commit.

Pattern 3: Conversation Memory (Multi-Turn Interactions)

Chatbots are one example. Multi-step reasoning, guided forms, iterative content refinement — all of these are multi-turn workflows. The key challenge is state management, not prompt engineering.

Representing Conversation State

Schema::create('conversations', function (Blueprint $table) {
    $table->id();
    $table->json('context')->nullable();
    $table->timestamps();
});

Schema::create('conversation_messages', function (Blueprint $table) {
    $table->id();
    $table->foreignId('conversation_id');
    $table->string('role'); // 'user' or 'assistant'
    $table->text('content');
    $table->timestamps();
});

Rehydrating Context

Before sending a request, load the full message history via Eloquent and build the message array:

$messages = $conversation->messages()
    ->orderBy('created_at')
    ->get()
    ->map(fn ($msg) => [
        'role'    => $msg->role,
        'content' => $msg->content,
    ])
    ->toArray();

Then append the new user input and send the full list to OpenAI. Large language models do not remember anything between API calls. Memory is entirely your responsibility. Persisting messages also lets you truncate context intelligently and summarize older turns when token counts grow large.

Pattern 4: Background Processing (Queued AI Work)

Not every AI task belongs in the request–response cycle. Bulk processing, document analysis, scheduled generation — these belong in queued jobs.

class GenerateSummaryJob implements ShouldQueue
{
    public function handle(OpenAiService $openAi): void
    {
        $summary = $openAi->summarize($this->document);

        $this->document->update(['summary' => $summary]);
    }
}

Laravel’s queue system provides retry semantics, failure hooks, and concurrency control — all essential when working with a rate-limited, cost-based API. A background job that silently retries three times on a large document can triple your bill. Jobs must cap token usage, log cost metadata, and fail deterministically.

Connecting the Patterns

These four patterns combine in real applications: a Livewire UI streams output, backed by a conversation record, which occasionally offloads heavy steps to queues, all of which reuse the same OpenAI service layer. That consistency is the real win — one abstraction, composable in every direction.

Building a Document Q&A Feature

Chatbots are not the most interesting application of LLMs in production. The real leverage comes from augmenting your existing data — documents, records, internal knowledge — with natural language access.

We’ll implement a Document Q&A feature: a user uploads a document, the system extracts and stores its content, and the user can ask questions grounded strictly in that document’s content. This exercises nearly every core pattern: one-shot generation, multi-turn context, background processing, and defensive prompt design.

We deliberately skip embeddings and vector databases here. That’s a scoped decision, not a shortcut. The simpler architecture works today and degrades predictably when it eventually outgrows itself.

Problem Definition and Constraints

What we want: natural language questions over uploaded documents, answers limited to document content, reasonable performance for mid-sized files, and clear failure modes.

What we’re not solving yet: semantic search across large corpora, cross-document reasoning, or high-recall retrieval at scale. Many “RAG tutorials” prematurely introduce vector databases when a simpler architecture would suffice.

High-Level Architecture

Four stages: ingestion (document uploaded and stored), extraction (text extracted and normalized), context preparation (content transformed into prompt context), and question answering (user questions answered via OpenAI). Each maps cleanly to a Laravel responsibility.

Stage 1: Document Upload and Persistence

Schema::create('documents', function (Blueprint $table) {
    $table->id();
    $table->string('original_name');
    $table->string('path');
    $table->longText('content')->nullable();
    $table->timestamps();
});

At upload time, do not call OpenAI:

public function store(Request $request): JsonResponse
{
    $file = $request->file('document');
    $path = $file->store('documents');

    $document = Document::create([
        'original_name' => $file->getClientOriginalName(),
        'path'          => $path,
    ]);

    ExtractDocumentContent::dispatch($document);

    return response()->json(['id' => $document->id]);
}

Dispatch a job. Extraction may be slow. Upload requests should return quickly. Extraction failures should be isolated — not cascade into upload failures.

Stage 2: Text Extraction as a Background Job

class ExtractDocumentContent implements ShouldQueue
{
    public function handle(): void
    {
        $text = $this->extractText(
            Storage::get($this->document->path)
        );

        $this->document->update([
            'content' => $this->normalize($text),
        ]);
    }
}

No AI calls, no tokens consumed. Failures are deterministic. You want extraction bugs to be boring.

Stage 3: Chunking Document Content

Here’s where many implementations fail. The naive approach — stuffing the entire document into a prompt — works briefly, then collapses under token limits and cost pressure.

Split the document into chunks based on size:

function chunkText(string $text, int $size = 800): array
{
    $chunks = [];
    $length = mb_strlen($text);
    $offset = 0;

    while ($offset < $length) {
        $chunks[] = mb_substr($text, $offset, $size);
        $offset  += $size;
    }

    return $chunks;
}

⚠️ Gotcha — str_split() Will Corrupt Multibyte Text

The PHP standard library’s str_split($text, $size) splits by bytes, not characters. Any document with UTF-8 content — French accents, German umlauts, Arabic, CJK — will produce chunks with broken characters at boundaries. Always use mb_substr() and mb_strlen() for text chunking. This is a silent production bug that only surfaces with real-world content.

Stage 4: Answering Questions

public function answerQuestion(Document $document, string $question): string
{
    $chunks = chunkText($document->content);
    $prompt = $this->buildPrompt($chunks, $question);

    return $this->openAi->generateText($prompt);
}

Prompt Construction

protected function buildPrompt(array $chunks, string $question): string
{
    $context = implode("\n\n", $chunks);

    return <<<PROMPT
You are answering questions based ONLY on the following document content.
If the answer cannot be found in the document, respond with:
"I don't know based on the provided document."

Document:
{$context}

Question:
{$question}
PROMPT;
}

This prompt constrains hallucination, defines a fallback response, and makes incorrect answers detectable. The goal is not perfection — it’s bounded failure. You’ll eventually outgrow this approach, but it fails gracefully, which makes it an excellent foundation.

Handling Multi-Turn Follow-Ups

Users rarely ask one question. Instead of re-sending the entire document every time, summarize prior context, append follow-ups as messages, and re-inject only relevant chunks. The conversation schema from Pattern 3 handles this directly.

Observability

Every OpenAI call in this feature should log document ID, token count, model, and latency. Not for analytics — for debugging. When a user reports “the AI gave a wrong answer,” you need the exact inputs that produced it. Without that log, you’re guessing.

Production Concerns

By the time an OpenAI integration reaches production, the question shifts from “Can we generate an answer?” to “How often does this fail? How expensive is it over time? What happens under load?”

AI integrations are probabilistic, rate-limited, and cost-based. Treating them like traditional REST APIs produces brittle systems.

Rate Limiting as a System Behaviour

OpenAI enforces rate limits at the account and model level. Most developers respond to sporadic 429s by adding retries. That’s the wrong mental model.

Rate limits are not an error condition. They’re a signal that your system’s demand exceeds its contract.

Centralizing Rate Awareness

Your OpenAI service layer should be the only place where rate-limit responses are interpreted:

if ($response->status() === 429) {
    throw new OpenAiRateLimited(
        retryAfter: (int) $response->header('Retry-After', 30)
    );
}

This exception should not be caught silently. Its consumers — controllers, jobs — must decide what to do next.

Different Strategies for Different Contexts

User-facing requests should fail fast with a clear message or degraded experience. Background jobs can be delayed and rescheduled:

public function handle(): void
{
    try {
        // OpenAI call
    } catch (OpenAiRateLimited $e) {
        $this->release($e->retryAfter ?? 30);
    }
}

This aligns retries with system capacity instead of guessing at backoff timing.

Cost as an Operational Metric

Token usage is not an abstract billing detail. It’s a runtime characteristic of your application. Two requests that look identical in code can differ by orders of magnitude in cost depending on prompt length, document size, conversation history, and model choice.

Recording Token Usage

Every OpenAI response includes token metadata. Capture it:

$usage = data_get($response->json(), 'usage');

OpenAiUsage::create([
    'model'             => $model,
    'prompt_tokens'     => $usage['prompt_tokens']     ?? 0,
    'completion_tokens' => $usage['completion_tokens'] ?? 0,
    'total_tokens'      => $usage['total_tokens']      ?? 0,
]);

This data is not for dashboards at first. It’s for answering questions like: Why did costs spike yesterday? Which feature is expensive? Which users generate the most load? You cannot optimize what you do not measure.

Cost-Aware Design Decisions

Once usage is visible, architectural decisions become concrete: summarizing older conversation turns to stay within token budgets, truncating document context for cheaper models, and routing simple tasks to gpt-4o-mini instead of gpt-4o. These are the difference between sustainable and runaway costs.

Caching AI Outputs (Carefully)

Caching AI responses is tempting. It’s also dangerous if done naively.

What Is Safe to Cache

Caching works well for deterministic prompts, non-user-specific inputs, and expensive but repeatable operations — document summaries, extracted metadata, classification labels:

Cache::remember(
    "doc:{$document->id}:summary",
    now()->addDay(),
    fn () => $openAi->summarize($document->content)
);

What Should Not Be Cached

Avoid caching conversational replies, user-personalized outputs, or anything derived from mutable state. If you cannot confidently answer “When should this be invalidated?”, don’t cache it.

Senior Dev Tip: Use Redis for AI output caching, not the file driver. Redis gives you atomic expiry, explicit invalidation by key pattern, and the ability to inspect what’s cached without reading from disk. File-based caches become unmanageable quickly when you’re caching per-document, per-user outputs.

Testing OpenAI Integrations Without Lying

This is where many teams get stuck. Mocking the OpenAI client to return a fixed string proves nothing useful. You need two distinct kinds of tests.

1. Contract Tests (Fast, Deterministic)

These verify that your service layer calls OpenAI correctly, that failures are handled, and that parsing logic works. Mocking is appropriate here:

Http::fake([
    'api.openai.com/*' => Http::response([
        'choices' => [
            ['message' => ['content' => 'Test output']],
        ],
        'usage' => [
            'prompt_tokens'     => 10,
            'completion_tokens' => 5,
            'total_tokens'      => 15,
        ],
    ]),
]);

You’re testing plumbing, not intelligence.

2. Behavioural Tests (Slow, Real)

These hit the real OpenAI API, use real prompts, and validate constraints — not specific wording:

$response = $service->answerQuestion($doc, 'What is the refund policy?');

$this->assertStringContainsString('refund', strtolower($response));

These tests should run infrequently, be clearly labelled, use dedicated API keys, and have cost limits. They catch failures no mock ever will.

Dealing With Non-Determinism

AI output is not stable across runs. Tests must reflect that. Assert structure, presence or absence, and refusal behaviour. If a model is instructed to say “I don’t know” under certain conditions, test for that branch explicitly. You’re testing behavioural boundaries, not string values.

Deployment Considerations

Increase PHP and proxy timeouts for streaming routes. Ensure queue workers can handle long-running jobs. If you haven’t locked down your production environment yet, deploy Laravel to production covers the infrastructure baseline these integrations depend on. Separate API keys across local, staging, and production — never reuse them. Wrap major prompt changes behind feature flags; a prompt change can alter behaviour as dramatically as a code change.

At minimum, you should be able to answer: Which OpenAI calls failed today? Which features are most expensive? Which prompts changed recently? This doesn’t require a full observability stack. It requires discipline: structured logs, request IDs, and consistent metadata.

Advanced Patterns

Your Laravel application now has a clean OpenAI service layer with request/response, streaming, memory, queues, cost tracking, rate limiting, testability, and observability. Let’s extend it.

Livewire Integration: Real-Time AI Without a Frontend Framework

Not every Laravel team wants a JavaScript SPA. Livewire allows server-driven interactivity that pairs naturally with AI workflows.

Livewire is request-driven, not socket-driven. Each interaction is an HTTP round trip, and long-running calls block the request lifecycle unless handled intentionally. For a complete working implementation, see Laravel Livewire + Claude Integration — the architectural patterns are identical for OpenAI.

Basic Livewire AI Component

use Livewire\Component;

class AiAssistant extends Component
{
    public string $prompt   = '';
    public string $response = '';
    public bool   $loading  = false;

    public function generate(OpenAiService $openAi): void
    {
        $this->loading  = true;
        $this->response = $openAi->generateText($this->prompt);
        $this->loading  = false;
    }

    public function render()
    {
        return view('livewire.ai-assistant');
    }
}

This works immediately but it’s blocking. If generation takes 10 seconds, the request takes 10 seconds. For short responses, acceptable. For longer outputs, you have three realistic strategies:

Polling: Trigger generation in a queued job, persist partial output, and poll from Livewire until complete. Simple and resilient.
Hybrid Route for Streaming: Livewire initiates the request, a separate streaming endpoint handles chunked output, and a thin JavaScript layer listens and updates the DOM.
Accept Blocking: For short, predictable outputs, it may be entirely acceptable.

The decision depends on latency tolerance, not preference.

Inertia + Vue: SPA-Style AI Interfaces

If your application already uses Inertia, you can leverage SSE natively on the client side.

Laravel Route:

Route::get('/ai/stream', function (Request $request, OpenAiService $openAi) {
    return response()->stream(function () use ($request, $openAi) {
        $openAi->streamText(
            $request->query('prompt'),
            function ($chunk) {
                echo "data: " . json_encode($chunk) . "\n\n";
                ob_flush();
                flush();
            }
        );
    }, 200, [
        'Content-Type'      => 'text/event-stream',
        'Cache-Control'     => 'no-cache',
        'X-Accel-Buffering' => 'no',
    ]);
});

Vue Client:

const eventSource = new EventSource(`/ai/stream?prompt=${encodeURIComponent(prompt)}`);

eventSource.onmessage = (event) => {
    response.value += JSON.parse(event.data);
};

Streaming is an architectural decision, not a cosmetic enhancement. It changes how users perceive your application’s responsiveness.

Retrieval-Augmented Generation (RAG) Fundamentals

Eventually, chunking entire documents into prompts stops scaling. You need selective context retrieval.

RAG solves this by converting text into embeddings, storing them in a vector store, and injecting only semantically relevant chunks into the prompt at query time — rather than the full document.

Step 1: Generating Embeddings

$response = $this->client->post('/embeddings', [
    'model' => 'text-embedding-3-small',
    'input' => $textChunk,
]);

$vector = data_get($response->json(), 'data.0.embedding');

Step 2: Storing Vectors

At small scale, PostgreSQL with the pgvector extension is sufficient. If you need a deeper introduction to how embeddings actually work before choosing your storage strategy, Embeddings and Vector Databases: The Foundation of Semantic AI Systems covers the concepts and storage trade-offs in detail.

Step 3: Retrieving Relevant Chunks

At query time, convert the question to an embedding, perform a cosine similarity search, and retrieve the top N chunks. Only those chunks enter the OpenAI prompt. This reduces token usage, latency, and hallucination. RAG is not magic — it’s controlled context injection.

Multi-Model Strategies

Assuming one model should do everything is one of the most expensive architectural mistakes in LLM-backed applications. Large reasoning models are slow and costly. Smaller models are fast and cheap. Embedding models serve an entirely different purpose.

Intent-Based Model Routing

$model = match ($taskType) {
    'summarization'    => 'gpt-4o-mini',
    'complex_reasoning' => 'gpt-4o',
    'embedding'        => 'text-embedding-3-small',
    default            => 'gpt-4o-mini',
};

This prevents overpaying for simple tasks. A document classification job does not need the same model as a multi-step reasoning pipeline.

Fallback Strategies

You can also implement primary → fallback model routing, high-accuracy → fast mode switching, and user-tier-based model selection. These are business decisions with direct cost and SLA implications, not technical hacks.

Senior Dev Tip: If you want to go further and treat your AI layer as a genuinely provider-agnostic abstraction — with tool calling, multi-model orchestration, and structured outputs built in — take a look at Building Agentic Laravel Apps with Prism PHP. Prism handles the abstraction layer so you’re not maintaining it yourself.

Where This Leaves You

Your Laravel + OpenAI system now supports real-time interfaces, retrieval-based document reasoning, multi-model routing, and a structured architecture ready to scale.

The service boundary protects you from API changes, model shifts, and provider swaps. OpenAI becomes infrastructure — replaceable, measurable, and contained. That’s the outcome worth building toward.

Next Steps

Ready to build with real conversation memory and streaming? Building a Claude API Chatbot in Laravel demonstrates both patterns in a complete, working implementation.

Hit rate limits in production? The queue-based throttling and backpressure strategies in Laravel Claude API Integration Guide apply directly to OpenAI workloads.

Want a server-driven real-time UI without JavaScript frameworks? Laravel Livewire + Claude Integration shows how to build AI features that stay in PHP.

Questions? Ask in our Developer Q&A or reach out directly.

Official References:

RAG-less Architecture in Laravel: Long-Context Caching with Gemini

Dewald Hugo — Thu, 23 Apr 2026 03:01:13 +0000

For the past few years, RAG has been presented as the only serious approach to grounding AI on large private datasets. Build a vector database, chunk your documents, embed the chunks, retrieve the closest matches, inject them into the prompt. Repeat for every query. It works, and for certain use cases it is genuinely the right call.

But for others — particularly repeated, deep-analysis tasks on the same corpus — RAG is an expensive workaround for a context window problem that the models have largely outgrown.

Google’s Gemini 2.5 Flash and 2.5 Pro both ship with a 1-million-token context window. That is approximately 750,000 words, or a mid-sized Laravel monolith with room to spare. More importantly, the Gemini API now supports Explicit Context Caching: you upload that corpus once, pay full input price once, and reference the cached context on every subsequent query at up to 75% off the standard input rate. For workloads that involve many questions against the same codebase, document set, or knowledge base, this is a fundamentally different cost model — and a fundamentally different architecture.

This guide walks you through building a GeminiContextCacheService in Laravel 11/12, binding it to the Service Container, and building a production-ready audit endpoint around it. We will cover the full cache lifecycle, including the storage cost trap that most tutorials skip entirely.

RAG vs. Long-Context Caching with Gemini: The Architectural Trade-Off

Before touching code, we need to be honest about what each approach actually is. Neither is universally superior.

RAG trades context completeness for cost predictability. You pay only for the retrieved chunks, not the full corpus — which is excellent when your dataset is enormous (hundreds of GB) and any single query only needs a narrow slice of it. The problem is that retrieval is fundamentally lossy. A semantic search might pull the right controller method and miss the BaseModel that overrides save(). It will find the public interface and skip the private helper that silently transforms the input. That retrieval gap is not a bug you can patch; it is the architecture.

Long-context caching with Gemini trades retrieval precision for corpus completeness. The model sees everything. There is no retrieval step, no chunking pipeline, no embedding model to maintain. The trade-off is cost structure: the first query is expensive (you pay to ingest the full context), and you carry a per-hour storage fee while the cache lives. Beyond the break-even point — which arrives faster than you expect — repeated queries cost significantly less than the RAG equivalent.

The Cost Curve: Why Caching Is the Financial Lever

Let us put real numbers on this. Using Gemini 2.5 Flash, which offers the best price-to-context ratio for this workload:

Standard input rate (long context >128K tokens): $0.30/MTok
Cached input rate: $0.03/MTok (90% reduction at Flash long-context tier)
Cache storage: billed per hour per MTok while the cache lives

For a 500K-token codebase audit with 20 questions:

Without caching: Each query sends 500K tokens to the API. That is $0.15 per question × 20 = $3.00 total.

With caching: Initial cache creation: $0.15 (one-time). Each subsequent query references the cache at $0.015. Twenty queries: $0.15 + ($0.015 × 19) = $0.435 total — plus a small storage fee for however long the cache lives.

The break-even is query 2. Every question after that is 90% cheaper than the RAG alternative that re-sends the same context.

When RAG Still Wins

We should be direct about this before anyone rips out a working vector pipeline.

If your corpus exceeds 1M tokens — a multi-year ticket archive, a large documentation set, a multi-repo monorepo — the long-context window is not a solution. You physically cannot fit the data. RAG is the architecture. Our existing guide on Laravel Embeddings, Vector Databases, and RAG covers pgvector and the full embedding pipeline if that is your situation.

Similarly, if your corpus changes frequently — user-generated content, live database records — cache invalidation becomes expensive. Re-creating a 1M-token cache on every data change defeats the cost benefit.

Long-context caching targets a specific sweet spot: a corpus that is large enough to matter (above RAG’s retrieval quality threshold), small enough to fit in 1M tokens, and stable enough that you will run many queries before the next update. B2B codebase auditing, legal document analysis, product catalog deep-dives — these are the use cases.

Implementation: The Stateful Expert Pattern

Prerequisites

Add your Gemini API key to .env:

GEMINI_API_KEY=your_key_here
GEMINI_MODEL=gemini-2.5-flash
GEMINI_CACHE_TTL=7200

Then register the configuration values in config/services.php:

'gemini' => [
    'api_key'   => env('GEMINI_API_KEY'),
    'base_url'  => 'https://generativelanguage.googleapis.com/v1beta',
    'model'     => env('GEMINI_MODEL', 'gemini-2.5-flash'),
    'cache_ttl' => (int) env('GEMINI_CACHE_TTL', 7200),
],

The minimum content size for Gemini context caching is 32,768 tokens — roughly 25,000 words. You cannot cache a handful of files. You need to be pushing a full module, a full codebase, or a substantial document set. Keep that constraint in mind when designing your corpus preparation step.

If you have not yet set up your Gemini client or worked through model selection on your Laravel stack, our Integrating Gemini into the Laravel AI SDK guide covers the foundational setup decisions before you start hitting the API directly.

Step 1: The Context Caching Service

This service owns two responsibilities: cache lifecycle (create, retrieve, delete) and query dispatch. We keep them in one class because the cache name is required for query — splitting them would force unnecessary dependency injection.

<?php

namespace App\Services;

use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Log;
use RuntimeException;

class GeminiContextCacheService
{
    private string $apiKey;
    private string $baseUrl;
    private string $model;
    private int    $ttl;

    public function __construct()
    {
        $this->apiKey  = config('services.gemini.api_key');
        $this->baseUrl = config('services.gemini.base_url');
        $this->model   = config('services.gemini.model');
        $this->ttl     = config('services.gemini.cache_ttl');
    }

    /**
     * Return an existing live cache name from Redis, or create a new one.
     * The cache key is your application's identifier (e.g. "audit:project-42").
     * The Gemini cache name is what the API issues back — it looks like
     * "cachedContents/abc123xyz" and is what you reference in generation requests.
     */
    public function getOrCreateCache(
        string $cacheKey,
        string $systemPrompt,
        array  $documents
    ): string {
        $geminiCacheName = Cache::get("gemini_cache:{$cacheKey}");

        if ($geminiCacheName) {
            Log::debug('Gemini cache hit', ['key' => $cacheKey]);
            return $geminiCacheName;
        }

        return $this->createCache($cacheKey, $systemPrompt, $documents);
    }

    private function createCache(
        string $cacheKey,
        string $systemPrompt,
        array  $documents
    ): string {
        // Each document becomes a separate part in the cached content
        $parts = array_map(
            fn(string $doc) => ['text' => $doc],
            $documents
        );

        $response = Http::withHeaders([
            'x-goog-api-key' => $this->apiKey,
            'Content-Type'   => 'application/json',
        ])->post("{$this->baseUrl}/cachedContents", [
            'model'            => "models/{$this->model}",
            'displayName'      => $cacheKey,
            'systemInstruction' => [
                'parts' => [['text' => $systemPrompt]],
            ],
            'contents' => [
                ['role' => 'user', 'parts' => $parts],
            ],
            'ttl' => "{$this->ttl}s",
        ]);

        if ($response->failed()) {
            Log::error('Gemini cache creation failed', [
                'status' => $response->status(),
                'body'   => $response->json(),
                'key'    => $cacheKey,
            ]);

            throw new RuntimeException(
                'Failed to create Gemini context cache: ' . $response->body()
            );
        }

        $cacheName = $response->json('name');

        // Store with a 60-second buffer before the Gemini TTL expires
        // so we never reference a cache the API has already deleted
        Cache::put(
            "gemini_cache:{$cacheKey}",
            $cacheName,
            $this->ttl - 60
        );

        Log::info('Gemini context cache created', [
            'name' => $cacheName,
            'key'  => $cacheKey,
        ]);

        return $cacheName;
    }

    /**
     * Run a user question against an existing cached context.
     */
    public function query(string $cacheName, string $question): string
    {
        $response = Http::withHeaders([
            'x-goog-api-key' => $this->apiKey,
            'Content-Type'   => 'application/json',
        ])->post("{$this->baseUrl}/models/{$this->model}:generateContent", [
            'cachedContent' => $cacheName,
            'contents'      => [
                ['role' => 'user', 'parts' => [['text' => $question]]],
            ],
        ]);

        if ($response->failed()) {
            if ($response->status() === 429) {
                throw new RuntimeException(
                    'Gemini rate limit exceeded. Implement exponential backoff before retrying.'
                );
            }

            Log::error('Gemini query failed', [
                'status' => $response->status(),
                'cache'  => $cacheName,
            ]);

            throw new RuntimeException('Gemini query failed: ' . $response->body());
        }

        $text = $response->json('candidates.0.content.parts.0.text');

        if ($text === null) {
            throw new RuntimeException(
                'Unexpected Gemini response structure: ' . $response->body()
            );
        }

        return $text;
    }

    /**
     * Explicitly delete a cache when the session ends.
     * Do not leave caches running — storage costs accumulate per hour.
     */
    public function deleteCache(string $cacheKey): void
    {
        $geminiCacheName = Cache::get("gemini_cache:{$cacheKey}");

        if (! $geminiCacheName) {
            return;
        }

        Http::withHeaders(['x-goog-api-key' => $this->apiKey])
            ->delete("{$this->baseUrl}/{$geminiCacheName}");

        Cache::forget("gemini_cache:{$cacheKey}");

        Log::info('Gemini context cache deleted', ['key' => $cacheKey]);
    }
}

[Architect’s Note] The 60-second TTL buffer on the Redis key is not optional. If your Laravel cache expiry aligns exactly with Gemini’s TTL, a query that arrives in the final seconds will find a valid Redis key but hit a deleted Gemini cache. The API returns a 404, your query() method throws, and your user gets a 503. Build the buffer in. If you are concerned about token governance and want to log every token consumed by cached vs. standard queries separately, our Laravel AI Middleware: Token Tracking & Rate Limiting guide covers the instrumentation layer you would wrap around this service.

Step 2: Binding to the Service Container

Register the service as a singleton in AppServiceProvider. Because the service reads from config in its constructor, a singleton is correct here — we want one instance per request cycle, not one per injection.

<?php

namespace App\Providers;

use App\Services\GeminiContextCacheService;
use Illuminate\Support\ServiceProvider;

class AppServiceProvider extends ServiceProvider
{
    public function register(): void
    {
        $this->app->singleton(GeminiContextCacheService::class);
    }
}

Laravel’s Service Container handles the rest. Any controller, job, or command that type-hints GeminiContextCacheService receives the same instance within a given request.

Step 3: The Audit Controller

<?php

namespace App\Http\Controllers;

use App\Services\GeminiContextCacheService;
use Illuminate\Http\JsonResponse;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Log;
use RuntimeException;

class CodeAuditController extends Controller
{
    public function __construct(
        private readonly GeminiContextCacheService $gemini
    ) {}

    public function audit(Request $request): JsonResponse
    {
        $validated = $request->validate([
            'project_id' => ['required', 'string', 'max:100'],
            'question'   => ['required', 'string', 'max:2000'],
        ]);

        try {
            $documents = $this->loadProjectDocuments($validated['project_id']);

            $systemPrompt  = 'You are a senior Laravel architect reviewing a production codebase. '
                . 'When answering questions, cite specific files and line numbers. '
                . 'Flag security risks, legacy patterns, and architectural concerns explicitly.';

            $cacheName = $this->gemini->getOrCreateCache(
                cacheKey:     "audit:{$validated['project_id']}",
                systemPrompt: $systemPrompt,
                documents:    $documents,
            );

            $answer = $this->gemini->query($cacheName, $validated['question']);

            return response()->json(['answer' => $answer]);

        } catch (RuntimeException $e) {
            Log::error('Code audit failed', [
                'project_id' => $validated['project_id'],
                'error'      => $e->getMessage(),
            ]);

            return response()->json(
                ['error' => 'Audit service temporarily unavailable.'],
                503
            );
        }
    }

    private function loadProjectDocuments(string $projectId): array
    {
        $path = storage_path("app/projects/{$projectId}/codebase.txt");

        if (! file_exists($path)) {
            throw new RuntimeException(
                "Codebase not found for project: {$projectId}"
            );
        }

        // Return as array — each element becomes a separate Part in the cached content.
        // Split large codebases into logical chunks (per-module) if approaching 1M tokens.
        return [file_get_contents($path)];
    }
}

Register the route with Sanctum auth and a conservative throttle. Twenty questions per minute is generous for a codebase audit endpoint — adjust to match your billing ceiling.

// routes/api.php
use App\Http\Controllers\CodeAuditController;

Route::middleware(['auth:sanctum', 'throttle:20,1'])
    ->post('/audit', [CodeAuditController::class, 'audit']);

Securing this endpoint properly is not optional — an unauthenticated audit endpoint is a direct vector for API cost abuse. Our Laravel Sanctum API Authentication guide covers token scoping and rate-limit configuration if you need to tighten this down further.

Cache Lifecycle Management

This is the section most tutorials omit. Skip it and you will find $800 in unexpected Gemini storage charges at the end of the month.

Gemini charges per MTok per hour for every active cache. The pricing varies by model — check Google’s official API pricing page for current rates. The pattern you want is: create the cache when the audit session starts, delete it when the session ends, and run a scheduled cleanup job to catch anything that leaked through.

The cleanup command:

<?php

namespace App\Console\Commands;

use Carbon\Carbon;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Log;

class CleanExpiredGeminiCaches extends Command
{
    protected $signature   = 'gemini:clean-caches';
    protected $description = 'Delete expired or stale Gemini context caches';

    public function handle(): int
    {
        $apiKey  = config('services.gemini.api_key');
        $baseUrl = config('services.gemini.base_url');

        $response = Http::withHeaders(['x-goog-api-key' => $apiKey])
            ->get("{$baseUrl}/cachedContents");

        if ($response->failed()) {
            $this->error('Failed to list caches: ' . $response->body());
            return self::FAILURE;
        }

        $caches = $response->json('cachedContents', []);

        foreach ($caches as $cache) {
            $expireTime = Carbon::parse($cache['expireTime']);

            if ($expireTime->isPast()) {
                Http::withHeaders(['x-goog-api-key' => $apiKey])
                    ->delete("{$baseUrl}/{$cache['name']}");

                Log::info('Deleted stale Gemini cache', ['name' => $cache['name']]);
                $this->info("Deleted: {$cache['name']}");
            }
        }

        return self::SUCCESS;
    }
}

Schedule it in bootstrap/app.php:

->withSchedule(function (Schedule $schedule) {
    $schedule->command('gemini:clean-caches')->hourly();
})

[Edge Case Alert] If a queue worker crashes mid-audit before deleteCache() is called, the Gemini cache lives until its TTL expires and storage charges accumulate. The scheduled cleanup command is your safety net, not a replacement for explicit deletion. Consider also dispatching a DeleteGeminiCacheJob via defer() at the end of each audit session as a belt-and-suspenders measure.

Use Case: The Persistent Auditor in Practice

Here is where the architecture earns its keep. You are auditing a legacy Laravel 6 application for a client. The codebase is 400K tokens — controllers, models, migrations, the lot.

The RAG approach: You ask about UserController::store(). The retrieval pulls three relevant chunks. The answer looks correct. But the client’s BaseModel has an overridden save() method that intercepts every Eloquent write and transforms certain fields. RAG did not retrieve BaseModel — it was not semantically close enough to the query. The AI gives you an answer that describes the surface behaviour and misses the hack underneath. The audit is wrong.

The cached-context approach: You upload the full codebase. When you ask about UserController::store(), Gemini has already read BaseModel. It answers that the apparent save logic is intercepted downstream, identifies the override, and flags it as a maintenance risk. The context is complete. The answer is complete.

This is what we mean by Architectural Integrity. RAG optimises for retrieval cost. Long-context caching optimises for answer correctness. For an audit product where a wrong answer has professional liability implications, correctness wins.

The cost math, using our earlier figures: 20 audit questions on a 500K-token codebase costs $3.00 without caching and approximately $0.44 with caching — plus storage for however long the session runs. For a B2B audit priced at $200 per report, the infrastructure cost is negligible either way. But the quality difference is not.

For production agentic pipelines — where the “questions” are not human prompts but tool calls from an orchestrated agent — the cost advantage compounds quickly. Our guide on Hardening Laravel Agentic Workflows: Schema Validation Against LLM Hallucinations covers the validation layer you will want around those agent outputs, regardless of which AI architecture you choose.

Preparing Your Corpus for the Cache

The code examples above load a pre-processed codebase.txt file. In production, you need a pipeline that generates this file from your actual source. A simple Artisan command handles it:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\File;
use Symfony\Component\Finder\Finder;

class PrepareCodebaseCorpus extends Command
{
    protected $signature   = 'corpus:prepare {project_id} {path}';
    protected $description = 'Concatenate a project codebase into a single corpus file';

    public function handle(): int
    {
        $projectId = $this->argument('project_id');
        $path      = $this->argument('path');

        $finder = (new Finder())
            ->files()
            ->in($path)
            ->name(['*.php', '*.json', '*.yaml', '*.env.example'])
            ->notPath(['vendor', 'node_modules', 'storage', '.git'])
            ->sortByName();

        $corpus  = '';
        $counter = 0;

        foreach ($finder as $file) {
            $relativePath = str_replace($path . '/', '', $file->getRealPath());
            $corpus .= "\n\n// FILE: {$relativePath}\n";
            $corpus .= $file->getContents();
            $counter++;
        }

        $outputDir  = storage_path("app/projects/{$projectId}");
        $outputPath = $outputDir . '/codebase.txt';

        File::ensureDirectoryExists($outputDir);
        File::put($outputPath, $corpus);

        $sizeKb = round(strlen($corpus) / 1024, 1);
        $this->info("Corpus prepared: {$counter} files, {$sizeKb} KB → {$outputPath}");

        return self::SUCCESS;
    }
}

Run it before starting an audit session: php artisan corpus:prepare project-42 /path/to/legacy-app. The file headers (// FILE: app/Models/BaseModel.php) give Gemini the file path context it needs to cite locations in its answers.

[Efficiency Gain] Strip comments and docblocks from PHP files before creating the corpus. A typical Laravel application loses 20–30% of its token count after comment stripping, which can push a borderline 1.2M-token codebase under the 1M limit and save meaningfully on cache creation costs. PHP’s token_get_all() makes this straightforward; a simple token filter removes T_COMMENT and T_DOC_COMMENT tokens before concatenation.

Conclusion: Context Is the New Database

RAG will not disappear. For certain corpus shapes — enormous, dynamic, multi-tenant — it remains the correct architecture. But it has been treated as the default answer for too long, applied reflexively to problems that long-context models can now solve more cleanly and more accurately.

Explicit context caching in Gemini changes the trade-off. The vector database, the chunking strategy, the embedding model, the retrieval tuning — for a well-bounded corpus, all of that overhead collapses into a single cached upload and a TTL you manage in Redis. The model sees the whole codebase. The answers reflect the whole codebase.

The architecture we have built here — GeminiContextCacheService bound as a singleton, TTL-buffered cache keys in Redis, explicit lifecycle management, a corpus preparation command — is production-ready. Wire up your Sanctum-protected audit endpoint, run the corpus preparation pipeline, and you have a stateful expert that knows your codebase better than any retrieval pipeline can.

For official context caching documentation and current pricing, refer directly to Google’s Gemini API Context Caching guide — pricing has shifted frequently in 2026 and the source of truth is Google’s pricing page, not any third-party aggregator.

Integrating Gemini into the Laravel AI SDK

Dewald Hugo — Thu, 16 Apr 2026 13:45:05 +0000

If your Laravel application already speaks to OpenAI or Anthropic through the first-party laravel/ai SDK, adding Gemini feels like it should be a five-minute job. Set a new API key. Point to a new provider. Deploy. Done. That assumption is mostly correct, but the gap between “mostly” and “done” is where production systems fall apart. This guide closes that gap.

We will cover the full Laravel Gemini integration path: the OpenAI-compatible soft entry that lets you validate the connection in under ten minutes, the native Gemini provider configuration that actually belongs in a long-lived codebase, and a GeminiManager service pattern that gives you model failover, rate-limit awareness, and a response contract your Blade templates never have to think about.

By the end, Gemini will be a first-class citizen in your AI stack, not an afterthought bolted to a controller.

Why Add Gemini At All?

Fair question. If you are already running claude-sonnet-4-6 or gpt-4o in production, the case for a third provider needs to be more compelling than “Google exists.” Here is the honest pitch.

Gemini 2.5 Flash is fast, cheap, and ships with a one-million-token context window as standard. For high-throughput tasks—document summarisation, classification pipelines, RAG over large corpora—it undercuts the competition on cost without giving up much capability. Gemini 2.5 Pro sits at the other end: serious reasoning, multimodal by default, and Vertex AI integration if you are already inside the Google Cloud ecosystem.

The second reason is resilience. Single-provider AI architectures have a hard failure mode. When OpenAI rate-limits your 2 AM batch job, the entire feature goes dark. Gemini as a fallback provider changes that calculation. We are not talking about a hot standby you never test; we are talking about a properly designed failover that routes intelligently.

The third reason is the laravel/ai SDK itself. As covered in detail in What Laravel 13 Actually Changes for AI Development, the SDK was designed from the start to be provider-agnostic. Gemini is a first-class supported provider. The architecture already expects you to run multiple models.

Route 1: The OpenAI-Compatible Endpoint (The Soft Entry)

Before committing to the native integration, you can validate that Gemini works for your use case using Google’s OpenAI-compatible endpoint. This is genuinely useful for a quick proof-of-concept, or when you are migrating from an existing openai-php/client setup rather than the laravel/ai SDK.

Google maintains an endpoint at https://generativelanguage.googleapis.com/v1beta/openai/ that accepts Chat Completions API requests in the standard OpenAI format. The swap is mechanical: change the base URL, swap the API key, and change the model string.

Start with your .env:

GEMINI_API_KEY=your-google-ai-studio-key
GEMINI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/
GEMINI_MODEL=gemini-2.5-flash

If you are on an existing OpenAI configuration in config/ai.php, you can add a second connection:

// config/ai.php
'connections' => [
    'openai' => [
        'driver'  => 'openai',
        'api_key' => env('OPENAI_API_KEY'),
    ],

    'gemini_compat' => [
        'driver'    => 'openai',           // still using the OpenAI driver
        'api_key'   => env('GEMINI_API_KEY'),
        'base_url'  => env('GEMINI_BASE_URL'),
    ],
],

You can now call Gemini through any code that already uses the openai driver, just by swapping the connection name:

use Illuminate\Support\Facades\AI;

$response = AI::connection('gemini_compat')->text(
    model: 'gemini-2.5-flash',
    messages: [
        ['role' => 'user', 'content' => $prompt],
    ]
);

This works well enough for a smoke test. Resist the urge to ship it as your production integration.

[Production Pitfall] The OpenAI-compatible endpoint has real coverage gaps. Gemini-native features—grounding via Google Search, the thinking budget parameter, native multimodal tool use, and fine-grained safety settings—are not available through the compat layer. If your feature depends on any of those, the compat approach will silently drop them and return a degraded response with no error. You will not find out until QA, or worse, production.

Route 2: The Native Gemini Provider (The Right Way)

The laravel/ai SDK ships with a first-party Gemini driver. Use it. The compat approach is a migration crutch; the native driver is what belongs in your config/ai.php long-term.

Install the SDK if you have not already:

composer require laravel/ai
php artisan vendor:publish --provider="Laravel\Ai\AiServiceProvider"
php artisan migrate

Then register the native Gemini connection:

// config/ai.php
'connections' => [
    'openai' => [
        'driver'  => 'openai',
        'api_key' => env('OPENAI_API_KEY'),
    ],

    'anthropic' => [
        'driver'  => 'anthropic',
        'api_key' => env('ANTHROPIC_API_KEY'),
    ],

    'gemini' => [
        'driver'  => 'gemini',
        'api_key' => env('GEMINI_API_KEY'),
    ],
],

'default' => env('AI_DEFAULT_CONNECTION', 'openai'),

Your .env stays clean:

GEMINI_API_KEY=your-google-ai-studio-key

A basic request through the native driver looks identical to your existing OpenAI calls:

use Illuminate\Support\Facades\AI;

$response = AI::connection('gemini')->text(
    model: 'gemini-2.5-flash',
    messages: [
        ['role' => 'user', 'content' => $prompt],
    ]
);

return $response->text;

That’s the surface. Now let’s build something that actually runs in production.

Model Selection: Which Gemini Model?

Not all Gemini models are created equal, and Google’s naming conventions have become genuinely confusing over the last year. Here is the practical breakdown for production use.

gemini-2.5-flash is your default. It hit general availability in June 2025, it is fast, it is cost-efficient, and it handles up to one million tokens of context. For most text generation, summarisation, and classification tasks, it is the correct choice.

gemini-2.5-pro is for heavy lifting. Complex reasoning chains, code generation over large codebases, long-form synthesis tasks. It is meaningfully more expensive than Flash and slower under load, so do not reach for it by default.

[Edge Case Alert] Do not use gemini-2.0-flash in new integrations. Google has announced that Gemini 2.0 Flash and Flash-Lite models will be shut down on June 1, 2026. Any production system pointing at those model strings will start throwing 404 errors at that date with no warning beyond the deprecation announcement. If you have any existing references to gemini-2.0-flash in your codebase, change them now.

For preview models in the Gemini 3.x family, treat them exactly as you would treat preview Claude models: useful for internal testing, not for customer-facing production paths until they have a stable designation.

The GeminiManager Pattern

A single AI connection configured in config/ai.php is fine for prototyping. Production systems need something more deliberate: model selection logic, rate-limit handling, failover between providers, and a response contract that the rest of your application can depend on regardless of which model actually ran.

The GeminiManager service class is that layer. It sits between the laravel/ai facade and your application code, owns the failover logic, and returns a typed response object your Blade templates or API controllers can render without knowing which provider answered the request.

Start by creating the service in app/Services/AI/:

<?php

namespace App\Services\AI;

use Illuminate\Support\Facades\AI;
use Illuminate\Support\Facades\Log;
use App\DTOs\AiResponse;

class GeminiManager
{
    private array $modelPriority = [
        'gemini-2.5-flash',
        'gemini-2.5-pro',
    ];

    private string $fallbackConnection = 'openai';
    private string $fallbackModel      = 'gpt-4o-mini';

    public function generate(string $prompt, array $options = []): AiResponse
    {
        foreach ($this->modelPriority as $model) {
            try {
                $response = $this->attemptGemini($prompt, $model, $options);

                return new AiResponse(
                    content:    $response->text,
                    model:      $model,
                    provider:   'gemini',
                    inputTokens:  $response->usage?->inputTokens ?? 0,
                    outputTokens: $response->usage?->outputTokens ?? 0,
                );

            } catch (\Exception $e) {
                Log::warning('Gemini model failed', [
                    'model'   => $model,
                    'error'   => $e->getMessage(),
                    'prompt_length' => strlen($prompt),
                ]);

                continue;
            }
        }

        // All Gemini models exhausted. Fall back to configured provider.
        return $this->fallback($prompt, $options);
    }

    private function attemptGemini(string $prompt, string $model, array $options): mixed
    {
        return AI::connection('gemini')->text(
            model: $model,
            messages: [['role' => 'user', 'content' => $prompt]],
            ...$options
        );
    }

    private function fallback(string $prompt, array $options): AiResponse
    {
        Log::error('All Gemini models exhausted. Falling back.', [
            'fallback_connection' => $this->fallbackConnection,
            'fallback_model'      => $this->fallbackModel,
        ]);

        $response = AI::connection($this->fallbackConnection)->text(
            model: $this->fallbackModel,
            messages: [['role' => 'user', 'content' => $prompt]],
        );

        return new AiResponse(
            content:     $response->text,
            model:       $this->fallbackModel,
            provider:    $this->fallbackConnection,
            inputTokens:  $response->usage?->inputTokens ?? 0,
            outputTokens: $response->usage?->outputTokens ?? 0,
        );
    }
}

The AiResponse DTO keeps your response contract stable:

<?php

namespace App\DTOs;

readonly class AiResponse
{
    public function __construct(
        public string $content,
        public string $model,
        public string $provider,
        public int    $inputTokens,
        public int    $outputTokens,
    ) {}
}

Bind it in your bootstrap/app.php, using Laravel 11/12 syntax:

use App\Services\AI\GeminiManager;

->withProviders([
    function () {
        app()->singleton(GeminiManager::class);
    },
])

[Architect’s Note] This pattern mirrors exactly what we described in Production-Grade AI Architecture in Laravel: Contracts, Governance & Telemetry—the response DTO is your contract, and nothing outside the service layer should care which provider produced the response. The moment your Blade template starts conditionally rendering based on $response->provider, you have an architecture problem, not a template problem.

Rate Limit Handling

The GeminiManager above catches all exceptions and continues to the next model. That is fine for a first pass but too broad for production. You want to distinguish between rate-limit errors, which should trigger immediate failover, and authentication or configuration errors, which should not silently retry.

Gemini rate-limit responses return HTTP 429. A tighter catch block:

use Illuminate\Http\Client\RequestException;

private function attemptGemini(string $prompt, string $model, array $options): mixed
{
    try {
        return AI::connection('gemini')->text(
            model: $model,
            messages: [['role' => 'user', 'content' => $prompt]],
            ...$options
        );

    } catch (RequestException $e) {
        $status = $e->response->status();

        if ($status === 429) {
            // Rate limited. Try next model.
            throw $e;
        }

        if ($status === 401 || $status === 403) {
            // Auth failure. Do not retry silently.
            Log::critical('Gemini authentication failure', ['status' => $status]);
            throw new \RuntimeException('Gemini authentication failed. Check GEMINI_API_KEY.');
        }

        // Anything else: surface it.
        throw $e;
    }
}

For consistent rate-limit and token tracking across all your AI providers, the middleware pattern we detailed in Laravel AI Middleware: Token Tracking & Rate Limiting plugs directly into this architecture. Wire it as HTTP middleware on your AI routes and your AiResponse DTO’s token counts feed it automatically.

Keeping Your Blade Templates Clean

This is the part most tutorials ignore. It is also where the design falls apart in real applications.

Your Blade templates should receive an AiResponse object and render $response->content. That is it. They should not know the provider is Gemini, they should not branch on $response->model, and they should never contain try/catch logic. If your failover logic is working correctly, the view never knows a failover happened.

In a controller:

use App\Services\AI\GeminiManager;

class ArticleSummaryController extends Controller
{
    public function __construct(
        private readonly GeminiManager $ai
    ) {}

    public function show(Article $article): View
    {
        $summary = $this->ai->generate(
            prompt: "Summarise this article in three sentences: {$article->body}"
        );

        return view('articles.show', [
            'article' => $article,
            'summary' => $summary,
        ]);
    }
}

The Blade template:

<div class="summary">
    {{ $summary->content }}
</div>

Done. No provider logic. No model names. No error handling. The controller injects the service, the service handles everything messy, and the view renders a string.

[Word to the Wise] The discipline to keep AI provider logic out of your views pays off when you swap models six months from now. And you will swap models. Gemini 2.5 Flash will eventually give way to whatever Google releases next. If your template has @if ($summary->model === 'gemini-2.5-flash') buried inside it, that migration becomes a grep exercise across your entire view layer. Design the abstraction correctly now.

Registering Gemini as a Queued Job Provider

Synchronous AI calls in web requests are a bad idea at scale. Wrap your GeminiManager calls in a queued job for anything that is not a real-time user interaction:

<?php

namespace App\Jobs;

use App\Services\AI\GeminiManager;
use App\Models\Article;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class GenerateArticleSummary implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public int $tries    = 3;
    public int $backoff  = 30; // seconds

    public function __construct(
        private readonly int $articleId
    ) {}

    public function handle(GeminiManager $ai): void
    {
        $article = Article::findOrFail($this->articleId);

        $summary = $ai->generate(
            prompt: "Summarise this article in three sentences: {$article->body}"
        );

        $article->update([
            'summary'          => $summary->content,
            'summary_model'    => $summary->model,
            'summary_provider' => $summary->provider,
        ]);
    }

    public function failed(\Throwable $e): void
    {
        Log::error('Article summary generation failed', [
            'article_id' => $this->articleId,
            'error'      => $e->getMessage(),
        ]);
    }
}

The $tries = 3 and $backoff = 30 give you three attempts with a 30-second gap between them, which is usually enough to ride out a transient Gemini rate limit without wasting queue resources. The failed hook gives you an audit trail.

Official Documentation

Two references worth bookmarking as you build this:

Google Gemini OpenAI Compatibility Documentation — the canonical reference for what is and is not supported through the compat layer, including current model strings and endpoint coverage.
Laravel AI SDK Documentation — the official introduction covering provider setup, agent workflows, and the provider-agnostic design philosophy.

Production Deployment Checklist

A few things that catch teams off-guard when they move this from staging to production.

Google AI Studio API keys are rate-limited by default. For production workloads, you need to request increased quota through Google Cloud, or switch to Vertex AI. The Vertex AI endpoint requires a different authentication mechanism (service account credentials rather than a plain API key) and a slightly different base URL, so plan for that migration if your volume grows.

Environment-specific model configuration matters. You might run gemini-2.5-flash in production for cost reasons and gemini-2.5-pro in staging for testing purposes. Keep that in your .env and out of your GeminiManager:

# .env.production
GEMINI_MODEL=gemini-2.5-flash

# .env.staging
GEMINI_MODEL=gemini-2.5-pro

Read it in the service:

private array $modelPriority = [];

public function __construct()
{
    $this->modelPriority = [
        config('services.gemini.model', 'gemini-2.5-flash'),
        'gemini-2.5-pro',    // fallback within Gemini
    ];
}

And add to config/services.php:

'gemini' => [
    'model' => env('GEMINI_MODEL', 'gemini-2.5-flash'),
],

If you are deploying to Laravel Forge or a similar managed environment, confirm your queue workers restart after deployment. A worker that caches the old GeminiManager configuration will ignore model changes until it restarts. This is not Gemini-specific, but it catches teams every time they change provider configuration without restarting workers.

[Efficiency Gain] Gemini 2.5 Flash’s one-million-token context window is genuinely useful for RAG pipelines. If you are building document search or retrieval-augmented generation, the ability to stuff a much larger context window means fewer chunking edge cases and simpler retrieval logic. If you are already running embeddings and vector search in Laravel, the integration pattern described in Laravel Embeddings, Vector Databases, and RAG: A Production Implementation Guide maps directly to Gemini’s native multimodal context handling.

Where to Go From Here

The GeminiManager pattern above is a solid foundation. The logical next step is versioning your prompts so that model upgrades do not introduce silent regressions—something that becomes critical the moment you start comparing output quality between gemini-2.5-flash and whatever Gemini 3.x stable lands in your production window. The Prompt Migrations: Bringing Determinism to AI in Laravel tutorial covers exactly that workflow.

The Google AI ecosystem moves fast. gemini-3.1-flash-lite-preview is already in the preview tier as of this writing, and Google is actively iterating. The architecture we have built here isolates model selection to a single config value and a single service class. When the next stable Gemini model lands, your migration is a one-line change.

Laravel Filament Admin Dashboard for AI Applications: Token Costs, Prompt Management, and Agent Audit Trails

Dewald Hugo — Thu, 16 Apr 2026 03:34:17 +0000

You have the AI pipeline. You have the service classes, the middleware, the queued jobs. The backend is instrumented. What you do not have is any way to look at it without writing a SQL query at 2am because costs just spiked.

That is the gap this article closes.

A Laravel Filament admin dashboard is not the glamorous part of AI development. But the teams that skip it are the ones who discover they have been over-spending on tokens for six weeks, or that an agent ran a destructive tool call nobody authorised, or that a prompt change from last Tuesday quietly broke output quality across the board. You cannot govern what you cannot see.

The Laravel Filament admin dashboard built in this guide surfaces four things: real-time token cost monitoring, usage analytics broken down by user and model, a prompt version management panel, and a full agent audit trail. Each section is a Filament v3 Resource or Widget, built on top of the telemetry data your existing middleware and service layer already generate.

Why Filament, and Why Now

There is no serious competition in the Laravel ecosystem for building admin interfaces right now. Filament v3 ships with a full panel system, Livewire-powered tables, stats widgets, chart widgets, custom pages, and a role-based authorization layer. It works inside your existing Laravel application. It does not require a separate process or a Node frontend.

The Livewire-driven rendering keeps things reactive without JavaScript complexity. The Eloquent-backed Resources mean your existing models map directly to admin panels with almost no boilerplate. And because it is all Laravel under the hood, your Service Container, policies, events, and observers all still apply — you are not learning a new paradigm, just adding a UI layer on top of architecture you have already built.

Filament v3 requires PHP 8.1+ and Laravel 10+. On Laravel 11 or 12 you will be fine.

Installation and Panel Setup

Install Filament with the panel scaffolding command:

composer require filament/filament:"^3.0" -W
php artisan filament:install --panels

This generates app/Providers/Filament/AdminPanelProvider.php and registers it via bootstrap/app.php. The panel provider is where you wire discovery, colours, middleware, and authentication. Here is a clean starting configuration:

// app/Providers/Filament/AdminPanelProvider.php

namespace App\Providers\Filament;

use Filament\Http\Middleware\Authenticate;
use Filament\Http\Middleware\DisableBladeIconComponents;
use Filament\Http\Middleware\DispatchServingFilamentEvent;
use Filament\Panel;
use Filament\PanelProvider;
use Filament\Support\Colors\Color;
use Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse;
use Illuminate\Cookie\Middleware\EncryptCookies;
use Illuminate\Foundation\Http\Middleware\VerifyCsrfToken;
use Illuminate\Routing\Middleware\SubstituteBindings;
use Illuminate\Session\Middleware\AuthenticateSession;
use Illuminate\Session\Middleware\StartSession;
use Illuminate\View\Middleware\ShareErrorsFromSession;

class AdminPanelProvider extends PanelProvider
{
    public function panel(Panel $panel): Panel
    {
        return $panel
            ->default()
            ->id('admin')
            ->path('admin')
            ->login()
            ->colors(['primary' => Color::Violet])
            ->discoverResources(
                in: app_path('Filament/Resources'),
                for: 'App\\Filament\\Resources'
            )
            ->discoverPages(
                in: app_path('Filament/Pages'),
                for: 'App\\Filament\\Pages'
            )
            ->discoverWidgets(
                in: app_path('Filament/Widgets'),
                for: 'App\\Filament\\Widgets'
            )
            ->middleware([
                EncryptCookies::class,
                AddQueuedCookiesToResponse::class,
                StartSession::class,
                AuthenticateSession::class,
                ShareErrorsFromSession::class,
                VerifyCsrfToken::class,
                SubstituteBindings::class,
                DisableBladeIconComponents::class,
                DispatchServingFilamentEvent::class,
            ])
            ->authMiddleware([
                Authenticate::class,
            ])
            ->authorization(fn () => auth()->user()?->is_admin ?? false);
    }
}

That last ->authorization() call is the line most tutorials omit. Without it, any authenticated user can reach your admin panel. We will come back to access control later — but set that guard from day one.

Modeling the Data Your Dashboard Will Surface

Before you build a single Filament Resource, you need to decide what data you are collecting. On a production AI application, there are two primary log types worth persisting.

AI Usage Logs capture every inference call: which provider, which model, how many tokens, what it cost, which feature triggered it, and whether it succeeded.

Agent Audit Logs capture agentic actions: which agent class ran, what action it performed, which tool it called, what the input and output were, and how long it took.

Both should be Eloquent models backed by indexed tables. Start with the usage log migration:

// database/migrations/2026_04_15_000001_create_ai_usage_logs_table.php

Schema::create('ai_usage_logs', function (Blueprint $table) {
    $table->id();
    $table->foreignId('user_id')->nullable()->constrained()->nullOnDelete();
    $table->string('provider');           // 'anthropic', 'openai'
    $table->string('model');
    $table->string('feature')->nullable(); // 'chat', 'summarize', 'agent'
    $table->integer('prompt_tokens');
    $table->integer('completion_tokens');
    $table->integer('total_tokens');
    $table->decimal('cost_usd', 10, 6)->nullable();
    $table->string('prompt_version')->nullable();
    $table->json('metadata')->nullable();
    $table->boolean('success')->default(true);
    $table->text('error_message')->nullable();
    $table->timestamps();

    $table->index(['user_id', 'created_at']);
    $table->index(['provider', 'model', 'created_at']);
    $table->index(['feature', 'created_at']);
});

The compound indexes on created_at are non-negotiable. Dashboard widgets run aggregation queries constantly, and without them you will feel the difference under real traffic.

The agent audit log has a slightly different shape:

// database/migrations/2026_04_15_000002_create_agent_audit_logs_table.php

Schema::create('agent_audit_logs', function (Blueprint $table) {
    $table->id();
    $table->foreignId('user_id')->nullable()->constrained()->nullOnDelete();
    $table->string('agent_class');
    $table->string('action');
    $table->string('tool_used')->nullable();
    $table->json('input_payload')->nullable();
    $table->json('output_payload')->nullable();
    $table->integer('tokens_used')->default(0);
    $table->float('duration_ms')->nullable();
    $table->boolean('success')->default(true);
    $table->text('error')->nullable();
    $table->timestamps();

    $table->index(['agent_class', 'created_at']);
    $table->index(['user_id', 'created_at']);
    $table->index(['success', 'created_at']);
});

Both models should cast their JSON columns and use standard Eloquent relationships back to User. Nothing exotic required.

Token Cost Monitoring: The Stats Overview Widget

The first thing an operator needs to see when they open the dashboard is a top-level summary of what the AI layer has cost in the last 24 hours. Filament’s StatsOverviewWidget handles this cleanly.

php artisan make:filament-widget AiStatsOverview --stats-overview

// app/Filament/Widgets/AiStatsOverview.php

namespace App\Filament\Widgets;

use App\Models\AiUsageLog;
use Filament\Widgets\StatsOverviewWidget as BaseWidget;
use Filament\Widgets\StatsOverviewWidget\Stat;

class AiStatsOverview extends BaseWidget
{
    protected static ?int $sort = 1;

    protected function getStats(): array
    {
        $today  = AiUsageLog::whereDate('created_at', today());
        $month  = AiUsageLog::whereMonth('created_at', now()->month)
                            ->whereYear('created_at', now()->year);

        return [
            Stat::make("Today's Tokens", number_format($today->sum('total_tokens')))
                ->description('Across all providers and models')
                ->color('primary'),

            Stat::make("Today's Cost", '$' . number_format($today->sum('cost_usd'), 4))
                ->description('USD — live tally')
                ->color('warning'),

            Stat::make('Monthly Spend', '$' . number_format($month->sum('cost_usd'), 2))
                ->description(now()->format('F Y'))
                ->color('danger'),

            Stat::make('Failures Today', number_format($today->where('success', false)->count()))
                ->description('Inference errors across all features')
                ->color('gray'),
        ];
    }
}

Register this widget on your dashboard in the panel provider’s ->widgets() array. It polls on a configurable interval using Livewire’s polling mechanism — add protected static ?string $pollingInterval = '30s'; if you want it to refresh automatically.

If you have not yet built the middleware layer that feeds these logs, the Laravel AI Middleware guide on token tracking and rate limiting covers the exact pattern for intercepting inference calls and persisting usage data before the response reaches the controller.

Daily Token Usage: The Chart Widget

Raw totals are useful. Trend lines are what actually surface problems — a cost spike on Tuesday, a failure rate that spiked after a deployment. Add a chart widget to give operators a 30-day view:

php artisan make:filament-widget TokenUsageChart --chart

// app/Filament/Widgets/TokenUsageChart.php

namespace App\Filament\Widgets;

use App\Models\AiUsageLog;
use Filament\Widgets\ChartWidget;

class TokenUsageChart extends ChartWidget
{
    protected static ?string $heading = 'Daily Token Usage — Last 30 Days';
    protected static ?int $sort = 2;
    public ?string $filter = 'tokens';

    protected function getFilters(): ?array
    {
        return [
            'tokens' => 'Total Tokens',
            'cost'   => 'Cost (USD)',
        ];
    }

    protected function getData(): array
    {
        $column = $this->filter === 'cost' ? 'cost_usd' : 'total_tokens';
        $label  = $this->filter === 'cost' ? 'Cost (USD)' : 'Total Tokens';

        $data = AiUsageLog::selectRaw("DATE(created_at) as date, SUM({$column}) as value")
            ->where('created_at', '>=', now()->subDays(30))
            ->groupBy('date')
            ->orderBy('date')
            ->get();

        return [
            'datasets' => [
                [
                    'label'       => $label,
                    'data'        => $data->pluck('value')->toArray(),
                    'borderColor' => '#8b5cf6',
                    'tension'     => 0.3,
                    'fill'        => false,
                ],
            ],
            'labels' => $data->pluck('date')->toArray(),
        ];
    }

    protected function getType(): string
    {
        return 'line';
    }
}

The $filter toggle lets operators switch between token volume and cost in a single widget, which keeps the dashboard clean without multiplying panels.

The Usage Log Resource: Per-User and Per-Model Breakdown

Stats and charts give you the summary. When you need to investigate — which user ran 400,000 tokens in one session, which model is responsible for the cost spike — you need a sortable, filterable table.

php artisan make:filament-resource AiUsageLog --generate

Then replace the generated table definition with something that has real operator value:

// app/Filament/Resources/AiUsageLogResource.php

namespace App\Filament\Resources;

use App\Filament\Resources\AiUsageLogResource\Pages;
use App\Models\AiUsageLog;
use Filament\Resources\Resource;
use Filament\Tables;
use Filament\Tables\Table;
use Filament\Tables\Columns\IconColumn;
use Filament\Tables\Columns\TextColumn;
use Filament\Tables\Filters\SelectFilter;
use Filament\Tables\Filters\Filter;
use Illuminate\Database\Eloquent\Builder;

class AiUsageLogResource extends Resource
{
    protected static ?string $model           = AiUsageLog::class;
    protected static ?string $navigationIcon  = 'heroicon-o-chart-bar';
    protected static ?string $navigationGroup = 'AI Governance';
    protected static ?string $navigationLabel = 'Usage Logs';

    public static function table(Table $table): Table
    {
        return $table
            ->columns([
                TextColumn::make('user.name')
                    ->label('User')
                    ->searchable()
                    ->sortable()
                    ->placeholder('—'),

                TextColumn::make('provider')
                    ->badge()
                    ->sortable(),

                TextColumn::make('model')
                    ->searchable()
                    ->copyable(),

                TextColumn::make('feature')
                    ->badge()
                    ->color('info')
                    ->placeholder('—'),

                TextColumn::make('total_tokens')
                    ->numeric()
                    ->sortable()
                    ->summarize(Tables\Columns\Summarizers\Sum::make()->label('Total')),

                TextColumn::make('cost_usd')
                    ->money('usd')
                    ->sortable()
                    ->summarize(Tables\Columns\Summarizers\Sum::make()->label('Total')),

                TextColumn::make('prompt_version')
                    ->badge()
                    ->color('warning')
                    ->placeholder('—'),

                IconColumn::make('success')
                    ->boolean(),

                TextColumn::make('created_at')
                    ->dateTime()
                    ->sortable()
                    ->since()
                    ->toggleable(),
            ])
            ->filters([
                SelectFilter::make('provider')
                    ->options([
                        'anthropic' => 'Anthropic',
                        'openai'    => 'OpenAI',
                    ]),

                SelectFilter::make('success')
                    ->options([
                        '1' => 'Successful',
                        '0' => 'Failed',
                    ]),

                Filter::make('today')
                    ->label('Today only')
                    ->query(fn (Builder $query) => $query->whereDate('created_at', today())),
            ])
            ->defaultSort('created_at', 'desc')
            ->paginated([25, 50, 100]);
    }

    public static function getPages(): array
    {
        return [
            'index' => Pages\ListAiUsageLogs::route('/'),
        ];
    }

    public static function canCreate(): bool
    {
        return false;
    }
}

Two things to note here. First, ->summarize() on the token and cost columns gives you running totals in the table footer — incredibly useful when you have applied filters to isolate a user or a date range. Second, canCreate(): false disables the “New record” button. Usage logs are system-generated. Nobody should be hand-entering them.

Production-Grade AI Architecture in Laravel: Contracts, Governance & Telemetry details the telemetry layer that populates this data. If your AI services are not already emitting structured usage events to an observable logging interface, that article is the right place to start before wiring up this dashboard.

Prompt Version Management

Operators need to know which prompt is active. They also need to see when it changed, what the previous version said, and whether cost or quality shifted after a change. This is where the dashboard and your version control system connect.

Prompt Migrations: Bringing Determinism to AI in Laravel covers building the underlying prompt migration system. Assuming you have a prompt_versions table from that guide, the Filament Resource surfaces it with minimal effort:

php artisan make:filament-resource PromptVersion --generate

// app/Filament/Resources/PromptVersionResource.php

namespace App\Filament\Resources;

use App\Filament\Resources\PromptVersionResource\Pages;
use App\Models\PromptVersion;
use Filament\Forms\Components\Textarea;
use Filament\Forms\Components\TextInput;
use Filament\Forms\Components\Toggle;
use Filament\Forms\Form;
use Filament\Resources\Resource;
use Filament\Tables;
use Filament\Tables\Table;
use Filament\Tables\Columns\IconColumn;
use Filament\Tables\Columns\TextColumn;

class PromptVersionResource extends Resource
{
    protected static ?string $model           = PromptVersion::class;
    protected static ?string $navigationIcon  = 'heroicon-o-document-text';
    protected static ?string $navigationGroup = 'AI Governance';
    protected static ?string $navigationLabel = 'Prompt Versions';

    public static function form(Form $form): Form
    {
        return $form
            ->schema([
                TextInput::make('key')->required()->disabled(),
                TextInput::make('version')->required()->disabled(),
                Textarea::make('system_prompt')->rows(8)->columnSpanFull(),
                Textarea::make('description')->rows(3)->columnSpanFull(),
                Toggle::make('is_active')->label('Active')->disabled(),
            ]);
    }

    public static function table(Table $table): Table
    {
        return $table
            ->columns([
                TextColumn::make('key')->searchable()->sortable(),
                TextColumn::make('version')->badge()->color('primary'),
                TextColumn::make('description')->limit(60)->placeholder('—'),
                IconColumn::make('is_active')->boolean()->label('Active'),
                TextColumn::make('created_at')->dateTime()->sortable(),
            ])
            ->defaultSort('created_at', 'desc');
    }

    public static function getPages(): array
    {
        return [
            'index' => Pages\ListPromptVersions::route('/'),
            'view'  => Pages\ViewPromptVersion::route('/{record}'),
        ];
    }

    public static function canCreate(): bool { return false; }
    public static function canEdit($record): bool { return false; }
    public static function canDelete($record): bool { return false; }
}

All three can* methods return false. Prompt changes belong in migrations under version control. The admin panel gives you read access only. This is intentional. The moment you allow operators to edit prompt content via a UI form, you have created an unversioned, un-reviewable, un-rollbackable change path that will cause you pain.

[Architect’s Note] Keep your admin panel read-only for prompt content. The dashboard is a governance tool, not a content editor. Every prompt mutation should flow through a migration, be committed to version control, and be deployable like code. If your team is asking for a UI to edit prompts live, that is a process problem masquerading as a tooling request. Fix the process.

Agent Audit Trails

This is the section most teams skip. It is also the one that saves them when something goes wrong.

Agentic workflows make decisions. They call tools. They read data and write data. When a user complains that the agent deleted something it should not have, or an integration breaks because the agent started formatting its output differently, you need a record of exactly what happened.

If you are building agentic pipelines, the Hardening Laravel Agentic Workflows guide covers schema validation and output hardening. The audit trail built here is the operational companion to that — it answers “what did it do?” as opposed to “did it do it correctly?”

Emit an audit log from your base agent class on every action:

// app/AI/Agents/BaseAgent.php

namespace App\AI\Agents;

use App\Models\AgentAuditLog;

abstract class BaseAgent
{
    protected function logAction(
        string $action,
        array $input,
        array $output,
        ?string $toolUsed = null,
        int $tokensUsed = 0,
        float $durationMs = 0.0,
        bool $success = true,
        ?string $error = null
    ): void {
        AgentAuditLog::create([
            'user_id'        => auth()->id(),
            'agent_class'    => static::class,
            'action'         => $action,
            'tool_used'      => $toolUsed,
            'input_payload'  => $input,
            'output_payload' => $output,
            'tokens_used'    => $tokensUsed,
            'duration_ms'    => $durationMs,
            'success'        => $success,
            'error'          => $error,
        ]);
    }
}

The Filament Resource for the audit log is where the detail view earns its keep. Operators need to be able to click into a single agent run and see the full input/output payload:

// app/Filament/Resources/AgentAuditLogResource.php

namespace App\Filament\Resources;

use App\Filament\Resources\AgentAuditLogResource\Pages;
use App\Models\AgentAuditLog;
use Filament\Infolists\Components\KeyValueEntry;
use Filament\Infolists\Components\Section;
use Filament\Infolists\Components\TextEntry;
use Filament\Infolists\Infolist;
use Filament\Resources\Resource;
use Filament\Tables;
use Filament\Tables\Table;
use Filament\Tables\Columns\IconColumn;
use Filament\Tables\Columns\TextColumn;
use Filament\Tables\Filters\SelectFilter;

class AgentAuditLogResource extends Resource
{
    protected static ?string $model           = AgentAuditLog::class;
    protected static ?string $navigationIcon  = 'heroicon-o-cpu-chip';
    protected static ?string $navigationGroup = 'AI Governance';
    protected static ?string $navigationLabel = 'Agent Audit Trail';

    public static function table(Table $table): Table
    {
        return $table
            ->columns([
                TextColumn::make('user.name')->label('User')->searchable()->placeholder('System'),
                TextColumn::make('agent_class')->badge()->searchable(),
                TextColumn::make('action')->searchable(),
                TextColumn::make('tool_used')->badge()->color('info')->placeholder('—'),
                TextColumn::make('tokens_used')->numeric()->sortable(),
                TextColumn::make('duration_ms')->label('ms')->numeric()->sortable(),
                IconColumn::make('success')->boolean(),
                TextColumn::make('created_at')->dateTime()->sortable()->since(),
            ])
            ->filters([
                SelectFilter::make('success')->options(['1' => 'Successful', '0' => 'Failed']),
                SelectFilter::make('agent_class')->relationship('agentClass', 'agent_class'),
            ])
            ->defaultSort('created_at', 'desc')
            ->recordUrl(fn ($record) => static::getUrl('view', ['record' => $record]));
    }

    public static function infolist(Infolist $infolist): Infolist
    {
        return $infolist
            ->schema([
                Section::make('Run Details')
                    ->columns(2)
                    ->schema([
                        TextEntry::make('agent_class')->badge(),
                        TextEntry::make('action'),
                        TextEntry::make('tool_used')->badge()->color('info')->placeholder('None'),
                        TextEntry::make('tokens_used')->numeric(),
                        TextEntry::make('duration_ms')->label('Duration (ms)')->numeric(),
                        TextEntry::make('success')->badge()->getStateUsing(
                            fn ($record) => $record->success ? 'Success' : 'Failed'
                        )->color(fn ($state) => $state === 'Success' ? 'success' : 'danger'),
                    ]),

                Section::make('Input Payload')
                    ->schema([
                        KeyValueEntry::make('input_payload')->label(''),
                    ]),

                Section::make('Output Payload')
                    ->schema([
                        KeyValueEntry::make('output_payload')->label(''),
                    ]),

                Section::make('Error')
                    ->schema([TextEntry::make('error')->placeholder('None')])
                    ->visible(fn ($record) => !$record->success),
            ]);
    }

    public static function getPages(): array
    {
        return [
            'index' => Pages\ListAgentAuditLogs::route('/'),
            'view'  => Pages\ViewAgentAuditLog::route('/{record}'),
        ];
    }

    public static function canCreate(): bool { return false; }
    public static function canEdit($record): bool { return false; }
}

The Infolist view is the right choice here over a form, because you are displaying data, not editing it. The KeyValueEntry component renders JSON payloads cleanly without any custom Blade work.

Locking Down the Admin Panel

Your Filament admin is a high-value target. It contains cost data, user data, prompt intellectual property, and agent decision history. Basic access control is not optional.

The ->authorization() callback in your panel provider is your first line of defence. The example earlier used is_admin on the User model. In practice, most teams reach for Spatie’s Laravel Permission package for this. Your panel provider call then becomes:

->authorization(fn () => auth()->user()?->hasRole('admin') ?? false)

For more granular control over individual Resources, use Laravel Policies. Filament automatically discovers and respects a policy for any model that has one. A AiUsageLogPolicy with viewAny returning $user->hasRole('admin') gives you fine-grained control without putting access logic inside the Resource class itself.

One pattern worth enforcing: scope your AI admin panel to a separate admin guard, and ensure it sits behind your Sanctum token authentication if your application issues API tokens to admin users. The Laravel Sanctum API Authentication guide covers token scoping and ability verification, and the same principles apply when you are protecting admin panel access programmatically.

Do not expose the Filament panel under the default /admin path in production. Change the path in your panel provider:

->path('ops-internal') // or any non-obvious string

This is not security through obscurity as a replacement for authentication. It is a minor but free win against automated scanning.

Deploying Filament Alongside Your AI Stack

Filament adds Livewire, Alpine.js, and its own asset pipeline to your application. Running php artisan filament:assets publishes vendor assets to your public/ directory. Make sure your deployment script runs this command after every Filament update — if you skip it, operators will see unstyled panels or JavaScript errors after a version bump.

php artisan filament:assets
php artisan filament:optimize  # Filament v3.1+

filament:optimize caches component discovery. On applications with many Resources and Widgets, it makes a measurable difference to panel load time.

If your application uses Redis for caching, you may want to cache the stats queries powering your overview widgets. Filament’s widgets do not cache their data by default. For a stats widget that runs four SUM() aggregations on every page load, wrapping them in cache()->remember() with a 60-second TTL is worth doing before you go to production.

The complete Laravel production deployment guide covers the full deployment pipeline in detail. When you add Filament, extend the deploy script to include filament:assets and filament:optimize alongside your standard migrate, config:cache, and queue:restart steps. Missing any of these on a Filament upgrade is a common source of post-deploy surprises.

Organising the Navigation: The AI Governance Group

With three or more Filament Resources all related to AI observability, you want them grouped in the sidebar. Every Resource in this article already carries protected static ?string $navigationGroup = 'AI Governance';. That single property is enough to group them under a collapsible sidebar section.

You can control group ordering in the panel provider:

->navigationGroups([
    'AI Governance',
    'User Management',
    'System',
])

Groups listed here appear in that order in the sidebar. Groups not listed appear at the bottom. Keep AI Governance at the top — it is why operators are opening this panel.

What to Cache, What to Queue

Two performance considerations you will hit once real traffic arrives.

The stats overview widget runs database queries on every page render. If 10 admins have this dashboard open, that is 10 concurrent aggregation queries on your ai_usage_logs table. Cache the results. Override getStats() to use cache()->remember('ai_stats_today', 60, fn() => ...) for each query group.

Bulk insert your usage logs via a queued job rather than synchronously inside the middleware response cycle. A single queued insert dispatched after the AI response returns adds zero latency to the user-facing request and prevents your ai_usage_logs table from becoming a write bottleneck under load. This is especially relevant if you are logging agent audit trails — agent runs that call multiple tools can generate a burst of log writes in a short window.

[Production Pitfall] If you are writing AiUsageLog::create() synchronously inside an AI middleware or a service class, you will see that call become a bottleneck under sustained load. A single slow database write can add 20–50ms to every AI response. Dispatch a queued job instead. Your usage logs do not need to be written before the response returns — they need to be written eventually and reliably.

Extending the Dashboard Over Time

This guide builds the foundation. Here is what comes next.

Budget alerts. Add an Eloquent Observer on AiUsageLog that triggers a notification when daily spend crosses a configurable threshold. Filament’s database notifications work well here — the panel will show the alert in the notification bell without any additional UI work.

Per-feature cost attribution. The feature column on ai_usage_logs lets you break cost down by product feature. A second chart widget comparing cost by feature over time gives product teams the data they need to make informed decisions about which AI features are worth their current token spend.

Prompt performance correlation. Once you have both usage logs (with prompt_version) and an output quality signal — whether that is a user rating, a downstream validation pass/fail, or a schema validation score from your agentic pipeline — you can join them in a custom Filament page and show quality versus cost per prompt version. That is a genuinely differentiated internal tool that most teams never build.

Model comparison. If you are testing multiple models for the same feature (Claude versus GPT-4o, for example), the model column combined with a quality signal gives you a cost-per-quality-point comparison you can act on. That decision gets much easier when you can see it in a table rather than inferring it from CloudWatch logs.

Refer to the official Filament v3 documentation for the full API surface on widgets, infolists, and panel configuration. The patterns in this guide are production-tested starting points, not exhaustive implementations.

Laravel Sanctum API Authentication: The Complete Production Guide

Dewald Hugo — Thu, 09 Apr 2026 22:06:40 +0000

There's a quiet assumption baked into almost every Laravel AI integration tutorial: authentication exists. Routes are protected. Tokens are issued. The API is locked down.

That assumption breaks the moment you sit down to build something real.

Laravel Sanctum is the framework's answer to lightweight API token authentication. It ships with Laravel, it integrates cleanly with Eloquent, and it handles the two most common authentication patterns - SPA cookie-based sessions and mobile/external API token issuance, without pulling in a full OAuth server. This guide covers both patterns, but it leans hard into the personal access token model, because that's what you need when you're building an API that your own frontend, mobile app, or third-party client will consume.

By the end, you'll have a production-ready authentication layer: token issuance with ability scoping, protected routes, revocation endpoints, rate limiting via Redis, and a multi-tenant token pattern that holds up under real load. We're also covering the Laravel 11/12 bootstrap/app.php configuration style throughout, no legacy Kernel.php references.

Let's get into it.

What Sanctum Actually Does

Before writing a single line of code, you need to understand where Sanctum fits in the ecosystem. Sanctum is not OAuth. It doesn't issue refresh tokens. It doesn't support third-party authorization flows. If you need those things, reach for Laravel Passport.

What Sanctum does exceptionally well is issue hashed personal access tokens tied to your users table via a polymorphic personal_access_tokens table. Each token can carry a set of abilities - scoped permissions that your application checks at runtime. The token itself is a random string; only the SHA-256 hash lives in the database. That's a sensible default.

For SPAs on the same domain, Sanctum piggybacks on Laravel's existing session authentication via a cookie. This is the EnsureFrontendRequestsAreStateful middleware doing its job. We'll touch on this briefly, but the majority of this guide focuses on token-based auth for API clients.

Installation and Setup (Laravel 11 / 12)

Sanctum ships with Laravel 11 and 12. If you're starting a fresh project, it's already in your composer.json. Confirm it's there:

composer show laravel/sanctum

If you're on an older install that upgraded to Laravel 11, you may need to install it:

composer require laravel/sanctum
php artisan vendor:publish --provider="Laravel\Sanctum\SanctumServiceProvider"
php artisan migrate

That publish step drops a config/sanctum.php file and a migration for the personal_access_tokens table. Run the migration before anything else.

bootstrap/app.php - The Laravel 11/12 Way

In Laravel 11, middleware registration moved out of Kernel.php and into bootstrap/app.php. This is where you register the Sanctum stateful middleware for SPA authentication:

// bootstrap/app.php
use Laravel\Sanctum\Http\Middleware\EnsureFrontendRequestsAreStateful;

->withMiddleware(function (Middleware $middleware) {
    $middleware->statefulApi();
})

Calling $middleware->statefulApi() is the clean Laravel 11 way to register Sanctum's SPA middleware. Avoid manually listing the middleware class inline, statefulApi() handles the correct ordering.

For token-only APIs (no SPA), you can skip this entirely. Your token-protected routes will use the auth:sanctum guard, which handles everything through the Authorization: Bearer header.

Configuring the Guard

In config/auth.php, make sure your api guard points to Sanctum:

'guards' => [
    'web' => [
        'driver'   => 'session',
        'provider' => 'users',
    ],

    'api' => [
        'driver'   => 'sanctum',
        'provider' => 'users',
    ],
],

This means auth:api and auth:sanctum both resolve to the same thing. Pick one and be consistent across your routes.

Preparing the User Model

Add the HasApiTokens trait to your User model:

use Laravel\Sanctum\HasApiTokens;

class User extends Authenticatable
{
    use HasApiTokens, HasFactory, Notifiable;
}

That trait gives Eloquent's User model the createToken(), tokens(), and currentAccessToken() methods. Everything downstream depends on this being in place.

Issuing Personal Access Tokens

Token issuance happens through a dedicated endpoint. Keep it clean: one controller, one responsibility.

// app/Http/Controllers/Auth/TokenController.php

namespace App\Http\Controllers\Auth;

use App\Http\Controllers\Controller;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Hash;
use Illuminate\Validation\ValidationException;
use App\Models\User;

class TokenController extends Controller
{
    public function issue(Request $request): \Illuminate\Http\JsonResponse
    {
        $request->validate([
            'email'       => ['required', 'email'],
            'password'    => ['required'],
            'device_name' => ['required', 'string', 'max:255'],
        ]);

        $user = User::where('email', $request->email)->first();

        if (! $user || ! Hash::check($request->password, $user->password)) {
            throw ValidationException::withMessages([
                'email' => ['The provided credentials are incorrect.'],
            ]);
        }

        $token = $user->createToken(
            $request->device_name,
            ['api:read', 'api:write']
        );

        return response()->json([
            'token' => $token->plainTextToken,
        ]);
    }

    public function revoke(Request $request): \Illuminate\Http\JsonResponse
    {
        $request->user()->currentAccessToken()->delete();

        return response()->json(['message' => 'Token revoked.']);
    }
}

The device_name field is part of Sanctum's design, it lets users see which device or client issued each token. The ValidationException approach returns a clean 422 with field-level errors rather than a 401, which is what most frontend clients expect.

Route registration:

// routes/api.php
use App\Http\Controllers\Auth\TokenController;

Route::post('/auth/token', [TokenController::class, 'issue']);
Route::delete('/auth/token', [TokenController::class, 'revoke'])->middleware('auth:sanctum');

Architect's Note: Never issue tokens from inside a route closure. The moment you need to add logging, rate limiting, or ability scoping, you're refactoring. Always use a dedicated controller from day one, the Service Container will thank you when you need to inject dependencies later.

Token Abilities: Scoped Permissions

Token abilities are Sanctum's lightweight alternative to OAuth scopes. You define them as strings at issuance time, then check them at the route or controller level.

// Read-only token
$token = $user->createToken('mobile-client', ['api:read']);

// Full-access token
$token = $user->createToken('admin-panel', ['api:read', 'api:write', 'api:delete']);

// AI feature token
$token = $user->createToken('ai-assistant', ['ai:query', 'api:read']);

Check abilities in your controllers using tokenCan():

public function store(Request $request): JsonResponse
{
    if (! $request->user()->tokenCan('api:write')) {
        return response()->json(['error' => 'Insufficient token abilities.'], 403);
    }

    // proceed with write operation
}

Or apply the abilities middleware directly on route groups:

Route::middleware(['auth:sanctum', 'abilities:api:read,api:write'])
    ->group(function () {
        Route::post('/documents', [DocumentController::class, 'store']);
    });

Route::middleware(['auth:sanctum', 'ability:api:read'])
    ->group(function () {
        Route::get('/documents', [DocumentController::class, 'index']);
    });

Note the difference: abilities (plural) requires the token to have all listed abilities. ability (singular) requires at least one. This distinction catches people off guard in production.

Protecting Routes

Standard protected route group for a JSON API:

// routes/api.php

Route::middleware('auth:sanctum')->group(function () {

    Route::apiResource('documents', DocumentController::class);

    Route::prefix('ai')->group(function () {
        Route::post('/query', [AiQueryController::class, 'query'])
            ->middleware('ability:ai:query');
        Route::get('/history', [AiQueryController::class, 'history'])
            ->middleware('ability:api:read');
    });

});

The auth:sanctum guard resolves the authenticated user from the Bearer token, making $request->user() available throughout the request lifecycle.

Token Revocation

You need at minimum three revocation endpoints:

// Revoke current token (logout this device)
Route::delete('/auth/token', [TokenController::class, 'revoke'])
    ->middleware('auth:sanctum');

// Revoke a specific token by ID
Route::delete('/auth/tokens/{tokenId}', [TokenController::class, 'revokeById'])
    ->middleware('auth:sanctum');

// Revoke all tokens (logout everywhere)
Route::delete('/auth/tokens', [TokenController::class, 'revokeAll'])
    ->middleware('auth:sanctum');

The controller methods:

public function revokeById(Request $request, int $tokenId): JsonResponse
{
    $deleted = $request->user()
        ->tokens()
        ->where('id', $tokenId)
        ->delete();

    if (! $deleted) {
        return response()->json(['error' => 'Token not found.'], 404);
    }

    return response()->json(['message' => 'Token revoked.']);
}

public function revokeAll(Request $request): JsonResponse
{
    $request->user()->tokens()->delete();

    return response()->json(['message' => 'All tokens revoked.']);
}

The ->where('id', $tokenId) scoping on the tokens relationship is critical. Without it, a user could delete another user's token by guessing IDs — an IDOR vulnerability. The relationship scope ensures only that user's tokens are in play before the delete fires.

Production Pitfall: Token revocation is only as fast as your database query. Under heavy concurrent traffic, a "revoke all" operation that hits an unindexed tokenable_id column will cause measurable latency spikes. Add an index to personal_access_tokens.tokenable_id, Laravel's migration doesn't include it by default in all versions.

// In a new migration
Schema::table('personal_access_tokens', function (Blueprint $table) {
    $table->index(['tokenable_type', 'tokenable_id']);
});

Token Expiry

By default, Sanctum tokens don't expire. Set an expiry in config/sanctum.php:

'expiration' => 60 * 24 * 30, // 30 days, in minutes

Pair this with a scheduled prune command:

// routes/console.php (Laravel 11)
Schedule::command('sanctum:prune-expired --hours=24')->daily();

Left unchecked, the personal_access_tokens table grows indefinitely and starts dragging down every authenticated request.

Rate Limiting with Redis

The auth:sanctum middleware doesn't include rate limiting. You're responsible for adding it. This is not optional for any API that calls an external AI provider - without it, a single misbehaving client can burn through your budget in minutes.

Define rate limiters in bootstrap/app.php:

use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Support\Facades\RateLimiter;

RateLimiter::for('api', function (Request $request) {
    return $request->user()
        ? Limit::perMinute(60)->by($request->user()->id)
        : Limit::perMinute(10)->by($request->ip());
});

RateLimiter::for('ai', function (Request $request) {
    return $request->user()
        ? Limit::perMinute(10)->by($request->user()->id)
        : Limit::perMinute(2)->by($request->ip());
});

Apply them to route groups:

Route::middleware(['auth:sanctum', 'throttle:api'])->group(function () {
    // standard API routes
});

Route::middleware(['auth:sanctum', 'throttle:ai'])->group(function () {
    // AI query routes
});

Switch your cache driver to Redis. The default file driver won't share rate limit counts across multiple server instances.

CACHE_DRIVER=redis
REDIS_HOST=127.0.0.1
REDIS_PORT=6379

Efficiency Gain: Redis handles both rate limiting and Sanctum's token lookup cache on the same connection. At scale, Sanctum resolves the user on every request by hashing the incoming token and hitting the database. If you're seeing measurable auth overhead in production profiling, an application-level cache keyed to the token hash eliminates that DB hit - but profile first, optimize second.

Multi-Tenant Token Scoping

A user who belongs to multiple organizations shouldn't be able to use a token issued for Organization A to access Organization B's data. The cleanest fix: store a team_id on the token itself using a custom token model.

Extend the default token model:

// app/Models/PersonalAccessToken.php

namespace App\Models;

use Laravel\Sanctum\PersonalAccessToken as SanctumToken;

class PersonalAccessToken extends SanctumToken
{
    protected $fillable = [
        'name',
        'token',
        'abilities',
        'expires_at',
        'team_id',
    ];
}

use Laravel\Sanctum\Sanctum;
use App\Models\PersonalAccessToken;

public function boot(): void
{
    Sanctum::usePersonalAccessTokenModel(PersonalAccessToken::class);
}

Add the column:

Schema::table('personal_access_tokens', function (Blueprint $table) {
    $table->unsignedBigInteger('team_id')->nullable()->index();
});

Issue tokens with team context:

$token = $user->createToken($request->device_name, ['api:read', 'api:write']);
$token->accessToken->forceFill(['team_id' => $currentTeam->id])->save();

Authorize against both user and team on every request:

$tokenTeamId = $request->user()->currentAccessToken()->team_id;

if ($tokenTeamId !== $resource->team_id) {
    abort(403, 'Token not authorized for this team.');
}

Listing Tokens for the Authenticated User

Give users visibility into their active tokens:

public function index(Request $request): JsonResponse
{
    $tokens = $request->user()->tokens()
        ->select(['id', 'name', 'abilities', 'last_used_at', 'expires_at', 'created_at'])
        ->latest()
        ->get()
        ->map(function ($token) {
            return [
                'id'           => $token->id,
                'name'         => $token->name,
                'abilities'    => $token->abilities,
                'last_used_at' => $token->last_used_at?->toDateTimeString(),
                'expires_at'   => $token->expires_at?->toDateTimeString(),
                'created_at'   => $token->created_at->toDateTimeString(),
            ];
        });

    return response()->json(['tokens' => $tokens]);
}

The raw token value is never returned here, that's only available at issuance via plainTextToken. After that, only the hash is stored.

Testing Sanctum Authentication

Sanctum ships with a clean testing helper:

use Laravel\Sanctum\Sanctum;

class DocumentApiTest extends TestCase
{
    use RefreshDatabase;

    public function test_authenticated_user_can_list_documents(): void
    {
        $user = User::factory()->create();
        Sanctum::actingAs($user, ['api:read']);

        $this->getJson('/api/documents')->assertStatus(200);
    }

    public function test_token_without_write_ability_cannot_create_document(): void
    {
        $user = User::factory()->create();
        Sanctum::actingAs($user, ['api:read']); // no api:write

        $this->postJson('/api/documents', [
            'title'   => 'Test',
            'content' => 'Test content',
        ])->assertStatus(403);
    }

    public function test_unauthenticated_request_is_rejected(): void
    {
        $this->getJson('/api/documents')->assertStatus(401);
    }
}

Sanctum::actingAs() bypasses the actual token lookup and tells the guard to treat the request as authenticated with those abilities. Your tests stay fast; no real tokens are issued.

Word to the Wise: Test the ability checks explicitly. Developers regularly ship applications where the happy path tests pass but the authorization boundaries are never verified. A token with api:read silently passing a api:write endpoint is a data integrity problem, not just a security one. Write the negative cases, they catch the middleware misconfiguration you didn't know you'd made.

Returning Consistent Auth Error Responses

By default, an unauthenticated request to a Sanctum-protected route returns a redirect to the login page. That's wrong for a JSON API. Fix it in bootstrap/app.php:

->withExceptions(function (Exceptions $exceptions) {
    $exceptions->render(function (AuthenticationException $e, Request $request) {
        if ($request->expectsJson()) {
            return response()->json(['error' => 'Unauthenticated.'], 401);
        }
    });

    $exceptions->render(function (AuthorizationException $e, Request $request) {
        if ($request->expectsJson()) {
            return response()->json(['error' => 'Forbidden.'], 403);
        }
    });
})

A 302 redirect from an API client that expected a 401 is a confusing failure mode that causes hours of frontend debugging.

SPA Authentication (Cookie-Based)

For a Vue or React SPA on the same top-level domain, Sanctum's stateful middleware handles auth through the session cookie — no tokens needed.

The flow: the SPA GETs /sanctum/csrf-cookie to initialize the CSRF cookie, POSTs credentials to your login endpoint, then includes the X-XSRF-TOKEN header on subsequent requests.

Configure your stateful domains in config/sanctum.php:

'stateful' => explode(',', env('SANCTUM_STATEFUL_DOMAINS', sprintf(
    '%s%s',
    'localhost,localhost:3000,127.0.0.1,127.0.0.1:8000,::1',
    Sanctum::currentApplicationUrlWithPort()
))),

Set SANCTUM_STATEFUL_DOMAINS explicitly in production. Wildcard domains are not supported.

Edge Case Alert: If your SPA and API are on different subdomains (app.example.com and api.example.com), the session cookie won't work across domains due to browser SameSite restrictions. Use token-based auth even for your SPA in this case. The cookie approach only works when both are served from the same domain.

Production Deployment Checklist

Database

personal_access_tokens migration applied
Index on tokenable_type and tokenable_id exists
Token expiry configured and sanctum:prune-expired scheduled

Configuration

APP_KEY is set and rotated from the default
SESSION_DRIVER is redis or database in multi-server setups
CACHE_DRIVER is redis for rate limiting across instances
SANCTUM_STATEFUL_DOMAINS explicitly set if using SPA auth

Rate Limiting

Named rate limiters defined for all API route groups
Stricter limiters applied to AI query endpoints
Rate limit responses return Retry-After headers

Security

Authorization header stripped at the CDN for non-API routes
HTTPS enforced — tokens over HTTP is a non-starter
Tokens are never logged — audit your logging configuration

Monitoring

Track personal_access_tokens table row count as a metric
Alert on unusual token issuance spikes
Log token revocation events for audit trails

What Sanctum Doesn't Cover

Sanctum doesn't do: OAuth2 authorization flows, refresh tokens, third-party client authorization, dynamic scope negotiation, or JWT issuance. If any of those are requirements, use Passport.

For the overwhelming majority of Laravel APIs — including every AI integration pattern — Sanctum's personal access tokens, SPA sessions, ability scoping, and revocation model cover the full authentication lifecycle. Don't reach for Passport's complexity if Sanctum's model fits your use case. The operational overhead isn't trivial.

Official Documentation

Originally published at origin-main.com — Laravel + AI integration guides for production developers.

Laravel AI Middleware: Token Tracking & Rate Limiting

Dewald Hugo — Sun, 05 Apr 2026 00:10:17 +0000

Integrating Claude or GPT-4o into a Laravel app is deceptively easy. One Http::post() and you feel like a genius. Then a single user loops your endpoint, or a bot scripts it, and your OpenAI bill climbs $300 before you wake up. The answer is a dedicated Laravel AI middleware layer, and most tutorials skip it entirely.

That’s not a hypothetical. It happens.

The fix isn’t smarter prompts, it’s infrastructure. In this guide, we build a proper AI management layer: tiered rate limiting, per-user token tracking with async logging, and a Pest test suite that actually validates the guards are working. We’re not demoing a toy. We’re building something you can ship. For prompt versioning and deployment discipline on the prompt side of that infrastructure, see the prompt migrations guide.

1. The Strategy: Why a Laravel AI Middleware Layer?

Middleware in the Laravel lifecycle is the right place for this because it separates infrastructure concerns (cost, rate limits) from business logic (the prompt itself). This decoupling means you can swap gpt-4o for a self-hosted Llama instance without touching a single Controller.

For the output side of that separation, validating what the model returns before your application acts on it, see the guide on schema validation against LLM hallucinations.

We’re tracking cost using the standard token pricing formula:

$$Total Cost = \sum_{i=1}^{n} (Tokens_{in} \cdot Price_{in} + Tokens_{out} \cdot Price_{out})$$

Centralise that calculation once. Never scatter it across Controllers.

2. Database Schema: An Audit Trail, Not Just a Counter

Don’t fall into the trap of incrementing a single total_tokens integer on your users table. You need a row per request so you can audit, refund, and debug.

// database/migrations/xxxx_xx_xx_create_ai_usage_logs_table.php
public function up(): void
{
    Schema::create('ai_usage_logs', function (Blueprint $table) {
        $table->id();
        $table->foreignId('user_id')->constrained()->onDelete('cascade');
        $table->string('model_used');        // e.g., 'gpt-4o', 'claude-sonnet-4-6'
        $table->string('feature_name');      // e.g., 'blog_generator', 'chat'
        $table->integer('prompt_tokens')->default(0);
        $table->integer('completion_tokens')->default(0);
        $table->decimal('estimated_cost', 10, 6)->default(0);
        $table->timestamps();
    });
}

3. Tiered Rate Limiting

Define your tiers in AppServiceProvider. Free users get 5 requests per minute. Premium gets 100. Adjust to whatever your unit economics demand.

// app/Providers/AppServiceProvider.php
use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Support\Facades\RateLimiter;
use Illuminate\Http\Request;

public function boot(): void
{
    RateLimiter::for('ai-api', function (Request $request) {
        $user = $request->user();

        return $user?->is_premium
            ? Limit::perMinute(100)->by($user->id)
            : Limit::perMinute(5)->by($user->id);
    });
}

Senior Dev Tip: Back this with Redis, not your database. Set CACHE_DRIVER=redis in .env. If you’re running the default file or database driver, every AI request triggers a DB read-write cycle under the rate limiter — that’s a quiet bottleneck that won’t show up until you’re under load. See the Laravel Cache documentation for driver configuration.

4. Building the Middleware

Generate the class:

php artisan make:middleware EnsureUserHasAiCredits

This middleware enforces auth and your per-user spending ceiling before the request ever reaches the Controller:

// app/Http/Middleware/EnsureUserHasAiCredits.php
namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;
use Symfony\Component\HttpFoundation\Response;

class EnsureUserHasAiCredits
{
    public function handle(Request $request, Closure $next): Response
    {
        $user = $request->user();

        if (! $user) {
            return response()->json(['error' => 'Unauthorized'], 401);
        }

        if ($user->monthly_spend >= $user->spending_limit) {
            return response()->json([
                'error'   => 'Monthly spending limit reached.',
                'upgrade' => route('billing.upgrade'),
            ], 402);
        }

        return $next($request);
    }
}

Registering the Middleware (Laravel 11+)

Laravel 11 dropped app/Http/Kernel.php. Register your middleware in bootstrap/app.php:

// bootstrap/app.php
->withMiddleware(function (Middleware $middleware) {
    $middleware->appendToGroup('api', [
        \App\Http\Middleware\EnsureUserHasAiCredits::class,
    ]);
})

Then apply the rate limiter on your route:

// routes/api.php
Route::post('/generate', [AiController::class, 'generate'])
    ->middleware('throttle:ai-api');

5. Externalising Model Pricing

Hardcoding 0.000005 per token inside your Job is a maintenance trap. One pricing change from OpenAI and your cost accounting silently breaks. Put rates in config/ai.php:

// config/ai.php
return [
    'models' => [
        'gpt-4o' => [
            'prompt_rate'     => 0.000005,
            'completion_rate' => 0.000015,
        ],
        'claude-sonnet-4-5' => [
            'prompt_rate'     => 0.000003,
            'completion_rate' => 0.000015,
        ],
    ],
];

6. Async Usage Logging with a Queued Job

Once the API responds, log asynchronously. Don’t make the user wait for a DB write.

// app/Jobs/LogAiUsage.php
namespace App\Jobs;

use App\Models\AiUsageLog;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class LogAiUsage implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function __construct(
        public readonly int    $userId,
        public readonly array  $usageData,
    ) {}

    public function handle(): void
    {
        $model  = $this->usageData['model'];
        $rates  = config("ai.models.{$model}", [
            'prompt_rate'     => 0.000005,
            'completion_rate' => 0.000015,
        ]);

        $cost = ($this->usageData['prompt_tokens'] * $rates['prompt_rate'])
              + ($this->usageData['completion_tokens'] * $rates['completion_rate']);

        AiUsageLog::create([
            'user_id'           => $this->userId,
            'model_used'        => $model,
            'feature_name'      => $this->usageData['feature_name'],
            'prompt_tokens'     => $this->usageData['prompt_tokens'],
            'completion_tokens' => $this->usageData['completion_tokens'],
            'estimated_cost'    => $cost,
        ]);
    }
}

Notice feature_name is included in both the Job payload and the create() call. The original migration defines that column, if you omit it from the insert, Laravel will either throw a NOT NULL constraint violation or silently ignore it if it’s nullable. Either way, you’ve lost attribution data. Always match your payload to your schema.

7. The Controller: Gluing It Together

// app/Http/Controllers/AiController.php
namespace App\Http\Controllers;

use App\Jobs\LogAiUsage;
use Illuminate\Http\Request;
use Illuminate\Http\JsonResponse;
use OpenAI\Laravel\Facades\OpenAI;
use Illuminate\Validation\ValidationException;

class AiController extends Controller
{
    public function generate(Request $request): JsonResponse
    {
        $validated = $request->validate([
            'prompt'  => ['required', 'string', 'max:4000'],
            'feature' => ['required', 'string', 'in:blog_generator,chat,summariser'],
        ]);

        try {
            $result = OpenAI::chat()->create([
                'model'    => 'gpt-4o',
                'messages' => [['role' => 'user', 'content' => $validated['prompt']]],
            ]);
        } catch (\OpenAI\Exceptions\TransporterException $e) {
            report($e);
            return response()->json(['error' => 'AI service unavailable. Please try again.'], 503);
        } catch (\OpenAI\Exceptions\ErrorException $e) {
            // Catches 429s, quota exceeded, invalid requests from OpenAI
            report($e);
            return response()->json(['error' => $e->getMessage()], $e->getCode() ?: 500);
        }

        LogAiUsage::dispatch(auth()->id(), [
            'model'             => 'gpt-4o',
            'feature_name'      => $validated['feature'],
            'prompt_tokens'     => $result->usage->promptTokens,
            'completion_tokens' => $result->usage->completionTokens,
        ]);

        return response()->json([
            'content' => $result->choices[0]->message->content,
        ]);
    }
}

If you’re integrating Anthropic’s SDK instead, the pattern is identical — swap the client call and update the model string to claude-sonnet-4-5. See the Anthropic API documentation for the current SDK reference.

8. Testing the Architecture with Pest

A production system is only as solid as the test suite backing it. We need to verify three things independently:

The rate limiter blocks users who’ve exhausted their quota.
The middleware halts users who’ve hit their spending ceiling.
The LogAiUsage Job is dispatched with the correct payload — without hitting the real API.

That last point is where most AI testing tutorials fail you. Bus::fake() queues the Job but does nothing about the OpenAI facade. If you don’t mock it, your test suite is burning real API credits.

// tests/Feature/AiGenerationTest.php
use App\Jobs\LogAiUsage;
use App\Models\User;
use Illuminate\Support\Facades\Bus;
use Illuminate\Support\Facades\RateLimiter;
use OpenAI\Laravel\Facades\OpenAI;
use OpenAI\Responses\Chat\CreateResponse;

beforeEach(function () {
    Bus::fake();
});

it('blocks a free user who has hit their rate limit', function () {
    $user = User::factory()->create(['is_premium' => false]);

    // Simulate 5 prior requests — the free-tier ceiling
    foreach (range(1, 5) as $i) {
        RateLimiter::hit('ai-api:' . $user->id);
    }

    $this->actingAs($user)
        ->postJson('/api/generate', [
            'prompt'  => 'Hello AI',
            'feature' => 'chat',
        ])
        ->assertStatus(429);
});

it('blocks a user who has exceeded their monthly spending limit', function () {
    $user = User::factory()->create([
        'monthly_spend'  => 50.00,
        'spending_limit' => 50.00,
    ]);

    $this->actingAs($user)
        ->postJson('/api/generate', [
            'prompt'  => 'Hello AI',
            'feature' => 'chat',
        ])
        ->assertStatus(402)
        ->assertJsonPath('error', 'Monthly spending limit reached.');
});

it('dispatches a LogAiUsage job with correct token data on success', function () {
    $user = User::factory()->create(['is_premium' => true]);

    // Mock the OpenAI facade — never hit the real API in tests
    OpenAI::fake([
        CreateResponse::fake([
            'choices' => [
                ['message' => ['role' => 'assistant', 'content' => 'Mocked response']],
            ],
            'usage' => [
                'prompt_tokens'     => 15,
                'completion_tokens' => 42,
                'total_tokens'      => 57,
            ],
        ]),
    ]);

    $this->actingAs($user)
        ->postJson('/api/generate', [
            'prompt'  => 'Test prompt',
            'feature' => 'chat',
        ])
        ->assertOk()
        ->assertJsonPath('content', 'Mocked response');

    Bus::assertDispatched(LogAiUsage::class, function (LogAiUsage $job) use ($user) {
        return $job->userId === $user->id
            && $job->usageData['prompt_tokens'] === 15
            && $job->usageData['completion_tokens'] === 42
            && $job->usageData['feature_name'] === 'chat';
    });
});

If you’re building out the Eloquent side of this and want to understand how to test model relationships and factory states at scale, the patterns covered in building robust Laravel test factories translate directly to the User and AiUsageLog models here.

9. What You Now Have

Shipping this architecture gives you:

Cost protection — tiered rate limiting via the Laravel RateLimiter facade backed by Redis, plus a hard monthly ceiling enforced in middleware.
Full audit trail — every token consumed is recorded per-user, per-feature, per-model via an Eloquent model in the Service Container-resolved LogAiUsage Job.
Zero latency impact — async logging means your users never wait for a DB write.
A test suite that doesn’t cheat — the OpenAI facade is mocked, so your CI pipeline stays free and your assertions are actually trustworthy.

The gap between a weekend project and a production system isn’t the AI integration. It’s everything around it. This is that layer.

References: