Process 10k Tasks Safely: Job Batching in Laravel 🚂

The Long-Running Request Problem

Imagine your application needs to generate end-of-month financial reports for 10,000 users. If you try to process this synchronously in an HTTP controller, the server will time out, leaving the user staring at a broken page and half of the reports ungenerated. Even if you push them to a standard queue, tracking when the entire job of 10,000 reports is finished is notoriously difficult.

At Smart Tech Devs, we handle massive, resource-intensive workflows using Laravel Job Batching. This allows us to group thousands of asynchronous jobs together, process them in parallel, and execute callbacks when the entire batch is complete.

Harnessing Job Batching

Instead of dispatching jobs blindly into the void, batching gives you a tracking ID. This allows your frontend to show a real-time progress bar to the user.

Step 1: Dispatching the Batch

We use the Bus::batch() facade to array our jobs and define what happens when the batch succeeds, fails, or finishes completely.

use App\Jobs\GenerateMonthlyReport;
use Illuminate\Support\Facades\Bus;
use Throwable;

$users = User::active()->pluck('id');
$jobs = $users->map(fn($id) => new GenerateMonthlyReport($id))->toArray();

$batch = Bus::batch($jobs)->then(function (Batch $batch) {
    // All jobs completed successfully
    Notification::route('mail', '[e-mail]')
        ->notify(new ReportsCompletedNotification());
})->catch(function (Batch $batch, Throwable $e) {
    // First batch job failure detected
    Log::error('Batch processing failed: ' . $e->getMessage());
})->finally(function (Batch $batch) {
    // Executed when the batch has finished (regardless of success/failure)
})->name('Monthly Financial Reports')->dispatch();

// Return the batch ID to the frontend to poll for progress
return response()->json(['batch_id' => $batch->id]);

Step 2: Processing the Individual Job

Inside the individual job class, we use the Batchable trait. If a specific user's report fails, we can mark the job as failed, which triggers the batch's catch callback.

namespace App\Jobs;

use Illuminate\Bus\Batchable;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;

class GenerateMonthlyReport implements ShouldQueue
{
    use Batchable, Queueable;

    public function __construct(public int $userId) {}

    public function handle()
    {
        if ($this->batch()->cancelled()) {
            // Determine if the batch has been cancelled...
            return;
        }

        // Heavy processing logic here...
    }
}

The Engineering ROI

By implementing Job Batching, you transform fragile, timeout-prone endpoints into robust, highly scalable background processes. You gain the ability to process tasks in parallel across multiple queue workers, handle partial failures gracefully, and provide users with transparent, real-time progress tracking.