DEV Community

Cover image for Laravel Queues: Redis vs Database Driver
devTalk
devTalk

Posted on Originally published at dev-talk.com

Laravel Queues: Redis vs Database Driver

Most advice on this topic stops at "Redis is faster". That is true, and it is also the least useful thing to know when you choose a queue driver. Speed rarely decides it. What decides it is what happens to a job when a transaction rolls back, when the server restarts, when Redis runs out of memory, or when a worker takes three seconds to notice new work. To get real answers instead of repeating folklore, I ran Laravel's own DatabaseQueue and RedisQueue classes (framework tag v13.35.0) against MariaDB 10.11 and Redis 7.0 and measured the differences. This article gives you the short answer, the evidence behind it, and the configuration that makes each driver behave properly.

The short answer

Your situation Start with
Small to medium app, a few thousand jobs a day, no Redis yet Database. One less service to run, and jobs follow your transactions.
Jobs must never exist if the surrounding database change rolled back Database on the same connection as your data (or Redis with after_commit).
Hundreds of jobs per second, or latency matters to users Redis with block_for set.
You want Horizon dashboards, auto-scaling workers and metrics Redis. Horizon only works with Redis queues.
Your primary database is already the bottleneck Redis, or at minimum a separate database connection for the jobs table.
Redis exists only as a cache and has an eviction policy Database, or a separate Redis instance configured for queues.

Laravel agrees that the database driver is a reasonable default: in the framework's shipped config/queue.php at 13.35, QUEUE_CONNECTION defaults to database. The rest of this article explains when to leave that default.

How each driver stores a job

Database driver

Jobs live in one table, created by the default 0001_01_01_000002_create_jobs_table.php migration (or php artisan make:queue-table if your app lacks it). The columns are id, queue (indexed), payload, attempts, reserved_at, available_at and created_at.

When a worker asks for a job, the driver runs a query like this inside a transaction. This is the real SQL, captured from a run against MariaDB 10.11:

select * from `jobs`
where `queue` = ?
  and ((`reserved_at` is null and `available_at` <= ?) or (`reserved_at` <= ?))
order by `id` asc
limit 1
FOR UPDATE SKIP LOCKED;

update `jobs` set `reserved_at` = ?, `attempts` = ? where `id` = ?;
Enter fullscreen mode Exit fullscreen mode

The FOR UPDATE SKIP LOCKED part is what lets many workers pull jobs at the same time without waiting for each other: a row another worker has locked is skipped rather than waited on. The framework only uses it when your server supports it. From the source of DatabaseQueue: MySQL 8.0.1 or newer, MariaDB 10.6 or newer, PostgreSQL 9.5 or newer, and recent Vitess. On older versions it falls back to a plain FOR UPDATE lock, which still works but makes workers queue up behind each other. When a job succeeds, the row is deleted. The same query also picks up rows whose reserved_at is older than retry_after, which is how a job from a crashed worker comes back.

Redis driver

Each queue is a small set of Redis keys. I pushed two jobs and one delayed job, popped one, and listed the keys:

Key Redis type Holds
queues:default list Jobs ready to run
queues:default:delayed sorted set Jobs scheduled for later, scored by time
queues:default:reserved sorted set Jobs a worker has taken, scored by when they expire
queues:default:notify list One marker per ready job, used by blocking pops

Pop, release and migration of expired jobs each run as a Lua script (see LuaScripts.php), so the move from "ready" to "reserved" is atomic on the Redis side. No row locks, no transaction, no table to bloat.

What I measured

The setup: a 2-vCPU container with MariaDB 10.11, Redis 7.0.15 and PHP all on the same machine, empty jobs, and the real framework queue classes. These numbers are from my environment, not a universal benchmark. Network latency to a managed database or Redis host will change them, and your jobs do real work. Read them as a comparison of the two drivers under identical conditions.

Throughput: pop and delete 8,000 empty jobs

Workers Redis (jobs/s) Database (jobs/s)
1 about 3,500 about 530
4 about 7,400 about 870
8 (on 2 cores) about 5,900 about 830

Every run processed every job exactly once, with zero duplicates, on both drivers. The timings include PHP process start-up, and the 8-worker rows are limited by only having two CPU cores, so look at the 1 and 4 worker rows for the clean comparison.

What this means in practice: with one worker, queue overhead was about 0.3 ms per job on Redis and about 1.9 ms per job on the database. If your job sends an email, calls an API or renders a PDF, that takes tens or hundreds of milliseconds, and the driver's overhead disappears into it. The database driver becomes the limiting factor only when you have very many very short jobs, in the hundreds per second.

Old database versions pay for locking

I forced the database driver to use the plain-lock fallback (what you get on MySQL older than 8.0.1 or MariaDB older than 10.6) and ran 8 workers over 4,000 jobs, twice for each mode. With SKIP LOCKED both runs took 5.3 s; with the plain lock they took 6.6 s and 6.4 s. In an earlier run with the plain lock one of my workers crashed with MySQL error 1213 (deadlock). My test script did not retry, whereas Laravel's worker catches the exception, reports it and sleeps one second before trying again, so you would see slowdowns and log noise rather than a dead worker. If you run the database driver on an old server, upgrade it before blaming Laravel.

Latency: how long a new job waits for a worker

Throughput hides a more visible difference. An idle worker is not waiting for a push; by default it asks for work, finds nothing, and sleeps for --sleep seconds (3 by default). I started four idle workers and pushed 8 jobs at random times, then measured the delay before each was picked up:

Configuration Median wait Longest wait
Database, --sleep=3 0.85 s 2.29 s
Redis polling (no block_for), --sleep=3 1.00 s 2.98 s
Redis block_for=5, --sleep=3 0.00 s 2.73 s
Redis block_for=5, --sleep=0 0.00 s 0.03 s

Two things stand out. First, Redis only feels instant after you configure it: out of the box it polls just like the database. Second, block_for alone is not enough. In the worker loop (Worker::daemon), when a blocking pop times out with no job, the worker still sleeps for --sleep seconds, and a job pushed during that sleep waits. That is the 2.73 second maximum in the third row. Setting --sleep=0 together with block_for removes the gap. Only do that with block_for set to a positive number: without blocking, --sleep=0 turns the worker into a busy loop that hammers Redis.

// config/queue.php
'redis' => [
    'driver' => 'redis',
    'connection' => env('REDIS_QUEUE_CONNECTION', 'default'),
    'queue' => env('REDIS_QUEUE', 'default'),
    'retry_after' => (int) env('REDIS_QUEUE_RETRY_AFTER', 90),
    'block_for' => 5,   // was null: wait up to 5 seconds for a job instead of polling
    'after_commit' => false,
],
Enter fullscreen mode Exit fullscreen mode
php artisan queue:work redis --sleep=0 --tries=3 --max-time=3600
Enter fullscreen mode Exit fullscreen mode

The Laravel documentation warns that block_for set to 0 blocks indefinitely and prevents signals such as SIGTERM from being handled until the next job is processed, so use a positive value like 5.

The database driver has no equivalent of blocking. Its latency is bounded by --sleep. You can lower it to 1, but every idle worker then runs a query every second. With ten idle workers that is ten queries per second doing nothing, which is the hidden cost of the database driver on a busy primary.

The differences that matter more than speed

1. Transactions: which driver gives you a job that should not exist

This is the most important behavioral difference. I dispatched a job to each driver inside a database transaction and then rolled the transaction back:

Scenario Database driver Redis driver
Job pushed inside a transaction that rolls back Job gone (queue size 0) Job still queued (queue size 1)
A worker polls before the transaction commits Does not see the job Sees and runs the job
A worker polls after the commit Sees the job Already ran it

With the database driver, the job row is written on the same connection as your data, so it commits or rolls back with it. With Redis, the job is visible the instant you push it, whatever the transaction does. The classic consequence: you create an order, dispatch SendReceipt, the worker picks it up in two milliseconds, and the order row it looks for is not committed yet. Or the transaction rolls back and a receipt goes out for an order that does not exist.

Laravel's answer for Redis (and any non-transactional driver) is the after_commit option. The documentation says that with it set to true, Laravel waits until open parent transactions have committed before dispatching the job, and discards jobs dispatched in a transaction that rolls back. You can also set it per job with ->afterCommit() or ->beforeCommit():

// config/queue.php, redis connection
'after_commit' => true,

// or per dispatch
ProcessOrder::dispatch($order)->afterCommit();
Enter fullscreen mode Exit fullscreen mode

A caveat for the database driver: the guarantee holds only when the queue uses the same database connection as your application data. The queue connection is chosen with DB_QUEUE_CONNECTION (the connection key of the queue config). Point it at a different database to take load off your primary, and you lose the transactional behavior and need after_commit there too.

2. Durability: what survives a restart or a memory spike

Database jobs are exactly as durable as your database: backups, replication and crash recovery all apply. Redis is an in-memory store, and what survives depends on its configuration. In Redis 7.0.15's default configuration (I checked with CONFIG GET), snapshots are taken on a schedule (save 3600 1 300 100 60 10000), the append-only file is off (appendonly no), and appendfsync is everysec. A crash can therefore lose everything pushed since the last snapshot. If your jobs matter, enable the append-only file in redis.conf:

appendonly yes
appendfsync everysec
maxmemory-policy noeviction
Enter fullscreen mode Exit fullscreen mode

That last line deserves its own test, because it is one of the easiest ways to lose jobs without noticing. I started Redis with a 3 MB memory limit, pushed 20,000 jobs while some cache keys also filled memory, and compared the two eviction policies:

Policy What dispatch() reported Jobs left in the queue
noeviction Error after 3,411 jobs: OOM command not allowed when used memory > 'maxmemory' 3,411 (nothing lost; the app got an error)
allkeys-lru All 20,000 pushes reported success 243 (the rest silently evicted)

With allkeys-lru, which is a common setting for a Redis used as a cache, Redis evicted the queue's own keys and the application never found out. With noeviction the dispatch fails loudly, which you can catch, alert on and fix. This is why a queue should not share a Redis instance with a cache that needs eviction. Run a separate instance, or use the database driver. The exact numbers depend on my tiny memory limit, but the behavior is the point.

3. Crash recovery works the same on both

Both drivers use a lease. A worker that takes a job marks it reserved until retry_after seconds pass; if the worker dies before deleting it, the job becomes available again with attempts incremented. I tested it with retry_after set to 2 seconds on each driver: the job could not be taken again immediately, and 3 seconds later it came back with attempts = 2 on both.

The setting to watch is the relationship with your job timeout. The Laravel documentation says a job's timeout should always be less than its retry_after value, otherwise the job may be started again before it has finished or timed out. The default job timeout is 60 seconds and the default retry_after in the shipped config is 90. If you raise a job's timeout to five minutes, raise retry_after past it, or two workers will run the same job.

4. Features that only work (or work better) on one side

  • Horizon is Redis only. The documentation states that Horizon requires your queue connection to be redis, and it is not compatible with Redis Cluster at this time. It adds a dashboard with throughput, runtime and wait-time metrics, plus worker balancing (auto, simple, false). With the database driver you monitor through logs, your own queries and the failed jobs table.
  • Unique jobs andWithoutOverlapping need a cache driver with atomic locks, but that list includes database and file, not only Redis. You do not need the Redis queue for them. Redis variants exist (RateLimitedWithRedis, ThrottlesExceptionsWithRedis) and the documentation calls them more efficient when you already use Redis.
  • Redis Cluster needs queue names that contain a hash tag, such as {default}, so that all of a queue's keys land in one slot. Also, the Redis queue driver does not support the serializer and compression Redis options.
  • Failover. Laravel 13.35's shipped config includes a failover queue connection that tries database and then deferred. If you use Horizon with a failover list that contains database, the documentation tells you to run a regular queue:work database process alongside Horizon, because Horizon only manages Redis queues.
  • Batches and failed jobs are stored in database tables (job_batches, failed_jobs) with either driver, so you need those migrations regardless. Prune batches with queue:prune-batches.

5. Operational cost

  • Database driver: nothing new to run, but job traffic competes with your application queries for connections and I/O. Polling workers add constant background load. Watch your connection limit: every worker holds a connection.
  • Redis driver: a new service to run, monitor, secure and size, with its own memory and persistence settings. In return, queue traffic stays off your primary database.

Setting up the database driver properly

# .env
QUEUE_CONNECTION=database
# Optional: keep job traffic off the primary database
# (this gives up transactional dispatch; use after_commit)
# DB_QUEUE_CONNECTION=queue_db
DB_QUEUE_RETRY_AFTER=90
Enter fullscreen mode Exit fullscreen mode
php artisan make:queue-table   # only if your app lacks the jobs migration
php artisan migrate
php artisan queue:work --tries=3 --max-time=3600
Enter fullscreen mode Exit fullscreen mode

Make sure the database server is new enough for SKIP LOCKED (MySQL 8.0.1+, MariaDB 10.6+, PostgreSQL 9.5+), run a number of workers that fits your connection limit, and use --max-time or --max-jobs so Supervisor restarts workers regularly and they release accumulated memory, as the documentation suggests.

Setting up the Redis driver properly

# .env
QUEUE_CONNECTION=redis
REDIS_QUEUE_CONNECTION=queue        # a Redis connection defined in config/database.php
REDIS_QUEUE_RETRY_AFTER=120
# phpredis (ext-redis) or predis/predis ~3.0 must be installed
Enter fullscreen mode Exit fullscreen mode
// config/database.php: a dedicated Redis connection for queues
'redis' => [
    'client' => env('REDIS_CLIENT', 'phpredis'),
    'queue' => [
        'host' => env('REDIS_QUEUE_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD'),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_QUEUE_DB', '2'),
    ],
],
Enter fullscreen mode Exit fullscreen mode

On the Redis server: appendonly yes if jobs matter, maxmemory-policy noeviction, and a memory limit with headroom and an alert before it. Set block_for to a positive number and run workers with a low --sleep, as shown above. Choose retry_after larger than your longest job timeout. If you use Horizon, run php artisan horizon:terminate on each deploy so your process monitor restarts it with the new code.

Switching from one driver to the other

Changing QUEUE_CONNECTION only affects where new jobs go. Jobs already sitting in the old store stay there and nobody works them. A safe sequence:

  1. Deploy with the new QUEUE_CONNECTION, so new dispatches go to the new driver.
  2. Start workers for the new connection.
  3. Keep one worker on the old connection until it is empty, using the connection argument: php artisan queue:work database --stop-when-empty.
  4. Restart workers with php artisan queue:restart (or php artisan reload on Laravel 13) so every process runs the new configuration.

Common mistakes

  • Using the Redis cache instance for queues with an eviction policy. Jobs vanish silently, as shown above.
  • Leavingblock_forunset and wondering why Redis is not faster to react. By default Redis polls like the database.
  • Setting a long job timeout without raisingretry_after. The job runs twice.
  • Dispatching to Redis inside a transaction and reading the row in the job. Use after_commit.
  • Not restarting workers after a deploy. They keep running old code (covered in my production mistakes article).
  • Running the database driver on an old MySQL or MariaDB. Without SKIP LOCKED concurrent workers contend for the same row.

Frequently asked questions

Is the database queue driver okay for production?

Yes. It is a supported driver and the default in Laravel's shipped config. It is a good fit until you need hundreds of short jobs per second, Horizon, or lower load on your primary database.

How many jobs per second can the database driver handle?

In my test, one worker on a local MariaDB popped and deleted about 530 empty jobs per second, and four workers about 870. Your hardware, network and job duration will change this. If your jobs take real time to run, the driver is rarely the limit.

Can I use Redis for cache and queues at the same time?

You can, but use separate Redis connections or databases, and set an eviction policy that cannot touch queue keys, which in practice means a separate instance for the queue or noeviction for both.

Will switching to Redis make my jobs run faster?

It makes picking up and finishing jobs cheaper (about 0.3 ms against 1.9 ms per job in my test), and with block_for it removes the polling delay. It does not make your job code any faster.

References: Laravel: Queues, Laravel: Horizon, Laravel: Deployment and DatabaseQueue source.


Originally published on DEV Talk.

Top comments (0)