We moved a small job queue from Redis onto Postgres two months ago, mostly to get transactions and a single backup story. Here's what actually changed.
Throughput was never the problem. We run about 40 jobs a second at peak, and SELECT ... FOR UPDATE SKIP LOCKED handles that without breaking a sweat. The counter arguments about Postgres queues are usually about ten thousand a second, which we will never see.
What did need thought was stuck jobs. A worker that dies mid-job leaves a locked row, and SKIP LOCKED will skip it forever. We added a locked_until column and a reaper that clears expired locks every thirty seconds. That reaper is the actual queue implementation; the select is easy.
The other surprise was vacuum. A table that churns through rows fast needs its autovacuum settings tuned, or the table bloats and the queue gets slower for no obvious reason.
Two months in: fewer moving parts, one backup, and a reaper job I check on Mondays. I'd make the same trade again.
Top comments (2)
The reaper being the real queue implementation matches what we've seen too. One thing that helped us was a per-job heartbeat that extends locked_until while the worker is alive, so long jobs are not reclaimed mid-run just because the lease was sized for the average case. For the vacuum side, lowering autovacuum_vacuum_scale_factor on that one table through a storage parameter keeps bloat in check without touching global settings. Do you keep finished jobs in the same table, or move them out?
Every job queue is just a cron job that got ideas above its station.