PostgreSQL as a Celery Broker

#postgres #queue #celery

I’m currently using Redis as the broker for Celery, but since I already have PostgreSQL running, I wondered if I could use it as a broker instead. This would eliminate the need for a separate Redis server, saving cost a bit.

I came across this GitHub issue discussing PostgreSQL as a potential Celery broker. This led me to discover an interesting alternative task queue called BQ, which is built specifically to use PostgreSQL for task management.

What caught my attention with BQ is that it use mix way to process tasks: the traditional polling method and a pub-sub model, which is pretty unique. For those unfamiliar, BQ’s pub-sub system leverages PostgreSQL’s NOTIFY function. The way it works is that when a new task is added, NOTIFY sends a signal to listening clients, so they can act immediately—no more constant polling to check for new tasks, which can be more efficient.

One thing I’ve been mindful of when working with task queues is the noisy neighbor problem, where one task queue can consume too many resources and affect the performance of others. I haven’t fully explored how BQ handles this, but it’s definitely something to look into if you’re considering a PostgreSQL-based task queue.