DEV Community

Kamal Mustafa
Kamal Mustafa

Posted on

2

PostgreSQL as a Celery Broker

I’m currently using Redis as the broker for Celery, but since I already have PostgreSQL running, I wondered if I could use it as a broker instead. This would eliminate the need for a separate Redis server, saving cost a bit.

I came across this GitHub issue discussing PostgreSQL as a potential Celery broker. This led me to discover an interesting alternative task queue called BQ, which is built specifically to use PostgreSQL for task management.

What caught my attention with BQ is that it use mix way to process tasks: the traditional polling method and a pub-sub model, which is pretty unique. For those unfamiliar, BQ’s pub-sub system leverages PostgreSQL’s NOTIFY function. The way it works is that when a new task is added, NOTIFY sends a signal to listening clients, so they can act immediately—no more constant polling to check for new tasks, which can be more efficient.

One thing I’ve been mindful of when working with task queues is the noisy neighbor problem, where one task queue can consume too many resources and affect the performance of others. I haven’t fully explored how BQ handles this, but it’s definitely something to look into if you’re considering a PostgreSQL-based task queue.

AWS Security LIVE!

Join us for AWS Security LIVE!

Discover the future of cloud security. Tune in live for trends, tips, and solutions from AWS and AWS Partners.

Learn More

Top comments (1)

Collapse
 
k4ml profile image
Kamal Mustafa

It turns out that django has an active DEP for background tasks queue implementation and there's PoC already.

github.com/django/deps/blob/main/a...

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay