DEV Community

Cover image for Using BullMQ with FastAPI: A Practical Guide to Queues and Workers
Subhamoy Datta
Subhamoy Datta

Posted on Edited on AI-assisted

Using BullMQ with FastAPI: A Practical Guide to Queues and Workers

Recently, I've been working on a FastAPI project that analyzes GitHub profiles using AI. It examines repositories, tech stacks, and README files to identify developers' strengths, weaknesses, areas for improvement, and suitable job roles.

While implementing the AI analysis functionality, I needed a better way to handle long-running tasks without keeping API requests open.

That's when I started exploring BullMQ, a job queue system that helps manage background tasks using Redis.

In this article, I'll share what I learned about queues, workers, and how to integrate BullMQ into a FastAPI application.

Why Do We Need a Job Queue?

Consider a simple FastAPI endpoint:

@router.post("/analyze")
async def analyze_profile(data: AIInput):
    return await analyze(data)
Enter fullscreen mode Exit fullscreen mode

This works, but what happens when the analysis takes 30–60 seconds?

The client must wait until the entire process finishes. As more users make requests, managing these long-running operations becomes increasingly challenging.

Although FastAPI supports asynchronous programming, async/await alone doesn't provide persistent job queues, automatic retries, or independent workers.

A job queue helps us separate request handling from task execution.

Understanding Queues, Workers, and BullMQ

Before implementing anything, let's understand three important components.

  • Queue: Stores jobs waiting to be processed.
  • Worker: A separate process that retrieves and executes jobs.
  • BullMQ: A job queue library that manages queues, workers, retries, concurrency, and job states.

We'll also use Redis, which stores and coordinates our jobs.

Here's the overall architecture:

           Frontend
              |
              v
        FastAPI Server
              |
         Add Job
              |
              v
       BullMQ Queue
              |
            Redis
              |
              v
        BullMQ Worker
              |
              v
         AI Analysis
              |
              v
     Store Result in Redis
              |
              v
     FastAPI → Frontend
Enter fullscreen mode Exit fullscreen mode

Instead of waiting for the analysis to finish, FastAPI returns a job ID immediately. The worker handles the actual analysis independently.

Implementing BullMQ in FastAPI

Let's build a simple implementation.

Step 1: Install BullMQ and Set Up Redis

Install BullMQ using uv:

uv add bullmq python-dotenv
Enter fullscreen mode Exit fullscreen mode

Or using pip:

pip install bullmq python-dotenv
Enter fullscreen mode Exit fullscreen mode

BullMQ requires Redis. We can run Redis locally using Docker:

docker run -d --name redis -p 6379:6379 redis:7
Enter fullscreen mode Exit fullscreen mode

Verify that Redis is running:

docker exec redis redis-cli ping
Enter fullscreen mode Exit fullscreen mode

You should receive:

PONG
Enter fullscreen mode Exit fullscreen mode

Finally, add the Redis URL to your .env file:

REDIS_URL=redis://localhost:6379
Enter fullscreen mode Exit fullscreen mode

Step 2: Create a Queue

Create app/jobs/queue.py:

import os
from dotenv import load_dotenv
from bullmq import Queue

load_dotenv()

QUEUE_NAME = "portfolio-analysis"

connection = {
    "connection": os.environ["REDIS_URL"]
}

queue = Queue(QUEUE_NAME, connection)
Enter fullscreen mode Exit fullscreen mode

Here, we create a queue named portfolio-analysis and configure its Redis connection.

The queue holds our analysis jobs until a worker is available.

Step 3: Define the Job Processor

Create app/jobs/tasks.py:

from app.schemas import AIInput
from app.services import analyze

async def process(job, token):
    content = AIInput.model_validate(job.data)
    return await analyze(content)
Enter fullscreen mode Exit fullscreen mode

Here's what's happening:

  • job.data contains the information submitted when the job was created.
  • model_validate() validates that information using our Pydantic AIInput model.
  • analyze() executes the actual AI analysis.

The important part is:

return await analyze(content)
Enter fullscreen mode Exit fullscreen mode

BullMQ stores the returned value as the job's result, which we can retrieve later.

For example, our analyze() function might return:

{
  "strengths": ["FastAPI", "TypeScript"],
  "weaknesses": ["Testing"],
  "recommended_role": "Backend Engineer"
}
Enter fullscreen mode Exit fullscreen mode

Step 4: Create a Worker

Now create app/jobs/worker.py:

import asyncio
import signal
from bullmq import Worker

from app.jobs.queue import QUEUE_NAME, connection
from app.jobs.tasks import process

async def main():
    stop = asyncio.Event()
    loop = asyncio.get_running_loop()

    loop.add_signal_handler(signal.SIGINT, stop.set)
    loop.add_signal_handler(signal.SIGTERM, stop.set)

    worker = Worker(
        QUEUE_NAME,
        process,
        {**connection, "concurrency": 1}
    )

    try:
        await stop.wait()
    finally:
        await worker.close()

if __name__ == "__main__":
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

The worker listens for jobs in our queue and calls process() whenever a job becomes available.

A few things worth understanding:

  • concurrency: 1 limits this worker instance to one active job at a time.
  • asyncio.Event() keeps the main coroutine waiting while the worker processes jobs.
  • Signal handlers allow graceful shutdown.
  • worker.close() stops fetching new jobs and allows active jobs to finish before closing.

This worker runs independently of our FastAPI server.

Step 5: Add Jobs Through FastAPI

Now let's create an endpoint that adds analysis requests to the queue.

@router.post("/analysis", status_code=202)
async def start_analysis(data: AIInput):
    job = await queue.add(
        "analyze-profile",
        data.model_dump()
    )

    return {
        "job_id": job.id,
        "status": "queued"
    }
Enter fullscreen mode Exit fullscreen mode

Here, queue.add() creates a new job containing our analysis input.

data.model_dump() converts the Pydantic model into a dictionary.

Instead of waiting for the analysis, the endpoint immediately returns a response containing the job ID.

Step 6: Check Job Status and Retrieve Results

Now we need a way for the frontend to check whether the analysis has finished.

from fastapi import HTTPException

@router.get("/analysis/jobs/{job_id}")
async def get_analysis_job(job_id: str):
    job = await queue.getJob(job_id)

    if job is None:
        raise HTTPException(
            status_code=404,
            detail="Analysis job not found."
        )

    status = await job.getState()

    if status == "completed":
        return {
            "status": status,
            "result": job.returnvalue
        }

    if status == "failed":
        return {
            "status": status,
            "message": "Analysis failed."
        }

    return {"status": status}
Enter fullscreen mode Exit fullscreen mode

Three important methods and properties:

Method / Property Purpose
queue.getJob(job_id) Retrieves the job object
job.getState() Returns the current job status
job.returnvalue Contains the completed job's result

The frontend can periodically request this endpoint to check the status. This approach is known as polling.

The complete job lifecycle looks like:

POST /analysis
      |
      v
   queue.add()
      |
      v
   Job Waiting
      |
      v
   Worker Active
      |
      v
   process(job)
      |
      v
   analyze(content)
      |
      v
   Job Completed
      |
      v
   Result Stored
      |
      v
GET /analysis/jobs/{job_id}
      |
      v
   job.returnvalue
Enter fullscreen mode Exit fullscreen mode

Completed jobs must remain stored long enough for the frontend to retrieve their results.

Step 7: Run the Application

Start the FastAPI server:

uvicorn app.main:app --reload
Enter fullscreen mode Exit fullscreen mode

In another terminal, start the worker:

python -m app.jobs.worker
Enter fullscreen mode Exit fullscreen mode

Both processes connect to the same Redis instance.

Now, when a request reaches the analysis endpoint, FastAPI creates a job, and the worker processes it independently.

In a production application, you'll also want to configure retries, job retention, error logging, and authorization before exposing job results.

What I Learned

Integrating BullMQ helped me better understand how background job processing works beyond simply using async/await.

The main takeaway is that a job queue doesn't necessarily make our operations faster; it makes them easier to manage and scale independently of API requests.

For small background operations, FastAPI's built-in BackgroundTasks can be sufficient. However, when tasks require independent workers, retries, concurrency control, and persistent job tracking, a queue system like BullMQ becomes useful.

This was another valuable learning experience while building my GitHub analytics project, and it helped me understand how queues and workers fit into real-world backend architecture.


References

Top comments (0)