Recently, I've been working on a FastAPI project that analyzes GitHub profiles using AI. It examines repositories, tech stacks, and README files to identify developers' strengths, weaknesses, areas for improvement, and suitable job roles.
While implementing the AI analysis functionality, I needed a better way to handle long-running tasks without keeping API requests open.
That's when I started exploring BullMQ, a job queue system that helps manage background tasks using Redis.
In this article, I'll share what I learned about queues, workers, and how to integrate BullMQ into a FastAPI application.
Why Do We Need a Job Queue?
Consider a simple FastAPI endpoint:
@router.post("/analyze")
async def analyze_profile(data: AIInput):
return await analyze(data)
This works, but what happens when the analysis takes 30–60 seconds?
The client must wait until the entire process finishes. As more users make requests, managing these long-running operations becomes increasingly challenging.
Although FastAPI supports asynchronous programming, async/await alone doesn't provide persistent job queues, automatic retries, or independent workers.
A job queue helps us separate request handling from task execution.
Understanding Queues, Workers, and BullMQ
Before implementing anything, let's understand three important components.
- Queue: Stores jobs waiting to be processed.
- Worker: A separate process that retrieves and executes jobs.
- BullMQ: A job queue library that manages queues, workers, retries, concurrency, and job states.
We'll also use Redis, which stores and coordinates our jobs.
Here's the overall architecture:
Frontend
|
v
FastAPI Server
|
Add Job
|
v
BullMQ Queue
|
Redis
|
v
BullMQ Worker
|
v
AI Analysis
|
v
Store Result in Redis
|
v
FastAPI → Frontend
Instead of waiting for the analysis to finish, FastAPI returns a job ID immediately. The worker handles the actual analysis independently.
Implementing BullMQ in FastAPI
Let's build a simple implementation.
Step 1: Install BullMQ and Set Up Redis
Install BullMQ using uv:
uv add bullmq python-dotenv
Or using pip:
pip install bullmq python-dotenv
BullMQ requires Redis. We can run Redis locally using Docker:
docker run -d --name redis -p 6379:6379 redis:7
Verify that Redis is running:
docker exec redis redis-cli ping
You should receive:
PONG
Finally, add the Redis URL to your .env file:
REDIS_URL=redis://localhost:6379
Step 2: Create a Queue
Create app/jobs/queue.py:
import os
from dotenv import load_dotenv
from bullmq import Queue
load_dotenv()
QUEUE_NAME = "portfolio-analysis"
connection = {
"connection": os.environ["REDIS_URL"]
}
queue = Queue(QUEUE_NAME, connection)
Here, we create a queue named portfolio-analysis and configure its Redis connection.
The queue holds our analysis jobs until a worker is available.
Step 3: Define the Job Processor
Create app/jobs/tasks.py:
from app.schemas import AIInput
from app.services import analyze
async def process(job, token):
content = AIInput.model_validate(job.data)
return await analyze(content)
Here's what's happening:
-
job.datacontains the information submitted when the job was created. -
model_validate()validates that information using our PydanticAIInputmodel. -
analyze()executes the actual AI analysis.
The important part is:
return await analyze(content)
BullMQ stores the returned value as the job's result, which we can retrieve later.
For example, our analyze() function might return:
{
"strengths": ["FastAPI", "TypeScript"],
"weaknesses": ["Testing"],
"recommended_role": "Backend Engineer"
}
Step 4: Create a Worker
Now create app/jobs/worker.py:
import asyncio
import signal
from bullmq import Worker
from app.jobs.queue import QUEUE_NAME, connection
from app.jobs.tasks import process
async def main():
stop = asyncio.Event()
loop = asyncio.get_running_loop()
loop.add_signal_handler(signal.SIGINT, stop.set)
loop.add_signal_handler(signal.SIGTERM, stop.set)
worker = Worker(
QUEUE_NAME,
process,
{**connection, "concurrency": 1}
)
try:
await stop.wait()
finally:
await worker.close()
if __name__ == "__main__":
asyncio.run(main())
The worker listens for jobs in our queue and calls process() whenever a job becomes available.
A few things worth understanding:
-
concurrency: 1limits this worker instance to one active job at a time. -
asyncio.Event()keeps the main coroutine waiting while the worker processes jobs. - Signal handlers allow graceful shutdown.
-
worker.close()stops fetching new jobs and allows active jobs to finish before closing.
This worker runs independently of our FastAPI server.
Step 5: Add Jobs Through FastAPI
Now let's create an endpoint that adds analysis requests to the queue.
@router.post("/analysis", status_code=202)
async def start_analysis(data: AIInput):
job = await queue.add(
"analyze-profile",
data.model_dump()
)
return {
"job_id": job.id,
"status": "queued"
}
Here, queue.add() creates a new job containing our analysis input.
data.model_dump() converts the Pydantic model into a dictionary.
Instead of waiting for the analysis, the endpoint immediately returns a response containing the job ID.
Step 6: Check Job Status and Retrieve Results
Now we need a way for the frontend to check whether the analysis has finished.
from fastapi import HTTPException
@router.get("/analysis/jobs/{job_id}")
async def get_analysis_job(job_id: str):
job = await queue.getJob(job_id)
if job is None:
raise HTTPException(
status_code=404,
detail="Analysis job not found."
)
status = await job.getState()
if status == "completed":
return {
"status": status,
"result": job.returnvalue
}
if status == "failed":
return {
"status": status,
"message": "Analysis failed."
}
return {"status": status}
Three important methods and properties:
| Method / Property | Purpose |
|---|---|
queue.getJob(job_id) |
Retrieves the job object |
job.getState() |
Returns the current job status |
job.returnvalue |
Contains the completed job's result |
The frontend can periodically request this endpoint to check the status. This approach is known as polling.
The complete job lifecycle looks like:
POST /analysis
|
v
queue.add()
|
v
Job Waiting
|
v
Worker Active
|
v
process(job)
|
v
analyze(content)
|
v
Job Completed
|
v
Result Stored
|
v
GET /analysis/jobs/{job_id}
|
v
job.returnvalue
Completed jobs must remain stored long enough for the frontend to retrieve their results.
Step 7: Run the Application
Start the FastAPI server:
uvicorn app.main:app --reload
In another terminal, start the worker:
python -m app.jobs.worker
Both processes connect to the same Redis instance.
Now, when a request reaches the analysis endpoint, FastAPI creates a job, and the worker processes it independently.
In a production application, you'll also want to configure retries, job retention, error logging, and authorization before exposing job results.
What I Learned
Integrating BullMQ helped me better understand how background job processing works beyond simply using async/await.
The main takeaway is that a job queue doesn't necessarily make our operations faster; it makes them easier to manage and scale independently of API requests.
For small background operations, FastAPI's built-in BackgroundTasks can be sufficient. However, when tasks require independent workers, retries, concurrency control, and persistent job tracking, a queue system like BullMQ becomes useful.
This was another valuable learning experience while building my GitHub analytics project, and it helped me understand how queues and workers fit into real-world backend architecture.
References
Top comments (0)