Trigger a task with FastAPI, Celery, Docker, and Docker Compose: A Step-by-Step Guide

MD. Shahnawaz Hossan — Wed, 21 Jun 2023 18:30:08 +0000

Introduction
Project Structure
Dockerization
Up and Run
Trigger a task

Introduction

In today's digital landscape, background tasks have become a crucial aspect of many applications. Whether it's automating repetitive processes or achieving instant responses, one popular solution that comes to the rescue is Celery. In this article, we'll guide you through the process of creating and managing background tasks using Celery. By seamlessly integrating it with FastAPI, we'll show you how to containerize your project. Get ready for a straightforward and exciting exploration!

FastAPI: It's is a modern web framework for building APIs with Python. Its utilization of asynchronous programming techniques combined with the power of the Starlette framework allows FastAPI to achieve remarkable performance levels.

Celery: A python-based distributed ask queue system with built-in support for task scheduling, result tracking and fault tolerance.

Docker: An open-source platform, empowers you to bundle applications and their dependencies into compact, portable containers. These containers guarantee a consistent runtime environment, enabling your application to run seamlessly on any host system.

Docker Compose: A tool that simplifies the management of multi-container applications. It allows you to define and configure multiple Docker containers as a single application using a YAML file. It also specifies the dependencies and relationships between different services.

Project Structure

celery-with-fastapi/
├── app/
│   ├── celery/
│   │   ├── app.py
│   │   ├── worker.py
│   │   └── __init__.py
│   ├── core/
│   │   ├── config.py
│   │   └── __init__.py
│   ├── __init__.py
│   └── main.py
├── .env
├── .gitignore
├── docker-compose.yml
├── Dockerfile
├── README.md
├── requirements.txt
└── run.sh

Dockerization

Make sure you've installed Docker and docker-compose in your pc. Your Dockerfile and docker-compose.yml file would be like as follows

Dockerfile

# Base image
FROM python:3.10-buster

# set work directory
WORKDIR /app

# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# install dependencies
COPY [ "requirements.txt", "run.sh", "./"]
RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt && chmod +x ./run.sh

# copy project
COPY . .

# Set entrypoint
ENTRYPOINT bash ./run.sh

run.sh

celery -A app.celery.app worker  -l info --concurrency=2 &
celery -A  app.celery.app flower -l info & celery -A app.celery.app beat -l INFO &
uvicorn app.main:app --host 0.0.0.0 --reload --workers 2

docker-compose.yml

version: "3.8"

services:
  web:
    build: .
    ports:
      - 8001:8000
      - 5556:5555
    volumes:
      - .:/app
    depends_on:
      - celerybackend

  celerybackend:
    image: redis:latest
    ports:
      - 6379:6379
    healthcheck:
      test: [ "CMD", "redis-cli", "ping" ]
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always

Up and Run

This is the final code. Before getting started, clone the repository first.

Create a .env file and copy these lines after cloning the project:

# .env file
CELERY_BROKER_URL=redis://celerybackend:6379/0
CELERY_RESULT_BACKEND=redis://celerybackend:6379/0

git clone https://github.com/shahnawaz-pabon/celery-with-fastapi.git
cd celery-with-fastapi
docker-compose up --build

You'll be able to see the backend project running at http://0.0.0.0:8001/ and the flower dashboard running at http://0.0.0.0:5556/

Trigger a task

Open a terminal and trigger a new task

curl http://localhost:8001/tasks -H "Content-Type: application/json" --data '{"type": 0}'

To know more about celery, please visit this.

I would like to express my heartfelt gratitude to each and every one of you who took the time to read my articles. Your support and engagement mean the world to me. 🙏🙏🙏