Setup Celery Worker with Supervisord on elastic beanstalk via .ebextensions

Introduction: The Backbone of Scalable Applications

Building a robust, scalable application often means dealing with tasks that require more than a single server or thread can handle efficiently. Whether it's processing images, sending emails, or performing data-heavy computations, offloading these tasks to a task queue is a best practice. For Text2Infographic, my AI-powered infographic generator, the challenge was clear: I needed to handle numerous simultaneous job submissions efficiently while maintaining a smooth user experience. This led me to adopt Celery, a powerful distributed task queue, and Supervisord, a process management system, all deployed seamlessly on AWS Elastic Beanstalk using the power of .ebextensions.

Here’s a step-by-step guide to how I set up a Celery worker with Supervisord on Elastic Beanstalk. But first, let’s unpack the key components of this setup and why they are essential.

What Is Celery?

At its core, Celery is a distributed task queue system that allows you to offload time-consuming tasks to separate processes or servers. It's widely used in Python applications to execute background jobs asynchronously or on a schedule. For Text2Infographic, Celery was the perfect solution for handling the computationally intensive process of generating custom infographics from user input.

Some benefits of using Celery:

Asynchronous Execution: Tasks can run in the background without blocking the main application.
Scalability: Easily add more workers to handle increased load.
Extensibility: Integrates with various message brokers like RabbitMQ or Redis.

What Is Supervisord?

Managing processes like Celery workers manually can become a hassle, especially when you need them to restart automatically after a crash or during deployments. Supervisord is a lightweight process control system that solves this problem by keeping an eye on your processes and ensuring they stay up and running.

With Supervisord, you can:

Automatically restart Celery workers if they fail.
Simplify process management with a single configuration file.
Log process activity for better debugging and monitoring.

What Is AWS Elastic Beanstalk?

AWS Elastic Beanstalk is a fully managed service that automates the deployment, scaling, and management of applications. It abstracts much of the complexity of infrastructure management, allowing developers to focus on writing code instead of configuring servers. Elastic Beanstalk supports various environments, from simple web servers to more complex setups like Celery workers.

For Text2Infographic, Elastic Beanstalk's scalability and simplicity were invaluable. As user demand fluctuates, the ability to scale worker instances dynamically ensures that jobs are processed efficiently, even during peak times.

What Are .ebextensions?

.ebextensions is a feature of Elastic Beanstalk that allows you to customize your environment during deployment. With .ebextensions configuration files, you can:

Install necessary software and dependencies.
Configure services like Supervisord and Celery workers.
Add environment variables and manage permissions.
This makes it possible to seamlessly integrate Celery and Supervisord into your Elastic Beanstalk deployment without manual intervention every time you deploy.

Why Celery for Text2Infographic?

Text2Infographic is designed to help marketers and content creators transform blog posts into stunning infographics. Each infographic generation request is computationally intensive, involving AI-based topic research, design optimization, and sourcing vector graphics. To maintain a seamless user experience, these tasks must be offloaded to a background worker that can handle multiple requests concurrently. Celery’s asynchronous task handling and scalability made it the obvious choice.

Why Supervisord?

While Elastic Beanstalk can manage web servers natively, it doesn’t have built-in support for background processes like Celery workers. Enter Supervisord. It acts as a supervisor for the Celery worker process, ensuring that it runs continuously and restarts automatically if it fails. This reliability is crucial for processing infographic generation requests without interruptions.

With the stage set, let’s dive into the technical details of configuring Celery, Supervisord, and eb_extensions on Elastic Beanstalk to create a scalable and efficient task queue for your application.

Step-by-Step: Setting Up Celery with Supervisord on Elastic Beanstalk

In this section, we'll walk through the .ebextensions files required to set up Celery with Supervisord on Elastic Beanstalk. Each step is explained in detail, with tips to help you avoid common pitfalls.

1. Installing Supervisord
File: 01_install_supervisord.config

This file installs Supervisord and sets up a non-root user for running processes securely.

commands:
  01_install_pip:
    command: "yum install -y python3-pip"
    ignoreErrors: true
  02_install_supervisor:
    command: "/usr/bin/pip3 install supervisor"
  03_create_nonroot_user:
    command: "useradd -r -M -s /sbin/nologin nonrootuser || true"
    ignoreErrors: true

Explanation:

Install pip: Ensures Python's package manager is available.
Install Supervisor: Uses pip to install Supervisord, a lightweight and powerful process manager.
Create non-root user: Adds a restricted user (nonrootuser) with no login shell or home directory. Running processes as a non-root user is a security best practice.

💡 Tip: Always use ignoreErrors: true when commands might fail during repeated deployments. This ensures your deployment won’t fail if the user or package already exists.

2. Cleaning Up Stale Processes
File: 02_cleanup_existing_supervisord.config

This file handles cleanup of old Supervisord instances and socket files that might linger between deployments.

commands:
  kill_existing_supervisord:
    command: "pkill supervisord || true"
    ignoreErrors: true
  remove_stale_socket:
    command: "rm -f /tmp/supervisor.sock"
    ignoreErrors: true

Explanation:

Kill existing Supervisord: Ensures no stray Supervisord processes are running. The || true part ensures this command won't throw errors if no process is found.
Remove stale socket: Deletes any old Supervisord socket files, which could prevent Supervisord from starting.

💡 Tip: Cleaning up sockets and processes is essential in environments like Elastic Beanstalk, where deployments can sometimes leave behind remnants of previous configurations.

3. Configuring Celery with Supervisord
File: 03_celery_configuration.config

This file creates the Supervisord configuration file and starts the Celery worker process.

files:
  "/etc/supervisord.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      [unix_http_server]
      file=/tmp/supervisor.sock
      chmod=0770
      chown=root:nonrootuser

      [supervisord]
      logfile=/var/log/supervisord.log
      logfile_maxbytes=50MB
      logfile_backups=10
      loglevel=info
      pidfile=/tmp/supervisord.pid
      nodaemon=false
      minfds=1024
      minprocs=200
      user=root

      [rpcinterface:supervisor]
      supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

      [supervisorctl]
      serverurl=unix:///tmp/supervisor.sock

      [program:celery]
      command=celery -A application.celery worker --loglevel=INFO
      directory=/var/app/current
      autostart=true
      autorestart=true
      startsecs=10
      stopwaitsecs=600
      stdout_logfile=/var/log/celery_worker.log
      stderr_logfile=/var/log/celery_worker.err.log
      environment=PATH="/var/app/venv/staging-LQM1lest/bin:$PATH"
      user=nonrootuser

Explanation:

Unix socket for control: The unix_http_server section creates a secure socket for interacting with Supervisord.
Logging: Logs are stored in /var/log/supervisord.log, with a rotation policy to prevent disk usage from spiraling out of control.
Celery program block:
Command: Runs the Celery worker with the application configuration.
Autostart and autorestart: Ensures Celery starts automatically on deployment and restarts if it fails.
Logs: Logs Celery’s output to /var/log/celery_worker.log and /var/log/celery_worker.err.log.
Environment: Ensures the correct Python virtual environment is used.

💡 Tip: Use directory=/var/app/current to point Supervisord to the application’s deployment directory, which is updated with each Elastic Beanstalk deployment.

4. Starting Supervisord
File: 03_celery_configuration.config (continued)

container_commands:
  01_start_supervisor:
    command: "supervisord -c /etc/supervisord.conf"

Explanation:

Container commands: These run after your application is deployed but before the environment is marked as ready. Starting Supervisord here ensures your Celery worker is running when the app goes live.

💡 Tip: Elastic Beanstalk processes container commands in alphabetical order, so prefix your commands with numbers like 01_ to control the execution order.

Fun Tricks with eb_extensions

Debugging Made Easy: If something doesn’t work, add a temporary container command to print environment variables or list directory contents:

container_commands:
  99_debug:
    command: "env > /tmp/env_vars.log && ls -al /var/app/current > /tmp/deployment_files.log"

Check the logs in /var/log/eb-activity.log.

Reuse Common Configs: Store shared configuration snippets in a separate YAML file, then include them in multiple .ebextensions files using the include directive (unofficially supported).

This setup ensures your Celery workers are managed efficiently with Supervisord, scaling alongside your Elastic Beanstalk application. Whether you're handling infographic generation or any other background task, this approach offers reliability, scalability, and peace of mind.