DEV Community

Alex Spinov
Alex Spinov

Posted on

Celery Has a Free Distributed Task Queue for Python Background Jobs

#python #celery #backend #devops

Celery processes millions of tasks per day with automatic retries, scheduling, and distributed workers. The standard for Python background processing.

When You Need a Task Queue

Your web request takes 30 seconds (sending emails, generating reports, processing images). Users wait. Timeouts happen.

Celery: offload long tasks to background workers. Return response immediately.

What You Get for Free

Define a task:

from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379')

@app.task(bind=True, max_retries=3)
def send_email(self, to, subject, body):
    try:
        smtp.send(to, subject, body)
    except Exception as exc:
        self.retry(exc=exc, countdown=60)  # retry in 60s
Enter fullscreen mode Exit fullscreen mode

Call it from your web app:

# This returns IMMEDIATELY
result = send_email.delay('user@test.com', 'Welcome!', 'Hello...')

# Check status later
print(result.status)  # PENDING, STARTED, SUCCESS, FAILURE
print(result.result)   # return value when done
Enter fullscreen mode Exit fullscreen mode

Features:

  • Automatic retries with exponential backoff
  • Rate limiting — @app.task(rate_limit='10/m')
  • Task routing — send different tasks to different workers
  • Scheduling — periodic tasks with Celery Beat (like cron)
  • Chaining — chain(task1.s(), task2.s(), task3.s())()
  • Groups — run tasks in parallel: group(task.s(i) for i in range(10))()

Periodic Tasks (Celery Beat) 

app.conf.beat_schedule = {
    'cleanup-every-hour': {
        'task': 'tasks.cleanup_old_data',
        'schedule': 3600.0,  # every hour
    },
    'daily-report': {
        'task': 'tasks.generate_report',
        'schedule': crontab(hour=9, minute=0),  # 9 AM daily
    },
}
Enter fullscreen mode Exit fullscreen mode

Quick Start 

pip install celery redis
celery -A tasks worker --loglevel=info
celery -A tasks beat  # for scheduled tasks
Enter fullscreen mode Exit fullscreen mode

Common Use Cases

  • Email sending (don't block the request)
  • Image/video processing
  • PDF generation
  • Data import/export
  • Webhook delivery
  • ML model inference
  • Report generation

If your Python app has any operation taking more than 2 seconds — Celery moves it to the background.

Need web scraping or data extraction? Check out my tools on Apify — get structured data from any website in minutes.

Custom solution? Email spinov001@gmail.com — quote in 2 hours.

Top comments (0)