DEV Community

Vigilmon
Vigilmon

Posted on

How to Monitor Your Celery Workers with Vigilmon

#python #celery #monitoring #backend

How to Monitor Your Celery Workers with Vigilmon

Celery is the standard Python background task queue — used for sending emails, processing uploads, running ML inference, and handling any work that shouldn't block an HTTP response. When Celery workers crash or stall, jobs queue up silently. No HTTP error page. No 500 status. Just a growing backlog and eventually, users wondering why their invoice never arrived.

This guide shows how to monitor Celery workers with Vigilmon — catching worker failures before they become user-facing problems.

How Celery Fails (And Why It's Hard to Detect)

Celery worker failures are sneaky:

  • Worker crash — the worker process exits silently; pending tasks stay queued
  • Task timeout — a task hangs indefinitely, consuming a worker slot and blocking all other tasks
  • Memory leak — workers OOM after processing many tasks; max_tasks_per_child fixes this but isn't always configured
  • Beat not running — Celery Beat (the scheduler) crashes; scheduled tasks stop firing
  • Broker connection lost — workers disconnect from Redis/RabbitMQ and fail to reconnect

Step 1: Add a Celery Health Heartbeat Check

The most reliable Celery health monitor is a scheduled task that pings a heartbeat URL:

# tasks.py
from celery import Celery
import requests
import os

app = Celery('myapp', broker=os.getenv('CELERY_BROKER_URL', 'redis://localhost:6379/0'))

@app.task
def health_heartbeat():
    """Ping Vigilmon heartbeat — proves Celery Beat + workers are functional."""
    heartbeat_url = os.getenv('VIGILMON_HEARTBEAT_URL')
    if heartbeat_url:
        requests.get(heartbeat_url, timeout=5)
Enter fullscreen mode Exit fullscreen mode 
# celery_beat_schedule.py
from celery.schedules import crontab

app.conf.beat_schedule = {
    'health-heartbeat': {
        'task': 'tasks.health_heartbeat',
        'schedule': 60.0,  # Every 60 seconds
    },
}
Enter fullscreen mode Exit fullscreen mode

In Vigilmon, create a Heartbeat monitor with a 2-minute expected interval and a 90-second grace period. If the heartbeat stops firing, you know either Beat crashed or the workers are down.

Step 2: Expose a Celery Worker Health Endpoint

For immediate health checks, add an HTTP endpoint that pings the broker and inspects active workers:

# health_endpoint.py
from flask import Flask, jsonify
from celery import Celery
import os

flask_app = Flask(__name__)
celery_app = Celery(broker=os.getenv('CELERY_BROKER_URL', 'redis://localhost:6379/0'))

@flask_app.route('/health/celery')
def celery_health():
    try:
        # Ping broker
        conn = celery_app.connection()
        conn.ensure_connection(max_retries=1, timeout=5)
        conn.close()

        # Check for active workers
        inspect = celery_app.control.inspect(timeout=2.0)
        stats = inspect.stats()
        worker_count = len(stats) if stats else 0

        if worker_count == 0:
            return jsonify({'status': 'degraded', 'workers': 0, 'error': 'No workers responding'}), 503

        return jsonify({
            'status': 'ok',
            'workers': worker_count,
            'broker': 'connected'
        }), 200
    except Exception as e:
        return jsonify({'status': 'error', 'error': str(e)}), 503

if __name__ == '__main__':
    flask_app.run(host='0.0.0.0', port=9100)
Enter fullscreen mode Exit fullscreen mode

Monitor http://your-app:9100/health/celery with Vigilmon.

Step 3: Monitor Task Failure Rates (Django Celery Results)

If you use django-celery-results, you can query the failure rate:

# management/commands/celery_health_check.py
from django.core.management.base import BaseCommand
from django_celery_results.models import TaskResult
from django.utils import timezone
from datetime import timedelta
import requests
import os

class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        # Check failure rate in last 10 minutes
        ten_min_ago = timezone.now() - timedelta(minutes=10)
        recent = TaskResult.objects.filter(date_done__gte=ten_min_ago)
        total = recent.count()
        failures = recent.filter(status='FAILURE').count()

        failure_rate = (failures / total * 100) if total > 0 else 0

        # Only ping heartbeat if failure rate is acceptable
        if failure_rate < 20:  # < 20% failure rate
            url = os.getenv('VIGILMON_HEARTBEAT_URL')
            if url:
                requests.get(url, timeout=5)
Enter fullscreen mode Exit fullscreen mode

Schedule this as a management command cron:

*/5 * * * * cd /app && python manage.py celery_health_check
Enter fullscreen mode Exit fullscreen mode

Step 4: Monitor Celery Flower Dashboard

If you run Celery Flower (the monitoring UI), Vigilmon can monitor it directly:

# Start Flower
celery -A myapp flower --port=5555
Enter fullscreen mode Exit fullscreen mode

Monitor http://your-host:5555/ — Flower returns 200 when it's up and connected to the broker.

Celery Monitoring Coverage Table

Monitor Type Target Alert Condition
Heartbeat Beat + worker health task No ping in > 2 min
HTTP(S) Worker health endpoint Status != 200
HTTP(S) Celery Flower dashboard Status != 200
Heartbeat Failure rate check cron No ping if failure rate > 20%

Conclusion

Celery workers are invisible until they fail — and when they fail, the impact is delayed and insidious. A heartbeat that fires only when workers are healthy is the most reliable way to know your Celery setup is running.

Set up Celery worker monitoring free at vigilmon.online

Top comments (0)