The Celery Lifecycle: How a Task Gets Registered, Queued, and Run

If you have ever needed to send an email, process a payment, or generate a report without making your user wait, you have probably run into Celery. Celery is a tool that lets you run jobs in the background, away from your main app. This article breaks down how it works, step by step, in plain language.

What Is Celery, In Simple Terms

Think of Celery like a restaurant kitchen.

• Your app (the waiter) takes an order from a customer.
• Instead of cooking the food itself, the waiter drops the order into a queue (the kitchen order rail).
• A cook (the worker) picks up the order from the rail and prepares it.
• When the food is ready, it goes to a pickup counter (the result backend) where anyone can come check if it's done.

Celery has four main players:

1. The Producer - your app, the one that creates tasks.
2. The Broker - the message queue that holds tasks until a worker is free.
3. The Worker - the process that picks up and runs the tasks.
4. The Result Backend - where results are stored, if you need them later.

In short: your app sends a task message to the broker. The broker holds it until a worker is free. The worker picks it up, runs the actual function, and (if you set one up) writes the result to the result backend. Your app can then go back and check that result backend to see what happened.

Now let's go through each part.

1. How Tasks Get Registered

Before Celery can run a task, it needs to know the task exists. This is called registration, and it happens the moment your Python code is imported - not when the task runs.

The @app.task decorator

You create a Celery app instance, then decorate any function with @app.task. That decorator does not run the function immediately. Instead, it wraps the function and adds it to a task registry - basically a dictionary that Celery keeps internally, mapping a task name to the actual function.

from celery import Celery

app = Celery("myproject")

@app.task
def send_welcome_email(user_id):
    # logic to send an email
    print(f"Sending welcome email to user {user_id}")

The moment Python imports this file, send_welcome_email gets registered under the name myproject.tasks.send_welcome_email (module path + function name, by default).

Why registration matters

Here is the key thing people miss: the producer and the worker are often two separate processes, sometimes on two separate machines. When your app calls send_welcome_email.delay(5), it does NOT run the function. It just creates a message like this:

{
  "task": "myproject.tasks.send_welcome_email",
  "args": [5],
  "kwargs": {},
  "id": "a1b2c3d4-uuid"
}

That message is just a name and some arguments - a string, basically. The actual Python function only needs to exist on the worker's side, because the worker is the one that looks up the name in its own registry and calls the real function.

This is why, if a task is not registered on the worker (say, you forgot to import that module in the worker's app), you will get a KeyError or NotRegistered error even if the producer sent the message fine. The message got sent, but nobody on the worker side knew what myproject.tasks.send_welcome_email meant.

Autodiscovery

In bigger apps (like Django), you don't manually import every task file. Celery gives you app.autodiscover_tasks(), which scans your installed apps for a tasks.py file and imports them automatically, registering every @app.task function it finds inside.

So the registration flow, in words, goes like this: Python imports the file that contains your task → the @app.task decorator fires → the function's name and a reference to it get added to the Celery app's internal registry → from that point on, the app knows this task exists and can hand it work whenever it's called.

2. The Broker URL - The Middleman

The broker is a message queue. Its only job is to hold task messages until a worker is ready to grab one. Celery does not process anything itself - it just passes messages through the broker.

You configure it with a single connection string, the broker URL:

app = Celery(
    "myproject",
    broker="redis://localhost:6379/0"
)

Common broker choices:

Broker	Example URL	Notes
Redis	redis://localhost:6379/0	Fast, simple, good for most small-to-medium apps
RabbitMQ	amqp://guest:guest@localhost:5672//	More robust, built for messaging, heavier to run
Amazon SQS	sqs://	Managed, good if you're already on AWS

What the broker URL actually controls

The broker URL tells Celery:

• Which service to talk to (Redis, RabbitMQ, etc.)
• Where it lives (host and port)
• How to authenticate (username/password, if needed)
• Which database/vhost to use (the /0 at the end of a Redis URL picks database 0, for example)

Nothing more. The broker does not know what a task "means." It just stores and forwards messages, like a mailbox. This is why Celery can swap Redis for RabbitMQ without changing a single line of your task code - only the broker URL changes.

Picture it this way: your app pushes a task message into the broker's queue. The broker just sits there holding it. On the other side, one or more workers are constantly listening to that same queue. Whichever worker is free at that moment grabs the next message in line. If you have five workers all pointed at the same broker URL, they are all pulling from the same line - so Celery scales horizontally just by adding more workers behind the same broker.

3. Execution - What Happens When a Task Runs

Once a worker pulls a task message off the broker, here is what happens, step by step:

1. Deserialize - the worker reads the JSON (or whatever serializer you're using) message and pulls out the task name, args, and kwargs.
2. Lookup - it checks its local task registry for a function matching that name.
3. Acknowledge (ack) - by default, Celery acknowledges the message either right before or right after running it, telling the broker "I've got this, you can remove it from the queue." This matters for reliability - if a worker crashes before acking, the broker can redeliver the task to another worker.
4. Run - the actual Python function executes, using the args/kwargs from the message.
5. Handle outcome - if it succeeds, the return value is captured. If it raises an exception, Celery marks the task as FAILURE and can optionally retry it, depending on your task's retry settings.
6. Store result - if a result backend is configured, the worker writes the outcome there.

Put in a single sentence: the broker hands a message to the worker, the worker matches that message to a real function in its registry, runs the function, tells the broker the message is done (so it disappears from the queue), and then, separately, writes whatever the function returned into the result backend.

Concurrency - how one worker runs many tasks

A single worker process does not run tasks one at a time by default. It has a pool of child processes or threads (the default is prefork, which uses multiple OS processes). This is controlled by the --concurrency flag when you start a worker:

celery -A myproject worker --loglevel=info --concurrency=4

This starts 4 child processes, each capable of running a task independently. So if 4 tasks arrive at once, all 4 can run in parallel, not one after another.

4. The Result Backend - Where Answers Go

Not every task needs a result backend. If you just want to fire off an email and forget about it, you don't need one. But if you need to know "did the task succeed?" or "what did it return?", you need a result backend.

app = Celery(
    "myproject",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1"
)

Notice the backend can be a different database index than the broker - /1 instead of /0. You can even use a completely different technology, like storing results in Postgres while your broker stays on Redis.

How you actually use it

result = send_welcome_email.delay(5)

result.status      # 'PENDING', 'STARTED', 'SUCCESS', 'FAILURE'
result.result      # the return value, once it's done
result.get(timeout=10)  # blocks and waits for the result (use carefully)

Behind the scenes, result.status is just Celery checking a key in the result backend (e.g., a Redis key like celery-task-meta-a1b2c3d4) to see what the worker wrote there.

Common mistake: confusing broker and backend

A lot of beginners think the broker stores results. It does not. Once a worker acks a message, the broker deletes it - it has no memory of what happened to that task. The result backend is a completely separate storage system whose only job is to hold outcomes, so your app can look them up later.

	Broker	Result Backend
Purpose	Deliver task messages to workers	Store task outcomes
Lifespan of data	Deleted once task is picked up (acked)	Kept until it expires (result_expires) or you check it
Required?	Yes, always	Only if you need to check status/results
Common choice	Redis, RabbitMQ	Redis, Postgres, MongoDB

Putting It All Together

Here is the complete lifecycle of a single task, in order:

1. You define a task with @app.task. As soon as that file is imported, the task gets added to Celery's internal registry.
2. Your app calls task.delay(...). Celery serializes the task name plus its arguments into a message - it does not run the function yet.
3. That message is pushed into the broker's queue, using whatever broker URL you configured.
4. A free worker, listening on that same broker, pulls the message off the queue.
5. The worker looks up the task name in its own registry to find the matching function.
6. The worker runs the real function, using the arguments from the message.
7. Once the function finishes (or fails), the worker acknowledges the message, so the broker removes it from the queue.
8. If a result backend is configured, the worker writes the outcome (success, failure, return value) there.
9. Your app, at any point, can check the result backend by task ID to see the status or grab the return value.

A Quick Practical Example

Since you already work with FastAPI, here's a minimal setup showing all four pieces together:

# celery_app.py
from celery import Celery

app = Celery(
    "reddo_tasks",
    broker="redis://localhost:6379/0",   # where tasks are queued
    backend="redis://localhost:6379/1",  # where results are stored
)

@app.task(bind=True, max_retries=3)
def process_payment_webhook(self, payload: dict):
    try:
        # heavy work: verify signature, update DB, notify user
        return {"status": "processed", "reference": payload.get("reference")}
    except Exception as exc:
        raise self.retry(exc=exc, countdown=10)

# main.py (FastAPI)
from celery_app import process_payment_webhook

@app.post("/webhook")
def handle_webhook(payload: dict):
    task = process_payment_webhook.delay(payload)
    return {"task_id": task.id}

# check status later
from celery.result import AsyncResult
from celery_app import app

def check_task(task_id: str):
    result = AsyncResult(task_id, app=app)
    return {"status": result.status, "result": result.result}

Run the worker separately:

celery -A celery_app worker --loglevel=info --concurrency=4

Summary

• Registration happens at import time - @app.task adds the function to an internal registry, and the worker needs that same code imported to run it.
• The broker URL tells Celery where the message queue lives. It only moves messages; it does not know what tasks mean or remember their results.
• Execution happens inside worker processes, which pull messages, look up the function, run it, and acknowledge the message so it's removed from the queue.
• The result backend is a separate store for outcomes. It's optional, and it's how your app checks status or gets return values after the fact.

Once these four pieces click, the rest of Celery - retries, chains, periodic tasks (Celery Beat), routing - is just extra features built on top of this same basic flow.