Quick answer
Celery is a Python task queue: your web code sends a job to a broker, and a separate worker runs it, so slow work does not block the web request. This post sends three shop orders to a real Celery worker, charges the two good ones, and watches the third fail and retry because its total is missing. It fits emails, charges, and image processing.
Picture a small online shop in Karachi. A customer clicks buy, and the checkout code charges the card, updates stock, and sends a confirmation email, all inside one web request. Then the email service stalls, the request times out, and the customer clicks buy again. The card is charged twice. A task queue breaks the chain: the web request only records the order and hands it to Celery, and a worker charges it and sends the email in its own time. If the worker crashes halfway, the order is still in the queue.
What I would check before using it
My first check: does the work need to outlive the web request? An email, a charge, or a resize can take seconds, while a web request should answer fast. When the slow step can fail on its own, a queue earns its place. A quick database write needs only plain code in the handler. For same-time work in one process, Python asyncio may be simpler.
My second check is who runs the broker. Celery needs a broker to hold the messages, usually Redis or RabbitMQ, one more service to run. If nobody owns it, a managed queue from your cloud provider is the honest alternative. The demo uses the filesystem broker, so it runs with no server.
My third check is what happens when a task fails. Every queue needs a plan for the job that can never succeed: how many retries, and where the failure is recorded. Without one, a bad task retries forever while good orders wait. The demo allows two retries, then fails for good.
My fourth check: are the tasks safe to run twice? A worker can crash after charging a card but before recording it, and Celery may run the task again. If charging twice is a disaster, the task needs an idempotency key, an order id checked before charging.
My verdict, as judgement: adopt Celery when background work must not block the user and must survive a crash. Skip it for a small script or a prototype. The recommendation flips the moment nobody owns the broker, or the queue costs more than the job.
Three orders, one worker, and a missing total
Three orders arrive at the same small shop. Ayesha buys a kurta and a dupatta for Rs 12,400, Bilal buys sneakers for Rs 7,800, and Hina buys a handbag and a wallet with no total. That bad record breaks the first real order.
One file defines the Celery app and the task. The broker is the filesystem transport, so the queue is a folder on disk and results go to SQLite. The task checks the total, waits one second, and returns a receipt string.
"""Order tasks for The Practical Developer (Celery demo).
A small online shop sends each order to a Celery worker instead of
charging the card inside the web request. The worker charges the order
and returns a receipt string. One order arrives without a total, so the
worker raises ValueError, retries twice, then fails for good.
Broker: the filesystem transport, so no Redis or RabbitMQ server is
needed. Result backend: a local SQLite database through Celery's
database backend, because Celery 5.7 has no working filesystem result
backend (backend="filesystem://" fails with ModuleNotFoundError).
Filesystem transport quirk, found in kombu's source: the producer
writes messages to data_folder_out and the consumer reads from
data_folder_in, so the two folders must point at the same directory
(or the worker needs the folders swapped). Both are set to
"data/queue" here. Run from this folder:
celery -A tasks worker --loglevel=INFO
"""
import time
from celery import Celery
app = Celery(
"shop_tasks",
broker="filesystem://",
backend="db+sqlite:///results.db",
broker_transport_options={
"data_folder_in": "data/queue",
"data_folder_out": "data/queue",
"data_folder_processed": "data/processed",
},
task_serializer="json",
result_serializer="json",
accept_content=["json"],
)
@app.task(
bind=True,
autoretry_for=(ValueError,),
retry_kwargs={"max_retries": 2, "countdown": 1},
)
def process_order(self, order_id, customer, items, total):
"""Charge one order and return a receipt string."""
if total is None:
raise ValueError(f"order {order_id} has no total to charge")
if total <= 0:
raise ValueError(f"order {order_id} has an invalid total: {total}")
time.sleep(1)
joined = ", ".join(items)
return (
f"Order {order_id} for {customer}: charged Rs {total} "
f"for {joined}; confirmation email queued"
)
Two small scripts send the orders: the first sends order 101 before any worker is running, and the second sends orders 102 and 103 after the worker starts, then collects the results.
# dispatch_first.py: send one order BEFORE any worker is running.
from tasks import process_order
result = process_order.delay(
order_id=101, customer="Ayesha", items=["kurta", "dupatta"], total=12400
)
print(f"Sent order 101 for Ayesha (Rs 12400, kurta, dupatta)")
print(f"Task id: {result.id}")
print(f"State right after sending: {result.state}")
print("No worker is running yet, so the order waits in the queue.")
# dispatch_rest.py: send the rest, then collect every result.
from tasks import app, process_order
def _first_id():
with open("first_task_id.txt") as handle:
return handle.read().strip()
sent = {}
sent[102] = process_order.delay(
order_id=102, customer="Bilal", items=["sneakers"], total=7800
)
print("Sent order 102 for Bilal (Rs 7800, sneakers)")
sent[103] = process_order.delay(
order_id=103, customer="Hina", items=["handbag", "wallet"], total=None
)
print("Sent order 103 for Hina (handbag, wallet, NO total)")
print("Waiting for results...")
for order_id in (101, 102, 103):
result = sent[order_id] if order_id != 101 else app.AsyncResult(_first_id())
try:
receipt = result.get(timeout=20)
print(f"Order {order_id}: {result.state} - {receipt}")
except Exception as exc:
print(f"Order {order_id}: {result.state} - {type(exc).__name__}: {exc}")
Here is the real output. Order 101 was sent with no worker running, so its state was PENDING, until the worker started and charged it.
$ python3 dispatch_first.py
Sent order 101 for Ayesha (Rs 12400, kurta, dupatta)
Task id: 2c76fcc1-9567-468c-8fb8-1a8406a1ed4c
State right after sending: PENDING
No worker is running yet, so the order waits in the queue.
$ celery -A tasks worker --loglevel=INFO --concurrency=1
-------------- celery@htch-runtime v5.7.0 (collider)
celery@htch-runtime ready.
$ python3 dispatch_rest.py
Sent order 102 for Bilal (Rs 7800, sneakers)
Sent order 103 for Hina (handbag, wallet, NO total)
Waiting for results...
Order 101: SUCCESS - Order 101 for Ayesha: charged Rs 12400 for kurta, dupatta; confirmation email queued
Order 102: SUCCESS - Order 102 for Bilal: charged Rs 7800 for sneakers; confirmation email queued
Order 103: FAILURE - ValueError: order 103 has no total to charge
The worker log agrees. Order 101 succeeded in about 1.1 seconds, and order 103 was retried twice, one second apart, before failing for good. Task ids are shortened for readability.
Task tasks.process_order[...] succeeded in 1.10s: 'Order 101 for Ayesha: charged Rs 12400 for kurta, dupatta; confirmation email queued'
Task tasks.process_order[...] succeeded in 1.07s: 'Order 102 for Bilal: charged Rs 7800 for sneakers; confirmation email queued'
Task tasks.process_order[...] retry: Retry in 1s: ValueError('order 103 has no total to charge')
Task tasks.process_order[...] retry: Retry in 1s: ValueError('order 103 has no total to charge')
Task tasks.process_order[...] raised unexpected: ValueError('order 103 has no total to charge')
Order 103 went wrong: with no total, the task raised ValueError, retried twice, and then failed. A missing total never appears on its own, so retrying it is pointless. Retry only failures that time can fix, and fail fast on bad data. Validate the total before the order reaches the queue, and dead-letter failed tasks instead of dropping them. The RabbitMQ post shows that pattern with the same three orders.
Common mistakes and how to fix them
First, sending a task before the worker starts and assuming it is lost. The symptom is a result stuck at PENDING. The fix is to start the worker: the task sits in the broker and runs as soon as one starts, as order 101 did here.
Second, retrying bad data forever. The symptom is a worker log full of retries for a task that can never succeed, like order 103, while good orders wait. The fix is to retry only transient errors, and fail fast on bad data.
Third, using the filesystem transport with separate in and out folders. The symptom is tasks stuck at PENDING forever even with a worker running, because kombu writes to data_folder_out and reads from data_folder_in. The fix is to point both folders at the same directory. For real use, switch the broker to Redis.
Fourth, setting the result backend to filesystem. The symptom is ModuleNotFoundError: No module named 'filesystem' the moment a task is sent, because Celery 5.7 has no working filesystem result backend. The fix is a real backend: db+sqlite:///results.db for learning, Redis for production.
Frequently Asked Questions
Do I need Redis to use Celery? No, not to learn it. This demo runs on the filesystem broker with a SQLite backend and no server. For real traffic, use Redis or RabbitMQ.
What happens to a task if no worker is running? It waits in the broker. Order 101 stayed PENDING until the worker started, then was processed. Nothing was lost.
How is Celery different from asyncio? Asyncio runs jobs at the same time inside one process, and everything is lost if that process dies. Celery hands jobs to separate workers through a broker, so they survive a crash and can run on another machine.
Can I use Celery with Django or Flask? Yes. Celery has first-class Django support, and Flask apps call tasks the same way this demo does. The Django and Flask posts cover those frameworks.
What to remember
- Send slow work to a worker with .delay(), and let the web request answer fast.
- A task sent with no worker running waits in the broker, it is not lost.
- Retry only failures that time can fix, and fail fast on bad data.
- Make tasks safe to run twice, with an idempotency check before charging.
- Own the broker: the filesystem transport is for learning, Redis is for real traffic.
Where this leaves you
Celery earns its place the day a slow step starts breaking web requests that should answer in a blink. The demo proved the three things that matter: an order sent before the worker started was processed anyway, good orders were charged and returned receipts, and the broken order retried twice and then failed loudly instead of vanishing. That is the whole job. Start with one task, one worker, and the filesystem broker on your own machine, then move the broker to Redis when real users arrive. An empty queue means fast requests. A full one is a list of real jobs.
Further reading
- Celery docs, getting started, https://docs.celeryq.dev/en/stable/getting-started/introduction.html
- Celery docs, tasks, https://docs.celeryq.dev/en/stable/userguide/tasks.html
- Celery docs, choosing a broker, https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/index.html
- RabbitMQ Tutorial: An Order Queue That Never Loses Work, https://practicaldevblog.blogspot.com/2026/11/rabbitmq-python-order-queue.html
- Redis Explained: The In-Memory Store That Speeds Things Up, https://practicaldevblog.blogspot.com/2026/10/redis-explained-in-memory-store-that.html
- Python asyncio: Run Jobs at the Same Time Without Threads, https://practicaldevblog.blogspot.com/2026/11/python-asyncio-running-jobs-at-same.html
- FastAPI Explained: Python APIs at Full Speed, https://practicaldevblog.blogspot.com/2026/10/fastapi-explained-python-apis-at-full.html
- Django Explained: Python's Batteries-Included Framework, https://practicaldevblog.blogspot.com/2026/10/django-explained-pythons-batteries.html
- Flask Explained: The Small Framework That Does Just Enough, https://practicaldevblog.blogspot.com/2026/10/flask-explained-small-framework-that.html
Waqas Shahid is a software engineer from Karachi with more than 12 years of experience, including 6+ years in engineering leadership. He writes The Practical Developer.
Top comments (0)