Processing 5 Million Records with Django and Celery: A Practical Pattern

A simple for loop over Order.objects.all() works fine until it meets 5 million rows. Then you get memory spikes, timeouts, a slow database, and no idea where the job stopped.

Here is the pattern that turns that into a restartable pipeline.

The core idea

Don't run one big job. Run thousands of small, independent Celery tasks, each handling one ID range.

CHUNK_SIZE = 2000

@shared_task
def dispatch_order_recalculation():
    bounds = Order.objects.aggregate(lo=Min("id"), hi=Max("id"))
    if bounds["lo"] is None:
        return 0
    start = bounds["lo"]
    while start <= bounds["hi"]:
        end = start + CHUNK_SIZE - 1
        process_order_chunk.delay(start, end)
        start = end + 1

@shared_task(bind=True, max_retries=3, acks_late=True)
def process_order_chunk(self, start_id, end_id):
    try:
        orders = list(
            Order.objects.filter(id__gte=start_id, id__lte=end_id)
            .only("id", "subtotal", "tax_rate", "discount")
        )
        for order in orders:
            order.total = calculate_total(order)
        Order.objects.bulk_update(orders, ["total"], batch_size=500)
    except Exception as exc:
        raise self.retry(exc=exc, countdown=30 * (self.request.retries + 1))

Checklist

• Keyset pagination, not OFFSET. Range queries on an indexed column stay fast across the whole table.
• .only() + bulk_update(). Fewer bytes read, far fewer queries. It skips save() and signals, so set updated_at yourself if needed.
• Idempotent tasks. Compute from source data, use upserts, and keep processed flags for emails or payments.
• Celery settings. acks_late=True, prefetch_multiplier=1, time limits, and a dedicated bulk queue.
• Protect Postgres. Limit concurrency, keep one chunk per transaction, index filter columns, and consider PgBouncer.
• Track progress. Use a BulkJob model with F() counters so failed ranges can be requeued alone.

BulkJob.objects.filter(pk=job_id).update(completed_chunks=F("completed_chunks") + 1)

Chunk size

There's no universal number. Start with 1,000 to 5,000 records, benchmark 500 / 2,000 / 5,000 on your real data, and aim for tasks that take a few seconds to a minute.

Full guide

The complete version, with Celery settings, common mistakes, a production checklist, and an FAQ, is on my blog:

👉 https://blog.reiteachaigean.com/blog/process-5-million-records-django-celery

What's the largest dataset you've processed with Celery? Share your setup below.