Exporting Large CSV Files in Django Without Memory Errors

#django #postgres #performance

Most CSV exports in Django work great until the table grows. Then the server runs out of memory, because the code builds the entire file before sending it.

The problem

response = HttpResponse(content_type="text/csv")
writer = csv.writer(response)
for order in Order.objects.all():
    writer.writerow([order.id, order.customer.name, order.total])

The queryset is cached in memory
The whole CSV is held in one response
order.customer.name causes N+1 queries

The fix: stream it

import csv
from django.http import StreamingHttpResponse

class Echo:
    def write(self, value):
        return value

def export_orders(request):
    writer = csv.writer(Echo())
    queryset = (
        Order.objects
        .values_list("id", "customer__name", "total", "created_at")
        .order_by("id")
    )
    response = StreamingHttpResponse(
        (writer.writerow(row) for row in queryset.iterator(chunk_size=2000)),
        content_type="text/csv",
    )
    response["Content-Disposition"] = 'attachment; filename="orders.csv"'
    return response

Why it works:

StreamingHttpResponse sends data as it's generated
iterator(chunk_size=2000) avoids caching the queryset
values_list() returns light tuples instead of model instances
Joined fields avoid N+1 queries

Gotchas

Server-side cursors: iterator() uses them on PostgreSQL, which can break behind PgBouncer in transaction mode. Check DISABLE_SERVER_SIDE_CURSORS.
Buffering: Nginx may buffer the response. You can send X-Accel-Buffering: no.
Timeouts: Long streams can hit the Gunicorn worker timeout.
ASGI: a sync iterator in StreamingHttpResponse makes Django load everything and warn you. Use an async iterator there.

Choosing an approach

Export size	Approach
Up to ~50k rows	Streaming with `values_list()`
50k to a few million	Streaming with `iterator()` or Celery
Millions, long-running	Celery task + storage + download link
Tens of millions	PostgreSQL `COPY`, usually in a background task

CSV injection

def safe_cell(value):
    if isinstance(value, str) and value.startswith(("=", "+", "-", "@")):
        return "'" + value
    return value

Full guide

The complete post covers the Celery export task, PostgreSQL COPY, Excel encoding, a checklist, and an FAQ:

👉 https://blog.reiteachaigean.com/blog/export-large-csv-files-django-without-memory-errors

DEV Community