Stream 50 million rows without OOM: Lazy chunked Query Streaming.
Day 09 of the WClickHouse Open-Source Engineering Series.
You don't need 64GB of RAM to process millions of ClickHouse records in Python. WClickHouse query_stream() keeps your memory footprint below 80MB.
The Pain Points We Faced
- Out-Of-Memory (OOM) killer crashing Python containers when fetching large results
- Cursor fetchall() loading 10GB of query data into Python RAM at once
- Slow pipeline startup waiting for entire query results before processing first row
The Implementation
db = WClickHouse(SensorReading, db_config)
# Stream 50 million rows in 50,000-row chunks: RAM never exceeds 80MB!
for chunk in db.query_stream("SELECT * FROM sensorreading", chunk_size=50000):
process_batch(chunk)
print(f"Processed chunk of {len(chunk)} rows cleanly.")
Why This Architecture Wins
- Lazy Generator: query_stream() yields chunks (e.g., 50,000 rows) on demand.
- Constant RAM Footprint: Memory stays under 80MB whether scanning 10k or 50M rows.
- Immediate First Row: Start processing pipeline logic immediately as first chunk arrives.
Verification & Status
Tested and verified against live ClickHouse server instances with 95%+ test coverage. Built for Python 3.9 through 3.14 with Apache Arrow and Pydantic v2.
Top comments (0)