Polars Lazy vs Eager: 50M Row Memory & Speed Benchmark

Lazy evaluation cut memory usage by 73% on a 50M row dataset. Here's what actually happened.

I ran the same aggregation pipeline on 50 million rows of synthetic sensor data — once in eager mode, once in lazy mode. Eager mode peaked at 6.2GB RAM. Lazy mode stayed under 1.7GB. Execution time dropped from 18.4 seconds to 11.2 seconds.

This isn't theoretical. The benchmark uses realistic CSV data with timestamps, sensor IDs, temperature readings, and missing values. The kind of data you'd see from IoT devices, financial tick data, or server logs.

But here's the twist: lazy mode doesn't always win. I'll show you exactly when it falls apart.

Photo by Pixabay on Pexels

What Eager and Lazy Mode Actually Mean

Eager execution runs each operation immediately. You call .filter(), Polars filters the entire dataset right then. You chain .group_by().agg(), each step materializes a new DataFrame in memory.

---

Continue reading the full article on TildAlice