hey dev.to

Kenzy Zero — Sun, 19 Jul 2026 07:45:43 +0000

Hey everyone 👋
I'm Kenzy. I work with big data & location intelligence, basically turning huge, messy datasets (and "where people actually go") into things you can make decisions with.
Lately I've been building my own tools on the side, so I figured I'd stop lurking and finally post here. First thing I've shipped is kenze... for cleaning data files that are too big for pandas, without your RAM tapping out. Proper writeup coming soon.
Honest heads-up: I'm a data guy, not a career dev, and I build a lot with AI helping me write the code. I'll be sharing what I make, what breaks, and whatever I pick up along the way.
Glad to be here - say hi

Medium Data: when your file is too big for pandas but too small for Spark

Kenzy Zero — Sun, 19 Jul 2026 07:34:59 +0000

There's an awkward size of data nobody wants to own.

Your file is a few GB. You open it in pandas… and your machine runs out of RAM and dies. Okay, so you reach for Spark, and now you're standing up a cluster and a whole distributed-systems mindset to do what is basically "filter some rows and drop a column." :(

I hit that wall constantly, so I built a tool for it: kenze. This is the story of it.

Quick honesty first

I'm not a software engineer. I work with big data files every day, that's my actual world... but I'm not a tooling/systems dev by trade. And I built kenze with a lot of AI help (Claude wrote a big chunk of the code while I drove the design, tested everything, and used it on my own real files).

I think that's just what building looks like now, and I'd rather be upfront than have someone "catch" me. The stuff I'm claiming here, especially the "never-runs-out-of-memory" part is tested (58-test suite, CI green on Linux + Windows) and benchmarked, and I use it daily. Real tool, honest about how it got made.

Enter kenze
One install: pip install kenze

Then either run one-liners:
kenze filter sales.parquet --where "amount > 0" -o clean.csv
kenze keep users.csv --cols id,city,email -o small.parquet
kenze dedup users.csv --on id -o unique.parquet

…or just run kenze with no arguments and land in an interactive shell, think a Claude-Code-style REPL, but for data prep.

The one real promise: it doesn't run out of memory
This is the whole reason kenze exists.

Under the hood it's DuckDB doing the heavy lifting. kenze caps memory to a fraction of your free RAM and spills to disk instead of dying. So you can point it at a file bigger than your RAM and it just… works, one streaming pass, no crash.

And don't take my word for it: the repo has a reproducible benchmark (it also runs in CI) that gives pandas and kenze the same memory budget on a file bigger than it. pandas OOMs. kenze spills to disk and finishes. On my laptop, a 60M-row Parquet opens instantly and a filter + dedup down to ~27M rows runs in about 3.5 seconds.

The interactive shell...
My favourite part. Run kenze, load a file once, and stack steps that each preview live:

kenze > load sales.parquet # 60M rows, opens instantly
kenze > filter amount > 0 # each step previews the result
kenze > plot amount by city # ascii bar chart, right in the terminal
kenze > keep id, city, amount
kenze > dedup id
kenze > run clean.csv # streamed through DuckDB, no OOM

It has a / command menu, it TAB-autocompletes your file's actual column names, and it can draw quick ASCII histograms / bar charts so you can eyeball a column without leaving the terminal. The whole session is really a recipe, you can save it and re-run it later.

No SQL required... but no lock-in either

You don't need SQL for the everyday stuff (simple verbs, or a readable recipe file). But if you want it, there's a sql escape hatch straight to DuckDB. And you can eject any recipe to plain SQL or Python, so if you outgrow kenze, you walk away with the code, not a dependency.

It reads/writes CSV, Parquet, JSON, gzip and Excel (.xlsx), local or on s3:// / gs://. It also does joins, pivots, PII masking, schema validation, and data-quality asserts that abort a run before it writes bad output.

Where it stops, ""on purpose""

kenze is one machine. That's a feature, not a gap. If you've genuinely outgrown a single box, multi-TB, distributed, real SLAs - that's Spark/Dask territory, and you should use them. kenze is what you reach for before you need those.

The mental model I like: pandas < kenze < Spark.

Try it / break it

Open-source (MIT), early, and I'm actively improving it.

PyPI: https://pypi.org/project/kenze/
GitHub: https://github.com/Kenzy-Zero/kenze

If you live in that "too big for pandas, too small for Spark" middle, I'd genuinely love for you to try it and tell me what breaks. What do you reach for when a file's in that awkward size?

DEV Community: Kenzy Zero

hey dev.to

Medium Data: when your file is too big for pandas but too small for Spark