There's an awkward size of data nobody wants to own.
Your file is a few GB. You open it in pandas… and your machine runs out of RAM and dies. Okay, so you reach for Spark, and now you're standing up a cluster and a whole distributed-systems mindset to do what is basically "filter some rows and drop a column." :(
I hit that wall constantly, so I built a tool for it: kenze. This is the story of it.
Quick honesty first
I'm not a software engineer. I work with big data files every day, that's my actual world... but I'm not a tooling/systems dev by trade. And I built kenze with a lot of AI help (Claude wrote a big chunk of the code while I drove the design, tested everything, and used it on my own real files).
I think that's just what building looks like now, and I'd rather be upfront than have someone "catch" me. The stuff I'm claiming here, especially the "never-runs-out-of-memory" part is tested (58-test suite, CI green on Linux + Windows) and benchmarked, and I use it daily. Real tool, honest about how it got made.
Enter kenze
One install: pip install kenze
Then either run one-liners:
kenze filter sales.parquet --where "amount > 0" -o clean.csv
kenze keep users.csv --cols id,city,email -o small.parquet
kenze dedup users.csv --on id -o unique.parquet
…or just run kenze with no arguments and land in an interactive shell, think a Claude-Code-style REPL, but for data prep.
The one real promise: it doesn't run out of memory
This is the whole reason kenze exists.
Under the hood it's DuckDB doing the heavy lifting. kenze caps memory to a fraction of your free RAM and spills to disk instead of dying. So you can point it at a file bigger than your RAM and it just… works, one streaming pass, no crash.
And don't take my word for it: the repo has a reproducible benchmark (it also runs in CI) that gives pandas and kenze the same memory budget on a file bigger than it. pandas OOMs. kenze spills to disk and finishes. On my laptop, a 60M-row Parquet opens instantly and a filter + dedup down to ~27M rows runs in about 3.5 seconds.
The interactive shell...
My favourite part. Run kenze, load a file once, and stack steps that each preview live:
kenze > load sales.parquet # 60M rows, opens instantly
kenze > filter amount > 0 # each step previews the result
kenze > plot amount by city # ascii bar chart, right in the terminal
kenze > keep id, city, amount
kenze > dedup id
kenze > run clean.csv # streamed through DuckDB, no OOM
It has a / command menu, it TAB-autocompletes your file's actual column names, and it can draw quick ASCII histograms / bar charts so you can eyeball a column without leaving the terminal. The whole session is really a recipe, you can save it and re-run it later.
No SQL required... but no lock-in either
You don't need SQL for the everyday stuff (simple verbs, or a readable recipe file). But if you want it, there's a sql escape hatch straight to DuckDB. And you can eject any recipe to plain SQL or Python, so if you outgrow kenze, you walk away with the code, not a dependency.
It reads/writes CSV, Parquet, JSON, gzip and Excel (.xlsx), local or on s3:// / gs://. It also does joins, pivots, PII masking, schema validation, and data-quality asserts that abort a run before it writes bad output.
Where it stops, ""on purpose""
kenze is one machine. That's a feature, not a gap. If you've genuinely outgrown a single box, multi-TB, distributed, real SLAs - that's Spark/Dask territory, and you should use them. kenze is what you reach for before you need those.
The mental model I like: pandas < kenze < Spark.
Try it / break it
Open-source (MIT), early, and I'm actively improving it.
If you live in that "too big for pandas, too small for Spark" middle, I'd genuinely love for you to try it and tell me what breaks. What do you reach for when a file's in that awkward size?
Top comments (0)