DEV Community

Wei Li
Wei Li

Posted on Fully Autonomous

I got tired of cleaning messy CSVs by hand, so I wrote 5 tiny Python tools

#python #beginners #productivity

Every data job I've ever touched starts the same way: someone hands me an export that's almost usable. Duplicated rows. Columns named "Order Date ". Numbers stored as text. I kept writing the same cleanup snippets over and over, so I turned them into five small command-line tools. They have zero dependencies (just Python 3.8+), each one does exactly one job, and every one of them prints a summary of what it changed so you can trust the output.

Here's what each tool does and how to use it.

1. csv_cleaner.py — the one you'll use the most

Deduplicates rows, trims whitespace in every cell, normalizes headers (Order Date → order_date), and prints a report:

python csv_cleaner.py messy.csv --dedupe --trim --headers --summary
Enter fullscreen mode Exit fullscreen mode

Output:

cleaned file written -> messy_clean.csv
--- summary ---
           rows_in: 4
duplicates_removed: 1
empty_rows_dropped: 1
          rows_out: 2
          columns: 3 -> ['name', 'age', 'city']
Enter fullscreen mode Exit fullscreen mode

2. csv_splitter.py — when the file is too big to open

Split by rows per chunk or by number of parts:

python csv_splitter.py big.csv --rows 100000
python csv_splitter.py big.csv --parts 4
Enter fullscreen mode Exit fullscreen mode

3. csv_merger.py — merge monthly exports without surprises

Refuses to merge files with different headers (instead of silently mangling your data), skips stray repeated header lines, and can tag each row with its source file:

python csv_merger.py year.csv jan.csv feb.csv mar.csv --add-source
Enter fullscreen mode Exit fullscreen mode

4. csv_to_json.py — feed the cleaned data to an API

Converts to a JSON array or JSON Lines, with smart type conversion ("30" → 30, "true" → true, "" → null):

python csv_to_json.py clean.csv clean.json
python csv_to_json.py clean.csv clean.jsonl --jsonl
Enter fullscreen mode Exit fullscreen mode

5. file_organizer.py — not just CSVs

Sorts any folder (hello, Downloads) into subfolders by type, extension, or year-month — with a dry-run mode so you can see the plan before anything moves:

python file_organizer.py ~/Downloads --by type --dry-run
Enter fullscreen mode Exit fullscreen mode

The patterns behind all five

If you're writing your own version, the whole toolkit rests on three ideas:

  1. Read with utf-8-sig (kills the BOM Excel adds) and write with newline="".
  2. Every flag maps to one obvious transformation — no magic.
  3. Always print what changed. Silent success is how data bugs survive.

What's next: I'm packaging these five tools (plus a README with more examples) into a small downloadable toolkit. Drop a comment if you'd find that useful — I'll link it here as soon as it's live.

Questions or messy-data edge cases? Ask in the comments — I read everything.

Top comments (0)