DEV Community

OmUniyal
OmUniyal

Posted on

I built a Python package to diff large data files — here's why existing tools weren't enough

#python #programming #opensource #beginners

Every few months at work I run into the same problem.

Two systems are supposed to produce identical data exports. A CSV from the old pipeline, a CSV from the new one. Simple enough to check — until the file has 500,000 rows, three sources to compare instead of two, and columns that were renamed somewhere along the way.

The usual tools fall apart fast:

  • Excel — opens maybe 100k rows before giving up
  • diff / fc — order-dependent, one mismatch per line, useless for structured data
  • pandas — fine for two files that fit in memory, painful for anything larger, no built-in N-way support
  • Custom scripts — I've written three. None of them were reusable.

So I built duckdiff.

What it does

duckdiff is a Python package for N-way, order-independent comparison of large structured files — CSV, TSV, and Parquet. It's powered by DuckDB, which means comparisons stream off disk and aren't bounded by RAM.

Install it:

pip install duckdiff
Enter fullscreen mode Exit fullscreen mode

Compare two files from the command line:

duckdiff compare old=export_v1.csv new=export_v2.csv --key transaction_id
Enter fullscreen mode Exit fullscreen mode

Output:

Sources:
  old: 547,823 rows, 14 columns
  new: 547,823 rows, 14 columns

Matched:     541,200
Mismatched:  6,123
Only in old: 500
Only in new: 0
Enter fullscreen mode Exit fullscreen mode

Not sure which columns to use as --key? There's a subcommand for that:

duckdiff keys a=export_v1.csv
Enter fullscreen mode Exit fullscreen mode

It scans the file and tells you which column combinations uniquely identify each row — printing results as it goes so you're not staring at a blank screen:

Key column suggestions for 'a':

  ✓  transaction_id  (unique)

  Suggested: duckdiff compare ... --key "transaction_id"
Enter fullscreen mode Exit fullscreen mode

A few things that make it different

N-way comparison. Compare 2, 3, or 20 sources in one pass. Not N pairwise diffs.

Order-independent. Rows don't need to be sorted. DuckDB handles it.

Fuzzy column mapping. If the new pipeline renamed cust_id to customer_id, duckdiff can suggest a mapping — but never applies one silently. You opt in explicitly.

Schema flexibility. If sources don't share all columns, --auto-intersect compares only the shared ones and tells you what was dropped.

Pre-flight dry-run. --dry-run checks schema compatibility and file sizes without scanning a single row. Useful before running a comparison on a large file.

Python API

The CLI is a thin wrapper around a clean Python API:

from duckdiff import ComparisonSession, ComparisonConfig

config = ComparisonConfig(key_columns=["transaction_id"])
with ComparisonSession(config) as session:
    session.add_source("old", "export_v1.csv")
    session.add_source("new", "export_v2.csv")
    result = session.compare()

print(result.matched_row_count)
print(result.mismatched_row_count)
print(result.only_in)
Enter fullscreen mode Exit fullscreen mode

Where to find it

It's at v0.1.0 — early, but tested (161 tests) and usable. Feedback and contributions are welcome — feel free to open an issue or star the repo on GitHub

Top comments (0)