My CSV diff tool reported a new row as removed, and its self-test cannot run

Last week I compared two CSV snapshots of a price table, one export from staging, one from production, with a tool from my lab: csv-key-diff. It takes two CSVs, a key column, and returns a JSON object with added, removed and changed arrays. It is 76 lines of Python, stdlib-only, nothing to install.

The production export had a row that staging did not have. That is an addition. The tool reported it under removed.

I only caught it because the row counts did not line up between the two exports, and the JSON said "added": [] while naming exactly one row in removed. One of those two statements had to be about my new row, and the array claiming "nothing appeared" was the wrong one.

What the code actually says

The whole comparison is three lines:

def compute_diff(data1, data2, key):
    added = [data1[k] for k in data1 if k not in data2]
    removed = [data2[k] for k in data2 if k not in data1]
    changed = [data1[k] for k in data1 if k in data2 and data1[k] != data2[k]]
    return {'added': added, 'removed': removed, 'changed': changed}

added iterates over data1, the first file argument. removed iterates over data2, the second. Read as an operation it is coherent: "what the first file holds that the second lost". Read as English, it is backwards. Every diff I use daily, diff old new, git diff a b, is oriented forward: what the new file brought.

Here is what that costs me on real files:

$ python3 csv_key_diff.py a.csv b.csv --key id
{"added": [], "removed": [{"id": "3", "name": "qux"}], "changed": [{"id": "2", "name": "bar"}]}

b.csv is the file where id 3 appeared and where id 2's name became BAZ. The tool calls the arrival a removal and reports the row it replaced.

That second part is a separate decision hiding in the same line: changed yields data1[k], the old row. A changed entry tells you which key moved and where it came from, never where it went. For a review workflow, the value you actually need is the half that is missing.

The README line for the tool is python csv_key_diff.py path/to/file1.csv path/to/file2.csv --key id. file1, file2. Nothing in the text says which one is the baseline, so the only way to learn the orientation is to be surprised by it once.

Two failures, one silent exit code

I fed it a file with a duplicated key to see what would happen. It exited 2 and printed nothing at all:

for row in reader:
    k = row[key]
    if k in data:
        sys.exit(2)

The same code 2 covers a missing key column:

if key not in reader.fieldnames:
    sys.exit(2)

Distinct causes, identical exit status, no message. Exit 2 is a fine signal for "halt the pipeline" and a useless one for "why did it halt at 2am". The other tool in this family I wrote, jsonl-cli, prints the line number of every broken record to stderr before it fails; that is the bar.

Everything is a string, compared as such

csv.DictReader hands back strings and the comparison is plain !=. Same key, quantity 2 against 2.0:

{"added": [], "removed": [], "changed": [{"id": "1", "qty": "2"}]}

A numeric no-op becomes a change. A row with fewer columns than the header comes back with a JSON null:

{"added": [], "removed": [], "changed": [{"id": "2", "name": "bar", "note": null}]}

So null means "this cell was empty" and also "this row never had that column" — the tool cannot tell those apart, and neither can the reader. An empty input file crashes outright, because fieldnames is None and key not in None raises a TypeError: exit 1, traceback, no explanation.

The self-test does not test the diff

The README advertises:

python csv_key_diff.py --self-test

That command has never worked. --key is declared required=True and both paths are positional, so argparse rejects it before test_self() is ever reached:

usage: csv_key_diff.py [-h] --key KEY [--self-test] csv1 csv2
csv_key_diff.py: error: the following arguments are required: csv1, csv2, --key

I had documented a command I never typed. So I typed it with the positional arguments filled in, and looked at what the harness does:

def run_test(a, b, expected_exit):
    result = subprocess.run(
        [sys.executable, __file__, a, b, '--key', 'id', '--self-test'],
        capture_output=True,
        text=True
    )
    if result.returncode != expected_exit:
        sys.exit(1)

The child process is launched with --self-test. It does not run the comparison it is supposed to be checking; it calls test_self() again, which spawns four children, each spawning four more. I ran it in a sandbox capped at 400 processes and a 6-second timeout: it never returned, the timeout killed it with 124.

Which means the four run_test calls in the file assert an exit code about a self-invocation, never about added, removed or changed. The happy path and the two error paths I describe above are all untested, and the recursion is what hid that from me: a suite that cannot finish looks like a suite that works if you never wait for it.

The repo ships tests/identical.csv and tests/missing.csv, 19 bytes each, and they are the same git blob — identical contents, despite the names. Neither filename appears anywhere in csv_key_diff.py. The CI workflow auto-detects checks by globbing test_*.py and *_test.py; this repo has neither, so what runs green is python3 -m py_compile over the one Python file. Single commit, f7348e1, 2026-09-14, MIT, zero dependencies.

I am not hiding that. A green CI, a self-test section in the README, and no behavioural coverage at all can absolutely coexist, because each of those signals is checked independently and none of them asks whether the tool answers the question you built it for.

What I would change, and what I left alone

Flipping two variable names would fix the orientation, and I would do it, because "forward like every other diff" is what readers assume. Printing one line to stderr before sys.exit(2) would make the failure diagnosable. Dropping --self-test from the child arguments would turn the harness into tests that actually reach compute_diff.

What I would not fix is the string comparison. Once you start coercing numbers you own a type guesser, and csv-inspect already exists for profiling a column's inferred type. This tool's job is the shape of the difference, not the meaning of the cell — but that boundary is worth stating on the tin, and right now it is not stated.

Repo: https://github.com/Raknaos/csv-key-diff

curl -sO https://raw.githubusercontent.com/Raknaos/csv-key-diff/main/csv_key_diff.py
python3 csv_key_diff.py old.csv new.csv --key id

Run it on two files whose answer you already know. If added lists rows that only exist in your second file, you have found the orientation on your own, and from then on you will read every output from this tool with the argument order in your head — which is the only defence that works when a tool's naming disagrees with its behaviour.