A CSV diff should reject duplicate IDs before it reports changes

yyygyf — Sat, 12 Sep 2026 20:19:08 +0000

Authorship: this article and the linked toolkit were prepared by Codex on behalf of the account owner. The examples use synthetic data. The execution results below come from actual local runs, not customer deployments.

Two CSV exports can contain the same number of rows and still represent different records. A line diff is also noisy when the exporting system changes the column order. A useful snapshot comparison needs a rule for identifying records before it can classify them as added, removed, changed or unchanged.

This walkthrough uses a small public Node.js toolkit, pinned to v0.2.0. Its standalone commands have no third-party dependencies. The implementation makes a deliberate choice: duplicate or blank keys stop the comparison, because there is no agreed rule for pairing those records.

Run a complete example

Download csv-toolkit-v0.2.0.zip from the release page and extract it. The attached ZIP includes synthetic inputs and pre-generated HTML reports in example-output. GitHub's automatic source archives contain the source without those generated reports.

With Node.js 22+ installed, run this from the extracted toolkit directory:

node demo.mjs my-demo-output

Use a directory name that does not already exist. The command creates nine files, including audit.html, changes.html and changes.json. The HTML reports are currently in Chinese; the identifiers and before/after values are visible directly.

The demo first merges four source records into three, keeping the first occurrence of a duplicate ID. It then audits the merged data and compares it with a later snapshot:

Result	Expected value
Added	`004`
Removed	`003`
Changed	`002`: amount string `200` becomes `250`
Unchanged	`001`

The audit finds one record with both a missing customer and an invalid numeric field. It reports those problems without repairing the source data.

Make identity rules executable

Save the following as article-repro.mjs alongside compare.mjs, then run node article-repro.mjs. This shorter example uses English column names.

import assert from 'node:assert/strict';
import { compareCsv } from './compare.mjs';

const before = 'id,name,amount\n001,A,100\n002,B,200\n003,C,80\n';
const after = 'amount,name,id\n100,A,001\n250,B,002\n90,D,004\n';
const result = compareCsv(before, after, ['id']);

assert.deepEqual(result.summary, {
  before: 3, after: 3, added: 1, removed: 1, changed: 1, unchanged: 1,
});
assert.deepEqual(result.changed, [{
  key: { id: '002' },
  fields: [{ column: 'amount', before: '200', after: '250' }],
}]);
assert.equal(result.added[0].id, '004');
assert.equal(result.removed[0].id, '003');

assert.throws(() => compareCsv(
  'id,amount\n002,200\n002,210\n', after, ['id'],
), /duplicate key/);

assert.throws(() => compareCsv(
  'id,amount\n,200\n', 'id,amount\n002,200\n', ['id'],
), /empty key/);

const formatted = compareCsv(
  'id,amount\n001,1.00\n', 'id,amount\n001,1\n', ['id'],
);
assert.equal(formatted.summary.changed, 1);

const composite = compareCsv(
  'customer,order,value\nx|y,z,old\nx,y|z,same\n',
  'order,value,customer\nz,new,x|y\ny|z,same,x\n',
  ['customer', 'order'],
);
assert.equal(composite.summary.changed, 1);
assert.equal(composite.summary.unchanged, 1);

console.log('PASS: categories, reordered columns, string IDs, invalid keys, exact values, composite keys');

There are four decisions behind these assertions:

Column position is not record identity. The two snapshots have the same column-name set in different orders. The comparison aligns fields by their exact names. A renamed, added or missing column is a schema mismatch and stops the run.
IDs stay strings. 001 remains 001; the comparator does not coerce it to the number 1. Other values also use exact string comparison, so 1.00 and 1 count as a change. Numeric equivalence would need a separately specified normalization rule.
Duplicate keys are ambiguous. If ID 002 occurs twice, the comparator refuses to guess which row survived. The merge command's explicit first-occurrence rule belongs to the merge operation; it is not silently applied inside the comparator.
Composite keys need boundaries. Joining ['x|y', 'z'] and ['x', 'y|z'] with | produces the same text. The implementation serializes the key-value array with JSON.stringify, retaining those boundaries. The last assertions exercise that case with reordered columns too.

Use the same contract in n8n

The release also includes n8n-csv-snapshot-v0.2.0.json. It contains a Manual Trigger and two JavaScript Code nodes. The comparison node expects one item with beforeText, afterText and a keys array; it returns record counts and field-level changes.

The included workflow was actually imported and executed through the n8n CLI on Windows with n8n 2.38.7 and Node.js 24.19.0. The workflow documentation includes reproduction commands and a saved execution result. Running node n8n/verify.mjs n8n/sample-execution.json checks the embedded code and that saved record; it does not start another n8n execution.

The sample has no external requests or credentials. Its per-snapshot limits are 10,000 data records, 100 columns and 1,000,000 UTF-16 code units. File attachment decoding, scheduling and production connectors are separate work. An n8n instance may retain input and output in its execution history, so use synthetic data when sharing a run.

What was checked, and what this comparison means

For the v0.2.0 release, six unit tests, three command self-tests and twelve n8n verification groups passed. The public ZIP was downloaded again, its checksum verified, and its demo rerun; all nine generated files matched the bundled examples byte for byte. The report was also inspected in Firefox at 1280px and 375px widths. The article's additional assertion script passed on Node.js 24.19.0 on Windows.

This is a comma-separated CSV comparison with in-memory processing. It does not read XLSX, reconcile monetary totals, infer business correctness or automatically fix data. Exact matching is useful when the export contract is stable. If identifiers, schemas or formatting rules change, specify that transformation first so that the resulting change report has a clear meaning.

DEV Community: yyygyf

A CSV diff should reject duplicate IDs before it reports changes

Run a complete example

Make identity rules executable

Use the same contract in n8n

What was checked, and what this comparison means