DEV Community

Dark Star
Dark Star

Posted on

How to Compare Two Large CSV Files Without Uploading the Data

#webdev #learning #beginners #programming

Comparing two CSV exports sounds simple until the files get large.

A common situation is having an older export and a newer export and wanting to answer four questions:

  • Which rows were added?
  • Which rows were removed?
  • Which existing rows were modified?
  • Which rows are unchanged?

A simple row-by-row comparison isn't reliable when the order of rows changes between exports. What you really want is to identify the same record in both files using a stable key, then compare the records associated with that key.

The basic approach

For each CSV:

  1. Parse the headers and rows.
  2. Identify a suitable column that can act as the record key.
  3. Build a lookup of key → row.
  4. Compare the keys present in each file.
  5. For keys present in both files, compare their values.
  6. Classify the result as modified or unchanged.
  7. Keys existing in only one file become added or removed records.

This turns the problem into a lookup/comparison problem instead of repeatedly scanning the entire second file for every row.

The browser-only part

There is another consideration when the CSV contains business or customer data.

Instead of uploading the files to a server, the entire comparison can happen in the browser. The files can be read locally, processed locally, and discarded when the page is closed.

That means there is no server-side CSV processing involved.

Then I hit another problem: large files

Testing with a small CSV is easy.

Testing with a few hundred thousand rows is where things get interesting.

The comparison itself can complete successfully, but rendering hundreds of thousands of result rows into the DOM is a terrible idea. The browser can become unresponsive simply because it is trying to create an enormous number of HTML elements.

The solution is to separate data processing from data rendering.

The complete comparison result can remain in memory, while the UI only renders a small portion of it at a time.

For example:

293,000 unchanged rows

Page 1 → rows 1–100
Page 2 → rows 101–200
Page 3 → rows 201–300
...
Page 2,930 → rows 292,901–293,000
Enter fullscreen mode Exit fullscreen mode

The user still has access to the complete result, but the browser only has to render the current page.

This also means exports can continue using the complete result rather than whatever happens to be visible on screen.

CSVCompare

I ended up building a small browser-based tool while working through these problems:

CSVCompare: https://csvcompare.pages.dev

It compares two CSV files, automatically detects the best key, and separates the results into modified, added, removed, and unchanged records.

The tool processes the CSV files locally in the browser.

This started as a small experiment, so I'm particularly interested in feedback from people who regularly work with CSV exports, data migrations, or recurring snapshots.

What approach do you use when you need to compare two large CSV exports?

Top comments (0)