Filtering a 2 GB CSV in the browser without uploading it: streams, a Web Worker and a 64-bit hash set

Excel stops at 1,048,576 rows. Most online CSV tools want you to upload the file first, which is a non-starter for a customer list or an accounting export. I wanted something in between: open a multi-gigabyte CSV in a normal browser tab, filter it, remove duplicates, split it into Excel-sized files, and never send a byte of it anywhere.

The result is CSV Géant, a static site with no framework and no runtime dependency. The UI is in French by default (there's an EN button in the header). This post covers the architecture, the actual code, and numbers I measured on my own machine, including the parts that don't stream.

The architecture

• Main thread: UI only. It sends the File object to a module Web Worker with postMessage. A File is a handle, so this doesn't copy the content.
• Worker: reads the file in 8 MB slices, decodes, parses, filters, deduplicates and writes the output. Progress messages go back to the UI.
• Output sinks: either an in-memory Blob that becomes a download, or, in Chrome/Edge, files written directly into a folder the user picks (File System Access API).
• Server: one Cloudflare Pages Function with three routes: anonymous counters, license check, and the price/purchase link. No CSV content, row or column name goes through it. The CSP has connect-src 'self', and once the page is loaded the tool keeps working offline.

The core (csv.js and traitement.js) has no DOM dependency, so the same code runs in Node for the tests (fs.openAsBlob gives a Blob-like file).

1. Reading: slice() plus a streaming TextDecoder

The whole read loop is short. Identifiers are in French; parcourir means "walk through":

export const TAILLE_BLOC = 8 << 20; // 8 Mo lus à la fois

export async function parcourir(fichier, { encodage, sep, surEnregistrement, surBloc }) {
  const decodeur = new TextDecoder(encodage); // retire le BOM éventuel
  let index = 0;
  const lecteur = new LecteurCSV(sep, (champs) => surEnregistrement(champs, index++));
  for (let o = 0; o < fichier.size; o += TAILLE_BLOC) {
    const tampon = await fichier.slice(o, o + TAILLE_BLOC).arrayBuffer();
    lecteur.pousser(decodeur.decode(new Uint8Array(tampon), { stream: true }));
    if (surBloc) await surBloc(Math.min(o + TAILLE_BLOC, fichier.size), index);
  }
  lecteur.pousser(decodeur.decode());
  lecteur.terminer();
  return index;
}

{ stream: true } matters: an 8 MB boundary can fall in the middle of a multi-byte UTF-8 character ("é" is two bytes). In streaming mode the decoder keeps the incomplete bytes and prepends them to the next chunk.

Encoding detection runs on the first 2 MB: BOM first, then a strict UTF-8 decode (new TextDecoder("utf-8", { fatal: true })), and Windows-1252 if that throws. Since the sample can itself end mid-character, a helper strips a truncated trailing sequence before the strict decode. Without it, a valid UTF-8 file would be misdetected as Windows-1252 whenever the 2 MB cut lands inside an accented character.

The delimiter (;, ,, tab, |) is guessed by parsing the sample with each candidate and keeping the one that gives the most regular column count (at least 2 columns), then the most columns.

2. A parser that can stop anywhere

The parser is fed arbitrary chunks of text, so a record can be cut anywhere, including inside a quoted field that contains newlines. The internal #analyser returns the index of the first character it could not consume as a complete record, and the caller keeps that tail for the next chunk:

pousser(morceau) {
  const s = this.reste ? this.reste + morceau : morceau;
  const consomme = this.#analyser(s, false);
  this.reste = consomme < s.length ? s.slice(consomme) : "";
  if (this.reste.length > LONGUEUR_MAX_ENREGISTREMENT) {
    throw new Error("ENREGISTREMENT_TROP_LONG");
  }
}

LONGUEUR_MAX_ENREGISTREMENT is 16 MB. Without it, a single unclosed quote (or a wrong delimiter/encoding choice) would make reste grow until the tab runs out of memory. With it, the user gets an error message that points at the likely cause.

The parser handles RFC 4180 quoting (doubled quotes, newlines inside quotes), \n, \r\n and \r, and tolerates text after a closing quote ("abc"def). Rows with an unexpected number of columns are counted and reported, not dropped.

3. Deduplication: 64-bit fingerprints in a Uint32Array

Storing every distinct row in a JS Set doesn't work at 20 million rows. Instead, each row (or the chosen column, optionally trimmed and lower-cased) is reduced to two independent 32-bit hashes, stored in a flat open-addressing table:

#inserer(h1, h2) {
  const masque = this.capacite - 1;
  let p = (h1 ^ Math.imul(h2, 0x9e3779b1)) & masque;
  const t = this.table;
  for (;;) {
    const a = t[p * 2];
    const b = t[p * 2 + 1];
    if (b === 0) {
      t[p * 2] = h1;
      t[p * 2 + 1] = h2;
      this.taille++;
      return true;
    }
    if (a === h1 && b === h2) return false;
    p = (p + 1) & masque;
  }
}

h2 is never 0, so 0 marks an empty slot. The table doubles when it would be more than half full.

Two honest consequences:

• Memory is 16 to 32 bytes per distinct key, depending on where you are in the doubling cycle (8 bytes per slot, load factor between 25% and 50%). For the 21.3 million distinct rows in my test file, the final table is 512 MiB, plus the previous 256 MiB table while it's being copied.
• It's probabilistic. Two different rows with the same 64-bit fingerprint would be treated as duplicates. The probability is about n²/2⁶⁵, so roughly 1 in 80,000 for 21 million distinct rows. For a deduplication tool, I decided that was acceptable; you may disagree.

4. Writing: BOM, CRLF, and two sinks

Output rows are serialized (quoted only when needed, CRLF line endings), accumulated in a string, and encoded with TextEncoder in batches of about 4 million characters. Each output file starts with a UTF-8 BOM and repeats the header row. The BOM is what makes Excel read accents correctly instead of guessing a legacy encoding. Splitting "for Excel" means 1,048,575 data rows per file, plus the header.

The download sink groups the encoded chunks into 64 MB Blobs as it goes:

async ecrire(octets) {
  const c = this.courant;
  c.morceaux.push(octets);
  c.enAttente += octets.length;
  c.taille += octets.length;
  if (c.enAttente >= 64 << 20) {
    c.blobs.push(new Blob(c.morceaux));
    c.morceaux = [];
    c.enAttente = 0;
  }
}

The whole result is still assembled before the download starts. For results over 1 GB, the UI recommends the other sink: showDirectoryPicker(), then one createWritable() stream per output file. That one only exists in Chromium browsers.

5. The part that doesn't stream: sorting

You can't sort a file you only see 8 MB at a time without an external merge sort, and I haven't built one. So sorting keeps the filtered result in memory, as compactly as I could manage: each row is stored already serialized in UTF-8 inside 16 MB blocks, numeric and date keys go into a Float64Array, and what gets sorted is a Uint32Array of indices:

const ordreIdx = new Uint32Array(this.n);
for (let i = 0; i < this.n; i++) ordreIdx[i] = i;
ordreIdx.sort((a, b) => cmp(cles[a], cles[b]) || a - b);

The || a - b makes the sort stable. Hard limits: 3 million rows or 600 million characters. Beyond that, the export stops with an explicit message (filter first, then sort).

Measured numbers

Conditions: AMD Ryzen 5 4500U (6 cores), 16 GB RAM, NVMe SSD, Windows 11. Chrome 154 headless driven by Playwright (the tests/e2e-tres-gros.mjs script from the repo), app served from localhost. Test file generated by the repo's scripts/generer-csv.mjs: 22 million rows, 9 columns, 2.22 GB, semicolon-delimited UTF-8 with accents, about 427,000 quoted fields containing line breaks, and 660,441 exact duplicates. Each result was checked against the expected counts written by the generator.

Operation (Chrome, 2.22 GB file)	Time
Preview (encoding + delimiter detection, first 100 rows)	0.3 s
Exact row count (22,000,000)	17.4 s / 17.8 s
Deduplicate everything + split into 21 Excel-sized files, written to a folder (2.18 GB out)	102.2 s / 106.9 s
Filter pays (country) = FR + deduplicate, single download (8,001,769 rows, 0.82 GB out)	71.8 s / 72.0 s

Two runs each: roughly 125 MB/s for counting, 20 to 30 MB/s once every row is hashed and re-serialized. In headless mode, the origin private file system stood in for the native folder picker.

On a second file (2 million rows, 198 MB): numeric sort of all rows, descending, in 9.8 s and 10.7 s.

The same library in Node 24 gives the same counts; full dedup peaked there at about 1 GB of resident memory, consistent with the hash table sizes above.

These are one laptop's numbers. I haven't measured anything above 2.22 GB: the reading loop has no size-dependent state, but deduplication and sorting do.

Free version and license

• Free: preview, encoding/delimiter detection and row counting with no limit. Exports (filter, dedup, sort, split) write at most 100,000 rows per operation. The worker still reads the whole file, so the result screen tells you how many rows the full result would contain.
• Pro: a €29 one-time license sold through Gumroad, which removes the row cap in the browser where the key is entered. The key goes to the Pages Function, which calls Gumroad's license verification API and refuses refunded or disputed purchases. The browser stores it in localStorage and re-checks it every 7 days; offline, the stored key stays valid.

Since everything runs client-side, the cap is enforced in client-side code (limite: m.pro ? Infinity : LIMITE_GRATUITE). It's a pricing boundary, not DRM.

What the server sees

Anonymous daily counters (a visit, a preview, an export, an error), the ?ref= tag of the visit if there is one, and the license key if you type one. No cookies and no third-party analytics. The easy way to verify it: load the page, cut the network, and run an export.

Limits

• Tested in Chrome only. Edge uses the same engine but I haven't tested it; Firefox and Safari aren't tested. Writing to a folder requires Chrome or Edge.
• Sorting is in memory: 3 million rows / 600 million characters max.
• Deduplication costs 16 to 32 bytes per distinct key and is probabilistic, as described above.
• One worker, one core. Parsing isn't parallelized.
• CSV in, CSV out. No .xlsx output.
• A single record over 16 MB stops processing (usually an unclosed quote).

Try it

csv-geant.pages.dev: no account, and it can generate sample files up to 2 million rows. I'd like to hear about timings on other hardware, and about CSV dialects that break the delimiter detection.

Comments

[sgaggjhkjh]

The 16 MB per-record cap is my favorite detail here — bounding the parser so one unclosed quote can't eat the tab's memory is exactly the kind of defensive thinking production systems need. Also appreciate the honesty about the dedup being probabilistic and the license cap being a pricing boundary rather than DRM. Refreshing write-up, thanks for sharing the measured numbers too.