DEV Community

Hamza Masood
Hamza Masood

Posted on

Big CSV files in JavaScript without freezing the tab

#javascript #node #webassembly #webdev

Open a CSV with a few million rows in a web app and the tab often freezes. In Node the same file can eat gigabytes of memory.

The usual cause: every row becomes a JavaScript object. Ten million rows means ten million small objects, each carrying its own keys, all parsed on one thread while the other cores sit idle.

A different layout

Vectro (vectrojs on npm) takes another route:

  • each column is stored as one typed array in WebAssembly memory, not as objects
  • the CSV is split into pieces and parsed on every core (worker threads in Node, Web Workers in the browser)
  • you query it with SQL or a small expression API

Node 

import { Engine, sql } from 'vectrojs';

const engine = new Engine();
const orders = engine.readCSV('orders.csv');

sql('SELECT region, SUM(price) AS revenue FROM orders GROUP BY region', { orders }).toRows();
Enter fullscreen mode Exit fullscreen mode

Browser 

import { Engine, sql } from 'vectrojs/browser';

const engine = new Engine();
const orders = await engine.loadCSV('/orders.csv');

await sql('SELECT region, SUM(price) AS revenue FROM orders GROUP BY region', { orders }).toRowsAsync();
Enter fullscreen mode Exit fullscreen mode

Threads in the browser need the page to be cross-origin isolated (COOP/COEP headers). Without them SplitEngine still uses every core.

Numbers

On an M1 Pro laptop (8 cores):

Task Result
Parsing a CSV stream 16.9M rows/s (plain Node: 3.26M)
GROUP BY region on 10M rows 6.5 ms (DuckDB on the same machine: 39.7 ms)

The scripts behind these are in the repo under compare/ and bench/.

Try it

  • In the browser, nothing to install: vectrojs.dev/playground. Drop in a CSV or generate up to 10M sample rows, then query it with SQL. The data stays in your tab.
  • npm i vectrojs

It's open source (pure JS + WebAssembly, no native build). Feedback and issues are welcome on GitHub.

Top comments (0)