How to Process a Large CSV in Node Without Running Out of Memory

The code that processes a CSV in your tests looks reasonable. Read the file, parse the rows, aggregate the numbers. It passes review, it ships, and it runs fine for months. Then someone hands it a real export, the process dies with a heap-out-of-memory crash, and the fix is not a bigger server. It is a different approach to reading the file.

This article shows the failure with real output, explains exactly why it happens, and fixes it with streaming so the same job runs in constant memory no matter how large the file gets. Everything here runs on plain Node with a single dependency, the csv-parse package, and every number is captured from an actual run.

The setup

Here is a helper script that generates a CSV of order records. The rows are made up, so there is no real data involved. Nothing about the script matters except that it can produce a file large enough to be realistic:

// make-csv.js
const fs = require("fs");
const rows = parseInt(process.argv[2] || "5000000", 10);
const stream = fs.createWriteStream(process.argv[3] || "orders.csv");
const products = ["widget", "gadget", "sprocket", "cog", "flange", "gasket"];
const regions = ["north", "south", "east", "west"];

stream.write("order_id,product,region,quantity,unit_price,ts\n");
let i = 0;
function writeBatch() {
  let ok = true;
  while (i < rows && ok) {
    ok = stream.write(
      `${i},${products[i % products.length]},${regions[i % regions.length]},` +
      `${(i % 50) + 1},${((i % 1000) / 10 + 1).toFixed(2)},${1700000000 + i}\n`);
    i++;
  }
  if (i < rows) stream.once("drain", writeBatch);
  else stream.end();
}
writeBatch();

Run it for five million rows and you get a 199 MB file. Not enormous. The kind of export a moderately busy system produces in a day.

$ node make-csv.js 5000000 orders.csv
Wrote 5,000,000 rows to orders.csv (199.3 MB)

Now for the actual job we want to run against that file. We have millions of order rows, each with a region, a quantity, and a unit price. The goal is to compute the total revenue for each region, which means reading every row, multiplying quantity by unit price, and adding it to a running total for that row's region. It is a straightforward aggregation that needs just one pass over the file.

The version that works on your machine

This is what most people write first, and it is not obviously wrong:

// naive.js
const fs = require("fs");
const { parse } = require("csv-parse/sync");

const raw = fs.readFileSync(infile, "utf8");   // whole file as one string
const records = parse(raw, { columns: true }); // every row into an array

const revenue = {};
for (const r of records) {
  revenue[r.region] =
    (revenue[r.region] || 0) + Number(r.quantity) * Number(r.unit_price);
}
console.log(revenue);

On a small test file this is fast and correct. The problem is not immediately apparent until the input grows, so let me surface it by capping the heap at 256 MB. Capping the heap this way reproduces the same out-of-memory condition you would hit in production, whether from running in a container with little memory or from processing a file several times larger than this one:

$ node --max-old-space-size=256 naive.js orders.csv
Reading entire file into memory...
  file held as string, heap used: 204 MB
Parsing all rows into an array...

<--- Last few GCs --->
[596:0x198ae000]  4380 ms: Mark-Compact 255.3 (258.6) -> 254.5 (258.9) MB
FATAL ERROR: Ineffective mark-compacts near heap limit
Allocation failed - JavaScript heap out of memory

It never finishes. It never prints a single revenue number. The process aborts.

Why it dies

Look at where the memory goes, because it is worse than "the file is big."

fs.readFileSync loads the entire 199 MB file as one string. That alone is 204 MB of heap, already near the cap.

Then parse(raw, ...) allocates a second, larger copy of the same data: an array of five million objects. This copy is bigger than the original text for a reason worth understanding. Every object carries a fixed memory overhead beyond the values it holds, and with columns: true each row also stores its own copy of the six column names as keys. Numbers like the quantity and price are kept as strings, not compact numeric types. Five million small objects with repeated keys and stringified values add up to noticeably more than the raw text they came from.

So at the moment of the crash you are holding the full file as a string and the full file again, larger, as an object array, at the same time.

That is the core problem. The naive approach needs memory proportional to the file size, plus the object overhead on top. Double the file and you double the memory. There is always a file large enough to exceed whatever limit you set, and you rarely control how large the inputs get.

The fix: stream the rows through

The fix starts with a simple observation about the task itself: a revenue total does not need all the rows in memory at once. It needs each row once. You read a row, add it to the running totals, and discard it. At any instant you are holding one row, not five million.

That is what streaming gives you, and it changes the shape of the code. Instead of one call that returns the finished array, you set up a pipeline that hands you rows one at a time as it reads them, and you react to each one. In Node that reaction is an event: the parser emits a data event per row, an end event when the file is done, and an error event if something goes wrong. You attach a handler to each.

Before the code, one trap to avoid, because it is the easiest way to accidentally keep the crash. The import must be csv-parse, not csv-parse/sync. The sync version is the one that builds the whole array in memory, so importing it here would defeat the entire point. Reach for the plain csv-parse, which is the streaming one.

// stream.js
const fs = require("fs");
const { parse } = require("csv-parse");   // streaming, not csv-parse/sync

const revenue = {};
let count = 0;

// Read the file as a stream and feed it through the parser.
const parser = fs.createReadStream(infile).pipe(parse({ columns: true }));

// Fires once per row. Aggregate here, then let the row go.
parser.on("data", (r) => {
  revenue[r.region] =
    (revenue[r.region] || 0) + Number(r.quantity) * Number(r.unit_price);
  count++;
});

// Fires once, after the last row.
parser.on("end", () => {
  console.log(`${count.toLocaleString()} rows total`);
  console.log(revenue);
});

// Streaming errors will not throw into a try/catch, so handle them here.
parser.on("error", (err) => {
  console.error("Parse error:", err.message);
  process.exit(1);
});

The important line is the data handler. Each row arrives, gets folded into the running totals, and is then eligible to be garbage-collected before the next one shows up. Nothing accumulates. That single change, from "collect all rows, then process" to "process each row as it arrives," is the whole fix.

One note on the error handler, since streaming makes it less obvious than it looks. A failure partway through the file is emitted as an event, not thrown as an exception, so a try/catch around this code would never catch it. You handle stream failures by listening for error, which is easy to forget when you are used to synchronous code.

Run it against the same 199 MB file, under the same 256 MB cap that just killed the naive version:

$ node --max-old-space-size=256 stream.js orders.csv
  1,000,000 rows processed, heap used: 6 MB
  2,000,000 rows processed, heap used: 7 MB
  3,000,000 rows processed, heap used: 6 MB
  4,000,000 rows processed, heap used: 8 MB
  5,000,000 rows processed, heap used: 7 MB

Done. 5,000,000 rows total.
Revenue by region:
  north: 1613500000.01
  south: 1680250000.01
  east: 1619750000.01
  west: 1686750000.01

peak heap used: 8 MB

It finishes, it produces the answer the naive version never reached, and the heap stays flat at around 7 MB the entire time. Not 200 MB. Not growing. Flat, because the amount of data in memory does not depend on the size of the file.

Proving the point: a much bigger file

If memory is truly constant, a far larger file should make no difference. Here is the same streaming script on an 810 MB file, four times the size of the one that crashed the naive version, under the same 256 MB heap:

$ node --max-old-space-size=256 stream.js big.csv
Done. 20,000,000 rows total.
Revenue by region:
  north: 6453999999.85
  ...

peak heap used: 9 MB

Twenty million rows, 810 MB on disk, peak heap of 9 MB. The file is four times larger and the memory is essentially unchanged. That is the whole advantage of streaming stated as a measurement: the memory tracks the size of one row, not the size of the file.

When the naive version is fine

Streaming is not always the right call, and reaching for it reflexively adds event-handler complexity you may not need.

If the file is small and bounded, and you control how large it can get, readFileSync plus a sync parse is simpler and perfectly fine. A config file, a lookup table of a few thousand rows, a test fixture: load it and move on. The sync parser also lets you use the parsed array directly with ordinary array methods, which is more convenient when the data genuinely fits.

The deciding factor is whether the file size is bounded by something you control. A file you generate internally with a known, small row count is safe. A file uploaded by a user, exported from a growing database, or received from a third party has no upper bound you can rely on, and that is where streaming stops being optional.

Takeaways

1. fs.readFileSync plus a sync parse holds the whole file, twice: once as a string, once as a bigger object array. Memory scales with file size.
2. There is always a file large enough to exceed your heap, and you rarely control how large inputs get.
3. Streaming holds one row at a time, so memory is constant regardless of file size. The proof is a flat heap while row count climbs into the tens of millions.
4. Watch the import: csv-parse/sync builds the full array, csv-parse streams. The difference is a crash or a flat 8 MB.
5. Use the simple version when the file size is bounded by something you control. Stream when it is not.

The generator and both versions are three short files and run on any recent Node with only csv-parse installed.