fs.readFileSync(path, "utf8").split("\n") works fine on a 500-row test fixture and falls over on a client's real 3-million-row export, either crashing with an out-of-memory error or, more insidiously, running slowly enough that a process manager kills it for exceeding a timeout nobody expected a "simple CSV import" to hit. The fix isn't a bigger server. It's not loading the whole file into memory in the first place.
Why the naive approach doesn't scale
Reading a file synchronously and splitting it on newlines requires the entire file's contents, plus the array of resulting lines, plus whatever intermediate parsed representation you build, to fit in memory simultaneously. For a small config file this is irrelevant. For a multi-gigabyte export, it's the difference between a script that runs in seconds and one that never completes at all on a machine with typical memory limits.
Node's streams exist specifically for this class of problem: process data in chunks as it arrives, never holding more than a small window of it in memory at once.
Setting up a streaming CSV parse
The csv-parse package (the streaming API from the csv.js.org project) is a solid default for this. Piped through a readable file stream, it emits parsed records one at a time as they become available, rather than waiting for the whole file.
const fs = require("fs");
const { parse } = require("csv-parse");
const parser = fs
.createReadStream("export.csv")
.pipe(parse({ columns: true, bom: true, skip_empty_lines: true }));
for await (const record of parser) {
await processRecord(record);
}
The bom: true option strips a byte-order mark automatically, the same issue covered in more depth in our main guide to CSV parsing, and columns: true uses the first row as field names so each record comes through as an object rather than a bare array.
Backpressure is the part most tutorials skip
Iterating with for await over the parser handles backpressure correctly by default, pausing the underlying read stream when your processing can't keep up. This matters more than it sounds like it should: if processRecord involves an async operation like a database write or an API call, and you don't respect backpressure, you can end up buffering tens of thousands of pending records in memory while waiting for a slow downstream operation, which defeats the entire purpose of streaming in the first place.
A version that gets this wrong looks tempting but is a trap:
// Don't do this on a large file - it ignores backpressure
parser.on("data", (record) => {
processRecord(record); // fires without waiting, can queue unboundedly
});
Firing off processRecord without awaiting it inside a data event handler kicks off every record's processing near-simultaneously as the parser emits them, which is exactly the unbounded-memory problem you were trying to avoid by streaming in the first place.
Batching writes without losing the memory benefit
Processing one row at a time is memory-safe but often too slow for the downstream write, especially against a database where a round trip per row adds up fast across millions of rows. The fix is batching a bounded number of records before flushing, not accumulating the whole file.
async function streamAndBatch(path, batchSize = 500) {
const parser = fs
.createReadStream(path)
.pipe(parse({ columns: true, bom: true }));
let batch = [];
for await (const record of parser) {
batch.push(record);
if (batch.length >= batchSize) {
await flushBatch(batch);
batch = [];
}
}
if (batch.length) {
await flushBatch(batch);
}
}
This keeps peak memory bounded to roughly batchSize records regardless of whether the file has ten thousand rows or ten million, while still getting the throughput benefit of batched writes over one-at-a-time inserts.
Handling malformed rows without killing the stream
By default, a single malformed row (wrong column count, an unescaped quote) will emit an error event and can terminate the stream if you're not handling it. For an import that needs to survive imperfect real-world files, listening for the parser's own error events and deciding row-by-row whether to skip or abort is usually the right call, following the same skip-and-log philosophy covered for malformed rows generally in the main guide linked above.
parser.on("skip", (err) => {
logSkippedRow(err.record, err.message);
});
Setting relax_column_count: true in the parse options tells the library to tolerate rows with the wrong number of fields rather than throwing, pairing it with your own downstream validation to decide whether a specific ragged row is usable or should be logged and skipped.
Handling a stream that errors out partway through
A file that's mostly fine but has one severely malformed section, a stretch of binary garbage from a corrupted export, say, can cause the parser to throw partway through a multi-million-row file, after you've already processed and committed a large batch of good rows. Deciding in advance whether a mid-stream failure should leave the already-processed rows in place or roll them back matters more here than in a smaller synchronous script, because the cost of redoing partial work on a multi-gigabyte file is much higher.
async function streamWithRecovery(path) {
let lastGoodBatch = [];
try {
await streamAndBatch(path);
} catch (err) {
await recordFailurePoint(path, err, lastGoodBatch);
throw err;
}
}
Recording enough context about where the stream failed, ideally including a byte offset or row number, means a retry can resume from roughly where it left off rather than reprocessing the entire file from scratch, which matters considerably more once a file is large enough that streaming was necessary in the first place.
Comparing this to a synchronous read on the same file
It's worth actually measuring the difference rather than assuming streaming is strictly better in every case. On a small file, a few thousand rows, fs.readFileSync plus a synchronous parse is often simpler code with negligible performance difference, and reaching for a streaming implementation adds complexity that isn't buying you anything. The crossover point where streaming starts to matter depends on your specific memory constraints and file sizes, but as a rough guide, once a file's uncompressed size approaches a meaningful fraction of your process's available memory, or once row counts move into the hundreds of thousands, the streaming approach's benefits stop being theoretical and start being the difference between a script that finishes and one that doesn't.
Measuring that it's actually working
Don't just assume streaming fixed the memory problem, verify it. Running the import with node --max-old-space-size=256 script.js against a large real file and watching whether it completes without hitting that artificially low memory ceiling is a fast, concrete way to confirm the streaming implementation is behaving the way you expect, rather than trusting that piping through csv-parse alone guarantees it.
Streaming plus bounded batching is the combination that actually matters here. Either one alone leaves a gap: streaming without batching is memory-safe but often too slow for high-volume writes, and batching without streaming just delays the memory problem until the whole file is buffered before batching starts.
For the rest of the CSV import pipeline, encoding detection, type coercion, and deciding what to do with malformed rows, see our full guide to CSV parsing code snippets from 137Foundry's data integration service.
Further reading: Node.js's own streams documentation covers backpressure semantics in more depth than fits in one article, and the Wikipedia entry on comma-separated values is a useful primer if you're explaining the format's quirks to a teammate who hasn't hit them yet.
Top comments (0)