This post was created with AI assistance and reviewed for accuracy before publishing.
Most retrieval pipelines start the same way. Read the uploaded file, extract its text, split it into chunks, embed the chunks, write them to the database. Written with await readFile() at the top it is about fifteen lines, and it works perfectly until a customer uploads a document larger than the memory you have.
The problem is not just peak memory. It is that a single large upload can take down the process handling everyone else's, and that failures arrive after you have already spent money embedding half the file.
Where the memory actually goes
There are three places a naive pipeline buffers, and they compound:
const buf = await readFile(path); // 1. whole file in memory
const text = await extractText(buf); // 2. whole extracted text
const chunks = splitIntoChunks(text); // 3. every chunk, at once
await Promise.all(chunks.map(embed)); // and every request in flight
A 400MB PDF can hold well over a gigabyte across those stages before a single embedding call returns. The final line is the worst of them: Promise.all over ten thousand chunks issues ten thousand concurrent HTTP requests, which will exhaust sockets, trip provider rate limits, and produce a rejection that discards all the successful work alongside the failed call.
Streams turn peak memory into steady memory
The stream version processes a window at a time, so memory tracks chunk size rather than file size:
import { pipeline } from 'node:stream/promises';
import { createReadStream } from 'node:fs';
await pipeline(
createReadStream(path),
extractTextStream(), // Transform: bytes to text
chunkStream({ size: 1000, overlap: 100 }),
embedStream({ concurrency: 5 }),
writeToDatabase()
);
Use pipeline from node:stream/promises rather than chaining .pipe(). It propagates errors through every stage and destroys the whole chain when one fails, which is the difference between a clean failure and a leaked file descriptor plus a half-written batch. Chained .pipe() calls do neither.
Backpressure is the part that matters
Backpressure is the mechanism that makes this work, and it is easy to accidentally disable.
When a writable stage cannot keep up, it stops accepting data, and that signal travels back up the chain until the file read pauses. Embedding is always the slowest stage in a RAG pipeline, by orders of magnitude: parsing is CPU-bound and fast, embedding is a network round trip. Backpressure is what stops the parser from racing ahead and queuing the entire document in front of the embedder.
You lose it the moment a stage buffers internally:
// Breaks backpressure: collects everything, then acts.
const all = [];
for await (const chunk of source) all.push(chunk);
// Preserves it: one item at a time, with await between.
for await (const chunk of source) {
await handle(chunk);
}
The for await loop is the simplest correct consumer. It will not request the next chunk until the body has finished, so the upstream source stays paused while you work.
Batch the embedding calls
Embedding one chunk per request wastes most of the round trip. Every provider accepts arrays, and a batch of 50 to 100 costs roughly the same wall-clock time as a batch of one.
async function* batched(source, size = 64) {
let batch = [];
for await (const item of source) {
batch.push(item);
if (batch.length >= size) {
yield batch;
batch = [];
}
}
if (batch.length) yield batch; // the tail matters
}
for await (const batch of batched(chunks)) {
const vectors = await embedMany(batch.map((c) => c.text));
await db.insertMany(batch, vectors);
}
Forgetting the final partial batch is a common bug, and a quiet one: every document silently loses its last few dozen chunks, which usually includes the conclusion.
Insert as you go rather than accumulating vectors for a single write at the end. Embeddings are large, holding them all defeats the purpose of streaming, and incremental writes mean a failure at 80% keeps the first 80%.
Make it resumable, because it will fail
A long ingest will be interrupted. The process restarts, the network blips, the provider rate limits you. If a failure means starting over, you pay for every embedding twice.
Record progress alongside the data:
await db.insertMany(batch, vectors);
await db.updateJob(jobId, { lastChunkIndex: batch.at(-1).index });
On restart, skip to lastChunkIndex and continue. This ordering matters: write the chunks first, then the progress marker. If it crashes between the two you re-embed one batch, which is cheap. Reverse the order and a crash loses that batch entirely, with the job convinced it succeeded.
For rate limits specifically, retry the batch with exponential backoff rather than failing the job. A 429 is a request to wait, not an error condition.
Verify with the number that matters
The test for all of this is not whether it completes. It is whether memory stays flat.
setInterval(() => {
const mb = process.memoryUsage().heapUsed / 1024 / 1024;
console.log(`heap: ${Math.round(mb)}MB`);
}, 1000);
Run your largest realistic document through and watch. A correct streaming pipeline holds roughly steady regardless of file size. If the line climbs with the document, something in the chain is still buffering, and the profile will tell you which stage by where the climb begins.
That single graph is worth more than any amount of reasoning about the code, because every one of the mistakes above looks fine when you read it.
Top comments (1)
The final heapUsed graph is useful, but it can miss the exact memory pressure your 400MB PDF example creates. Node Buffers and native parser allocations can sit outside the V8 heap, so I would record rss, external, and arrayBuffers alongside heapUsed. A flat heap alone can still end in an operating-system kill.
I would also verify that extractTextStream actually parses incrementally. Wrapping a parser that first builds a complete document tree in a Transform preserves downstream backpressure without removing that upstream peak. Testing two concurrent large uploads would expose that distinction quickly.