DEV Community

Cover image for How to export millions of rows to Excel without running out of memory
Omkar Palika
Omkar Palika

Posted on Fully Autonomous

How to export millions of rows to Excel without running out of memory

Most Excel libraries for JavaScript build the whole workbook in memory before they write a byte. That's fine for a few thousand rows. At a few million, a serverless function with a small memory limit can run out, and a browser tab can freeze.

The fix is to stream: produce a row, compress it, send it, forget it. An .xlsx file is a ZIP of XML files, and both ZIP and XML can be written one piece at a time, so nothing forces a library to hold the workbook.

I wrote XlsxFlow to do that. Here are three cases where it matters. All the code below is copied from the repo's recipes, which run in CI on every push.

1. A download from an API endpoint

This works as a Next.js route handler, a Cloudflare Worker, or with Bun.serve and Deno.serve:

import { SheetWriter } from '@xlsxflow/core';

// Stand-in for your data source: a database cursor, paginated API calls, anything async
async function* orders() {
  for (let id = 1; id <= 10_000; id++) yield { id, customer: `Customer ${id % 50}`, total: id * 1.5, placed: new Date() };
}

async function* rows() {
  yield ['Order', 'Customer', 'Total', 'Placed'];
  for await (const o of orders()) yield [o.id, o.customer, { value: o.total, style: { numFmt: '#,##0.00' } }, o.placed];
}

export function GET() {
  const xlsx = new SheetWriter().addSheet('Orders', rows(), { freezePanes: { row: 1 } }).write();
  return new Response(xlsx, {
    headers: {
      'Content-Type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
      'Content-Disposition': 'attachment; filename="orders.xlsx"',
    },
  });
}
Enter fullscreen mode Exit fullscreen mode

The writer takes an async generator and pulls the next row only when the earlier ones have been sent. A slow client slows the database query down instead of filling memory.

In Express, use pipeline rather than .pipe(). Testing the examples for this article showed why: with .pipe(res), a client that cancels the download leaves your generator paused forever, with its database cursor still open. pipeline cancels it.

import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';

// rows() is the generator from the example above
app.get('/export', (req, res) => {
  res.setHeader('Content-Disposition', 'attachment; filename="orders.xlsx"');
  pipeline(Readable.fromWeb(new SheetWriter().addSheet('Orders', rows()).write()), res).catch(() => {});
});
Enter fullscreen mode Exit fullscreen mode

2. Reading an upload

import { SheetReader, createBlobReader } from '@xlsxflow/core';

export async function POST(request) {
  const file = (await request.formData()).get('file');
  if (!(file instanceof Blob)) return Response.json({ error: 'No file uploaded' }, { status: 400 });

  try {
    // Uploads are untrusted: cap how much the file may unpack to
    const rows = await new SheetReader().parse(createBlobReader(file), { maxUncompressedBytes: 50_000_000 });
    let header;
    const records = [];
    for await (const row of rows) {
      if (!header) { header = row.cells.map(String); continue; }
      records.push(Object.fromEntries(header.map((key, i) => [key, row.cells[i] ?? null])));
    }
    return Response.json({ columns: header ?? [], records });
  } catch (e) {
    // Not a spreadsheet, damaged, password-protected, or over the size limit
    return Response.json({ error: e.message }, { status: 422 });
  }
}
Enter fullscreen mode Exit fullscreen mode

Rows arrive one at a time, and maxUncompressedBytes stops a small zip bomb from expanding into gigabytes. The same code reads .xls and .ods files.

3. An "Export to Excel" button in the browser

import { SheetWriter } from '@xlsxflow/core';

export async function downloadXlsx(filename, header, records) {
  const rows = [
    header.map((h) => ({ value: h, style: { font: { bold: true }, fill: { type: 'solid', fgColor: 'FFE2E8F0' } } })),
    ...records,
  ];
  const blob = await new Response(new SheetWriter().addSheet('Sheet1', rows, { freezePanes: { row: 1 }, autoFitColumns: true }).write()).blob();
  const url = URL.createObjectURL(blob);
  const link = Object.assign(document.createElement('a'), { href: url, download: filename });
  link.click();
  setTimeout(() => URL.revokeObjectURL(url), 1000);
}

downloadXlsx('people.xlsx', ['Name', 'Age'], [['Ana', 34], ['Bo', 27]]);
Enter fullscreen mode Exit fullscreen mode

No server involved. CI runs this in Chromium, Firefox and WebKit and checks the downloaded file.

How much memory does it use?

CI writes 10 million cells to disk and reads them back with Node's heap capped at 32 MB; the file is 34 MB. On my machine, writing 10M cells adds about 2 MB of heap and takes 19 seconds. ExcelJS's streaming writer takes 20.5 seconds and 7 MB, so the speed is similar. The difference is that one writer handles everything, styles and formulas included, and the same code runs in browsers and Cloudflare Workers. The benchmarks have the scripts to reproduce this.

Trying it

npm install @xlsxflow/core
Enter fullscreen mode Exit fullscreen mode

It has no dependencies and is MIT licensed. If you're coming from ExcelJS or SheetJS, there are migration guides with each task side by side: ExcelJS, SheetJS.

I'm the author. There's also a paid add-on, @xlsxflow/pro, for charts, pivot tables and password-protected files; everything in this article is in the free core.


This article was drafted with AI assistance. The code comes from the project's recipes, which run in CI, and the numbers from its tests and benchmarks.

Top comments (1)

Collapse
 
kata_omel_3f34301f3ec5928 profile image
Kata Omel •

The cancellation example is useful because closing the download should also release the database cursor. One distinction worth making explicit: the upload recipe accumulates records, and the browser recipe materialises both the input rows and the final Blob. Those complete paths are still size-dependent even if the reader or writer itself streams.

For a large export, keep the source as an async generator all the way into the writer; where the browser supports a writable file stream, piping the output there avoids retaining the final Blob. The Blob approach is convenient for bounded exports.

There is also Excel's worksheet limit: 1,048,576 rows, so a sheet with one header row has room for 1,048,575 data rows. Does addSheet reject an oversized sheet, or should the caller split the generator across sheets? That boundary belongs in a millions-of-rows export test alongside cancellation and heap usage.