DEV Community: Omkar Palika

How to export millions of rows to Excel without running out of memory

Omkar Palika — Fri, 09 Oct 2026 12:35:27 +0000

Most Excel libraries for JavaScript build the whole workbook in memory before they write a byte. That's fine for a few thousand rows. At a few million, a serverless function with a small memory limit can run out, and a browser tab can freeze.

The fix is to stream: produce a row, compress it, send it, forget it. An .xlsx file is a ZIP of XML files, and both ZIP and XML can be written one piece at a time, so nothing forces a library to hold the workbook.

I wrote XlsxFlow to do that. Here are three cases where it matters. All the code below is copied from the repo's recipes, which run in CI on every push.

1. A download from an API endpoint

This works as a Next.js route handler, a Cloudflare Worker, or with Bun.serve and Deno.serve:

import { SheetWriter } from '@xlsxflow/core';

// Stand-in for your data source: a database cursor, paginated API calls, anything async
async function* orders() {
  for (let id = 1; id <= 10_000; id++) yield { id, customer: `Customer ${id % 50}`, total: id * 1.5, placed: new Date() };
}

async function* rows() {
  yield ['Order', 'Customer', 'Total', 'Placed'];
  for await (const o of orders()) yield [o.id, o.customer, { value: o.total, style: { numFmt: '#,##0.00' } }, o.placed];
}

export function GET() {
  const xlsx = new SheetWriter().addSheet('Orders', rows(), { freezePanes: { row: 1 } }).write();
  return new Response(xlsx, {
    headers: {
      'Content-Type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
      'Content-Disposition': 'attachment; filename="orders.xlsx"',
    },
  });
}

The writer takes an async generator and pulls the next row only when the earlier ones have been sent. A slow client slows the database query down instead of filling memory.

In Express, use pipeline rather than .pipe(). Testing the examples for this article showed why: with .pipe(res), a client that cancels the download leaves your generator paused forever, with its database cursor still open. pipeline cancels it.

import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';

// rows() is the generator from the example above
app.get('/export', (req, res) => {
  res.setHeader('Content-Disposition', 'attachment; filename="orders.xlsx"');
  pipeline(Readable.fromWeb(new SheetWriter().addSheet('Orders', rows()).write()), res).catch(() => {});
});

2. Reading an upload

import { SheetReader, createBlobReader } from '@xlsxflow/core';

export async function POST(request) {
  const file = (await request.formData()).get('file');
  if (!(file instanceof Blob)) return Response.json({ error: 'No file uploaded' }, { status: 400 });

  try {
    // Uploads are untrusted: cap how much the file may unpack to
    const rows = await new SheetReader().parse(createBlobReader(file), { maxUncompressedBytes: 50_000_000 });
    let header;
    const records = [];
    for await (const row of rows) {
      if (!header) { header = row.cells.map(String); continue; }
      records.push(Object.fromEntries(header.map((key, i) => [key, row.cells[i] ?? null])));
    }
    return Response.json({ columns: header ?? [], records });
  } catch (e) {
    // Not a spreadsheet, damaged, password-protected, or over the size limit
    return Response.json({ error: e.message }, { status: 422 });
  }
}

Rows arrive one at a time, and maxUncompressedBytes stops a small zip bomb from expanding into gigabytes. The same code reads .xls and .ods files.

3. An "Export to Excel" button in the browser

import { SheetWriter } from '@xlsxflow/core';

export async function downloadXlsx(filename, header, records) {
  const rows = [
    header.map((h) => ({ value: h, style: { font: { bold: true }, fill: { type: 'solid', fgColor: 'FFE2E8F0' } } })),
    ...records,
  ];
  const blob = await new Response(new SheetWriter().addSheet('Sheet1', rows, { freezePanes: { row: 1 }, autoFitColumns: true }).write()).blob();
  const url = URL.createObjectURL(blob);
  const link = Object.assign(document.createElement('a'), { href: url, download: filename });
  link.click();
  setTimeout(() => URL.revokeObjectURL(url), 1000);
}

downloadXlsx('people.xlsx', ['Name', 'Age'], [['Ana', 34], ['Bo', 27]]);

No server involved. CI runs this in Chromium, Firefox and WebKit and checks the downloaded file.

How much memory does it use?

CI writes 10 million cells to disk and reads them back with Node's heap capped at 32 MB; the file is 34 MB. On my machine, writing 10M cells adds about 2 MB of heap and takes 19 seconds. ExcelJS's streaming writer takes 20.5 seconds and 7 MB, so the speed is similar. The difference is that one writer handles everything, styles and formulas included, and the same code runs in browsers and Cloudflare Workers. The benchmarks have the scripts to reproduce this.

Trying it

npm install @xlsxflow/core

It has no dependencies and is MIT licensed. If you're coming from ExcelJS or SheetJS, there are migration guides with each task side by side: ExcelJS, SheetJS.

I'm the author. There's also a paid add-on, @xlsxflow/pro, for charts, pivot tables and password-protected files; everything in this article is in the free core.

This article was drafted with AI assistance. The code comes from the project's recipes, which run in CI, and the numbers from its tests and benchmarks.

I sent an AI agent to buy from real stores. Here's what actually breaks.

Omkar Palika — Fri, 17 Jul 2026 17:20:28 +0000

AI agents are starting to shop for people. ChatGPT has an operator that clicks around the web. Perplexity ships a "buy" button. Anthropic and others are wiring agents into real checkout flows. If you run an online store, some fraction of your future customers won't be humans clicking — they'll be agents acting on a human's behalf.

Which raised a question I couldn't answer for my own projects: can an AI agent actually complete a purchase on this store?

Every "agent-readiness" tool I found just lints your markup — does your robots.txt allow agent crawlers, do you have structured data, that kind of thing. Useful, but it answers a different question. Clean markup doesn't mean an agent can navigate your funnel any more than a valid HTML resume means you can do the job. The only way to know is to send an agent through and watch where it dies.

So I built AgentiQA. It's open source (MIT). Here's how it works and — more interestingly — what I found when I pointed it at real stores.

Two layers

Static checks (stdlib only, no API key): does robots.txt allow agent user-agents (Claude-User, ChatGPT-User, OAI-SearchBot, PerplexityBot), is there JSON-LD Product/Offer data, OpenGraph tags, a reachable sitemap. This is the cheap "is the door unlocked" pass.

The live shopper agent: Claude driving a real headless Chromium browser through the actual funnel — find a product, add to cart, reach checkout — recording milestones as it goes. This is the part that answers the real question.

How the agent "sees" the page

No screenshots, no vision model. The agent reads a text snapshot of the DOM: the page URL and title, every interactive element indexed by a number, and the visible text. Roughly:

URL: https://store.example/
TITLE: All Products

INTERACTIVE ELEMENTS:
[0] <a href=/product/blue-top> Blue Top
[1] <button> Add to cart
[2] <input type=text name=search> Search
...

VISIBLE TEXT (truncated):
Blue Top  Rs. 500  In Stock ...

The agent gets a tiny toolset — read_page, click(element_id), type_text(element_id, text), goto(url), record_milestone(stage) — and loops: read the page, decide, act, read again. Text-only is cheaper than vision, deterministic, and it's roughly what an agent shopper actually operates on anyway.

The safety part that matters: the payment guard is code, not a prompt

Any tool that drives a browser through checkout has to never touch payment data. You cannot rely on a system prompt for that — prompts are suggestions, and a model under pressure to "complete the task" can rationalize its way past one.

So the refusal lives in the executor, not the instructions. Before any keystroke, the field is checked:

PAYMENT_FIELD_RE = re.compile(r"card|cvv|cvc|cc-|expir|security.?code|pan\b", re.IGNORECASE)

def is_payment_field(el) -> bool:
    for attr in ("name", "id", "autocomplete", "placeholder", "aria-label"):
        v = el.get_attribute(attr)
        if v and PAYMENT_FIELD_RE.search(v):
            return True
    return False

If the agent tries to type into anything that looks like a payment field, the tool returns a hard REFUSED and tells it to stop and summarize. It also never creates accounts and never submits a final order — it stops at the checkout page. The model's judgment is a second layer, not the only one.

Zero-cost driver: the Claude CLI, not just an API key

The obvious way to drive this is the Anthropic API, which costs ~$1–2 per full audit. But for a solo dev validating an idea, I didn't want a per-run bill. So there's a second driver that speaks to the claude CLI (the Claude Code subscription) over a small JSON protocol — one action per turn, resumed across steps:

python -m agentiqa https://your-store.com --driver cli   # subscription, no API key
python -m agentiqa https://your-store.com --driver api   # API key, faster
# default: api if ANTHROPIC_API_KEY is set, else cli

Same browser loop, same safety guards — just a different way to get the model's next move. It means you can run the whole thing on a subscription you already pay for.

What I actually found

Here's the honest part, and it wasn't what I expected going in.

I ran the benchmark against a set of real, public demo stores — the kind built to be tested against, with genuine cart and checkout flows. My assumption was that agents would faceplant everywhere. They didn't.

On clean, functional stores, the agent checked out fine. Product → cart → checkout, no drama. One "failure" in my batch turned out to be a dead server throwing a Cloudflare SSL error — not an agent problem at all.

The failures that are real are dumber and quieter than "the agent is dumb":

An "add to cart" button that works for a human click but produces no state an agent can detect — no cart-count change, no confirmation, no visible signal. A human sees the cart bump and moves on. The agent has no way to know it worked, so it retries or stalls.
No cart link anywhere in the nav. Humans know to look top-right. An agent needs a discoverable path.
Product data an agent can't parse — price and availability rendered in a way that's obvious visually but not in the DOM text.

None of these show up in a markup linter. And here's the kicker I found: a store scored 1/4 on static readiness and the agent still checked out perfectly. Static score did not predict agent success. The behavioral test and the markup test measure different things — you need the behavioral one.

Try it on your store

pip install -r requirements.txt
playwright install chromium

# free static check, no API key:
python -m agentiqa https://your-store.com --no-agent

# full run with the live agent:
python -m agentiqa https://your-store.com

You get an HTML report: a verdict banner (did the agent complete checkout?), a prioritized fix list, the funnel, and the full step-by-step transcript of what the agent did. There's also a batch mode that runs a list of stores and emits an aggregate leaderboard.

Only run it against stores you own or have permission to test.

Where this goes

Agentic commerce is early. Right now the interesting finding is that the gap between "works for humans" and "works for agents" is made of small, invisible things — and nobody's funnel-testing for them yet. The tool is open source; I'd genuinely like to know what breaks on your store, because my sample is small and demo stores are easier than messy production ones.

Repo, demo GIF, and the full report format: https://github.com/OmkarPalika/agentiqa

If you run an e-commerce store and want to know whether an AI agent can buy from you — or you just want to see an agent narrate its way through a checkout and give up at a broken button — point it at your site and tell me what you find.