DEV Community: KanuniLabs

A pivot table your users can drag, on a million rows, and how to tell whether it freezes the page

KanuniLabs — Sun, 11 Oct 2026 18:57:27 +0000

Every reporting screen gets the same request after its first week: "Can I see this by month instead?" Then by product, then by country and month together. A pivot table is the answer that stops the requests, because the user drags the fields where they want them and gets the report they were about to ask for.

The catch is the data. Every drag regroups and re-sums every row. With a few thousand rows nobody notices. With a million, the obvious implementation freezes the page while it works, and a page that freezes on every drag is a page people stop dragging in.

This post measures that on real data: the 1,067,371 invoice lines of the Online Retail II dataset, pivoted in the browser with our free PivotGrid. The demo times every recompute in your own browser, so you can check each number here against your machine.

The report a user builds

Start from revenue by country and year, then drag Month into the columns or Product into the rows. Each drop recomputes the whole table:

In React that is one component and a list of fields; area says where each field starts, and the user moves them from there:

import { BasePivotGrid } from '@kanunilabs/pivotgrid-react';
import '@kanunilabs/pivotgrid-react/styles.css';

const fields = [
  { id: 'country', dataField: 'country', caption: 'Country', dataType: 'string', area: 'row' },
  { id: 'year', dataField: 'year', caption: 'Year', dataType: 'string', area: 'column' },
  { id: 'revenue', dataField: 'revenue', caption: 'Revenue', dataType: 'number', area: 'data', summaryType: 'sum' },
  { id: 'month', dataField: 'month', caption: 'Month', dataType: 'string', area: 'filter' },  // waiting to be dragged
  { id: 'product', dataField: 'description', caption: 'Product', dataType: 'string', area: 'filter' },
];

<BasePivotGrid gridId="sales" data={rows} initialFields={fields} />;

What a drag costs

The demo measures two things on every recompute:

How long it takes, from the moment the pivot starts to the moment the new table is painted.
The longest pause on the page's main thread in that window. The browser's Long Tasks API reports every task of 50 ms or more, and the main thread is the one that also handles your clicks and keys. Rendering the new table counts too.

For country by year, or sales against cancellations by country, a recompute took 29 to 48 ms and no task on the main thread reached 50 ms. Drag Month into the columns and Year out, about 40 countries by 25 months, and it took 45 to 82 ms over three runs, with the longest pause between none and 54 ms:

The grouping and summing run in a Web Worker, a second thread the page starts for the pivot. The main thread only receives the finished totals and draws them.

The obvious way, measured

To see what the worker buys, the demo has a button that computes the same country-by-year revenue the way a quick implementation would: one pass over every row with a Map, on the main thread.

const totals = new Map<string, number>();
for (const r of rows) {
  const key = `${r.country}|${r.year}`;
  totals.set(key, (totals.get(key) ?? 0) + r.revenue);
}

That loop froze the page for about 110 ms, longer than the pivot's whole recompute. Its totals are the same as the pivot's: £7,511,064 for the United Kingdom in 2011 and £134,790 for France in 2010, which we also checked with an independent computation over the source file.

A tenth of a second does not sound like much until you type through it. The demo has a text box for that: type while you drag fields, and the letters keep appearing; type while the loop runs, and they arrive all at once when it ends.

The worker does not make the work free. It moves it off the thread that answers the user.

Where the page stopped, and the fix it led to

Put Product in the rows and Month in the columns, about 5,000 products by 25 months, sorted by name, and the measurement changed: on October 7 the recompute took 350 to 650 ms and the main thread paused for 290 to 510 ms.

The aggregation was still in the worker, so we looked at what was left on the main thread. Sorting the same table by revenue instead of by name cut the pause to 105 to 183 ms. The difference was how the 5,000 product names were compared:

// what the pivot's sort did before 1.7.2
names.sort((a, b) => a.localeCompare(b, locale, { numeric: true }));

// the same order, with the collator built once
const collator = new Intl.Collator(locale, { numeric: true });
names.sort(collator.compare);

Both give the same order. On 5,000 names, about 60,000 comparisons, the first took 206 to 226 ms and the second 4 to 5 ms, on the same machine. Called with an options object, localeCompare sets up its comparison rules again on every call; a collator sets them up once.

We shipped the second version in PivotGrid 1.7.2 on October 8 (@kanunilabs/pivotgrid-react 1.4.2). Measured again in the live demo, five loads each, the layout sorted by name now takes 151 to 196 ms with a longest pause of 97 to 118 ms, the same as sorted by revenue. You can check it at /demos/pivot?layout=product-month&sort=name.

If you sort strings anywhere a user waits, a table header, a filter list, a dropdown, this one is worth checking in your own code.

The remaining 100 or so milliseconds were in the React grid, not in the pivot. After the first paint it measured the panel widths and header heights and kept them in React state, so the roughly 1,400 visible cells were drawn a second time in the same task, only to change two paddings and a few CSS variables. It also wrote the row-header width variables on the grid's outer box, where every cell inherits them, so each write invalidated the style of every cell and the next measurement forced a full style recalculation. Now the measurements go through refs and the variables sit only on the panel that reads them. On a local production build of this demo, ten switches to Product × Month, the median longest pause went from 89 to 51 ms and the recompute from 155 to 102 ms. That fix shipped in @kanunilabs/pivotgrid-react 1.5.0 on October 9.

Measure your own pivot

You do not need our component to get these two numbers. Any pivot or table that recomputes on user input can be measured the same way:

const longTasks: PerformanceEntry[] = [];
new PerformanceObserver((list) => longTasks.push(...list.getEntries()))
  .observe({ type: 'longtask', buffered: true });

const t0 = performance.now();
await recompute();                       // whatever your table does on a drag
requestAnimationFrame(() => requestAnimationFrame(() => {
  const t1 = performance.now();          // the new table has been painted
  const pause = Math.max(0, ...longTasks
    .filter((t) => t.startTime + t.duration >= t0 && t.startTime <= t1)
    .map((t) => t.duration));
  console.log(`recompute ${t1 - t0} ms, longest main-thread pause ${pause} ms`);
}));

If the second number is large, your users feel it, whatever the first one says.

What this costs

It is free. Drag and drop, filters, sorting, % of totals and saved views are in the Community package, with no licence key and no watermark. Calculated fields in code and the rows behind a clicked cell are free too; the formula designer for end users, rendered charts, the ready-made drill-down window and styled Excel/PDF export are in the paid edition.
The data has to be in the browser. The demo downloads a 5.5 MB file once and decodes it into a million row objects before the first pivot. For data that should stay on the server, the paid edition can ask your server for the aggregates instead.
Wide layouts still pause the page. For 5,000 products by 25 months, after the two fixes above, the median longest pause was 51 ms on our local production build; slower machines see more. The panel in the demo shows what yours does.

Try it

The demo is at /demos/pivot; /demos/pivot?layout=product-month opens the wide layout, best sellers first, and adding &sort=name sorts it by name. The PivotGrid docs cover fields, filters, summary types and the React and plain JavaScript setups.

Data: Online Retail II, UCI Machine Learning Repository (Chen, 2019), CC BY 4.0.

Disclosure: this article was drafted with AI assistance. The demo and every number in it were measured against our packages and the published dataset in October 2026.

Select all in a million filtered rows, and send the server one line instead of a list

KanuniLabs — Mon, 05 Oct 2026 13:48:28 +0000

"How do I get the selected rows from a checkbox column?" has been read more than 82,000 times on Stack Overflow, and it is rarely about the checkbox. It comes up when a table has a filter on and a header checkbox, and someone asks:

When the user ticks "select all" on a filtered table, which rows are selected? The ones on screen, the ones the filter matches, or every row?
Which of them does my code get back?
What do I send to the server when "all" means a million rows?

This post answers each one on real data, with the React code. The demo loads the 1,067,371 invoice lines of the Online Retail II dataset into the free Community DataGrid. You can tick rows, filter by country, and see what the selection would look like as a request.

Selection belongs to rows, not positions

The first rule is the one that makes the others possible. A selection is a set of row keys, not row indexes:

import { DataGrid } from '@kanunilabs/datagrid-react';

<DataGrid
  dataSource={rows}
  rowKey="id"
  columns={columns}
  selection={{ mode: 'multiple', checkboxColumn: true }}
/>

Sort the table and the ticked rows move with their data. Filter them out of view and back, and they are still ticked. An index-based selection points at whichever row happens to be third after the sort, and that is a bug report waiting to happen.

What "select all" selects

With a filter on, the header checkbox selects the rows the filter matches. Not just the rows on screen, and not the rows the filter hides.

Filter to France and tick the header. That selects 14,330 lines:

Untick three of them and you have 14,327. So far this is what everyone expects. The interesting part is what that selection is made of.

A million keys is a lot to hold and more to send

The obvious way to select all is to put every matching key in a set. For France that is 14,330 keys and nobody notices. Clear the filter and tick the header, and it is 1,067,371 keys, built on the main thread at the moment of the click. Then the user presses "Archive selected" and you send them.

The demo has a button that does exactly that and times it in your browser. With three rows unticked from the whole table, the list is 1,067,368 ids and 7.1 MB of JSON. On our machine it took about a tenth of a second just to build, before the network.

That is the eager form. There is a second one.

Deferred: a rule and its exceptions

Switch on deferred and select-all stops building a list. It records one flag, "everything the current filter matches", and every row the user unticks afterwards goes into a list of exceptions:

<DataGrid
  dataSource={rows}
  rowKey="id"
  columns={columns}
  selection={{ mode: 'multiple', checkboxColumn: true, deferred: true }}
  onReady={setGrid}
/>

Read it back with getSelectionDescriptor():

grid.getSelectionDescriptor();
// { allMatching: true, keys: [73, 75, 77] }   ← keys are the EXCEPTIONS here

When allMatching is false, keys is the plain list of ticked rows, as before. The descriptor costs as much as the exceptions do, whether one row is selected or a million.

Send the rule, and the filter it was made under

A descriptor with allMatching: true means nothing without the filter, so send both:

const archive = async () => {
  const { allMatching, keys } = grid.getSelectionDescriptor();
  await fetch('/api/invoices/archive', {
    method: 'POST',
    body: JSON.stringify(
      allMatching
        ? {
            filter: grid.isFilterEnabled() ? grid.getEffectiveFilter() : null,
            search: grid.getSearchText(),
            except: keys,
          }
        : { ids: keys },
    ),
  });
};

The server turns it back into a query, something like:

UPDATE invoice_lines SET archived = true
WHERE country = 'France' AND id NOT IN (73, 75, 77);

Three things are worth knowing before you build on this:

Your server has to apply the same filter. getEffectiveFilter() gives you the conditions the grid ran, and getSearchText() the quick search, which narrows the rows too. Your endpoint is responsible for running them identically. A filter the server reads differently (case, time zone, a column it does not know) selects different rows.
The rule is evaluated when the server runs it. A line added in France between the click and the request is included, because it matches. That is usually what "all matching" means, but if your users expect "exactly the rows I saw", send ids.
The rule follows the filter. Select all in France, then switch the filter to Germany, and the selection is now Germany's 17,624 rows: it was always "everything the current filter matches". The three French exceptions travel along, harmlessly, since no German row has those ids. If your screen should not allow that, clear the selection when the filter changes.

Which rows does my code get back?

This is the part of the Stack Overflow question that still trips people up, because two methods answer it differently:

getSelectedRows() returns the selected rows that are in the current view, in view order. A ticked row that the filter now hides is not in it.
getSelectedKeys() returns every selected key, visible or not. Under deferred selection the rule only covers the current view, so it walks the view to expand it into a list, which is the cost the mode exists to avoid.
getSelectedCount() is the number to show the user.

With the eager form, the count includes ticked rows the filter now hides. So if you show "14,327 selected" and then act on getSelectedRows() after the user changed the filter, you are acting on fewer rows than you said. Decide which one you mean and use the same method for both.

All of it in one take: a filter, select all, three exceptions, a different filter, and the cost of turning it into a list:

Two bugs this post found

Preparing the screenshots for this post turned up two bugs in deferred select-all. Both are fixed in @kanunilabs/datagrid-react 1.3.0 (with @kanunilabs/datagrid-core 1.5.0), released on 5 October:

The selectionChanged event reported a select-all as no rows and a count of 0, and after the user unticked a row it reported that row as the selection. "Export selected rows only" read the selection the same way and wrote every row, the unticked ones included.
After a filter change, the count subtracted exceptions the new filter hides. The Germany step above showed 17,621 selected while every one of the 17,624 German rows was ticked: the three French exceptions were taken off a count they were never part of.

On an older version, read the selection from getSelectionDescriptor(), which was right all along, not from the event.

What this costs

It is free. Selection, the checkbox column and deferred select-all are in the Community package.
Your server does the selecting. A deferred selection is a query, and your endpoint has to run it. If you cannot express your filters on the server, use the eager form and send ids.
Everything here is in the browser. The demo holds all 1,067,371 lines in the page; the server-side data docs cover the case where the server holds them and the grid only asks for a page at a time.

Try it

The demo is at /demos/selection, and /demos/selection?country=France opens it filtered. The selection docs cover single and multiple modes, row-click behaviour and the keyboard.

Data: Online Retail II, UCI Machine Learning Repository (Chen, 2019), CC BY 4.0.

Disclosure: this article was drafted with AI assistance. The demo and every behaviour described were checked against our packages and the published dataset in October 2026.

The hard part of an editable table is knowing what changed

KanuniLabs — Sat, 03 Oct 2026 06:40:26 +0000

Editing is the feature people ask data grid libraries for most, and "how do I get the row's values after an edit?" is one of the most-read grid questions on Stack Overflow. The question is telling. Nobody struggles to put an input box in a cell. They struggle with what comes after it:

Which cells did the user change, and what were they before?
Is this value allowed, and how does the user find out it is not?
Can they take an edit back?
What do we send to the server, and in what shape?
What happens to their work when the server says no?

This post walks through each one on a real order, with the React code. The demo has 240 order lines a UK online retailer sold to customers in France in the first week of December 2011, from the Online Retail II dataset. Its "server" is a function in the page that logs every request, so you can watch what a save sends.

Keep edits out of the data until Save

The first decision is where an edit lives before it is saved. The tempting answer is "in the row": the user types 24, you write 24 into the row object. Then Cancel has to remember that it used to be 10, and so does undo, and so does any code that asks "what changed?".

The grid does the opposite. Pending edits are held in a separate change store, keyed by the row's key, and your data is not touched until a save succeeds. Cancel is then just "drop the pending edits". Keying by row rather than by position means an edit stays on its record when the user sorts, filters or pages away and back.

In batch mode, every change waits until the user presses Save, and the toolbar counts the rows that will be sent:

import { EnterpriseDataGrid } from '@kanunilabs/datagrid-react-enterprise';

<EnterpriseDataGrid
  dataSource={orders}
  rowKey="id"
  columns={columns}
  toolbar
  editing={{
    mode: 'batch',
    allowUpdating: (row, columnId) => ['description', 'quantity', 'price'].includes(columnId),
    rules,
    crud: { update },
  }}
/>

Refuse a bad value where it is typed

Rules are declared per column and checked as the user types, and again on save:

const rules = {
  description: [{ type: 'required', message: 'A product needs a name' }],
  quantity: [
    { type: 'range', min: 1, message: 'Quantity must be at least 1' },
    { type: 'custom', validate: ({ value }) => Number.isInteger(Number(value)) || 'Quantity must be a whole number' },
  ],
  price: [{ type: 'range', min: 0.01, message: 'Price must be above zero' }],
};

Type −5 into a quantity and the editor refuses to close, with the reason under the cell:

The message is the part we nearly got wrong. While taking the screenshots for this post we found that it was rendered in the right place and never seen: the cell clips anything that overflows it, and the next row is drawn on top of the one being edited. The user got a red underline and no reason. It is fixed in @kanunilabs/datagrid-react-enterprise 1.3.2, released on 1 October.

Take an edit back

Ctrl+Z undoes and Ctrl+Y redoes. Each step is one user action, not one cell: undoing a paste of 200 cells undoes the paste, not one cell of it. The history keeps the last 100 actions by default, and it is a public API (undo(), redo(), canUndo()), so a toolbar button can use it too.

Send only what changed

When the user presses Save, the grid calls your handler once per changed row, with the changed columns only:

const update = async (key, values, row) => {
  const res = await fetch(`/api/orders/${key}`, {
    method: 'PATCH',
    body: JSON.stringify(values), // { quantity: 24 }, not the whole row
  });
  return res.json(); // the stored row
};

That is the shape a REST endpoint or an UPDATE … SET wants. With three edits in three rows, the demo's log shows three requests, each carrying one field:

The handlers run one at a time: inserts first, then updates, then removes, so a back end with foreign keys sees a parent row before the child that refers to it. Saves themselves also run one at a time. Two saves in quick succession used to snapshot the same pending edit and send it twice; the second save now waits for the first.

The return value matters

The comment on that return res.json() line is there because the demo got it wrong first. The grid does not write your data on its own. If update returns the stored row, the grid adopts it, including anything the server computed. If it returns nothing, the grid clears the pending edit and leaves your data alone, so the cell shows the old value again. Our first version of the demo returned nothing, and every save looked like it had been thrown away.

If your endpoint answers with no body, return { ...row, ...values }, or update your own state after the save.

When the server says no

Tick "Make the next save fail" in the demo, change two rows and save. The first request comes back with a 500:

One rejected request fails the whole save. Every edit stays pending, still marked, still counted on the Save button, so the user can try again without retyping anything.

There is a limit to this, and it is worth stating plainly: requests that already succeeded before the failing one are not rolled back. The grid does not know how to undo a write on your server, and pretending otherwise would be worse than saying so. If a batch has to be all or nothing, use onSave, which receives every change in one call, and make your endpoint transactional.

All of it in one take: an edit, a second edit, a refused value, undo, and a save that sends only what was left:

What this costs

It is paid. Editing, validation, undo and the CRUD handlers are in the Enterprise edition of the grid. Without a licence key it runs with a watermark, so you can try everything here first. The read-only grid, sorting, filtering and the rest are in the free Community package.
Pending edits live in the page. They are held in memory until saved; this demo does nothing to keep them across a reload.
Partial batches are possible with per-row handlers, as above. That is the trade for sending the shape a REST API wants.

Try it

The demo is at /demos/order-editing, and /demos/order-editing?fail=1 opens it with the failing server switched on. The editing docs cover the other modes (cell, row, form and popup), the CRUD handlers and every rule type.

Data: Online Retail II, UCI Machine Learning Repository (Chen, 2019), CC BY 4.0.

How to find cancellations in a million invoice lines

KanuniLabs — Thu, 01 Oct 2026 11:10:53 +0000

"How much did cancellations cost us, and what got cancelled?" is one of the first questions anyone asks of sales data. In a grid it looks like a one-filter job: show the lines with a negative quantity and read the total.

On real data that filter gives the right amount of money and more than twice the real number of returned units. This post walks through why, using the same 1,067,371 transactions as the sales dashboard post: Online Retail II, two years of invoices from a UK online retailer, loaded into the browser once. Every number below was read off the live demo and checked against the source file.

The obvious filter

Filter the Qty column to values below zero. The grid keeps 22,950 of the 1,067,371 lines, and the footer totals them: −1,064,078 units and −£1,527,041.43.

In code, through the grid's onReady handle:

grid.setColumnFilter('quantity', {
  columnId: 'quantity',
  operator: '<',
  value: 0,
});

The filter that matches the data

The dataset's own description says how a cancellation is marked: its invoice number starts with a C. Filter on that instead:

grid.setColumnFilter('invoice', {
  columnId: 'invoice',
  operator: 'startswith',
  value: 'C',
});

Now the grid keeps 19,494 lines: −490,992 units and −£1,526,667.86.

The demo has both filters as buttons above the grid, so you can switch between them and watch the footer:

What the difference is

The two totals differ by £373.57 and by 573,085 units.

The £373.57 is one line: a "Manual" adjustment on cancellation invoice C496350 with a quantity of +1, which the quantity filter misses.

The 573,085 units are 3,457 lines that have a negative quantity, an ordinary invoice number and a unit price of £0. Sort the quantity-filtered view by revenue, largest first, and they come to the top, because every real return is negative:

These are stock write-offs. 2,689 of them have no description at all; the rest say things like "damages", "lost", "check" or "?". They move no money, which is why the revenue totals agree. They do move units, so a returns report built on quantity < 0 would count more than twice the goods customers actually sent back.

So the question decides the filter. For the money, either one is close enough. For anything about units, return rates or products, filter on the invoice.

The largest cancellations

With the invoice filter on, sort Revenue ascending. The first line is the order from the last post: 80,995 paper craft birds, cancelled on 9 December 2011 as C581484 for −£168,469.60. The second is 74,215 ceramic storage jars, −£77,183.60.

What actually got cancelled

Single lines mislead in the other direction, so the next step is a pivot: products in the rows, revenue and units summed, the ten largest by cancelled revenue.

To slice by "is this a cancellation", the rows need a field that says so. The source has none, so we derive one while decoding:

kind: invoice[0] === 'C' ? 'Cancellation'
    : invoice[0] === 'A' ? 'Adjustment'   // six bad-debt adjustments
    : 'Sale',

Then the pivot:

const fields = [
  { id: 'kind', dataField: 'kind', caption: 'Kind', dataType: 'string',
    area: 'column', filterValues: new Set(['Cancellation']) },
  { id: 'product', dataField: 'description', caption: 'Product', dataType: 'string',
    area: 'row', sortBySummaryField: 'revenue', sortOrder: 'asc', topN: 10 },
  { id: 'revenue', dataField: 'revenue', caption: 'Revenue', dataType: 'number',
    area: 'data', summaryType: 'sum' },
  { id: 'units', dataField: 'quantity', caption: 'Units', dataType: 'number',
    area: 'data', summaryType: 'sum' },
];

sortOrder: 'asc' because cancelled revenue is negative: the largest refund is the smallest number.

Product	Cancelled revenue	Units
Manual	−£423,513	−5,450
AMAZON FEE	−£294,773	−39
PAPER CRAFT , LITTLE BIRDIE	−£168,470	−80,995
MEDIUM CERAMIC TOP STORAGE JAR	−£77,480	−74,494
Bank Charges	−£36,097	−77
REGENCY CAKESTAND 3 TIER	−£16,750	−1,468
POSTAGE	−£15,256	−303
Discount	−£13,882	−3,068
WHITE HANGING HEART T-LIGHT HOLDER	−£9,390	−3,638
CRUK Commission	−£7,933	−16

The two largest are not products. "Manual" and "AMAZON FEE" together are −£718,285, 47% of everything cancelled. Six of the top ten are fees, charges, postage, discounts or manual corrections. A chart of "most returned products" that does not exclude them is mostly a chart of bookkeeping.

Kind sits in the column area rather than the filter area on purpose: with filterValues on it, the pivot shows a single "Cancellation" header, so the screen says what is being counted.

Which markets cancel most

Swap the layout: countries in the rows, Kind in the columns, revenue in the cells, the ten largest markets.

Across the whole dataset, cancellations are £1,526,668 against £20,961,532 of sales, 7.3%. Per market, as a share of that market's sales:

Spain: 15.9% (−£17,319 on £109,179)
France: 8.1%
United Kingdom: 7.4%
EIRE: 7.4%
Netherlands: 1.0%

Spain's rate is the highest and also the easiest to over-read: it is £17,319 across 91 cancellation lines, small enough for a handful of orders to move it.

What this approach costs

The convention is this dataset's. "Invoice starts with C" is how this retailer's export marks a cancellation. Your system will have its own field or code; the lesson is to filter on that, not on the sign of a number.
A cancellation is not always a customer return. Manual corrections, marketplace fees and bank charges are cancelled the same way. Decide what you are counting before you count it.
The file has exact duplicates. 34,335 lines appear more than once, 390 of them cancellations; you can see two pairs of AMAZON FEE lines in the sorted view above. We kept every row, as in the previous post, so the totals here match the published file. A real report would deduplicate first and come out slightly lower.
The data ends on 9 December 2011, so the last month is a week, and the largest cancellation in the set happened on its last day.
The pivot cuts long product names in its row header at the default width. The table above has them in full.

Try it

The demo opens in each state from a link: quantity below zero, cancellations only, cancellations, largest first, the most cancelled products and cancellations by country. The filters, the footer totals, sorting a pivot by a measure and Top-N are all in the free Community packages; the DataGrid and PivotGrid docs cover each prop used here.

Data: Online Retail II, UCI Machine Learning Repository (Chen, 2019), CC BY 4.0.

Disclosure: this article was drafted with AI assistance. The demo, the data processing and every number in it were produced and checked against the published dataset and our packages in October 2026.

How to build a sales dashboard on a million real rows in React DataGrid

KanuniLabs — Tue, 29 Sep 2026 08:32:59 +0000

Most sales dashboards I have worked on send every question to a server. You change a filter, a request goes out, a SQL query runs and a page of rows comes back. That is the right design when the data is huge or private. It is also a lot of machinery for a question like "which country bought the most in November?" when the whole dataset would fit in a browser tab.

So we tried it the other way round. We took a real sales dataset of just over a million rows, sent it to the browser once, and did everything else there: sorting, searching, grouping, pivoting and charting. This post walks through each part with the React code, and says where it gets slow.

The data

We used Online Retail II from the UCI Machine Learning Repository. It is two years of transactions from a UK online retailer that sells gifts and homeware, from December 2009 to December 2011: 1,067,371 rows, one per invoice line. Each row has an invoice number, a stock code, a product description, a quantity, a unit price in pounds, a timestamp, a customer ID and a country. It is published under CC BY 4.0, so you can build on it too.

It is real data, so it is untidy in the way sales data always is. Cancelled invoices start with a "C" and carry negative quantities. Some lines are not products at all: "Manual", "AMAZON FEE", "Adjust bad debt". We kept every row.

The finished page is live at /demos/online-retail. It has two parts, a data grid with every transaction and a pivot table with a chart, and we will build them in that order.

Getting a million rows into the browser

As a CSV the dataset is 91 MB, and 14.8 MB gzipped. That is too much to ask a visitor to download, but most of it is repetition. Five of the eight columns draw from a small set of values: 43 countries, about 5,300 products and about 53,600 invoices. We ship each of those columns as a dictionary plus one integer per row, which brings the download to 5.8 MB gzipped.

Decoding it back into plain row objects is a loop:

const res = await fetch('/demos/online-retail/retail.v1.json.gz');
const text = await new Response(
  res.body!.pipeThrough(new DecompressionStream('gzip')),
).text();
const { dicts, cols, rows: n } = JSON.parse(text);

const rows = new Array(n);
for (let i = 0; i < n; i++) {
  const quantity = cols.quantity[i];
  const price = cols.price[i];
  rows[i] = {
    id: i + 1,
    invoice: dicts.invoice[cols.invoice[i]],
    description: dicts.description[cols.description[i]],
    country: dicts.country[cols.country[i]],
    quantity,
    price,
    revenue: quantity * price,
  };
}

The version on the page decodes in slices of 50,000 rows and yields to the browser between slices, so the progress bar keeps moving while a million objects are built.

The grid

The grid takes those rows as they are:

import { DataGrid } from '@kanunilabs/datagrid-react';

const gbp = new Intl.NumberFormat('en-GB', { style: 'currency', currency: 'GBP' });

const columns = [
  { field: 'invoice', headerName: 'Invoice' },
  { field: 'description', headerName: 'Product', width: 300 },
  { field: 'quantity', headerName: 'Qty', dataType: 'number' },
  { field: 'revenue', headerName: 'Revenue', dataType: 'number',
    valueFormatter: (v) => gbp.format(v) },
  { field: 'country', headerName: 'Country' },
];

<DataGrid
  dataSource={rows}
  rowKey="id"
  columns={columns}
  toolbar
  statusBar
  groupPanel
  summaries={[
    { columnId: 'quantity', type: 'sum' },
    { columnId: 'revenue', type: 'sum' },
  ]}
/>

Only the rows on screen are in the DOM, so scrolling to the bottom of a million rows is the same work as scrolling to the top. The footer shows the totals for the whole result: 10,608,492 units and £19,287,250.55 over the two years.

Sorting by revenue

Click the Revenue header. The sort runs in a Web Worker, so the page keeps responding while a million rows are reordered. Our benchmarks page has the measured times on a million generated rows, and a script to run them yourself.

The top line is a single order for 80,995 paper craft birds at £2.08 each, £168,469.60 in one row. Sort the other way and the first row is the same order again, cancelled the same day as invoice C581484 at −£168,469.60. The second-largest line, 74,215 ceramic storage jars, has a matching cancellation too. Nobody would plan a dashboard around that, but it is the first thing a sort shows you on real data.

Searching

Type into the search box and the grid filters every column at once and highlights the match. "WHITE HANGING HEART" matches 5,918 of the 1,067,371 rows, and the footer updates to that result: 93,050 units and £257,533.90 in revenue.

The search matches what you see, so a search for "£2.95" works as well as a search for "2.95". That means the grid formats every row of every formatted column once, the first time you search, and this is where one line of our own code cost us a minute.

We first wrote the formatter as v.toLocaleString('en-GB', { style: 'currency', currency: 'GBP' }). With options, toLocaleString builds a new Intl.NumberFormat on every call, so the first search sat on "Preparing data" for over a minute in a development build. With one shared formatter, as in the code above, it took about six seconds on the same laptop. If you format numbers in a large grid, build the formatter once.

Grouping by country

Drag the Country header into the bar above the grid and the rows fold into one group per country, each with its own row count and totals.

Grouping, the group bar and the footer totals are in the free Community package. The per-group totals shown here are part of the paid EnterpriseDataGrid, which this page uses.

A pivot of country by year

A pivot table answers the next question, how each market did in each year, without writing a query. Three fields describe it:

import { BasePivotGrid as PivotGrid } from '@kanunilabs/pivotgrid-react';

const fields = [
  { id: 'country', dataField: 'country', caption: 'Country',
    dataType: 'string', area: 'row', areaIndex: 0 },
  { id: 'year', dataField: 'year', caption: 'Year',
    dataType: 'string', area: 'column', areaIndex: 0 },
  { id: 'revenue', dataField: 'revenue', caption: 'Revenue',
    dataType: 'number', area: 'data', areaIndex: 0, summaryType: 'sum' },
];

<div style={{ position: 'relative', height: 520 }}>
  <PivotGrid data={rows} initialFields={fields} />
</div>

Note the wrapper. The pivot lays itself out against its nearest positioned ancestor, so give it a container with a height and position: relative. On our page, without it, the pivot drew itself over the grid above.

Every field in the pivot can be dragged between the rows, columns, values and filter areas, and it re-aggregates the million rows on each drop.

Revenue by month

Swap the layout for months in the rows and revenue as the only measure, and the chart becomes readable. With countries in the rows it is not: the UK accounts for most of the revenue, so every other bar is a sliver.

November is the peak in both years, £1,422,655 in 2010 and £1,461,756 in 2011. December 2011 looks like a collapse at £433,686, but the dataset stops on 9 December. That is worth checking before anyone builds a forecast on it.

The chart follows the pivot, so filtering the pivot to one country redraws the chart for that country. Charts are part of the paid PivotGrid Enterprise package.

A dark theme

The grid ships six themes. This one is theme="kanuni-midnight":

What this approach costs

Sending the whole dataset to the browser is a trade, and it is not the right one for every dashboard.

The download. 5.8 MB, once per visit. Fine for an internal tool, heavy for a phone on a bad connection.
Memory. A million row objects live in the tab for as long as it is open. We did not measure the footprint for this post.
First-time costs. The first search indexes the formatted text, and the first pivot layout aggregates everything. Both are cached afterwards, but the first time is seconds rather than milliseconds.
It is all in one tab. If the data is private, changes every second, or is ten times bigger, a server is still the right place for it.

Try it

The dashboard is at /demos/online-retail. A few links open it in a given state: sorted by revenue, grouped by country, with the monthly chart and in the dark theme. The DataGrid docs and PivotGrid docs cover every prop used here.

Data: Online Retail II, UCI Machine Learning Repository (Chen, 2019), CC BY 4.0.

Sorting a million rows in JavaScript, and where the time actually goes

KanuniLabs — Mon, 28 Sep 2026 13:00:00 +0000

Our data grid can sort a million rows in the browser without freezing the page. Getting there took several separate changes, and each one removed a different kind of cost. This post goes through them in the order we made them, with numbers from one laptop (Intel i7-11390H, Node 24).

Numbers from a laptop wobble. Where I quote the grid's engine, the figures come from a benchmark script that runs against our npm package, and the full tables, including the slow first-query numbers, are on our benchmarks page. The small examples below are standalone and have no dependencies, so you can paste them into a file and check my numbers.

The starting point: a comparator over row objects

The obvious sort is the one everyone writes first:

rows.sort((a, b) => a.revenue - b.revenue);

A comparison sort of a million items makes roughly 20 million comparisons (n log₂ n, and log₂ of a million is about 20). Every one of those is a call into your comparator. Any work inside it happens 20 million times, so the comparator has to be almost free.

Ours was not. Our first date sort parsed the date strings inside the comparator. Here is that mistake next to the fix, on a million ISO date strings:

// 3.2 s
rows.sort((a, b) => Date.parse(a.orderedAt) - Date.parse(b.orderedAt));

// 0.53 s: parse once per row, then sort an index array
const keys = new Float64Array(n);
for (let i = 0; i < n; i++) keys[i] = Date.parse(rows[i].orderedAt);
const idx = new Uint32Array(n);
for (let i = 0; i < n; i++) idx[i] = i;
idx.sort((a, b) => keys[a] - keys[b] || a - b);

The trick is decorate-sort-undecorate: compute each row's sort key once, sort positions by key, then read rows in that order. The source array is never touched, which also matters for a grid, because the user's data is not ours to reorder. The || a - b keeps equal dates in their original order.

Text: rank once, then compare integers

Text is where comparators hurt most. Sorting strings for humans needs Intl.Collator, and a collator comparison is far more expensive than subtracting two numbers. In our engine, sorting a million product names this way takes about 2.2 seconds.

The fix is to notice that a text column usually repeats itself. We already store each text column as a dictionary of distinct strings plus an integer code per row. So we sort the dictionary once with the collator, give every distinct string a rank, and after that a text sort is an integer sort. On repeat sorts of the same column our engine takes about 90 ms.

There is a catch, and our first benchmark tables hid it. Ranking the dictionary is a collator sort. If a column has few distinct values (a region, a status, a country), that sort is tiny and the first sort costs almost nothing extra. If every value is distinct, as with the product names in our benchmark, the first sort has to rank a million strings and took 1.6 to 1.9 seconds. That is not much better than the plain comparator. The ranks are cached until the data changes, so only the first sort pays. We now publish both numbers side by side, because the fast one on its own is misleading.

One bug came out of this. Our collator uses sensitivity: 'base', so it treats "Berlin" and "berlin" as equal. Our first ranking code gave them different ranks anyway, because they were at different positions in the sorted dictionary. That quietly changed multi-column sorts: rows that should have tied on city and fallen through to the second sort key were ordered by capitalisation instead. Strings the collator calls equal have to share a rank.

Numbers: stop comparing at all

With keys in a Float64Array, the comparator is just keys[a] - keys[b]. It is still 20 million function calls. On the same million values:

	time
`Uint32Array.prototype.sort` with a comparator	357–427 ms
radix sort, no comparator	80–90 ms

A radix sort does not compare elements. It distributes them into 256 buckets by one byte of their key, then does it again for the next byte, eight times for a 64-bit key. Each pass is a straight loop over typed arrays. Because every pass is stable, sorting by the lowest byte first and the highest byte last gives the full order.

The awkward part is that the bytes of a double are not in numeric order. The sign bit is at the top, so every negative number looks larger than every positive one, and negative numbers are stored as magnitudes, so -5 looks larger than -1. Both can be fixed by rewriting the 64 bits before sorting:

const f64 = new Float64Array(1);
const u32 = new Uint32Array(f64.buffer); // little-endian: [low, high]

function radixSortIndices(values) {
  const n = values.length;
  const lo = new Uint32Array(n);
  const hi = new Uint32Array(n);
  for (let i = 0; i < n; i++) {
    f64[0] = values[i] === 0 ? 0 : values[i]; // -0 and +0: equal numbers, different bits
    let l = u32[0];
    let h = u32[1];
    if (h & 0x80000000) {
      l = ~l >>> 0; // negative: flip every bit, so -5 sorts below -1
      h = ~h >>> 0;
    } else {
      h = (h ^ 0x80000000) >>> 0; // positive: set the top bit, above all negatives
    }
    lo[i] = l;
    hi[i] = h;
  }

  let src = new Uint32Array(n);
  for (let i = 0; i < n; i++) src[i] = i;
  let dst = new Uint32Array(n);
  const counts = new Uint32Array(256);

  for (let pass = 0; pass < 8; pass++) {
    const key = pass < 4 ? lo : hi;
    const shift = (pass & 3) * 8;
    counts.fill(0);
    for (let i = 0; i < n; i++) counts[(key[src[i]] >>> shift) & 255]++;
    let sum = 0;
    for (let b = 0; b < 256; b++) {
      const c = counts[b];
      counts[b] = sum;
      sum += c;
    }
    for (let i = 0; i < n; i++) dst[counts[(key[src[i]] >>> shift) & 255]++] = src[i];
    [src, dst] = [dst, src];
  }
  return src;
}

I checked this against the stable comparator sort on a million random values, including negatives and a -0 next to a 0, and the two orders came out identical.

That -0 line is the second bug our tests caught. -0 === 0 is true, so a comparator calls them a tie and keeps their original order. Their bit patterns differ in the sign bit, so the radix sort put every -0 before every 0. It is rare in real data, until someone computes 0 * -1 in a formula column.

The version in our engine does a little more than this one. It parks empty cells (NaN in the key array) after everything else in both directions. It sorts descending by inverting the key bits rather than reversing the output, because reversing would also reverse tied rows and break stability. And it skips any pass where every key has the same byte, which happens often with real data. Below 50,000 rows it does not use radix at all: the setup costs more than it saves.

Sorting by several columns uses the same stability: one radix pass per column, starting from the last sort column and ending with the first. That is also why the two-column sort is our slowest columnar query on repeat runs, at about 210 ms.

Moving it off the main thread

Even at 100 ms, a sort on the main thread is a visible hitch if someone is scrolling while it runs. So above 20,000 rows the sort runs in a Web Worker. The worker holds the typed-array columns, runs the sort, and sends back a Uint32Array of row positions in the transfer list, so the result moves between threads without being copied.

Building those typed arrays still happens on the main thread, because it calls user-supplied value getters, and functions cannot be posted to a worker. For all nine columns of our benchmark at a million rows it takes about 3.3 seconds in total. Two things make that tolerable. Columns are encoded lazily, only when a query first needs them. And at 100,000 rows and above the encoder yields to the browser as it goes, so the page keeps painting. It is still the slowest thing in the pipeline, and I would rather say so than leave it out.

Keeping the fast path honest

Every one of these optimisations is a second implementation of something that already worked. So we never deleted the slow one. The simple row-based engine stays in the codebase as the reference, and a set of equivalence tests run the same queries through both and require identical row orders.

Those tests caught the collator tie and the -0 ordering above. The third bug they found was in filtering, not sorting. The reference engine converted values with Number() before comparing, and Number(null) is 0. So a filter for = 0, or for an empty string, matched every empty cell. The rule we settled on is that an empty cell equals only another empty cell, in both engines.

If you are writing a fast path for anything, keep the slow version around as the test oracle. In our case it found all three of these before a release did.

Try it

The standalone examples above run in any recent Node. To time the engine itself, the script on the benchmarks page installs from npm and prints the same table we publish, including the first query for each column:

npm install @kanunilabs/datagrid-core
curl -O https://kanunilabs.com/benchmarks/datagrid-pipeline.bench.mjs
node datagrid-pipeline.bench.mjs 1000000

Your numbers will be different. If they are very different, I would like to know about it.

Disclosure: this article was drafted with AI assistance. The code was run and the numbers were measured by us in September 2026; the engine figures come from the published benchmark script against @kanunilabs/datagrid-core 1.2.0.

What "free" means when a data grid also sells a paid edition

KanuniLabs — Thu, 24 Sep 2026 13:00:00 +0000

Plenty of data grid libraries with a paid edition also have a free one, and the word "free" on their homepages covers several different deals. The license field in package.json does not settle it either. Ours says SEE LICENSE IN LICENSE, which tells you nothing until you open the file.

We sell a grid, so we have a stake in this. I'll use our own licence as the worked example throughout, including the part where we got it wrong in public.

Three things "free" can mean

Open source. MIT, Apache-2.0 and similar. You can read the code, change it, fork it and ship it inside your own library. When a vendor with an open source grid also sells features, those features normally live in a separate package under a separate, commercial licence. The free package stays open source; the paid one does not.

Free of charge, but proprietary. No payment and no key, but the code is not yours to modify or redistribute on its own terms. Whether commercial use is allowed depends entirely on the licence text, and it varies. Our Community packages are this kind. The grant section says you may:

(a) install and use the Software as a dependency of your applications, including commercial applications; and
(b) reproduce and distribute the Software solely as an incorporated, compiled part of such applications.

So a commercial SaaS app is fine. The restrictions section is where people should slow down. You may not modify it, and you may not distribute it "on its own, or as part of any toolkit, component, library, or product that competes with KanuniLabs products". If you are building a UI kit to sell, our free licence does not cover you, and I would rather you learn that from this post than from an email.

Free to evaluate. All the features, but not licensed for production. The usual signals are a watermark, a console message or a time limit. Some vendors let the evaluation build keep working indefinitely with the watermark in place. Our paid package does exactly that when it has no key. It is easy to mistake for a free tier, but the licence does not cover shipping it that way.

What paid editions usually charge for

Once you look past the marketing pages, paid editions tend to draw their line along a few axes, often more than one at a time.

Features. Editing, Excel export, server-side data, charts and pivoting are common candidates. Sorting, filtering and virtual scrolling are nearly always in the free tier.
Seats. Priced per developer who writes code against the grid, not per end user. Check whether "developer" includes the person who only fixes a CSS bug once a year.
Deployment. Some licences count applications, domains or production deployments. "Unlimited apps" and "one key per domain" can both be true of the same product, so read the definitions.
Updates. A perpetual licence usually means you keep the version you bought and get updates for a fixed window. A subscription usually means the licence ends when you stop paying. What happens at that point matters more than the word used.

That last point deserves an actual test before you sign anything. Ask what your users see if the key is missing or expired. If the component throws, your app's uptime is now tied to your billing. If it renders with a watermark, the failure is ugly but survivable. We wrote about why our keys are verified offline in an earlier post, so I won't repeat it here.

Where our line is

For completeness, here is the whole deal, because a post like this is useless if the author stays vague about their own product.

Community is free on npm, needs no key and shows no watermark. It covers 98 of the 136 features across our two components (50 of 82 in DataGrid, 48 of 54 in PivotGrid). Commercial use is allowed under the terms above.
The paid edition adds editing with validation and undo, range selection and fill handle, master-detail and tree data, server-side data, a chart panel, and styled Excel and PDF export. It is a separate package on a private registry, not on public npm.
Without a key the paid package still runs. It shows a watermark instead of breaking, because you cannot evaluate a feature you cannot run.
With a key the watermark is gone. One key covers both components.
Prices are $19 a month or $185 a year for one developer, and $79 a month or $699 a year for up to five. If a subscription ends, what you have shipped keeps running without a watermark: the key covers every version released while you were paying. New versions and support stop.

The mistake on our pricing page

In August an outside audit of our site found that the Community card on our pricing page listed "Excel & PDF export" for DataGrid. The free package does not have that. If you call exportToExcel on the Community grid, you get:

[datagrid] Excel export is an Enterprise feature. Import EnterpriseDataGrid
from @kanunilabs/datagrid-react-enterprise (CSV export is available in Community).

Nobody did this on purpose. The feature list in the code and the bullet points on the pricing page were written in two different places, and they drifted. A developer who installed the free package because of that bullet would have been right to feel misled.

The fix was partly editing the card and partly making the build fail the next time it happens. A check now reads our feature inventory and refuses to build if any free-tier bullet names a paid feature. It is deliberately narrow: it only looks for names that can mean nothing else, like "range selection" or "master-detail", because a check that fires on innocent sentences gets turned off within a week.

If you are evaluating a grid, ours or anyone's, install the free package and call the feature you are planning around before you commit to it. It takes five minutes. We wrote up the checks we would run in a separate post.

Disclosure: this article was drafted with AI assistance and checked against our code, licence text and pricing by the KanuniLabs team. The numbers are from the repository as of September 2026.

Is that free data grid free for your project? Checks you can run in a terminal

KanuniLabs — Mon, 21 Sep 2026 08:26:40 +0000

A few weeks ago we found that our own pricing page listed a paid feature under the free tier. Nobody had noticed for a while. It made me wonder how many people pick a component because of a bullet point, and never check the thing they actually install.

So here are the checks I would run on any "free" grid before building on it. Every command below was run on our own Community package, and the output is pasted as it came out (one long npm error is trimmed, and marked as such), including the parts that do not flatter us. None of this is legal advice. It is the homework you do before asking a lawyer a better question.

1. Read the licence field, then ignore it

npm view @kanunilabs/datagrid-react license

SEE LICENSE IN LICENSE

That string is the npm convention for "the licence is custom, go read the file". An SPDX name like MIT is more informative, but it only describes that one package. If the features you need are in a second package, the second package has its own licence, and it is usually not MIT.

2. Read the LICENSE file from the tarball

Not the README, not the website. The file that ships with the version you install:

npm pack @kanunilabs/datagrid-react
tar -xzf kanunilabs-datagrid-react-*.tgz
cat package/LICENSE

Things to look for, roughly in order of how often they surprise people:

Commercial use. Is it granted in words, or just not forbidden?
Redistribution. Can it ship inside your app? Inside a library you publish?
Modification. Can you patch it if a bug blocks your release?
Competing products. Some free licences exclude people building component libraries or dev tools.
Termination. What ends the licence, and what you must do then.

Ours grants use "including commercial applications" and distribution "as an incorporated, compiled part of such applications". It forbids modification, and it forbids shipping the package as part of a library or toolkit that competes with ours. If you are building an internal dashboard or a SaaS product, that is fine. If you are building a component kit to sell, it is not.

Save a copy of this file next to your release notes. If a later version changes its terms, you want to be able to show which text you agreed to.

3. Find out where the features you need actually live

npm view @kanunilabs/datagrid-react dependencies peerDependencies --json

{
  "dependencies": {
    "@kanunilabs/datagrid-core": "1.2.0"
  },
  "peerDependencies": {
    "react": ">=18.0.0",
    "react-dom": ">=18.0.0"
  }
}

Nothing paid is pulled in. The paid edition is a different package, and it is not on the public registry at all:

npm view @kanunilabs/datagrid-react-enterprise version

npm error code E404
npm error 404 Not Found - GET https://registry.npmjs.org/@kanunilabs%2fdatagrid-react-enterprise - Not found
npm error 404
(trimmed)

This is a quick way to read a vendor's docs. If the example for the feature you need imports from a package that the first command did not list, that feature is probably not in the free tier, whatever the feature table says.

4. Call the feature from the free package

The fastest check of all. Take the one feature you would be most upset to lose, and call it on the free component. On ours, Excel export is a paid feature. You do not even need a browser to check, because the controller runs in Node:

import { DataGridController } from '@kanunilabs/datagrid-core';

const grid = new DataGridController({
  columns: [{ field: 'id', dataType: 'number' }, { field: 'name' }],
  dataSource: [{ id: 1, name: 'a' }, { id: 2, name: 'b' }],
  rowKey: 'id',
});

try {
  await grid.exportToExcel();
} catch (e) {
  console.log(e.message);
}

[datagrid] Excel export is an Enterprise feature. Import EnterpriseDataGrid from @kanunilabs/datagrid-react-enterprise (CSV export is available in Community).

That is the behaviour I would want from any library: a loud, specific error at the moment of the call. A method that silently does nothing is much worse, because you find out from a user.

If you end up on a paid edition, repeat the test somewhere that is not localhost. Many licence checks treat localhost as always allowed, and ours is one of them. A key limited to certain domains will look fine on your machine and show a watermark on a staging domain you forgot to add to the key.

5. Search the build for licence and watermark code

grep -rilE "watermark|unlicensed|trial|licen[cs]e key" package/dist

On our two Community packages this matched only the type declaration files (index.d.ts, index.d.cts), never the JavaScript. The matches are doc comments: one describes the overlay slot the paid edition uses for its watermark, one mentions the unlicensed state in an API description. The code that draws a watermark is not in the free packages.

A match in the JavaScript is not proof of anything bad. It is a pointer to read that part of the bundle and find out when it runs.

6. Search for network calls

grep -rEo "fetch\(|XMLHttpRequest|sendBeacon|new WebSocket" package/dist | sort | uniq -c

No output on ours. That means the Community packages do not phone home for licence checks or telemetry.

A static search can miss calls built at runtime, so do the dynamic version too: load your app with the grid, open the network tab, filter out your own API, and use the grid for a minute. If anything leaves the page that you did not ask for, find out what it is before production does it for you.

What these checks do not cover

They will not tell you whether a library is maintained, whether the vendor will still exist in three years, or what the paid tier costs once your team grows. They also cannot catch terms that live outside the package, such as a separate EULA on a website. For those, ask the vendor in writing.

What they do give you is the licence you would actually be installing and the behaviour you would actually get, checked in about twenty minutes.

Disclosure: this article was drafted with AI assistance. Every command was run by us against the published packages (@kanunilabs/datagrid-react 1.1.1, @kanunilabs/datagrid-core 1.2.0) in September 2026; outputs are unedited except where marked as trimmed.