Add Lucene-style search and highlighting to MUI DataGrid

MUI's DataGrid ships a filter panel: pick a column, pick an operator, type a value. It works, and it stops working the moment someone wants role:engineer AND joined:>=2021-01-01, or a wildcard, or two conditions on the same column.

This is a build-along. By the end you will have a DataGrid with a single search box that accepts a real query language, and every character that caused a match will be marked, using the offsets the filter itself produced.

Every snippet below is from a project I built and ran while writing this. There is also a live demo of the finished thing.

Why not just search the text again?

The obvious way to add highlighting is: filter the rows, then in each cell search the text again for the term and wrap it in <mark>.

That is two independent matches, and they disagree wherever their rules differ. Case folding is where it shows up first. A case-insensitive regex /s/iu marks ſ (long s) that a plain toLowerCase() comparison never matched. Drop the u flag to fix that, and now /k/i refuses the Kelvin sign K that your filter did match. You get rows on screen with nothing highlighted, and highlights on rows that should not be there.

The fix is to stop matching twice. If the engine that decided the row matched also tells you where it matched, there is nothing to re-derive and nothing to disagree about.

Two packages do that:

• @siftql/core, a Lucene-style query language for plain JavaScript objects with zero runtime dependencies
• @siftql/react-highlighter, which turns the offsets it reports into <mark> elements

What you need

npm install @mui/material @mui/x-data-grid \
  @emotion/react @emotion/styled \
  @siftql/core @siftql/react-highlighter

Step 1: some rows

Nothing siftql-specific yet. id matters, because DataGrid requires it and we will key the highlight index on it later.

// src/data.ts
export interface Person {
  id: number;
  name: string;
  email: string;
  role: string;
  joined: string;
}

export const ROWS: Person[] = [
  {
    id: 1,
    name: 'Ada Lovelace',
    email: '[e-mail]',
    role: 'engineer',
    joined: '2020-06-15',
  },
  {
    id: 2,
    name: 'Alan Turing',
    email: '[e-mail]',
    role: 'engineer',
    joined: '2019-01-04',
  },
  {
    id: 3,
    name: 'Grace Hopper',
    email: '[e-mail]',
    role: 'admiral',
    joined: '2021-03-01',
  },
  // four more of the same shape
];

Step 2: a search box that filters

Here is the whole idea in one function. Parse the query into an AST once, then hand that same AST to the filter now and to the highlighter later.

import { useMemo, useState } from 'react';
import { Box, TextField } from '@mui/material';
import { DataGrid, type GridColDef } from '@mui/x-data-grid';
import {
  parse,
  filter,
  isSiftQLError,
  type SiftQLAst,
} from '@siftql/core';

import { ROWS } from './data';

interface Parsed {
  ast: SiftQLAst | null;
  error: string | null;
}

export const App = () => {
  const [query, setQuery] = useState('name:*a*');

  const parsed = useMemo((): Parsed => {
    const trimmed = query.trim();
    if (trimmed === '') return { ast: null, error: null };

    try {
      return { ast: parse(trimmed), error: null };
    } catch (err) {
      const error = isSiftQLError(err)
        ? err.message
        : String(err);
      return { ast: null, error };
    }
  }, [query]);

  const rows = useMemo(
    () =>
      parsed.ast === null ? ROWS : filter(parsed.ast, ROWS),
    [parsed.ast],
  );

  const columns: GridColDef[] = [
    { field: 'name', headerName: 'Name', flex: 1 },
    { field: 'email', headerName: 'Email', flex: 1 },
    { field: 'role', headerName: 'Role', flex: 1 },
    { field: 'joined', headerName: 'Joined', flex: 1 },
  ];

  return (
    <Box sx={{ p: 4, maxWidth: 1100, mx: 'auto' }}>
      <TextField
        fullWidth
        size="small"
        label="query"
        value={query}
        onChange={(e) => setQuery(e.target.value)}
        error={parsed.error !== null}
        helperText={
          parsed.error ?? `${rows.length} of ${ROWS.length} rows`
        }
        slotProps={{
          input: { sx: { fontFamily: 'monospace' } },
        }}
      />

      <DataGrid
        rows={rows}
        columns={columns}
        autoHeight
        disableRowSelectionOnClick
        hideFooter
      />
    </Box>
  );
};

That already works. Type a query, the grid filters:

Two things in there are load bearing.

Parsing lives in a useMemo, and the throw is caught. A search box is mid-keystroke most of the time, so parse runs on every character, and role: is a syntax error until you finish typing the value. Catching it and putting the message in helperText beats letting it throw through your render.

parse and filter are separate calls. You could call filter(query, rows) with the string directly, but then the AST is thrown away and the highlighter would have to parse it again. Parsing once is what guarantees the marks and the rows come from the same match.

Step 3: what you can type in that box

Worth knowing before you go further, because this is the part MUI's filter panel cannot do:

Query	Matches
ada	any field containing "ada"
role:engineer	the role field, whole value
name:*ada*	name containing "ada"
name:"Ada Lovelace"	a quoted phrase, exactly
joined:>=2021-01-01	a real date comparison
joined:[2021-01-01 TO 2021-12-31]	an inclusive range
age:>30	numeric comparison
name:/^A.a/	a regular expression
role:engineer AND joined:>=2021-01-01	boolean composition
NOT role:admiral	negation
assignee.name:ada	a nested path

Dates are compared as instants rather than as strings, which is why joined:>=2021-01-01 does what you expect rather than a lexicographic comparison. And joined:>=2021-02-29 is refused with an error, because that date does not exist. new Date('2021-02-29') does not refuse it: it rolls over to March 1st UTC, which then prints as Feb 28 anywhere west of Greenwich. Two different wrong answers instead of one error.

Step 4: index the highlights, once per row

Now the highlighting. highlight(ast, row) returns which fields matched and at what offsets.

The tempting thing is to call it inside renderCell so each cell asks about itself. Do not. highlight() walks the entire record regardless of which field you are about to render, so calling it per cell re-matches the same row once for every column.

I measured it on a 500 row, 12 column grid:

highlight() per cell : 90.5 ms   (6000 calls)
indexed once per row :  7.3 ms   (500 calls)
ratio                : 12.4x

The penalty tracks your column count, not your row count: a 20 column grid pays 20x. And it lands on every keystroke.

So call it once per row and index the result:

// src/queryHighlights.ts
import { highlight, type SiftQLAst } from '@siftql/core';
import type {
  HighlightSpan,
  Spans,
} from '@siftql/react-highlighter';

type FieldSpans = readonly HighlightSpan[] | null;

export interface QueryHighlights {
  spansFor(rowId: string | number, field: string): Spans;
}

/** Stable identity, so memoised columns stay memoised. */
export const NO_HIGHLIGHTS: QueryHighlights = {
  spansFor: () => undefined,
};

export const indexHighlights = <
  R extends { id: string | number },
>(
  ast: SiftQLAst,
  rows: readonly R[],
): QueryHighlights => {
  const byRow = new Map<string, Map<string, FieldSpans>>();

  for (const row of rows) {
    const byField = new Map<string, FieldSpans>();

    for (const hit of highlight(ast, row)) {
      const field = hit.segments[0];
      if (typeof field !== 'string') continue;

      byField.set(field, hit.ranges ?? null);
    }

    byRow.set(String(row.id), byField);
  }

  return {
    spansFor: (rowId, field) =>
      byRow.get(String(rowId))?.get(field),
  };
};

Key by row.id, not by array position. The grid reorders on sort, and position stops meaning anything the moment it does.

Look up the field with hit.segments[0], not hit.path. path is a dotted display string and is lossy when a key contains a dot. segments is the canonical array form.

Step 5: three states, not two

hit.ranges ?? null above is the one semantic move in this integration.

There are three outcomes for a cell:

spans	Meaning	Render as
an array	these offsets matched	<mark> them
null	the value is why the row matched, but no substring of it is	mark the whole value
undefined	the value is not why the row matched	plain text

The middle one is the case people forget. Query joined:>=2021-01-01 and every returned row matched on that column, but there is no substring to underline. The whole value is the answer, and it renders as an underline rather than as marks:

There is not a single <mark> in that screenshot. Nothing was marked because nothing should be: no substring of 2021-03-01 is the reason the row is on screen.

siftql signals this by omitting ranges, and a missed Map.get also returns undefined. So one of them has to become null on the way to the component, which is what hit.ranges ?? null does.

Step 6: paint the cells

// src/highlightCell.tsx
import { useTheme } from '@mui/material';
import type { GridRenderCellParams } from '@mui/x-data-grid';
import { SiftQLHighlight } from '@siftql/react-highlighter';

import type { QueryHighlights } from './queryHighlights';

const ACCENT = { light: '#8D3B00', dark: '#FFA726' };

export const highlightCell = (
  field: string,
  highlights: QueryHighlights,
) => {
  const HighlightCell = (params: GridRenderCellParams) => {
    const mode = useTheme().palette.mode;
    const accent = ACCENT[mode];
    const row = params.row as Record<string, unknown>;
    const id = row.id as string | number;

    return (
      <SiftQLHighlight
        text={String(row[field] ?? '')}
        spans={highlights.spansFor(id, field)}
        markStyle={{
          backgroundColor: accent,
          color: mode === 'dark' ? '#000' : '#FFF',
          padding: 0,
        }}
        wholeValueStyle={{
          borderBottom: `2px solid ${accent}`,
        }}
      />
    );
  };

  HighlightCell.displayName = `HighlightCell(${field})`;
  return HighlightCell;
};

Then back in App.tsx, build the index and give each column a renderCell:

import {
  indexHighlights,
  NO_HIGHLIGHTS,
} from './queryHighlights';
import { highlightCell } from './highlightCell';

const highlights = useMemo(
  () =>
    parsed.ast === null
      ? NO_HIGHLIGHTS
      : indexHighlights(parsed.ast, rows),
  [parsed.ast, rows],
);

const columns = useMemo(
  (): GridColDef[] => [
    {
      field: 'name',
      headerName: 'Name',
      flex: 1,
      renderCell: highlightCell('name', highlights),
    },
    {
      field: 'email',
      headerName: 'Email',
      flex: 1,
      renderCell: highlightCell('email', highlights),
    },
    {
      field: 'role',
      headerName: 'Role',
      flex: 1,
      renderCell: highlightCell('role', highlights),
    },
  ],
  [highlights],
);

That is the whole build. The rest of this post is the four things that will bite you.

Gotcha 1: never use params.value on a date column

This one cost me real time.

A type: 'date' column needs a valueGetter so MUI receives a real Date:

{
  field: 'joined',
  headerName: 'Joined',
  flex: 1,
  type: 'date',
  valueGetter: (value: string) => new Date(value),
  renderCell: highlightCell('joined', highlights),
}

Now params.value is a Date, not your string. But siftql's offsets are indexes into the original string. Slice the Date instead and you get this:

// siftql said: mark [0, 4), the year
'2021-03-01'.slice(0, 4);
// "2021"

String(new Date('2021-03-01')).slice(0, 4);
// "Sun "

Look closely at that second line. There is a second bug hiding inside it. new Date('2021-03-01') parses as UTC midnight and stringifies in local time, so at any negative UTC offset it renders as Feb 28. You would be highlighting the wrong characters of the wrong date.

So read the raw field off the row, which is what row[field] does in Step 6, rather than params.value.

Gotcha 2: warning.main with getContrastText fails WCAG

The obvious way to theme the mark is MUI's warning palette with an automatic contrast foreground. It does not pass.

MUI's contrastThreshold defaults to 3, and 3:1 is the WCAG AA bar for large text. A grid cell is body text, which needs 4.5:1 for AA and 7:1 for AAA. getContrastText picks white in both light and dark mode, giving roughly 3.08:1 and 3.79:1, under AA in both.

Use dedicated tokens and measure them. Note also that light and dark need different accents, not one accent with two foregrounds: a dark mode amber sits at about 1.86:1 on a light paper surface, and a light mode brown at about 2.25:1 on dark. Both are effectively invisible. That is why ACCENT above is a pair.

Gotcha 3: <mark> carries user-agent padding

Browsers give <mark> a small horizontal padding. In prose you would never notice. In a dense grid a marked word sits wider than the text beside it and the column looks subtly misaligned, which reads as a spacing bug rather than as a highlight. Hence padding: 0.

Gotcha 4: do not tint the whole cell for the null state

It is tempting to render "matched, nothing to point at" as a background tint on the cell. Try that with joined:>=2021-01-01 and every visible cell in the column lights up, which conveys nothing, because they all matched.

An underline reads as "this column is why this row is here" without competing with marks that point at actual characters.

What I deliberately did not build

You will notice there is no @siftql/mui package here, and that is on purpose. An adapter would have to own an API for every grid library, forever, and the row and column index is application specific anyway. This one keys on row.id because that is what DataGrid guarantees. Yours might key on a composite, or on a nested path.

Roughly 40 lines in your own codebase beats a dependency that has to guess at your data shape.

Wrapping up

The load bearing idea is Step 2: parse once, use twice. Everything else follows from the marks and the filter coming from the same match. Nothing is re-searched, so nothing can disagree.

• Live demo: https://siftql-demo.vercel.app/
• Query playground with 70 worked examples: https://hussein-abdallah.github.io/siftql/
• @siftql/core and @siftql/react-highlighter on npm

Both packages are MIT and have zero runtime dependencies. If you try it and something is wrong, the issue trackers are open. I would rather hear about it.