DEV Community

duz52
duz52

Posted on

Your LLM Writes \(x\), Your Markdown Parser Wants $x$

#javascript #markdown #opensource #ai

TL;DR — LLMs love \(...\) and \[...\]. Most Markdown math plugins only speak $...$ and $$...$$. I tried to bridge that with a regex, failed in interesting ways, and ended up teaching the tokenizer instead. Two packages came out of it: micromark-extension-math-extended and remark-math-extended.

The setup

Sometimes the hard part of rendering math isn't the equation. It's agreeing on where the equation starts and stops.

I was piping Markdown from some OpenAI models into a rendering pipeline, and the output kept looking like this:

The lift coefficient is \(C_L\).

\[
L = \frac{1}{2} \rho v^2 S C_L
\]
Enter fullscreen mode Exit fullscreen mode

Nothing wrong here. This is textbook TeX:

  • \( ... \) → inline math
  • \[ ... \] → display math

But every Markdown math package in my toolchain wanted dollars:

The lift coefficient is $C_L$.

$$
L = \frac{1}{2} \rho v^2 S C_L
$$
Enter fullscreen mode Exit fullscreen mode

Two characters of difference. Weeks of my life. Let's go.

Attempt #1: "just swap the delimiters" 🙃

The obvious move is to preprocess the model output before it hits the parser:

\(x\)  ->  $x$
\[x\]  ->  $$x$$
Enter fullscreen mode Exit fullscreen mode

A regex handles the happy path. Real Markdown is not the happy path.

Your converter now has to not touch delimiters that live inside:

  • inline and fenced code
  • escaped Markdown punctuation
  • math that's already dollar-delimited
  • truncated model output
  • TeX commands that are themselves full of backslashes

That last one bites hard:

\begin{cases}
x \\[1em]
y
\end{cases}
Enter fullscreen mode Exit fullscreen mode

\\[1em] is a TeX line break with optional spacing. That [ is not the start of a display equation. Your regex does not know this. Your regex has never known anything.

And then there's the genuinely dangerous case:

\[
not closed

# Everything after this
Enter fullscreen mode Exit fullscreen mode

If the model forgets the closing \] — which happens constantly when you're streaming — a naive converter swallows the rest of the document into one enormous equation.

At the point where you've handled all of that, congratulations: you didn't write a preprocessor. You wrote a second Markdown parser, and now you maintain two.

Attempt #2: teach the tokenizer

So instead of rewriting the input before parsing, I added the TeX-style delimiters directly to the micromark tokenizer. Parse it properly once, and all of the above stops being your problem.

That turned into two packages.

micromark-extension-math-extended

The low-level one, for projects using micromark directly.

npm install micromark-extension-math-extended
Enter fullscreen mode Exit fullscreen mode 
import {micromark} from 'micromark'
import {math, mathHtml} from 'micromark-extension-math-extended'

const markdown = String.raw`
The lift coefficient is \(C_L\).

\[
L = \frac{1}{2} \rho v^2 S C_L
\]
`

const html = micromark(markdown, {
  extensions: [math()],
  htmlExtensions: [mathHtml()]
})

console.log(html)
Enter fullscreen mode Exit fullscreen mode

All three forms work, and they keep the meaning you'd expect:

Syntax Renders as
$C_L$ inline
\(C_L\) inline
$$C_L$$ display
\[C_L\] display

remark-math-extended

The higher-level one, for remark / unified. Drop-in replacement for remark-math:

npm install remark-math-extended
Enter fullscreen mode Exit fullscreen mode 
import rehypeKatex from 'rehype-katex'
import rehypeStringify from 'rehype-stringify'
import remarkMath from 'remark-math-extended'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import {unified} from 'unified'

const markdown = String.raw`
The lift coefficient is \(C_L\).

\[
L = \frac{1}{2} \rho v^2 S C_L
\]
`

const file = await unified()
  .use(remarkParse)
  .use(remarkMath)
  .use(remarkRehype)
  .use(rehypeKatex)
  .use(rehypeStringify)
  .process(markdown)

console.log(String(file))
Enter fullscreen mode Exit fullscreen mode

The streaming problem, specifically

This is the part I care about most, so it gets its own heading.

Rule: \[ must have a matching \]. When it doesn't, the parser falls back to ordinary Markdown instead of eating the remainder of your document.

If you're rendering token-by-token from an LLM, every single frame is technically malformed input. A response that's half-arrived shouldn't make your UI explode into one giant KaTeX block and then unexplode a second later.

The tokenizer also tells a real nested opener apart from legitimate TeX like \\[1em].

These two behaviors are the entire reason I went parser-level instead of regex-level.

The tradeoff you should know about ⚠️

One intentional incompatibility: in standard CommonMark, a backslash escapes punctuation, so \( and \[ mean literal ( and [. Turning on TeX-style delimiters changes that.

If you need the original behavior back:

math({backslashDelimiters: false})
Enter fullscreen mode Exit fullscreen mode

or with remark:

unified().use(remarkMath, {
  backslashDelimiters: false
})
Enter fullscreen mode Exit fullscreen mode

Dollar-delimited math keeps working either way.

One known limitation

remark-math-extended reuses the existing mdast-util-math tree and serializer. The math value and its inline/display meaning survive the round trip, but serialization normalizes delimiters back to dollars:

\(x\)  ->  $x$
\[x\]  ->  $$x$$
Enter fullscreen mode Exit fullscreen mode

Fine for most rendering pipelines. Not fine if you need byte-for-byte delimiter preservation. Open an issue if that's you — I'd like to know how common it is.

Links

If you work with LLM-generated Markdown, scientific writing, or anything that mixes Markdown and TeX: what edge cases have you hit? I'm collecting them. The \\[1em] one took me embarrassingly long to find, and I'm sure it's not the last.

If these saved you from maintaining yet another delimiter-conversion parser, you can buy me a coffee ☕

Top comments (0)