How a TSX bug report turned into a TextMate engine for Go

It started with a small bug report. A user of ttt, my terminal text editor, opened #670: in HTML, Angular templates, and JSX/TSX, when a tag's attributes are split over several lines, only the first line gets colours.

<button
  type={getType()}     ← plain text
  class="a"            ← plain text
  aria-hidden="true"   ← plain text
>

*: So the syntax colorization used here also has issues as you can see. Symbol "=" not highlighted as symbol and getType() not highlighted as identifier

That is Prettier's default formatting for long tags, so it affects a lot of real code. The reporter even guessed the cause: ttt highlights one line at a time, so every line starts from scratch and forgets it is still inside a tag.

What followed was a couple of days of work with an AI coding agent that ended somewhere I did not expect: a pure Go port of VS Code's syntax highlighting engine, a maintained fork of a regex library, and 124 languages highlighted the way VS Code highlights them. This is the story of how we got there, including the detours.

The problem was not new

ttt used chroma, the de facto pure Go syntax highlighter. Chroma is a port of Pygments and lexes a whole document at a time. An editor wants something different: it re-highlights the lines you see, one line at a time, as you type. Chroma has no API to start a line in the state the previous line ended in.

Multi-line constructs had bitten ttt before, and each fix was a hack:

• Re-lexing the whole buffer on every keystroke cost 68 ms per edit at 1,000 lines and 3.26 s at 50,000. Closed.
• A naive "am I inside a comment" pre-scan hard-coded /* */ for every language, so rm -rf build/* in a shell script greyed out the rest of the file. Closed.
• What shipped was cleverer: ask chroma itself where a block comment opens by appending a closer and lexing, then do the same for template literals, raw strings, and docstrings. Each construct needed its own probe, and each probe needed guards against false positives. Swift, Scala, and Java lost their """ support along the way, because their token streams look identical to Rust's.

So #670 was not really a new bug. It was the next item in a list that would never end.

Going back and forth

I asked the agent to walk me through the options, and we tried most of them for real before deciding.

One more chroma hack. We built it: detect that a line ends inside an open tag, then lex the next line behind a fake <x prefix so chroma starts in its tag state. It fixed the reported cases. It also broke a common one: a multi-line JSX handler, whose body was now coloured as more attributes. On top of that, TypeScript generics like Map<string, number lex as tags, and so does i <n. We shelved it.

Tree-sitter. It is the modern answer, so I wanted to try it. The official Go bindings need cgo, and ttt ships static binaries for five targets. There is a promising pure Go runtime, gotreesitter, but its grammars are 15.1 MB compressed. ttt's entire binary was 15.6 MB. That is not specific to one project: tree-sitter grammars are large generated parse tables. We dropped it.

Fork chroma to carry state. This is the most direct fix: the change to chroma itself is 19 lines. The agent measured it across chroma's own sample files: of 153 lexers with samples, 44 produce different output line by line than whole-file. Many match multi-line constructs with a single regex, so fixing them means rewriting lexers, and then maintaining a fork of ~250 of them. It turned a bug fix into a project.

Other Go highlighters. Micro's engine does not even colour tag names. The rest were cgo or abandoned.

Then the obvious thing clicked. VS Code is open source. Its highlighting engine, vscode-textmate, is MIT licensed and about 6,200 lines of TypeScript. TextMate grammars tokenize line by line and carry a stack of open rules to the next line, which is exactly what an editor needs. #670, block comments, template strings, and docstrings all just work when the grammar is right. The grammars are the same ones VS Code and Shiki use, about 260 of them. Being JSON, they compress well: 1.4 MB for all 260, against tree-sitter's 15.1 MB.

So we decided to port it.

Planning the port so it could be trusted

I asked the agent to write an agent-ready plan, and the part I cared most about was testing. vscode-textmate's tests are TypeScript, and much of their value is data: expected tokens for real grammars and real files.

I decided not to port the tests at all. Instead, the Go engine got a small CLI that speaks JSON over stdin, and a thin JavaScript adapter makes it look like vscode-textmate. VS Code's own test suites then run unchanged against the Go engine. The adapter also has a reference mode that runs the real vscode-textmate, and it must pass 100% before any Go result counts, so a bug in the adapter cannot pass for a bug in the engine.

The second idea I pushed the agent to implement was differential testing with the real vscode-textmate as the oracle: run both engines over real files and fuzzed edits, and treat any token difference as a bug. No hand-written expectations needed, and it scales to any grammar.

Another agent implemented the plan. The result, textmate-go, passes all 95 upstream tokenization fixtures and 72 real theme files, and matches vscode-textmate token for token on the differential corpus.

Wiring it into ttt

The integration was pleasantly boring. ttt's highlight package lost its state tables, probes, and region logic, and became a thin layer over the engine's incremental document helper: hand it the lines, ask for a line, map token categories to colours.

I ran the binary on the bug from #670 and it was just right: multi-line attributes, JSX handlers, block comments, template literals with ${}, Markdown with fenced code, all correct. Dropping chroma also made the binary smaller: 13.8 MB, down from 15.6 MB.

It was also noticeably slower. Compared with chroma, up to 10× slower per line.

Making it fast

My first question was how far off we really were. The agent ran three engines on the same corpus: textmate-go, chroma, and vscode-textmate itself in Node. Chroma turned out to be the wrong target. It does far less work per line than a TextMate grammar asks for. The fair comparison is vscode-textmate, which runs the same grammars through the same algorithm using the Oniguruma regex engine compiled to WebAssembly. Against that, we were 3 to 5× slower: TSX at 103 µs per line versus 20, HTML 72 versus 25.

Profiling showed about 80% of the time inside regexp2, the Go regex engine the port uses. Over several rounds, the agent found and fixed the causes, while I kept asking for proof at each step.

• Rebuilding scanners. Rules whose end pattern refers back to the start, like a JSX closing tag </\1>, recompiled every nested regex whenever the tag name changed. Cheap in C, expensive in Go. A per-grammar pattern cache fixed it.
• Locks and pools on every search. Each of the roughly 150 regex searches per line took a mutex, a map lookup, and a sync.Pool round trip, all redundant because the grammar already serializes tokenization.
• Searches that could never match. The most expensive patterns almost never matched, yet they ran the interpreter at every position of every line. Most of them need some specific character to match at all. We taught regexp2 to compute that set at compile time and fail immediately when the rest of the line does not contain it. This was the single biggest win: TSX went from 74 to 26 µs per line.
• A pool that forgot. Benchmark results were bimodal. sync.Pool is per CPU and cleared on every garbage collection, so a goroutine that moved between cores rebuilt regex runners constantly. Keeping one runner per regex outside the pool fixed it.
• C++ keyword lists. C++ was the outlier. Its grammar has 7 KB patterns with keyword lists of about 100 words, sorted by length, so regexp2 never merged common prefixes and tried each keyword in turn. Grouping branches by first letter turned each list into a trie: 415 to 272 µs per line.

Those changes lived in regexp2 itself, so I forked it. The fork is published as eugenioenko/regexp2.
I am not planing on keeping the fork, hopping taking some time later to push the fixes upstream.

Not everything worked, and I think that is worth saying. A jump table for the interpreter's opcode switch made things slightly slower. Packing flags and inlining helpers did nothing. A dedicated fast path for lookbehinds was 9 to 15% faster on the patterns it targeted, but only about 1% end to end, and it made unrelated short matches 4 to 10% slower. We measured it, wrote the numbers on the issue, and closed it. The agent's prediction for it had been wrong, and the data said so.

Measuring honestly turned out to be half the work. Early on, the same benchmark would give 25 µs per line in one run and 75 in the next. The agent first blamed thermal throttling, and I pushed back: this machine is not a laptop. The real cause was the power profile, which was set to battery saving. Switching it to performance made results stable. From then on every comparison ran base and candidate interleaved, many times, compared with benchstat and p-values, never one run after the other.

From 40 to 124 languages

Switching from chroma had a cost I did not want to hide: textmate-go embedded 40 grammars, and chroma highlights about 290 languages. Of chroma's languages without a grammar, 105 exist in the Shiki collection and 132 have no TextMate grammar anywhere.

Licensing was the real constraint, not size. I asked the agent to add only grammars with clear permissive licenses and to check the unclear ones I cared about by hand. That caution paid off immediately. YAML looked like it was covered by TextMate's permissive README grant, but its syntax folder ships its own MIT license file, which the README excludes from the grant. Elixir and Sass were marked as unknown only because GitHub's license detector did not recognise an Apache header and a two-part MIT file. GPL grammars stayed out of an MIT library. The generator now records every hand-reviewed license, with a link to the evidence, and appends the text to the NOTICE file.

The result is 124 languages for 471 KB of extra binary.

Adding the samples of every new grammar to the differential test paid off too. It caught five languages that tokenized differently from VS Code, all problems that predated our performance work:

• \N, Oniguruma's "any character except newline", was not translated, which disabled patterns in Hy, Shell sessions, and Stylus.
• Character-class intersection ("a letter, but not a vowel") disabled 16 Haskell patterns. Since a class matches exactly one character, it can be rewritten as a class plus a lookahead, which regexp2 does support.
• In Smalltalk, an operator pattern matched a period. That was a real bug in regexp2's parser, inherited from .NET: an escaped hyphen at the end of a character range left the range open. We fixed it in the fork.

After those fixes, all 166 files match vscode-textmate token for token.

Hardening

A few questions I asked late turned into important work. What happens if regexp2 panics? Nothing caught it, so one bad pattern would crash the editor. Now a panic disables that pattern with a diagnostic and highlighting continues. Does regexp2's CI test for stability? It runs its test suite on four platforms, but it has no race detector, fuzzing, or benchmark tracking. There is now an issue describing a CI we can trust, including a fuzzer that checks every optimization gives identical results when switched off, since the only way those optimizations can fail is by silently skipping a real match.

Finally, I do not want to maintain a regex fork forever. regexp2's author is active and focused on performance, and outside pull requests do get merged. So the plan is to benchmark each change on its own against upstream, drop what does not clearly help, and propose the rest one at a time, starting with an issue that asks how he would like to receive them.

Where it landed

Warm line-by-line tokenization, in microseconds per line (lower is better):

Language	textmate-go	vscode-textmate	vs vscode-textmate
SQL	13.6	41.9	0.32×
CSS	25.1	74.0	0.34×
PHP	14.9	24.6	0.60×
Go	9.6	14.0	0.69×
Ruby	42.0	59.0	0.71×
Python	45.7	48.4	0.94×
JavaScript	67.7	64.3	1.05×
TSX	22.8	20.8	1.10×
TypeScript	76.3	58.2	1.31×
Java	59.6	38.9	1.53×
C++	269.6	158.0	1.71×

Across all 20 benchmarked languages, textmate-go is faster than vscode-textmate in 12 and averages about 0.87× its time. It started at 3 to 5× slower. The three still clearly behind, TypeScript, Java, and C++, have the largest, most lookahead-heavy grammars.

In ttt, #670 is fixed properly, and so is every multi-line construct the old hacks covered, plus the ones they never could. The highlighting now matches VS Code, the binary is smaller than it was with chroma, and there is no language-specific code left in the highlight package.

What's next

• TypeScript and Java. Oniguruma rejects many searches early by checking for literal strings a pattern requires, like => or async. Before building that, we will measure how many failing searches it would actually skip. The lookbehind experiment taught us to measure first.
• VS Code themes. Every token carries a full scope stack, like source.ts meta.function.ts entity.name.function.ts. ttt currently collapses that into about 15 colours. Supporting VS Code themes would make that detail visible and let people bring the themes they already use.
• Upstreaming the regexp2 changes that earn their place.

Working with an agent

This was a collaboration in the real sense. I brought the questions and the judgment calls: measure against vscode-textmate rather than chroma, test before trusting a number, prove every performance claim, stay strict on licenses, and drop what does not pay for itself. The agent brought the exploration, the profiling, the implementation, and a steady stream of hypotheses, some right, some wrong. When it was wrong, like the throttling guess or the lookbehind estimate, the measurements we had agreed on caught it quickly.

The most valuable thing I did was keep asking "how do we know?". The most valuable thing the agent did was make answering that cheap.