Lexer design in Koi Editor

Michael Sjoberg — Sat, 19 Sep 2026 04:03:01 +0000

Koi uses small handwritten lexers instead of syntax definitions or syntax trees. This provides direct control over highlighting and folding, with very little work at runtime.

Sublime Text uses declarative regex/context syntax definitions.

A .sublime-syntax file describes matches, scopes, and state transitions, which Sublime's syntax engine then executes:

syntax definition -> generic syntax engine -> tokens and scopes

Zed uses Tree-sitter grammars to parse the document into a syntax tree, then runs Tree-sitter queries such as highlights.scm against that tree.

The same parsed structure can support highlighting, outlines, bracket matching, indentation, text objects, and other features:

Tree-sitter grammar -> Tree-sitter parser -> syntax tree -> queries -> tokens and structure

Xcode's syntax highlighting pipeline is proprietary, so I don't really know how it works.

Koi does something much more direct:

language-specific lexer -> tokens and fold levels

There are no generic grammars or regex definitions sitting between the text and the highlighting engine.

Basically, Koi's lexers are ordinary code. Each language has a small purpose-built lexer that directly decides how text should be styled and folded.

Syntax-aware folding

Syntax-aware folding is useful when folding can't be inferred from indentation alone.

For example, this code should still be foldable:

if (mem.nTitle) {
memcpy(z, mem.zTitle, mem.nTitle);
}

Sublime is interesting because its folding isn't really defined by .sublime-syntax.

It does correctly fold the flat code block above, although I'm not sure where or how this is implemented internally.

Zed still appears to use indentation-based folding by default. There has been work on syntax-aware folding using Tree-sitter folds.scm queries.

However, as of today, using the current version of Zed, the flat code block above does not fold for me.

Here's another set of examples to determine the folding logic used:

test {
    ...
}

test {
...
}

test {
}

In plain text, Sublime and Zed both folds first, but not second and third.

In Python, even if this is not valid syntax, Sublime folds first and second, but not third.

Zed still folds first, but not second and third.

Koi does not fold any of these examples in plain text, since it's plain text.

In Python, Koi folds all three since I have enabled folding on sets and do not really care if the surrounding syntax is valid or not.

Here's another indentation only example:

test
    foo

In C, Sublime and Zed both folds this indented code.
Koi does not fold this.

This makes me think Sublime and Zed folding is based on delimiters and indentation.

There seems to be some language-specific rules in Sublime on indentation in languages like Python.

Syntax rules as code

Implementing syntax rules as code avoids having to figure out how to express an unusual condition in a grammar, regex state system, Tree-sitter query, or generic folding configuration.

If I want some unusual contextual condition for highlighting or folding, I simply write it.

In recent benchmarks, Koi continues to apply syntax highlighting to files containing millions of lines, while Sublime and Zed become unusable or fail much earlier.

I think part of the reason is that Koi deliberately does very little work when lexing:

What kind of token is this?
What state should I carry forward?
What is the fold level?

Sometimes doing less work by default is the performance optimization.

For example, adding a triple quote near the top of a large Python file, in this case 100,000 lines:

"""
def main(): return
def main(): return
def main(): return
def main(): return
def main(): return
...

should immediately change the syntax state of every following line until the next triple quote.

Koi and Sublime both update the highlighting immediately, even on large files.

Zed does this correctly on smaller files, but at 100,000 lines I don't see the highlighting update throughout the document.

I don't know whether this is a Tree-sitter limitation, a Zed performance optimization, or something else in Zed's highlighting implementation.

The case for handwritten lexers

I like handwritten lexers because they're simple, fast, and give me complete control.

They also seem to provide great performance, but there is the obvious tradeoff where adding a new language takes more work.

I like performance, so I think the tradeoff is worth it.
The good thing is that I only need to implement each lexer once.

DEV Community: Michael Sjoberg

Lexer design in Koi Editor

Syntax-aware folding

Syntax rules as code

The case for handwritten lexers