Building a development environment for Monkey: Part 1 - Syntax highlighting with Tree-sitter

Intro

A few years ago on a quest of learning Go, I followed Writing An Interpreter In Go and built Monkey, a small interpreted programming language. Recently, I came back to it with a different goal: I wanted to see how far I could take the language as an actual development environment.

My plan is to build the tooling around Monkey step by step: first syntax highlighting, then LSP support, and eventually use the language to solve an Advent of Code problem.

As we begin writing code in Monkey, we immediately notice that it lacks the editor features we have come to take for granted: syntax highlighting, diagnostics, navigation, completion, and more.
There is a lot of machinery behind features we take for granted, and I wanted to understand it by building it. This post is the first step: teaching editors how to understand Monkey's syntax using Tree-sitter, primarily for highlighting purposes.

Before building the tooling, we should first understand the language we are building it for. Monkey is small, but it has enough language features to build simple tooling around it that would teach us all the basics of parsing. The core functionalities and features in Monkey are the following:

• Data Types: integers, booleans, strings, arrays, and hash maps.
• Variables: created using let statements (e.g., let foo = 1;).
• Operators: prefix (e.g., -, !) and infix operators (e.g., +, -, *, /, ==, !=, <, >) to evaluate arithmetic and boolean expressions.
• Functions as First-Class Citizens: Functions can be bound to names, passed as arguments, and returned from other functions, allowing for higher-order functions.
• Closures: fully supports lexical closures, meaning functions can capture and retain access to the variables in which they were defined.
• Built-in Functions: includes pre-defined utility functions, such as len() (to get the length of strings or arrays), puts() (for printing to standard output), first(), rest(), last(), and push().

You can find the full docs of Monkey and full Tree-sitter implementation in the Monkey repository.

By the end of this post we'll have a Tree-sitter grammar that parses every construct in Monkey, tested against a corpus, and wired into Neovim.

---

Why Tree-sitter?

I decided on using it instead of the approaches most editors used for syntax highlighting before Tree-sitter, such as regular expressions, custom parsers, and lexer-based systems, because Tree-sitter has become widely adopted for editor tooling and syntax-aware features.

There are a few other reasons:

• Error recovery: Tree-sitter can parse incomplete or syntactically invalid code while still producing a useful syntax tree. This is important for editors because code is often temporarily broken while we are in the process of writing it.
• A real syntax tree: Instead of treating source code as a collection of patterns to match, Tree-sitter produces a syntax tree, which the CLI displays as S-expressions. Editors can use this structural information to provide more accurate, context-aware highlighting. For example, an identifier can be distinguished based on whether it represents a type, variable, function, or something else.

And syntax highlighting is only one of the things we can build on top of a syntax tree. Once we have one, we can also implement features such as:

• Incremental parsing
• Code folding
• Structural selection
• Code navigation
• Semantic highlighting
• LSP integration and more

---

Project setup

Before we start writing the grammar, let's get a minimal Tree-sitter project running. Since the goal of this post isn't to explain the installation process, I'll keep this section short and link to the official setup guide.

Dependencies:

• A JavaScript runtime
• A C compiler
• Tree-sitter CLI (0.26.9)

Follow the getting started documentation to set up a similar project.

---

Building the grammar

With the Tree-sitter project set up, the next step is to teach it what Monkey looks like. We already have a parser and AST that define the language's structure in the interpreter, so rather than designing the grammar from scratch, we'll translate those existing concepts into Tree-sitter rules.

1. Starting with a basic grammar

The grammar skeleton

The following is the minimal skeleton of our grammar.
grammar.js

/**
 * @file Monkey grammar for tree-sitter
 * @author SegniAT <[e-mail]>
 * @license MIT
 */

/// <reference types="tree-sitter-cli/dsl" />
// @ts-check

export default grammar({
  name: "monkey",

  rules: {
    source_file: $ => repeat($._statement),

    _statement: $ => choice(
      $.expression_statement
    ),

    expression_statement: $ => seq(
      $.expression,
      optional(';')
    ),

    expression: $ => choice(
      $.identifier,
      $.integer,
    ),

    identifier: _ => /[a-zA-Z_]+/,
    integer: _ => /\d+/,
  }
});

The grammar function is the most important part of this file. It contains the declarative schema for our language. The name property is the name of the language we're writing the grammar for and the rules property allows us to define rules using built-in functions listed in the documentation.

We build the grammar top-down, mirroring how the interpreter's AST (Abstract Syntax Tree) is organized. One important fact to know up front is that the start rule for the grammar is the first property in the rules object. In the example above, that would correspond to source_file, but it can be named anything.

Every grammar rule is written as a JavaScript function that takes a parameter $. The syntax $.identifier is how you refer to another grammar symbol within a rule.

In the monkey parser, a program is a struct that has a slice of statements:

// ast/ast.go
type Program struct {
    Statements []Statement
}

We replicated this in our grammar as follows:

source_file: $ => repeat($._statement), // 'repeat(rule)' creates a rule that matches zero-or-more occurrences of a given rule.

Now we have to define _statement. In our Monkey parser, there are 3 types of Statements:

// ast/ast.go
type Statement interface {
    Node
    statementNode()
}

// 1. The LET statement 
type LetStatement struct {
    Token token.Token // the token.LET token
    Name  *Identifier
    Value Expression
}

// 2. The RETURN statement
type ReturnStatement struct {
    Token       token.Token // the 'return' token
    ReturnValue Expression
}

// 3. The ExpressionStatement statement
type ExpressionStatement struct {
    Token      token.Token // the first token of the expression
    Expression Expression
}

In our skeleton grammar we only define the Expression Statement for now:

// Starting a rule's name with an underscore causes the rule to be hidden in the syntax tree. This avoids depth and noise to the syntax tree.
// 'choice(rule1, rule2, ...)' function creates a rule that matches one of a set of possible rules.
_statement: $ => choice(
      $.expression_statement
      // We later add the LET and RETURN statements here.
),

Expression Statement is defined as the following in our grammar:

// `seq(rule1, rule2, ...)` function creates a rule that matches any number of other rules, in order.
expression_statement: $ => seq(
    $.expression,
    optional(';') // `optional(rule)` function creates a rule that matches zero or one occurrence of a given rule.
)

An Expression Statement is just an Expression followed by an optional semicolon (semicolons are optional in Monkey).
Let's look at Expression now:

expression: $ => choice(
      $.identifier,
      $.integer,
),

identifier: _ => /[a-zA-Z_]+/,
integer: _ => /\d+/,

An expression can be an identifier or an integer here, we will add much more to our final version. We describe identifier and integer as regular expressions, an identifier can only include letters and underscores, while an integer only includes numbers.

At this point we have enough grammar to parse something simple, so let's verify that Tree-sitter produces the tree we expect before adding more rules.

Testing the grammar

Create test/corpus/basics.txt with these two tests:

=================================
Expression statement (identifier)
=================================

foo

---

(source_file
  (expression_statement
    (identifier)))

==============================
Expression statement (integer)
==============================

42

---

(source_file
  (expression_statement
    (integer)))

Let's now generate a parser from the grammar and test it against the test.

tree-sitter generate
tree-sitter test

Output:

basics:
    1. ✓ Expression statement (identifier)
    2. ✓ Expression statement (integer)

Total parses: 2; successful parses: 2; failed parses: 0; success percentage: 100.00%; average speed: 593 bytes/ms

Our generated parser based on the grammar outputs an S-expression when parsing our target language source code, so we use that fact to write our tests. Let's look at the first tests.

• The name of each test is written between two lines containing only = (equal sign) characters.
• Then the input source code is written, followed by a line containing three or more - (dash) characters.
• Then, the expected output syntax tree is written as an S-expression. The exact whitespace in the S-expression doesn't matter.

As shown in the expected output, the root of our syntax tree is a named node called source_file, which directly corresponds to our root rule defined in grammar.js.

The next node we might expect would be _statement, but the underscore makes it hidden. So we move on to its children, for now we just have expression_statement which has either identifier or integer as its children. In our first test case, foo will be identified as identifier, but in the second 42 is an integer node.

2. Building out the grammar

The basic grammar works, so now we can start filling in the pieces we left out of the skeleton.

Statements: let and return

Our interpreter has two more Statement types in addition to the already defined Expression Statement. We add them as the other two alternatives of _statement rule:

// `field(name, rule)` function assigns a field name to the child node(s) matched by the given rule. We can use it to access specific children in the resulting syntax tree.
let_statement: $ => seq(
  'let',
  field('name', $.identifier),
  '=',
  field('value', $.expression),
  optional(';')
),

return_statement: $ => seq(
  'return',
  field('value', $.expression), 
  optional(';')
),

This mirrors the interpreter's AST: the LetStatement struct has a Name and a Value field.

With the remaining Statements in place, we can move on to the remaining Literals and Expressions. This is where the grammar starts getting more interesting. Monkey has several kinds of Literals and Expressions, and some Tree-sitter features become necessary to keep the resulting tree useful and less noisy.

Literals and Expressions

Some of the remaining Literals are as follows:

// `token(rule)` function marks the given rule as producing only a single token. Tree-sitter's default is to treat each `String` or `RegExp` literal in the grammar as a separate token. We don't want 3 separate tokens here, just one.
string: _ => token(seq('"', /[^"]*/, '"')), // We don't allow '"' in strings since we cannot escape characters in Monkey at the time of writing this.
boolean: _ => choice("true", "false"),

// ... function, hash, array literals

In the Monkey grammar, the expression rule is defined as a choice between many different kinds of Expressions:

    expression: $ => choice(
      $.identifier,
      $.integer,
      $.string,
      $.boolean,

      $.unary_expression,
      $.binary_expression,
      $.paren_expression,

      $.call_expression,
      $.index_expression,

      $.if_expression,
      $.function_literal,
      $.array_literal,
      $.hash_literal,
),

Keyword extraction using word

Consider the following snippet:

iffoo

Tree-sitter would lex this source code as follows:

• an if keyword
• an identifier foo

But if should only be matched if it appears as a whole word, on its own.

The word property solves this problem. It tells Tree-sitter which rule represents the language's "word" (its identifier). This is what drives keyword extraction: Tree-sitter scans the grammar for string literals that could collide with identifiers (let, fn, true, false, if, else, return) and turns them into keywords that only match as whole words.
Add word property as a grammar-level setting:

word: $ => $.identifier,

Abstract categories using supertypes

Some rules in your grammar will represent abstract categories of syntax nodes, such as "expression", "type", or "declaration". These rules are often defined as simple choices between several other rules. As shown above, in our grammar, the expression rule is a choice between 13 different rules.

By default Tree-sitter will generate a visible node type for each of these abstract category rules, which can lead to unnecessarily deep and complex syntax trees. To avoid this you can add these abstract category rules to the grammar's supertypes definition. Tree-sitter will then treat these rules as supertypes and will not generate visible node types for them in the syntax tree.

supertypes is a cleanup setting, we add it as a grammar-level setting:

supertypes: $ => [
  $.expression,
],

What's the difference between supertypes and hidden rules (using _ prefix)?
Both supertypes and hidden rules are used to keep the final syntax tree cleaner, they are both hidden. However, they serve different purposes and are used in distinct ways.

supertypes mark abstract categories as conceptual groups, so they don't appear as concrete nodes while hidden rules hide intermediate helper rules that exist only to structure the grammar, not to represent a meaningful language construct.

A query can target a supertype, and it will automatically match all of its sub-types. When it comes to hidden rules, since the node doesn't exist in the tree, you can't query for it.

Expression precedence (Pratt precedence)

This is the most interesting part of our parser, because the interpreter's Pratt parser and Tree-sitter's GLR parser solve the same problem in completely different ways.

Let's look at why we need to explicitly define precedence for rules that may cause conflict. Let's say our grammar has the following snippet:

{
  // ...
  expression: $ => choice(
    $.identifier,
    $.unary_expression,
    $.binary_expression,
    // ...
  ),

  unary_expression: $ => choice(
    seq('-', $.expression),
    seq('!', $.expression)
  ),

  binary_expression: $ => choice(
    seq($.expression, '*', $.expression),
    seq($.expression, '+', $.expression),
    // ...
  ),
}

This flat structure is highly ambiguous. If we try to generate a parser with the tree-sitter generate command Tree-sitter gives us an error message:

Error: Unresolved conflict for symbol sequence:

  '-'  _expression  •  '*'  …

Possible interpretations:

  1:  '-'  (binary_expression  _expression  •  '*'  _expression)
  2:  (unary_expression  '-'  _expression)  •  '*'  …

Possible resolutions:

  1:  Specify a higher precedence in `binary_expression` than in the other rules.
  2:  Specify a higher precedence in `unary_expression` than in the other rules.
  3:  Specify a left or right associativity in `unary_expression`
  4:  Add a conflict for these rules: `binary_expression` `unary_expression`

• in the error message indicates where exactly during parsing the conflict occurs. For an expression like -a * b, it's not clear whether the - operator applies to the a * b or just to the a.

This is where the prec function comes into play. By wrapping a rule with prec, we can indicate that certain sequence of symbols should bind to each other more tightly than others. For example, the -, $.expression sequence in unary_expression should bind more tightly than the $.expression, +, $.expression sequence in binary_expression.

Our interpreter resolves operator precedence in code at runtime, with a table of constants. They are numbered starting from 1 (LOWEST) to 8 (INDEX):
parser/parser.go

const (
    _ int = iota
    LOWEST
    EQUALS      // ==
    LESSGREATER // > or <
    SUM         // +                
    PRODUCT     // *
    PREFIX      // -X or !X
    CALL        // myFunction(X)
    INDEX       // array[index]
)

Tree-sitter handles precedence in the grammar, by attaching prec and prec.left to the rules themselves. The mapping is almost one-to-one:

Operator	Interpreter constant	Tree-sitter
==, !=	EQUALS	prec.left(1)
<, >	LESSGREATER	prec.left(2)
+, -	SUM	prec.left(3)
*, /	PRODUCT	prec.left(4)
-x, !x (prefix)	PREFIX	prec(5)
f(x) (call)	CALL	prec.left(7)
a[i] (index)	INDEX	prec.left(8)

{
  // ...

// `prec(number, rule)` function marks the given rule with a numerical precedence, which will be used to resolve LR(1) Conflicts (https://en.wikipedia.org/wiki/LR_parser#Conflicts_in_the_constructed_tables) at parser-generation time.
  unary_expression: $ =>
    prec(
      5,
      choice(
        seq("-", $.expression),
        seq("!", $.expression),
        // ...
      ),
    );

    // `prec.left([number], rule)` function marks the given rule as left-associative (and optionally applies a numerical precedence).
    binary_expression: $ => choice(
      prec.left(1, seq(field('left',$.expression), '==', field('right',$.expression))),
      prec.left(1, seq(field('left',$.expression), '!=', field('right',$.expression))),
      prec.left(2, seq(field('left',$.expression), '<', field('right',$.expression))),
      prec.left(2, seq(field('left',$.expression), '>', field('right',$.expression))),
      prec.left(3, seq(field('left',$.expression), '+', field('right',$.expression))),
      prec.left(3, seq(field('left',$.expression), '-', field('right',$.expression))),
      prec.left(4, seq(field('left',$.expression), '*', field('right',$.expression))),
      prec.left(4, seq(field('left',$.expression), '/', field('right',$.expression))),
    ),
}

prec.left(n) means: this operator binds with precedence n and is left-associative, so a - b - c parses as (a - b) - c, exactly like the interpreter's loop, which keeps parsing while the next token's precedence is higher. Prefix operators use plain prec(n), because associativity doesn't apply to them.

In the Pratt parser the constants only need to be relatively ordered. Tree-sitter's numbers do the same job, so the two tables line up almost exactly.

There is one main difference between the two styles worth knowing: the Pratt parser decides how to continue looking forward from an expression, but the GLR parser explores all parse branches in parallel and only uses precedence to resolve conflicts when they actually arise. That's why for example choice order doesn't matter in Tree-sitter.

Calls, indexing, and the remaining constructs

With operator precedence sorted out, we can finish the remaining Expressions. Calls and indexing are Infix Expressions with the highest binding as seen in the previous section.

call_expression: $ => prec.left(7, seq(
  field('function', $.expression),
  field('arguments', $.argument_list)
)),

index_expression: $ => prec.left(8, seq(
  field('object', $.expression),
  '[',
  field('index', $.expression),
  ']',
)),

The remaining constructs need no new concepts, they are combinations of the primitives we have already seen.

block: $ => seq('{', repeat($._statement), '}'),

if_expression: $ => seq(
  'if',
  '(',
  field('condition', $.expression),
  ')',
  field('consequence', $.block),
  optional(seq('else', field('alternative', $.block))),
),

function_literal: $ => seq(
  'fn',
  field('parameters', $.parameter_list),
  field('body', $.block)
),

hash_pair: $ => seq(
  field('key', $.expression),
  ':',
  field('value', $.expression),
)

3. The complete grammar

At this point we've covered all the Tree-sitter concepts that required explanation. The remaining rules are mostly combinations of the same primitives, so we can jump directly to the complete grammar. The final grammar.js is only ~145 lines.

Feeding it a complete program shows what we built. Create a file with the following content:

let x = 5;
let add = fn(a, b) { return a + b; };
let result = add(3, 4);
let arr = [1, 2, 3, 4];
let person = {"name": "Alice", "age": 30};
let age = person["age"];
if (age > 21) { let status = "adult"; } else { let status = "minor"; }
let makeAdder = fn(x) { return fn(y) { return x + y; }; };
let addFive = makeAdder(5);
addFive(10);

Running the command tree-sitter parse [your file name] should output the following:

(source_file
  (let_statement name: (identifier) value: (integer))
  (let_statement
    name: (identifier)
    value: (function_literal
      parameters: (parameter_list
        name: (identifier)
        name: (identifier))
      body: (block
        (return_statement
          value: (binary_expression
            left: (identifier)
            right: (identifier))))))
  (let_statement
    name: (identifier)
    value: (call_expression
      function: (identifier)
      arguments: (argument_list
        (integer)
        (integer))))
  (let_statement
    name: (identifier)
    value: (array_literal
      (integer) (integer) (integer) (integer)))
  (let_statement
    name: (identifier)
    value: (hash_literal
      (hash_pair
        key: (string)
        value: (string))
      (hash_pair
        key: (string)
        value: (integer))))
  (let_statement
    name: (identifier)
    value: (index_expression
      object: (identifier)
      index: (string)))
  (expression_statement
    (if_expression
      condition: (binary_expression
        left: (identifier)
        right: (integer))
      consequence: (block
        (let_statement
          name: (identifier)
          value: (string)))
      alternative: (block
        (let_statement
          name: (identifier)
          value: (string)))))
  (let_statement
    name: (identifier)
    value: (call_expression
      function: (identifier)
      arguments: (argument_list
        (integer))))
  (expression_statement
    (call_expression
      function: (identifier)
      arguments: (argument_list
        (integer)))))

Every construct shows up as a named node, with fields (name:, value:, left:, right:, condition:, consequence:, ...) carrying the same information the interpreter's AST carries in Go.

The grammar is now complete, but that does not mean that the generated parser outputs the correct syntax tree. Validation is our next undertaking.

---

The corpus as executable specification

tree-sitter generate compiles the grammar into a C parser, and tree-sitter test runs the corpus (test cases). There are 70 test cases in the project written alongside the grammar:

Total parses: 70; successful parses: 70; failed parses: 0; success percentage: 100.00%

The corpus is the spec. Because each test includes an input source code and the expected output tree, it also serves as documentation of the language.

Error recovery: Tree-sitter's superpower for editors is its behavior on broken code. Instead of stopping the parsing task, the parser emits an ERROR node and continues its parsing. The tree stays mostly correct while you type. We test these cases in the corpus with the :error directive:

===========================
Integer as let name
:error
===========================

let 5 = x;

---

============================
Missing value
:error
============================

let x = ;

---

============================
Unterminated list
:error
============================

[1, 2

These tests assert that the input contains a syntax error while still allowing Tree-sitter to produce a useful tree around it. This is why highlighting stays useful in an editor. Code is broken most of the time while we're writing it, and the parser still produces a useful tree around the errors.

At this point we have a parser that can turn Monkey source code into a syntax tree, and we can test that parser against a corpus to verify its validity. But our original goal wasn't just to parse Monkey, we wanted an editor to understand it well enough to highlight it.

---

Turning the parsed tree into highlighting

To achieve our main goal of syntax highlighting, we need to write queries that match nodes in the syntax tree and assign them captures, those are names that Neovim understands and maps to highlight groups it has already defined.

You can find the list of standard captures supported by Neovim in their Tree-sitter docs. These include captures such as @function, @variable, @keyword, and @string, which are highlighted according to the current colorscheme.

The queries assigned to captures live at queries/highlights.scm. The ordering inside it matters, more specific patterns need to come before generic ones. But we will set priorities using #set! priority N to be more explicit. Higher numbers win, which lets us keep related patterns grouped together instead of carefully ordering every rule.

In our case, we have different types of functions and identifiers:

; User-defined function calls
((call_expression
  function: (identifier) @function.call)
  (#set! priority 120))

; Built-in function calls
((call_expression
  function: (identifier) @function.builtin
  (#any-of? @function.builtin "len" "first" "last" "rest" "push" "puts"))
  (#set! priority 130))

; Function parameters
((function_literal
  parameters: (parameter_list
    (identifier) @variable.parameter))
  (#set! priority 120))

; Hash keys are properties (strings and bare identifiers)
((hash_pair
  key: (string) @property)
  (#set! priority 120))

((hash_pair
  key: (identifier) @property)
  (#set! priority 120))

; All-caps identifiers are constants by convention, which is a convention from other languages - Monkey has no reassignment, so every variable is effectively constant.
((identifier) @constant
  (#match? @constant "^[A-Z][A-Z_]*$")
  (#set! priority 120))

; more captures ...

; Catch-all: every other identifier is a variable
(identifier) @variable

Two predicates do the heavy lifting here:

• #any-of? checks the node's text against a list, that's how len and other builtin functions become @function.builtin while every other callee is @function.call. Monkey has no named function definitions (functions assigned to a variable with let), so @function itself stays unused.
• #match? applies a regex, all-caps identifiers become @constant by convention, before the catch-all below turns everything else into @variable. The Monkey language does not have the concept of constant variables, this is just for convention.

The rest is a straightforward mapping of literals and keywords:

; Keywords
"let" @keyword

[
  "if"
  "else"
] @keyword.conditional

"return" @keyword.return

"fn" @keyword.function

; Literals
[
  "true"
  "false"
] @boolean

(integer) @number

(string) @string

; ... more

Keywords and literals map directly, and the rest of queries/highlights.scm covers operators, brackets, and delimiters (,, ;, :). Check out the file to explore all the captures. The full list is 18 captures which is tiny compared to languages like TypeScript or Rust for example.

We now have both pieces we need to finalize our project: a parser that produces the tree and highlight queries that turn nodes into captures. The final step is connecting that grammar to Neovim so it knows when to use the parser for .monkey files.

---

Neovim integration and showcase

Now comes our final task, i.e. editor integration. We will follow the nvim-treesitter README.md, "Adding custom languages" section.

First, we need to let Neovim know that .monkey files are a thing, in our configuration file we add the following:

vim.filetype.add({ extension = { monkey = "monkey" } })

Then we need to register the parser with nvim-treesitter:

vim.api.nvim_create_autocmd('User', {
  pattern = 'TSUpdate',
  callback = function()
    require('nvim-treesitter.parsers').monkey = {
      install_info = {
        url = 'https://github.com/SegniAT/monkey-language-interpreter',
        location = 'tree-sitter-monkey',
        queries = 'tree-sitter-monkey/queries',
      },
    }
  end,
})

The interesting part is the monorepo layout: our grammar lives inside the interpreter's repository. nvim-treesitter downloads the repo tarball from url, builds the grammar found at location, and installs the queries found at queries.

This works, but it isn't really ideal since the grammar is only one directory inside a much larger repository of the Monkey interpreter (and soon, the LSP as well)! So installing the parser means downloading the entire interpreter repository.

After this, the update loop is:

tree-sitter generate && tree-sitter test   # locally
git commit && git push                     # to the repo
:TSUpdate monkey                           # in Neovim

That's enough to get the whole pipeline working. We can now open a .monkey file in Neovim, have Tree-sitter parse it, apply our queries, and get syntax highlighting.

Before:

After: (tokyonight-night theme)

Beautiful!

A look at the generated tree by the parser using :InspectTree in Neovim:

A few queries on the generated tree using :EditQuery in Neovim:

---

Conclusion

To add syntax highlighting and make our custom language prettier to look at and more convenient to work with, we had to go through a whole bunch of fun challenges: a hand-written grammar, a corpus that also serves as a spec, queries that map the tree to editor semantics, and finally editor integration.

The biggest lesson: the grammar is an executable spec and the corpus is its test suite. Together they make the language's actual behavior much harder to misunderstand than documentation alone.

A second lesson: a toy language is still a precise language. Monkey's grammar fits in ~145 lines, yet every rule had to be right: precedence levels, hidden rules, keyword extraction, fields, this is the same machinery that parses Go or Zig, just much smaller.

But syntax highlighting is only the beginning. We can now tell the editor what a piece of code looks like, but not much about what it means. The next step is to teach the editor about Monkey itself: diagnostics, definitions, completion, and eventually the other features we expect from a modern development environment. That's where the LSP comes in, in our second entry the series.

---

References

• Writing An Interpreter In Go by Thorsten Ball
• Tree-sitter documentation
• The Monkey interpreter + tree-sitter grammar repository
• TJ DeVries - Tree-sitter explained
• Tree-sitter: a new parsing system for programming tools - GitHub Universe 2017