When an LLM edits code, it usually works on text: it rewrites the file, or it finds a string and replaces it. I wanted to try the other way around. The model edits a tree, a checker validates it before anything is written, and a printer is the only thing that produces source.
That's Nodast (Node + AST). This post is about what I measured, including the part where my first conclusion turned out to be wrong.
The idea
The model writes short S-expressions:
(let x (add 1 2) x)
A Hindley-Milner checker accepts or rejects the tree, with a structured error the model can retry on. If it passes, a deterministic printer emits TypeScript:
const x = 1 + 2;
x;
To edit, every node carries an id, like n4:(add n5:v0 n6:7), and the model works through three tools: find, show and edit. A rejected edit changes nothing.
First result: a huge win (that wasn't fair)
My first benchmark compared Nodast with the model rewriting the whole TypeScript file. On a 500-line file, changing one constant:
- Nodast with tools: 1,122 input tokens, 57 output tokens, 4.5 s
- Full rewrite: 5,559 input, 5,513 output, 36 s
About a hundred times less output. Great headline. Wrong comparison.
Nobody serious edits by rewriting the file. Coding agents today use grep, read and a string replace. So I added that baseline: same TypeScript file, same tool loop, same call budget, replace with a replace-all option, and diagnostics after each edit, like an editor.
Second result: a tie
Against that fair text agent, on the same single-constant edit, Nodast tied. After trimming its prompt to one grammar line and taking edits as plain fields, it still used slightly more input (772 vs 581 tokens), the same output and the same time.
The lesson: almost all of the savings came from editing in pieces instead of rewriting, not from the tree format. A text agent with good tools gets that too.
Third result: where the tree actually wins
A tree should win where text tools struggle: changes that touch many places and need to understand scope. So I added two structural operations and a benchmark task for each.
rename renames a binding and every use that resolves to it, following scopes. The task: rename rate to scale in a 400-step program, where a variable called rate_cap sits right next to it.
The text agent's usual move is to replace rate everywhere, which turns rate_cap into scale_cap, and then fix it back. Nodast does it in one edit and never touches rate_cap.
param adds a parameter to a function and passes an argument at every call. The task: a function called 200 times, each call with a different argument, so replace-all doesn't help. The oracle also calls step(1, 5), so just hardcoding the value inside the function doesn't count.
Results with gpt-6.1-sol, 5 runs each, every variant correct in every run. Cost at 2 USD per million input tokens and 10 USD per million output tokens:
| Task | Nodast, tools | Text agent, tools | Full rewrite |
|---|---|---|---|
Rename in 400 steps, next to rate_cap
|
4.4 s, 907 in / 55 out | 6.7 s, 1,187 in / 80 out | 29 s, 4,975 in / 4,927 out |
| Add a parameter, 200 calls | 9.1 s, 2,182 in / 136 out | 13.7 s, 12,045 in / 174 out | 35 s, 7,432 in / 5,668 out |
| Change one constant, 500 lines | 4.6 s, 772 in / 61 out | 4.5 s, 581 in / 67 out | 33 s, 5,559 in / 5,513 out |
Adding a parameter at 200 call sites cost about 4.5 times less than with the text agent. The rename was faster and cheaper too, though the gap moved a lot between passes: the text agent sometimes gets it in 2 calls and sometimes needs 5.
A bug the benchmark found in my checker
Writing small programs from scratch, Nodast got 55/60 right on the first try against 60/60 for plain TypeScript. All five failures were the same: the model wrote (fn (f) (f 5)), which is a perfectly valid program, and my checker rejected it as "ambiguous type", because it had no generics.
I replaced the checker with Hindley-Milner inference: a function bound with let is generic, and a type nothing fixes stays a variable, so that program is ((number) -> a) -> a. First-try writes went to 60/60, with 23 output tokens against 25 for TypeScript.
The model still never writes a type annotation.
Limits
-
The language is small on purpose: numbers, bools,
let, functions, calls,if. No strings, lists or records yet, so no real-world programs yet. -
The benchmark is synthetic: long chains of
letand repeated calls, built to isolate one kind of edit at a time. - One strong model in the main results. MiniMax was used during development.
What's next
- More structural operations, like extracting an expression into a
let. - Growing the language toward real programs: strings, lists, records.
- Results on more models. A Claude provider for the benchmark is a good first issue.
The code, the benchmark and an offline demo (pnpm demo, no API key) are on GitHub: https://github.com/R4F405/Nodast. It's MIT, and the TypeScript reference is mirrored by a Rust engine.
If you build coding agents, I'd love to know which edits hurt the most with text tools. Those are the next benchmark tasks.
Top comments (0)