The agent submitted. The judge said no. Then the agent submitted again.
That was the one failed Vow submission in my November 2026 hard-task runs, and it is the reason the numbers below read "19" next to "18". Here is the full picture, including the parts where Vow did not win.
The setup
Vow is a programming language built for AI agents to write. It is designed around the loop every coding agent runs: write, run, read the error, fix. I'm Jake Burwick, a high school student, and I wanted to know whether a language built around that loop actually makes agents better at finishing work.
So I measured. Here are the results, with the sample sizes attached, because they are small.
October 2026, Claude Haiku 4.5
On three brand-new hard tasks, Vow had one failed submission in nine runs. TypeScript had zero.
Code that agents finished was then run against fresh hidden tests. It passed at 99.5% for Vow and 99.3% for TypeScript.
That is a tie. On a sample this small, the two perform evenly.
November 2026, Claude Sonnet 5.5 and Opus 5.5
Three more brand-new hard tasks. Both models solved every run in both languages.
Vow had one failed submission in 18 runs. TypeScript had zero.
Attempts count too. The 18 Vow runs took 19 judged submissions. The 18 TypeScript runs took 18. I have not measured token use, so I can't tell you what that costs.
Nobody is winning that table. If you came here for "Vow crushes TypeScript", I do not have it.
Where the design shows up
The results say little about how often agents finish. The design is about what happens on the way there. Every Vow error says what is wrong, where, and the likely fix, written as a repair instruction an agent can act on without a human:
x TypeError: can't add a num and a str
-> line 6, col 10
6 | return "Total: " + sum
^^^^^^^^^^^^^^^
Likely fix: convert the number to text
first: str(sum)
There are no nulls and no exceptions. A function that can fail returns a Result, and you handle it with match or the ? operator:
fn parsePrice(text) {
let n = num(text)?
if n < 0 { return err("can't be negative") }
return ok(n)
}
Some bad calls never get to run at all:
fn withdraw(bal, amt)
expects amt > 0 and amt <= bal
{ return bal - amt }
print(withdraw(100, 500))
x ContractError: this call can never
satisfy fn withdraw's contract
-> line 4, col 7
One more result, on bad data
The tie above is about finishing tasks. A separate benchmark asks whether a language catches bad inputs. vow test has a bad-input probe. After a program's tests pass, it corrupts the inputs those tests use (an empty input, a field that reads 12a, a missing JSON field) and flags any corrupted copy that still printed a normal answer. It only runs programs the checker proves cannot touch the network, files or environment.
Measured November 15-16, 2026 with Claude Haiku 5.5, on three fresh sets of 3 bad-data tasks (36 programs per language), the share of bad inputs flagged was:
- Vow with the probe: 68%
- Python: 53%
- Vow with the probe off: 51%
- TypeScript: 20%
On the third set, which had many logic problems in the data, Vow with the probe flagged 57% and Python 45%.
I reran the sixth set with Claude Sonnet 5.5 (12 programs per row, every program passed all valid-input tests). Vow with the probe flagged 85%. Python 39%, Vow with the probe off 35%, TypeScript 21%.
A row with the probe described in the guide but switched off also scored 35%. So the gain comes from what vow test prints back, not from the guide text.
The limits: this benchmark has only run on Claude Haiku and Sonnet. It has not been run on Opus.
Trying it
npm install -g github:JakeBurwick/vow
For Claude Code, add the MCP server. It gives the agent vow_guide, vow_check and vow_run tools:
claude mcp add vow -- vow-mcp
For Codex, Cursor or anything else, paste the one-paragraph prompt from the site and it points the agent at the guide. The site also has an llms.txt and an AGENTS.md you can drop into a project.
Vow is open source under the MIT license. The source is at github.com/JakeBurwick/vow, and the whole language is about 6,400 lines of JavaScript.
The playground, tutorial, docs, and an Errors page with five bugs side by side in Vow, TypeScript and Python are at https://vow-lang.vercel.app
If you run your own comparison, I would like to hear whether your ties look like mine.
Top comments (0)