Build Buddy, Day 2: two AI fixes compiled, only one was correct

Day 2 of MLH Global Hack Week: Builders Week. Yesterday my tool Build Buddy fixed 6 of 6 simple C# errors with a local model. Today I wanted to compare models on something harder, and to do it properly.

The setup

One C# file with three planted errors:

• CS0029: GetStringAsync returns Task<string>, assigned to string
• CS1061: body.Lenght instead of Length
• CS0161: Half() doesn't return on every path

Before asking any model, I committed a protocol: the exact prompt, the input file, the scoring rules, and the compiler's real output for the broken file. Then each answer was built in a clean copy of the project.

The baseline caught my mistake first

My first version used a missing semicolon instead of the typo. The baseline showed the compiler stopped at a syntax error (CS1003) and never reported the other two errors. A model "fixing three errors" would really have fixed one. I swapped it for the typo before running anything.

Results

	Grok 4.7 (Cursor)	Gemma 3 4B (local, Ollama)
Builds	✅	✅
Lines touched	3	6
Error codes named correctly	3/3	0/3
Time	33 s	17 s

Only Grok 4.7 was unlocked on my Cursor plan, so I added my local Gemma 3 4B as a comparison point. It's not a Cursor model.

Both compiled. Only one was right.

Grok 4.7 used await, fixed the typo, added return 0;, and changed nothing else. It named all three error codes correctly.

Gemma 3 4B was twice as fast, and its code compiled, but:

• it used .Result, which blocks the thread
• it rewrote Half(), so Half(5) now returns 3 instead of 2
• it added an unneeded using System;
• its explanation invented a "System.Linq was not used" error and never mentioned the Task/await problem

If I had only checked "does it build?", I'd have scored these the same.

What I'm taking away

• Compiling is necessary, not sufficient. Behavior and explanation need checking too.
• Record a baseline before testing fixes. My own test case was broken.
• Start every check from a clean folder. One of my manual checks failed with nine errors because I copied into an old folder, and none of the errors were the model's fault.

Protocol, raw answers, and results: benchmarks/cursor-models

Next

Loading Build Buddy's results into a Tiger Data hypertable, so I can query fix rates over time.

How it was built: protocol and analysis worked through with Claude; Build Buddy features built in Cursor; every result comes from a real build on my laptop.