This is a submission for the Monthly MLH × DEV Writing Challenge.
What I Built
AI tools answer every compiler error with confidence. I wanted to know how often they're actually right, measured offline on my own laptop, in C#, the language I work in.
So during MLH Global Hack Week: Builders Week I built Build Buddy, a .NET command-line tool that:
- runs
dotnet buildand collects every C# compiler error, - asks a local model (Gemma 3 4B through Ollama) to explain each error and fix it,
- checks the fix itself instead of trusting the model.
That third step is the whole project. Over the week, the check got stricter, and every step was forced by something I found.
Step 1: does it build?
On day one, Gemma fixed 6 of 6 single-error samples, and every patched project rebuilt cleanly. Average 11.0 seconds per error. It looked perfect.
Step 2: same bug, two models
I gave one broken file with three planted errors to two models: Grok 4.7 in Cursor, and Gemma on my laptop. Both fixes compiled. Only one was correct. Gemma's fix quietly rewrote a method so Half(5) returned 3 instead of 2. The compiler had no way to notice.
Step 3: does it behave?
A reader on my Day 2 post asked whether behaviour tests would catch that automatically. So Build Buddy now runs the patched program and compares its output with an expected.txt for each sample. A fix only counts if it builds and behaves.
The live result dropped from 6/6 to 5/6. The CS0161 fix compiled but printed three lines instead of one. At temperature 0, Gemma gives that same answer every run, so it was a real weakness, not bad luck.
Step 4: one sentence in the prompt
The same reader suggested treating CS0161 as a known failure case and changing the prompt. I added one sentence: "Keep the program's behaviour and console output exactly the same." That took it to 6/6.
Then I tested my own fix on four new samples the rule had never seen. 3/4 with the rule, 3/4 without it. It fixed the problem it was written for, and nothing more. I wrote the rule after seeing the failures and tested it on the same six samples, which is the classic way to fool yourself. The held-out test caught it.
| Check | Result |
|---|---|
| Builds only (day one) | 6/6 |
| Builds and behaves | 5/6 |
| With the one-sentence rule (after the week) | 6/6 |
| Held-out samples, with or without the rule | 3/4 |
What surprised me
Two models fixed the same bug, and both fixes compiled. Only one was correct. Later, one sentence in the prompt fixed the failing case and, as a side effect, two wrong explanations.
That second part is the one I can't stop thinking about. Gemma's code was often right while its explanation was wrong: it said a missing semicolon "wasn't required," and invented an extension method that doesn't exist. The behaviour check proves the code, not the words.
What I learned
- Run the code, not just the build.
- Time the model's cold start separately. My "slowest error" was really the model loading into memory.
- Give agents read-only access first.
- Vector search finds the right topic, not the true answer.
- Test a fix on cases it wasn't tuned on.
Demo
The video shows two real runs: the default prompt (6/6 build, 5/6 behave, with the CS0161 output mismatch) and the same run with --keep-output (6/6).
Code, samples, and every result file: github.com/GurminderKaur/build-buddy
The full build log is in six short posts: Day 1 · Day 2 · Day 3 · Day 4 · Day 5 · after the week
Partner Technologies
Tiger Data
I used Tiger Data three ways, and each one taught me something I wouldn't have found otherwise.
A hypertable for benchmark results. I loaded every Build Buddy run into a build_results hypertable partitioned by day, then queried it with time_bucket. CS0029 looked like the hardest error (15.0 s average). It isn't. It's the first sample in every run, so it pays for loading the model: 21.7 s and 23.1 s cold, 7.7 s and 7.6 s warm. One run would never have shown that. Comparing runs over time did, and it's why Build Buddy now times the cold start separately.
Tiger CLI and MCP, read-only first. Tiger's MCP server gives an agent tools that can delete services and reset passwords, so before connecting one I ran tiger config set read_only all. With that set, the write tools aren't even registered. My network blocked the database port, so the agent couldn't query the table. It read the service's live logs, saw that no connection ever arrived, and told me it was using my local file instead of inventing live numbers. That's the behaviour I want from an agent with database access.
pgvector for searching AI explanations. I embedded 12 real explanations locally with nomic-embed-text and stored them in a hypertable with a VECTOR(768) column and an HNSW index. Asking "async call assigned without await" found the right explanation clearly (0.245, next best 0.338). But for "missing semicolon," the best match was the wrong explanation (0.231). Vector search found the right topic and had no idea whether the content was true. Retrieval needs two separate checks: trusted sources, and a relevance threshold.
Cursor
I built Build Buddy's verify mode, its samples, and its error handling with Cursor's agent, and used Cursor for the model comparison in Step 2. Cursor's agent also taught me a lesson about checking AI work: when I tightened the definition of "fixed," it relabelled my old results instead of rerunning them. I threw those rows out and reran everything, so every row in the repo comes from a real run.
Hackathon Experience
I took part in Global Hack Week: Builders Week (Nov 6–12, online, solo), and completed twelve challenges along the way, including four from Tiger Data and two from Cursor.
The hardest parts weren't the AI. My network blocked the database port, and one of my own test cases was broken before any model touched it.
My favourite moment: a reader asked on my Day 2 post whether behaviour tests would catch the bad fix. That one comment became two features and a more honest number. And an AI agent that couldn't reach my database said so, instead of making up live numbers.
If I could tell another developer one thing: "It compiles" isn't "it works," and "it works" isn't "the explanation is right." Check each one separately.
What I'd do differently: run the AI's code in a sandbox, not just with a time limit, and write held-out samples on day one, before tuning anything.
Limits: six original samples plus four held-out, one model per run, and explanation verdicts are my own reading.
How it was built: Build Buddy's starter code, the later features, and this write-up were worked through with Claude; features were also built with Cursor's agent. I ran every test on my own laptop, and every number comes from those runs.
Thanks to MLH, the Builders Week team, and the reader whose questions shaped this project.
Top comments (0)