Three Things I Learned Building a Local-LLM Java Repair Tool

I built a tool that uses AI to fix broken Java projects. One of the biggest lessons? You can't just trust what the AI gives you.

Over the past few days, I've been working on CompileLoop, a Java project that uses local LLMs to try and fix broken Maven builds.

The idea is pretty simple. You give it a broken Java project, it sends the build errors to a local code model, gets back a proposed fix, applies that fix in a temporary copy of the project, and runs the build again to see if it actually worked.

If the fix passes validation, you get a patch to review. Your original source code stays untouched.

I'm building this because I want to explore how far we can take AI-assisted software development without blindly trusting the AI to get everything right. I also wanted to build something that goes beyond just sending prompts to a model and hoping for the best.

While working on CompileLoop, I ran into a few interesting problems. Some were things I hadn't really thought about before, and others made me realise that getting an AI model to produce code is only one part of the problem.

Here are three things I learned along the way.

1. The AI can get the fix right and still get the details wrong

One of the first interesting problems I ran into was with unified diffs.

For anyone who hasn't worked with them before, a unified diff is basically a way of describing the changes between two versions of a file. Git uses this format to show changes, and tools can use it to apply patches.

In one of my benchmark cases, the Java compiler complained about a missing semicolon. The model correctly identified what needed to change and produced a diff that looked something like this:

--- a/src/main/java/bench/Adder.java
+++ b/src/main/java/bench/Adder.java
@@ -5,3 +5,3 @@
 public class Adder {
     public int add(int a, int b) {
-        return a + b
+        return a + b;
     }
 }

The actual fix was correct. The problem was that the model got the line counts in the diff header wrong.

A unified diff uses those counts to tell the patch tool how many lines to expect before and after the change. My original implementation rejected the patch if those counts didn't match.

So here I was, looking at a model that had identified the correct fix, but my tool was rejecting it because of a formatting detail.

It happened repeatedly.

Eventually, I realised I needed to handle this differently. Instead of making the model try again every time it got the counts wrong, I changed the patch-handling logic to recalculate those counts from the diff body, similar to what git apply --recount does.

The important part was that I wasn't relaxing the validation of the actual changes. I was correcting a part of the diff that could be calculated deterministically.

My biggest takeaway here is that not every problem needs to be solved by asking the AI to try again.

Sometimes, the better solution is to identify which parts of the output can be corrected safely by normal code.

2. Just because the AI says it fixed something doesn't mean it actually did

This one was interesting because it showed me how important it is to separate what a model says it did from what it actually changed.

I had a Java method declared to return an integer, but its implementation was returning a string:

public int add(int a, int b) {
    return "x";
}

The compiler correctly complained that a String couldn't be converted to an int.

The model returned a summary saying it would fix the method's return type to int. However, the method was already declared to return int.

What the model actually changed was the method body:

public int add(int a, int b) {
    return a + b;
}

In this case, the diff was correct, but the explanation of the fix wasn't.

At first, this might seem like a small issue. The code was fixed, so why worry about the summary?

The problem is that if you build a tool around AI-generated output, you need to know which parts of that output you can actually trust.

A model can produce a correct patch and an incorrect explanation in the same response. If your tool relies on the explanation instead of examining the actual changes, you could end up making the wrong decision.

That's why CompileLoop validates the patch itself and checks the result by running the build. The model's summary is useful context, but it isn't proof that the fix worked.

For me, the lesson is simple: AI can suggest the solution, but the evidence needs to come from the actual code and the tests.

3. You need to know when to stop asking the AI for help

Another problem I encountered was the model repeatedly producing the same incorrect fix.

The first attempt would fail. CompileLoop would send the model feedback about the failure, and the model would return another patch.

Except the second patch was identical to the first.

Sometimes the third attempt was identical too.

Technically, the tool was doing what I had designed it to do. It had a maximum number of attempts, and it would eventually stop.

But when I looked at what was happening, I realised it was wasting time.

Each attempt involved another model call and another Maven build. If the model wasn't making progress, repeating the process wasn't giving me anything useful.

So I added two conditions to stop the repair loop early:

• Repeated patches: If the model produces a patch that has already been attempted, the loop stops instead of applying the same failed solution again.
• Unchanged errors: If two consecutive attempts fail with the same diagnostic information, the loop recognises that it isn't making progress.

For the patch fingerprints, I used Java's String.hashCode() rather than introducing a cryptographic hash. In this case, the fingerprint is only used to compare patches generated by the same process. A collision could cause the tool to stop early, but it wouldn't allow an unsafe patch through.

In my benchmark runs, these stop conditions were triggered 27 times across 60 runs. That included 21 cases where the tool detected a lack of progress and six where the AI provider was unavailable.

That showed me that stopping early wasn't just a theoretical improvement. It was something the tool was actually using.

The lesson I learned is that an AI repair loop needs more than a maximum attempt count. It needs a way to recognise when another attempt is unlikely to help.

Otherwise, you're just spending more time and computing resources asking the same question.

What happened when I tested different models?

I also wanted to see how different local code models performed rather than assuming that a model with a particular parameter count would automatically be better.

I created 20 small Maven projects, each containing one deliberate defect. These covered syntax errors, type mismatches, missing symbols, assertion failures, and bugs involving multiple files.

I tested three models, with three attempts per fixture.

Here are the results from my benchmark:

Model	Successful repairs	Repair rate	Average latency
Qwen2.5-Coder 7B	9/20	45%	29 seconds
Qwen2.5-Coder 1.5B	4/20	20%	12 seconds
DeepSeek-Coder 6.7B	1/20	5%	39 seconds

Qwen2.5-Coder 7B performed best overall in these tests.

What surprised me was the result for DeepSeek-Coder 6.7B. It often identified the correct fix in its summary, but its diffs contained context lines that didn't match the actual source files. CompileLoop rejected those patches because they couldn't be applied safely.

This was a useful reminder that choosing a model isn't just about its size or how good its explanations sound. It also matters whether it can produce output in the exact format your system requires.

I also noticed that Qwen2.5-Coder 7B performed better on the test-failure fixtures than on the compilation-failure fixtures. There were only five test fixtures, so I wouldn't make any big claims from that result, but it's an interesting observation worth investigating with a larger benchmark.

The benchmark isn't perfect, and the sample is small. That's exactly why I wanted the results to be available instead of only sharing the successful examples.

The fixtures, runner, and individual result files are committed to the repository. You can reproduce the benchmark by pulling the required Ollama models and running:

bash benchmarks/run.sh

What I actually enjoyed building

Something I've realised while working on CompileLoop is that the AI integration is only one part of the engineering.

A lot of the work is about making sure everything around the model behaves properly.

For example, I had to build:

• A process runner that executes Maven, handles its output without deadlocking, enforces timeouts, and terminates the process tree when necessary.
• A parser that turns Maven's output into structured compilation errors and test failures.
• A patch validator that rejects dangerous or unsupported changes, including path traversal, symlink escapes, and unexpected file operations.
• A temporary workspace where patches can be applied and tested without modifying the original project.
• A repair engine that records attempts, checks whether fixes work, and stops when it detects that progress has stalled.

I also wrote tests around these components, including tests that use a fake Ollama server and scripted model responses. That means the test suite can run without downloading a local model.

The project currently has 233 tests, with CI running on Ubuntu and Windows.

These are the parts that make the difference between a simple AI coding experiment and a tool that attempts to handle failures in a controlled way.

There are still things I want to improve. Regression detection is one of them, because fixing one failing test shouldn't mean accidentally breaking another. I also want to introduce stronger isolation for build execution and eventually support other languages beyond Java and Maven.

CompileLoop currently uses a temporary copy to protect the original source tree, but that isn't a security sandbox. Build scripts can execute arbitrary commands, so proper container isolation is still an important next step.

I've documented these limitations rather than pretending the project is more complete than it is.

Building with AI is still engineering

I used Claude during development for architecture discussions, reviewing code, and working through problems when I wasn't sure which approach made the most sense.

But using AI to help build a project doesn't remove the responsibility of understanding what you're building.

I still needed to test the implementation, investigate failures, compare different approaches, and decide which trade-offs were appropriate.

For example, I needed to understand why recalculating diff counts was safe, why the patch itself mattered more than the model's summary, and why stopping repeated attempts was better than blindly continuing.

Those decisions matter because I need to be able to explain and defend them, not just get the code to compile.

I'm still learning as a software engineer, and that's part of why I enjoy projects like this. They give me an opportunity to work through problems that go beyond implementing a feature and to understand how the different parts of a system fit together.

I also think this is one of the more interesting areas of AI-assisted development: not just getting models to write code, but building systems around them that can verify their work and handle their mistakes.

That's ultimately what I want CompileLoop to explore.

The model proposes a fix. The tool checks it. The developer reviews the result.

And if the model keeps getting it wrong, the tool should know when to stop.

Try CompileLoop

The source code, benchmark fixtures, and results are available on GitHub:

github.com/LambdaJedi/compileloop

You can build the project with Maven and run it using a local Ollama model:

git clone https://github.com/LambdaJedi/compileloop.git
cd compileloop

mvn -B -DskipTests package

ollama pull qwen2.5-coder:7b

java -jar target/compileloop-0.1.0-SNAPSHOT.jar repair \
  examples/broken-compilation-project \
  --model qwen2.5-coder:7b

If you're working on local LLMs, Java tooling, or AI-assisted software development, I'd be interested to hear what you've been building and what problems you've run into.

I'm still developing CompileLoop, so feedback and ideas for improving the approach are welcome.