DEV Community: Daniel

Claude Code made 6 mistakes upgrading a legacy Spring Boot service. None reached a commit.

Daniel — Tue, 06 Oct 2026 16:12:12 +0000

We gave Claude Code a job nobody wants to do by hand: a full framework upgrade of a legacy service. Java 11 and Spring Boot 2.7 at the start, with Springfox for the API docs, the old WebSecurityConfigurerAdapter that Spring Security 6 removed, and a TODO from 2019. Java 25 and Spring Boot 4.1 at the end.

It took about two hours of Claude time and roughly $24 at API prices. It ended green, 47 tests passing.

The result isn't the interesting part. On the way, Claude made six mistakes, and not one of them made it into a commit. Here they are, straight from the session transcripts, with whatever caught each one.

The service is a demo, built on purpose to look like the legacy code a lot of us keep alive (a parcel tracking REST API called ShipTrack). The sessions are real.

1. "56 tests pass." The build ran 47.

After the move to Java 21, Claude modernized the code and wrote a tidy summary. Everything in it was accurate except one number: it said 56 tests passed. The build had run 47.

Small, but it's the cleanest example of the whole problem. A model's summary describes the work. It doesn't prove it. The evidence is in the build output.

That's why the definition of done in our CLAUDE.md doesn't say "tests pass". It names a command:

## Definition of done
- `./mvnw verify` passes.
- No new deprecation warnings in the compiler output.
- UPGRADE_NOTES.md says what a reviewer should double-check.

Caught by: reading the build's own output, not the summary.

2. A good plan with two wrong claims in it

For the jump to Spring Boot 3.5, Claude didn't touch code at first. It went into plan mode, explored with subagents, and wrote a plan: every file to change, the order of operations, even the limits our hooks put on the work. A good plan.

It also contained two false statements. It said Spring Boot 3 no longer had a path-matching property (it still does). And it said server.max-http-header-size kept its name in 3.5. It didn't: since 3.0 it's server.max-http-request-header-size.

Checking took one command. Spring Boot jars ship their own configuration metadata, including every deprecated key and its replacement:

unzip -p path/to/spring-boot-autoconfigure-3.5.x.jar META-INF/spring-configuration-metadata.json \
  | jq '.properties[] | select(.name | test("header-size"))'

A long plan with two wrong claims is normal. That's exactly why you review a plan like a pull request, before any code changes. A wrong assumption costs one sentence to fix in a plan. After the edits, it costs an afternoon.

Caught by: plan review plus the configuration metadata.

3. The decision our own guardrail forced it to make

This one is my favorite, because it was partly our fault.

After the jump to 3.5, two tests failed. Since Spring Framework 6.1, unknown URLs go through the static resource handler, and the 404 body now says "No static resource". A real change to the public API.

Claude did the right thing: it laid out two options and asked us to choose. Then it tried to stop and wait for an answer.

Our Stop hook said no. Its job is to keep Claude working until the build is green, and the build was red. So Claude picked option B itself, got the build green, and flagged the choice for review.

The reasoning was sensible. But a product decision that belonged to a human was made by the model, and our guardrail pushed it there. The fix went into the hook: when the only failing tests are characterization tests, a red build means behavior changed, and that's a question for a person. Now the hook lets Claude stop and ask.

For reference, the wiring is just two entries in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      { "matcher": "Edit|Write",
        "hooks": [{ "type": "command", "command": ".claude/hooks/protect-characterization-tests.sh" }] }
    ],
    "Stop": [
      { "hooks": [{ "type": "command", "command": ".claude/hooks/require-green-build.sh" }] }
    ]
  }
}

The first one blocks edits to the tests that freeze today's behavior (exit code 2 blocks the tool call). The second one refuses to end the turn on a red build. If you copy the idea, give the Stop hook a way out for "this needs a human", or it will make decisions for you.

Caught by: reading the transcript. The reviewer subagent flagged it too.

4. Green build, ship verdict, and the API had changed anyway

Now the build was green, all 47 tests passed, and the reviewer subagent said "ship it".

Before committing we did one boring thing: ran the old version and the new one side by side, sent the same unknown URL to both, and compared the real responses.

Same status, same message. But the new body had an extra details field, and the timestamp was formatted differently. Two changes to the public API, with every test green.

Why? The test for unknown URLs checked four fields. It never checked that there was nothing else. A test only protects what it asserts.

The fix started with the contract, not the code. A human made that test stricter (exact number of fields, exact timestamp format) and ran it against the old version first, to prove it described what clients really got. Then Claude got the evidence and one instruction: go back to Spring Boot's default error body and leave the tests alone. It removed its own handler and overrode just the message, inside Spring Boot's error attributes. Five fields, old timestamp format. Then we committed.

Caught by: a human comparing real responses.

5. A confident diagnosis. Also wrong.

On to Spring Boot 4.1. Two characterization tests red again. In one, the error body for malformed JSON had lost its message field.

Claude investigated and gave a clear diagnosis: Spring Boot 4 no longer puts the message in the response, so we'd need extra code to bring it back.

Plausible. Wrong.

Spring Boot silently ignores properties it doesn't recognize. One more look at the configuration metadata showed the key had been renamed in 4.0: server.error.include-message became spring.web.error.include-message. The old key was ignored, so the message disappeared. No code needed.

The plan had also skipped spring-boot-properties-migrator, which our upgrade instructions explicitly ask for. It exists to report exactly this kind of rename.

Caught by: the configuration metadata, again.

6. The build was green. The compile had failed.

Last step, Java 25. Should be a one-line change. Claude switched the version, ran the build: green. Then it checked for warnings with something like mvn compile | grep -i warning. Nothing. Looks done.

Except that first build wasn't clean, so classes compiled earlier were still sitting in target/. And the compile had actually failed. The error lines just didn't contain the word "warning", and the pipe swallowed the exit code.

Before finishing, Claude ran one more full clean build, and the truth came out: cannot find symbol. The Lombok getters were missing. Since JDK 23, javac no longer runs annotation processors found on the class path unless you ask it to, so Lombok has to be declared in the compiler plugin.

Claude recognized it and said its earlier check was flawed. Output you filtered is not a result. True for humans, true for models. Our Stop hook now runs a clean build whenever the pom.xml changes.

Caught by: a clean build.

What I take from it

Look at the list of catches: the build output, a plan review, the transcript and the reviewer, a human comparing responses, the configuration metadata, a clean build. No single guardrail caught everything. Each one caught something the others missed.

And Claude did almost all of the work, and did it well. It found a real production bug before the upgrade even started, refused to work around a guardrail, and checked APIs in the actual jars. The human part was small. But it was the part that mattered: building the guardrails and making the decisions.

Every one of those guardrails is something you can add to your own repo this week.

The full walkthrough, with the transcripts on screen:

Personal project, views my own. The sessions, numbers and transcripts are real; this write-up was drafted with AI from the video and its transcripts.

How I review an AI agent's diff (the diff comes last)

Daniel — Thu, 01 Oct 2026 19:08:41 +0000

When an AI agent hands you a diff, reading it line by line is the last step, not the first.

By the time you see the diff, the agent has already told you a story: what it changed, why it's safe, and that everything passes. A senior review checks that story against evidence. Here's the order I use, walked through on one real step: a legacy demo service moving from Spring Boot 2.7 to 3.5, done by Claude Code and replayed from its transcripts.

Here's how that step ended, in the agent's own words:

The independent review confirms no hidden regressions. Green build. All 47 tests passing.

Those are claims. The question for each one is the same: can I check it against output the model didn't write?

Check 1: the claims

Scroll back to the final build. It ran in quiet mode, so a success prints almost nothing. Claude tried to read the exit code, and that command needed an approval nobody was there to give. So it concluded success from the absence of an error.

It was right this time. The build really was green. But silence is weak evidence, and the same reasoning in another run could hide a failure.

The fix is cheap: ask for the line the build itself prints, with the test count, pasted into the report. Something like:

Tests run: 47, Failures: 0, Errors: 0, Skipped: 0

If the agent can't show you that line, you don't have a green build. You have a sentence about one.

Check 2: the decisions

Not the changes, the choices.

There was one, flagged honestly at the end of the summary. A hook fired before a human could answer, so Claude defaulted to "option B": a choice about what the public API returns for unknown URLs.

Maybe it's the right choice. That's not the point. A decision about your public API was made by the model, so it needs an explicit yes or no from a person before the step is done. Make a list of every choice like that in the session, and answer each one.

Check 3: a second opinion, and what it based its verdict on

We use a reviewer subagent: one markdown file in .claude/agents/, its own context, and no edit or write tools. It sees the diff, not the reasoning behind it. Its checklist is ordered by damage: public API first, then security, the frozen tests, configuration, dependencies. Every finding gets a severity, and the report ends with one verdict: ship, or fix first.

The report earned its keep. It flagged the model's decision as an unreviewed change to the error body. It spotted deprecated annotations in two test classes that a hook had kept Claude from touching. And a security rule that was probably dead.

Then the verdict: ship.

Now read how it knows. The build: green, because no output means success. The public API: the error tests pass unmodified, so no drift. The reviewer trusted the same tests the author trusted. A second opinion built on the same evidence isn't independent. It's a filter. Useful, but read its reasons, not just its verdict.

Check 4: compare real behavior, old against new

This is the check that found the problem, and it's almost embarrassingly simple. Start the old version and the new one. Send both the same request. Compare.

# old version on :8080, new one on :8081 — adapt the paths and users to your API
for path in /api/parcels/does-not-exist /api/parcels/42 /api/stats; do
  curl -s -u clerk:clerk "http://localhost:8080$path" | jq -S . > old.json
  curl -s -u clerk:clerk "http://localhost:8081$path" | jq -S . > new.json
  diff -u old.json new.json || echo "^^^ $path changed"
done

An unknown URL on 2.7, then on the new build: a details field that was never there, and a timestamp in a different format. Forty-seven green tests, a ship verdict, and the public API had changed.

Why did the tests miss it? The test for that URL looked at four fields and ignored the rest of the body. A new field couldn't turn it red. Green meant those four fields were fine, not that the body was.

Build your own list of requests your clients depend on. Put at least one request that should fail in it, and one that tests who's allowed. Every difference gets a human decision: bring the old behavior back, or keep the new one deliberately, with a note that says why.

When a check like this finds a gap, fix the contract before the code. A person tightened the test (exactly five fields, the old timestamp format) and ran it against the old version first. It passed there, so it described what clients really got. Then Claude got both real responses as evidence, a note of what the human had already changed, and one instruction with a clear finish line: restore the default error body exactly as on 2.7, and leave the tests alone.

It took its own handler out and changed just the message through Spring Boot's error attributes. Same request again: five fields, old timestamp style, matches 2.7.

One more habit from that exchange. Claude's last line separated its edits from the human's. In a diff with two authors, require that every time. You need to know whose work you're approving.

Check 5: what the agent admits it didn't check

Claude's plan included calling the running app by hand. Launching it in the background needed an approval, and in a headless run there was no one to give it. So it dropped that check and wrote that the automated tests already covered it.

Mostly. But not one test in the suite ever requested the OpenAPI description or the Swagger page. So those went on the human's list:

no login, trailing slash added: still 401;
logged in as a clerk: the parcel is still found;
both documentation endpoints respond.

Everything the agent couldn't verify is your checklist. Look for the words "should", "already covered" and "assumed" in its report.

Check 6: the diff itself

Only now. Read it for scope, and for the next change, not just this one.

In a later step (Java 25), Claude declared Lombok for the compiler plugin with a hard-coded version, and left a note: a human should bump it by hand if Lombok is ever upgraded. The review changed one line, to use the version property the Spring Boot parent already manages. Now the next framework upgrade moves the annotation processor and the library together.

A note that asks for manual work later is a small bug with no date on it yet.

The order, in one place

Check the claims against output the model didn't write.
List its decisions and answer each one.
Get a second opinion, and read what it based its verdict on.
Compare real behavior, old against new.
Turn everything the agent couldn't verify into your own checklist.
Then read the diff, for this change and the one after it.

None of this is specific to Claude Code. Copilot, Cursor and Codex all document reviewer agents of their own, and the checks don't care which agent wrote the diff.

The video version, with the transcripts and both responses on screen:

Personal project, views my own. The sessions and transcripts are real; this write-up was drafted with AI from the video and its transcripts.