I built an AI agent that fixes the code a dependency upgrade breaks

Every Python developer knows this moment. Dependabot opens a PR, "Bump pydantic from 1.10 to 2.14", CI goes red, and the PR sits there for three months because fixing it is real work.

Bumping the version number was never the hard part. Fixing what breaks is. So I built patchpilot, an agent that does that part:

patchpilot upgrade . pydantic --pr

It upgrades the package inside a sandbox, runs your tests, reads the failures and the changelog, edits your code until the tests pass, and opens a pull request explaining what broke and how it was fixed.

That's a real run, sped up 2×. Here's the pull request it opened: the description, the diff and the verification note were all written by patchpilot.

You can try it yourself with pipx install patchpilot.

Here's what I learned building it.

An agent is just a loop

Before this project, "AI agent" sounded like magic to me. It turned out to be a for loop:

1. Send the model the task and a list of tools it may use (read_file, run_tests, pip_install, …).
2. The model replies: "call run_tests".
3. My code runs the tests and sends back the output.
4. Repeat until the model replies with text instead of a tool call.

The model decides what to do next. My code is the only thing that does anything. That split made everything else easier to reason about, especially safety. I wrote the loop by hand instead of using a framework, so I'd understand every step. The whole file is under 100 lines.

The first real run

I wrote a small app against pydantic v1 that breaks three ways on v2, and pointed patchpilot at it. Without being told what was wrong, the agent:

• installed pydantic 2, ran the tests, and saw that they couldn't even load
• worked out that BaseSettings had moved to a separate package, installed pydantic-settings, and added it to requirements.txt
• rewrote @root_validator as @model_validator(mode="after") and replaced the deprecated .dict() and .parse_obj()
• got all the tests passing

The part that impressed me most was in its report, under "Needs your attention". It pointed out a subtle behaviour change no test would catch: in v2, a form with a bad email and mismatched passwords now reports only the email error. That's exactly what a careful human reviewer would mention.

Lesson 1: verify, don't trust

My favourite bug came from the second real run. I ran the upgrade from the repo root instead of the demo folder. The agent upgraded pydantic, ran "the tests"… which were patchpilot's own tests. They never import pydantic, so they passed before and after. patchpilot proudly printed ✅ tests pass.

To its credit, the agent noticed and wrote in its report that the right tests were never run. But the green tick was still a lie. Two fixes came out of it:

• patchpilot now re-runs the tests itself after the agent says it's done. The ✅ is patchpilot's verdict, not the model's.
• patchpilot refuses to start if the package isn't a dependency of the folder you pointed it at, and tells you which folder declares it.

The general lesson: when an agent's output gets acted on automatically, check its claims with code, not with more prompting.

Lesson 2: the agent runs code you don't own

Upgrading a package means installing it (and package installs can run arbitrary code) and running tests. If you point patchpilot at a random open-source project, that's a stranger's code on your laptop.

So everything that executes runs in a Docker container that:

• mounts only a temporary git worktree, not your home folder or SSH keys
• runs as a normal user with all Linux capabilities dropped
• is capped at 2 CPUs, 2 GB of RAM and 512 processes
• is deleted at the end of the run

The agent loop and the API key stay outside the container. I also wrote integration tests that check these claims. For example, one asserts that the container can't see my home directory. Writing "it's sandboxed" in a README is easy; proving it is better.

Lesson 3: fail before spending, not after

An agent run costs money and minutes. Few things are more frustrating than watching it do five minutes of work and then fail at the last step because you weren't logged in to GitHub.

So patchpilot checks everything it can before the first API call: Docker is running, gh is logged in, the repo has a remote, the base branch is pushed, and the branch name is free. The same idea showed up when my API credit ran out mid-test: the SDK error was a 40-line traceback. Now it's one line saying what happened and where to add credit.

Lesson 4: a small tool surface beats a shell

The easy design is to give the model a bash tool and let it do anything. I deliberately didn't. patchpilot's agent gets narrow tools: pip_install takes exactly one requirement string and rejects flags, URLs and file paths, and edit_file refuses to touch .git/. Fewer tools mean fewer surprises, and they're much easier to sandbox and test.

What's next

• a GitHub Action that runs weekly and opens PRs automatically
• upgrading several packages in one run, ordered by risk
• cutting the sandbox's network access while the tests run
• support for uv and Poetry lockfiles

patchpilot is open source and on PyPI (pipx install patchpilot). The code is on GitHub at ilmeedesilva/patchpilot, and here's the PR it opened in the demo above. Issues and ideas are welcome, especially breaking upgrades you'd like to see it try.

I'm on LinkedIn if you'd like to connect.