DEV Community: Ilmee De Silva

My AI agent now opens its own pull requests every Monday

Ilmee De Silva — Tue, 03 Nov 2026 02:09:56 +0000

A few days ago I wrote about patchpilot, an agent that upgrades a Python dependency and fixes the code the new version breaks (part 1 is here). It worked, but it had one honest flaw: you had to remember to run it. And "I'll do the upgrade later" is exactly how dependencies end up two major versions behind.

So I made patchpilot a GitHub Action. You add one workflow file, and every Monday it picks an outdated package, upgrades it in a sandbox, fixes the code until the tests pass, and opens a pull request.

That's the log from its first real run, replayed.

Here's what happened in that run, with nobody at a keyboard:

skipped pydantic, which already had an open patchpilot PR
picked marshmallow 3 → 4
found that version 4 removed the missing= and default= field options, and the test module crashed on import
renamed them to load_default= and dump_default=, and all 9 tests passed
opened this pull request as github-actions[bot]

It also flagged a README line that still said "marshmallow 3", and deliberately left it alone to keep the PR focused. That's the judgement I'd want from a human reviewer.

Here's the whole setup:

name: patchpilot

on:
  schedule:
    - cron: "0 6 * * 1"  # every Monday at 06:00 UTC
  workflow_dispatch:      # adds a "Run workflow" button

permissions:
  contents: write
  pull-requests: write

jobs:
  upgrade:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: ilmeedesilva/patchpilot@v0.2.0
        with:
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

It's on the GitHub Marketplace. Getting there took more decisions than I expected. These are the ones worth sharing.

1. Don't use AI for the boring decision

The first question is which package to upgrade. I could have asked the model. I didn't, because this decision has a clear right answer, and code gives it for free, instantly, and the same way every time.

patchpilot's rule: a package is a candidate when its newest release on PyPI falls outside the range you declared. pydantic>=1.10,<2 while 2.x is out? Candidate. An unpinned requests? Never, because pip already installs the newest version. Among the candidates it picks the smallest jump first (patch, then minor, then major), because the safest upgrade is the one most likely to get merged.

One rule makes the weekly schedule work: skip any package that already has a patchpilot/upgrade-* branch. Without it, every Monday would redo the same upgrade. There was a catch, though. On a GitHub runner, actions/checkout fetches only the default branch, so a local branch check finds nothing. patchpilot asks the remote directly:

git ls-remote --heads origin 'patchpilot/upgrade-*'

My rule of thumb now: use the model for the parts that need judgement (reading a changelog, fixing code). Use code for everything else.

2. Quiet weeks should cost nothing

A scheduled job that spends money every week to find out there's nothing to do is a job people switch off. So when every outdated package already has an open PR, patchpilot prints "Nothing to upgrade" and exits successfully, without a single API call. The run takes about a second.

Exit code 0 matters here too. If "nothing to do" were an error, your Actions tab would turn red every quiet week, and people learn to ignore red.

3. A throwaway machine is not a sandbox

GitHub gives every job a fresh virtual machine that's deleted afterwards. So it was tempting to drop patchpilot's Docker sandbox in CI and run everything directly on the runner.

But while the job runs, that machine holds your Anthropic API key and a GitHub token that can push to your repo. Meanwhile it's installing a new version of a package and running a test suite: code you didn't write. A compromised release could read those secrets from the environment in one line.

So the sandbox stays. The project's code runs in a container that sees only a temporary copy of the repo, and never the environment the secrets live in. "Throwaway" protects the next job. It doesn't protect this one.

4. Small details that would have bitten users

Never paste inputs into shell scripts. A composite action that writes ${{ inputs.package }} straight into run: lets a crafted input run shell commands. Every input goes through an environment variable instead.
Pin the tool to the action. The action installs patchpilot from its own source (pipx install "$GITHUB_ACTION_PATH"), not "whatever's latest on PyPI". So @v0.2.0 always means patchpilot 0.2.0.
Logged in isn't the same as gh auth status. Locally, patchpilot checks gh auth status before doing any work. In Actions the token comes from an environment variable, and that check can fail even though pushing and opening PRs work fine. Now a token in GH_TOKEN counts as logged in.
Test the action in CI without a key. CI runs the real action on a tiny project where nothing is outdated. That exercises install, picking and the exit code on GitHub's machines, without ever calling Claude.

5. The demo taught me more than the code did

To test it I needed a second outdated package in the demo repo, one that really breaks. My first pick was semver, since version 3 was supposed to remove the old module functions. Before building anything, I installed both versions and checked. Version 3 had kept them, only deprecated. That demo would have been an upgrade where nothing broke, and nothing for patchpilot to fix. I switched to marshmallow 4, which really did remove missing=.

The other trap: adding marshmallow to requirements.txt would have given my live pydantic PR a merge conflict, since that PR edits the same lines. It went into a separate requirements-api.txt instead. Small, but a "live example" with a red conflict banner isn't much of an example.

Even publishing had a lesson. The Marketplace rejected the name "patchpilot" because a GitHub account already has it. The failed release still created the git tag, so it's worth checking for stray tags before you try again.

What's next

The next goal is the one that matters most: real users. I'll run patchpilot on a few open-source projects with dependencies stuck behind an old major version, review every diff by hand, and ask the maintainers before opening anything. A merged PR in someone else's project is a much better test than any demo.

If you'd like to try it, the README has the copy-paste setup. It needs an Anthropic API key, and you can still run it locally with pipx install patchpilot. Feedback is very welcome, especially an upgrade you'd like to see it attempt.

I'm on LinkedIn if you'd like to connect.

I built an AI agent that fixes the code a dependency upgrade breaks

Ilmee De Silva — Thu, 29 Oct 2026 08:58:40 +0000

Every Python developer knows this moment. Dependabot opens a PR, "Bump pydantic from 1.10 to 2.14", CI goes red, and the PR sits there for three months because fixing it is real work.

Bumping the version number was never the hard part. Fixing what breaks is. So I built patchpilot, an agent that does that part:

patchpilot upgrade . pydantic --pr

It upgrades the package inside a sandbox, runs your tests, reads the failures and the changelog, edits your code until the tests pass, and opens a pull request explaining what broke and how it was fixed.

That's a real run, sped up 2×. Here's the pull request it opened: the description, the diff and the verification note were all written by patchpilot.

You can try it yourself with pipx install patchpilot.

Here's what I learned building it.

An agent is just a loop

Before this project, "AI agent" sounded like magic to me. It turned out to be a for loop:

Send the model the task and a list of tools it may use (read_file, run_tests, pip_install, …).
The model replies: "call run_tests".
My code runs the tests and sends back the output.
Repeat until the model replies with text instead of a tool call.

The model decides what to do next. My code is the only thing that does anything. That split made everything else easier to reason about, especially safety. I wrote the loop by hand instead of using a framework, so I'd understand every step. The whole file is under 100 lines.

The first real run

I wrote a small app against pydantic v1 that breaks three ways on v2, and pointed patchpilot at it. Without being told what was wrong, the agent:

installed pydantic 2, ran the tests, and saw that they couldn't even load
worked out that BaseSettings had moved to a separate package, installed pydantic-settings, and added it to requirements.txt
rewrote @root_validator as @model_validator(mode="after") and replaced the deprecated .dict() and .parse_obj()
got all the tests passing

The part that impressed me most was in its report, under "Needs your attention". It pointed out a subtle behaviour change no test would catch: in v2, a form with a bad email and mismatched passwords now reports only the email error. That's exactly what a careful human reviewer would mention.

Lesson 1: verify, don't trust

My favourite bug came from the second real run. I ran the upgrade from the repo root instead of the demo folder. The agent upgraded pydantic, ran "the tests"… which were patchpilot's own tests. They never import pydantic, so they passed before and after. patchpilot proudly printed ✅ tests pass.

To its credit, the agent noticed and wrote in its report that the right tests were never run. But the green tick was still a lie. Two fixes came out of it:

patchpilot now re-runs the tests itself after the agent says it's done. The ✅ is patchpilot's verdict, not the model's.
patchpilot refuses to start if the package isn't a dependency of the folder you pointed it at, and tells you which folder declares it.

The general lesson: when an agent's output gets acted on automatically, check its claims with code, not with more prompting.

Lesson 2: the agent runs code you don't own

Upgrading a package means installing it (and package installs can run arbitrary code) and running tests. If you point patchpilot at a random open-source project, that's a stranger's code on your laptop.

So everything that executes runs in a Docker container that:

mounts only a temporary git worktree, not your home folder or SSH keys
runs as a normal user with all Linux capabilities dropped
is capped at 2 CPUs, 2 GB of RAM and 512 processes
is deleted at the end of the run

The agent loop and the API key stay outside the container. I also wrote integration tests that check these claims. For example, one asserts that the container can't see my home directory. Writing "it's sandboxed" in a README is easy; proving it is better.

Lesson 3: fail before spending, not after

An agent run costs money and minutes. Few things are more frustrating than watching it do five minutes of work and then fail at the last step because you weren't logged in to GitHub.

So patchpilot checks everything it can before the first API call: Docker is running, gh is logged in, the repo has a remote, the base branch is pushed, and the branch name is free. The same idea showed up when my API credit ran out mid-test: the SDK error was a 40-line traceback. Now it's one line saying what happened and where to add credit.

Lesson 4: a small tool surface beats a shell

The easy design is to give the model a bash tool and let it do anything. I deliberately didn't. patchpilot's agent gets narrow tools: pip_install takes exactly one requirement string and rejects flags, URLs and file paths, and edit_file refuses to touch .git/. Fewer tools mean fewer surprises, and they're much easier to sandbox and test.

What's next

a GitHub Action that runs weekly and opens PRs automatically
upgrading several packages in one run, ordered by risk
cutting the sandbox's network access while the tests run
support for uv and Poetry lockfiles

patchpilot is open source and on PyPI (pipx install patchpilot). The code is on GitHub at ilmeedesilva/patchpilot, and here's the PR it opened in the demo above. Issues and ideas are welcome, especially breaking upgrades you'd like to see it try.

I'm on LinkedIn if you'd like to connect.