Before I let an agent open PRs on strangers' repos, I made it earn it

Over the last six weeks I've had 44 bug fixes merged into other people's open-source projects, most of them MCP servers. Many came from the same loop: mcp-doctor, my static linter for MCP servers, flags something. I reproduce it, write the fix and a regression test, run the project's tests, and open a PR in their template.

That loop is mechanical enough that an agent could run it. It's also the kind of thing you shouldn't hand to an agent on vibes. A wrong PR on a stranger's repo costs more than no PR: it takes a maintainer's time, and it spends my reputation.

So before building the agent, I wrote down the bar it had to clear. This post covers how it cleared it, where it failed, and the bug it caught in my own tool on its first real run.

The bug class

The agent handles exactly one kind of fix. In a Python MCP server, a parameter written name: str = None produces a tool schema that says "type": "string", "default": null. A client that sends that advertised default back explicitly, as some agent frameworks do for every optional field, gets a validation error before the tool runs. The fix is to annotate it str | None.

It's a small bug, but a real one: telegram-mcp had it in 111 of 131 tools.

Gate 1: replay against what maintainers actually merged

The kill rule, written before any code: replay the agent on the code from just before each merged fix and compare its patch with what the maintainer merged. Under ~60% match, or any PR I wouldn't send, and I publish an eval write-up instead of shipping.

The first replay failed in an instructive way. The parameter changes matched the merged fix exactly, but --fix also added 57 TODO: describe this parameter docstring stubs, because it applied every fix type at once. A PR full of TODO stubs is a PR I wouldn't send. So I added --fix-only, which limits a run to one kind of fix, so each PR carries one change.

With that, across 15 merged null-default fixes:

• 14 of 15: the agent changed exactly the parameters the maintainer merged. On one repo the output was byte-identical to the merged file.
• The 15th: every registered tool matched. The differences were 15 functions that aren't registered as MCP tools (the human PR fixed them for consistency) and 1 LangChain tool with the same bug.
• 0 wrong edits.

That passed the kill rule. But I didn't trust it, for one reason.

The replay was partly circular

Those 15 bugs were found by mcp-doctor in the first place. A replay proves the patch equals what maintainers accepted. It doesn't prove the agent finds new bugs, or that it avoids bad ones on code it has never seen. Precision on unseen repos was the real question, and the replay couldn't answer it.

Gate 2: a forward test on repos it had never seen

I ran the full loop on unseen repos and saved drafts locally, posting nothing.

Finding them was the first lesson. In a random sample of about 160 active Python MCP repos, only 3 had this bug (about 2%). It's rarer in the wild than my earlier sweeps suggested. A targeted code search found 16 more, so I had 19.

Reproducing them safely was the second. Running a stranger's server to test a fix is a bad idea: one server's main() starts a login thread against a real fitness API. Instead, I copied each flagged tool's exact signature onto a stub, registered it with the same SDK the repo uses, and called it with null over an in-memory client. That runs none of the repo's code. All 19 reproduced. I also confirmed the bug on every SDK version in use: mcp 1.9, 1.30 and 2.3, and fastmcp 2.12, 3.0 and 4.1.

Then the raw loop failed the kill rule. Two of the 19 would have gone wrong:

1. A false PR. One repo is a security-detection benchmark. 61 of the 64 flagged files were synthetic "benign tool" servers that the benchmark uses as test data. The README says so. The scanner can't know that.
2. A PR that would land red. In another repo, the fix changes the tool schema on purpose, which breaks a schema snapshot test. Their CONTRIBUTING says to regenerate the snapshot and mention it in the PR.

Two more were just too big to send cold: 172 and 132 parameters.

Each failure became a rule in the agent's instructions:

• Fixture filter: read the README and the paths, and drop test data, benchmarks, examples and vendored code.
• Size routing: over about 30 parameters, draft an issue instead, so the maintainer chooses how big a change to take.
• Test step: run the repo's tests before and after. Regenerate a schema snapshot the way CONTRIBUTING says. Any other new failure drops the draft.

With those three rules, all 19 were handled correctly. Two of the rules need judgment, which a static linter can't supply, so they live in a Claude Code skill (fix-upstream) rather than in the CLI.

The rule that doesn't bend: a human sends every PR

The skill never pushes, opens a PR or comments. It stops at a local branch and a PR description, and I review it and decide. That isn't a temporary safety net. It's the design: the agent does the slow parts (reproduce, patch, test, write up), and the judgment call stays with the person whose name is on the PR.

The first real run caught a bug in my own tool

I shipped the skill and ran it on 5 real repos. 4 drafts were clean, and I sent them after reviewing each. They're waiting on maintainer review.

The 5th, a trading framework, the agent refused to send. It found that on the official MCP Python SDK, the fix itself breaks working calls. The SDK JSON-decodes a string argument before validating it whenever the annotation isn't exactly str. So after str → str | None:

• a parameter that takes a JSON string, like '{"BTC-USDT": "..."}', gets decoded into a dict and rejected;
• the literal text "null" silently becomes None.

The standalone fastmcp package doesn't do this, which is why the replay never saw it.

The agent also noticed the scanner had missed a whole file of tools: significance_test.py registers 8 tools but was skipped as a test file because of its name.

Both became real bugs in mcp-doctor (#108 and #109), and both were fixed and released the same afternoon. Before releasing, I checked every merged fix I'd already sent. None had changed a parameter that takes JSON text. On the 19-repo forward test, the new rule skips exactly 7 parameters, and all 7 really do take JSON.

The part of that run I'm proudest of is the PR it didn't send.

What I'd take from this to any agent you're putting near real users

• Write the kill rule before you build. Mine was "≥60% match with merged fixes, zero PRs I wouldn't send." Having it written down is what made the replay's TODO stubs a failure rather than a footnote.
• Know when your evidence is circular. A replay against your own tool's past finds is a regression test, not an eval. Precision comes from data the system has never seen.
• Turn each failure into a rule you can name. Fixtures, snapshots and size each started as one bad draft. "Be careful" doesn't scale. "Read the README and drop benchmark data" does.
• Design for the agent stopping. The most valuable output of the first real run was a refusal with a reason.

The skill is in the mcp-doctor plugin (/plugin install mcp-doctor --marketplace vishalhabib99/mcp-doctor). The replay and forward-test numbers are in the build log.

---

More of my open-source work: github.com/vishalhabib99