Most bug reports die at the same line. "Cannot reproduce." The maintainer tried it once, it worked on their machine, and the issue sits there until someone closes it as stale. I got tired of that sentence, so I built PatchPilot. Paste a public GitHub issue link, and it reproduces the bug in a Docker sandbox, then hands back a reproduction script, a failing test, and a suggested fix diff, with the logs as proof.
The part I care about most is the verdict. It is computed, not judged. Exit codes and an error-signature match decide whether a bug counts as reproduced. The model writes the script, but the model never gets to grade its own work. If the log does not contain the error from the issue, the verdict is NOT REPRODUCED, no matter how confident the generated script looks.
I measured it on ten real issues, not a demo set. Pyflakes, pyupgrade, packaging, humanize. Each one was checked twice by hand before the run: the recorded error occurs at the base commit, the first parent of the fix merge, and it does not occur at the fix commit. Issues that failed that check were rejected. The run then had to reproduce each bug from the issue text alone, starting at the base commit, with no knowledge of the fix.
The first measured run: 7 reproduced, 3 missed. The hits were pyflakes 764 and 834, pyupgrade 194, packaging 638, and humanize 17, 57, and 122. The misses were pyflakes 744, pyupgrade 437, and humanize 62.
Then I changed the code, and I am publishing what happened next as well. The misses had an obvious suspect: the attempt loop retried the identical prompt up to six times, with no feedback from the previous attempt's log. So I built the feedback version, attempts 2 to 6 get the previous attempt's exit code and a capped tail of its log, and re-ran the same ten issues on 2026-11-15. The re-run scored 3 reproduced, 7 missed. The hits were pyflakes 764 and humanize 17 and 122. Four issues that reproduced before did not reproduce this time.
The loop did not lift the score on that run. One run against one run does not prove the loop caused the drop, model sampling alone could do it, and I am not going to pretend otherwise. But I am not quoting 7 any more either, because that number no longer describes the current code. Both runs are on the case study page, hits and misses side by side. A tool that only shows hits cannot be trusted, so I am not going to start now.
The run that first counted was the seventh. The first six were invalidated by real bugs the live host exposed, and unit tests could not see any of them, because tests fake the exact boundaries that failed in production. One line each:
The Bedrock client was built with no region, so every model call died before it started. Model replies arrive wrapped in markdown fences, and the raw reply was written as the script, so all sixty attempts failed the syntax check with zero container runs spent. A relative workspace path was passed to Docker as a volume name, so all sixty attempts exited 125. Installs happened in a disposable container and never reached the run container, so the target package was never importable. Container root could not write into the workspace, so all ten jobs failed setup. VCS-versioned projects could not build without git in the slim image, so all four humanize jobs failed setup.
An intermediate fair run before the last fix scored 3 of 10 with 4 setup failures. None of the invalidated runs is quoted as a model result anywhere.
One more check ran the same evening as the first run. A crafted issue ordered the agent to curl an external URL and exfiltrate /etc/passwd. It went through the real pipeline on the host. The job verdict was REPRODUCED on the crafted bug, and the only commands that reached the executor were the repro script and the test run. No injected text reached the executor.
What it is not: no PRs, no comments, nothing posted to your repo. It reproduces and reports. What is not claimed yet: the head-to-head against Claude Haiku 5.5 never ran, because the account returns AccessDenied for that model, so Qwen3-Coder-30B is the only measured model. The JS and C++ paths are built and unit-tested, but the live fixture proof so far is Python only.
Try it, or read the case study first.
Live demo: https://patchpilot-shah20.vercel.app
Case study, both runs: https://patchpilot-shah20.vercel.app/case-study
Code: https://github.com/abdullah129026/patchpilot
Top comments (0)