DEV Community: CH

How I caught a crash by actually running someone's fix

CH — Wed, 28 Oct 2026 21:56:39 +0000

Most code review on open source PRs is reading. Someone looks at the diff, it makes sense, they leave a thumbs up. I've started doing something a bit different: I actually install the PR and run it. On Windows, which turns out to be the part that matters.

Today I tried it on three open PRs in simonw/llm, the command-line tool for talking to language models. Here's what happened.

PR 1: a clean exit that crashed on start

This PR made llm chat exit cleanly when you hit end-of-input, instead of printing "Aborted!". The diff was small and looked fine.

But it called click.termui._readline_prompt, a private function inside the click library. That function only exists in click 8.4 and newer, and llm doesn't pin a click version. So I set up a throwaway venv and ran it three ways:

main: "Aborted!" (the original bug)
the PR with click 8.5: clean exit, works
the PR with click 8.3.1: AttributeError the moment chat starts

So the fix works on the newest click and breaks chat completely on anything older. Reading the diff, you'd never see that. I posted the results and suggested either pinning click>=8.4 or catching EOFError around a plain input().

PR 2: a Windows fix that still fails on Windows

This one was about llm install / llm uninstall, running pip in a separate process so llm can update itself. On Windows I ran the self-reinstall case on main and on the PR.

main: pip refuses to modify itself
the PR: WinError 5 (access denied) on a .pyd file

The running llm process has that compiled extension loaded, and Windows locks loaded files, so the child pip can't replace it. Different error, same result: it still can't reinstall itself while running. Again, nothing in the diff tells you that. You have to run it on Windows.

PR 3: a cleanup that was just fine

The third PR removed some old Windows-only readline setup lines. The author said they couldn't reproduce the original problem, so nobody was sure it was safe. I ran it on main and on the PR, checked that chat opens and arrow keys still work, and both behaved the same. So I left a short "tested on Windows, looks good as a cleanup". Not every test finds a bug, and a confirmed "it's fine" is useful too.

What I took from it

Running a PR takes maybe 15 minutes with a throwaway venv. That's often more than the author could do, because they might not have your OS or your versions.
Private functions and unpinned dependencies are where "works on my machine" hides. Try the oldest version the project allows, not just the newest.
Windows is underrepresented among testers. If you're on Windows, you're already useful.
Post exactly what you ran and what happened. Let the maintainer decide what to do with it.

Disclosure: I used AI (Claude) to help pick PRs, write the little test scripts, and draft this post. I ran every test myself on my own PC, and the results above are what I saw.

What I learned fixing bugs in other people's code

CH — Wed, 28 Oct 2026 19:59:59 +0000

I'm fairly new to contributing to other people's projects. Lately a handful of my small fixes have been merged into Python projects I'd never touched before, and a lot more are still waiting on review. Here's what I've learned so far.

1. Prove the bug before you touch the fix

The fixes that went smoothly all started the same way: a small test that fails on the current code and passes with my change. Not a description of the bug, not a screenshot, a test.

It does two jobs. It shows the maintainer the problem in a way they can run in ten seconds, and it stops the bug from quietly coming back later. If I couldn't write that test, I usually didn't understand the bug yet.

2. "Fixed" in a commit message is a claim, not a fact

A lot of the bugs I found were in code where an earlier commit said it had already fixed that exact thing. The fix was partly there: it handled the case in the commit message, but not a slightly odd input next to it.

Now when I read "fixed X" in a history, I treat it as something to check, not something to trust. That includes my own commits, and the ones my AI tools write for me.

3. Small and focused wins

On one project I sent seven separate small PRs instead of one big one. The maintainer merged all seven and said small, focused PRs were nice to review. That stuck with me. A reviewer is a volunteer doing me a favour, and a ten-line change with one test is something they can say yes to on a lunch break.

4. Run the project's checks exactly the way CI does

Running the linter and tests "my way" isn't the same as running them the project's way. Now I open the CI workflow file first and run the same commands, with the same versions, before I push. And when CI does go red, I check whether it's red on the main branch too before assuming it's my fault. Cheap habit, saves an awkward "sorry, one more commit".

5. Read the contributing rules, even on tiny repos

Some projects want you to comment "I'd like to take this" on an issue before you start. Some want a sign-off line on every commit. Some say outright that they'd rather get an issue than a PR. Following the house rules costs two minutes and tells the maintainer you respect their time.

6. Say only what you checked

In PR descriptions I try to write exactly what I ran and what I didn't. "These 5 tests also fail on main for network reasons" is better than letting the reviewer find that out. Being plain about the limits seems to build more trust than sounding confident.

7. The best outcome isn't always the merge

On one project the owner merged my test-only PR and then opened a follow-up issue to close the rest of the gaps it pointed at. That felt better than the merge itself. A good bug report can be more useful than the fix.

I'm still new at this, and most of my PRs are still waiting on review. If you maintain a project and have advice for first-time contributors, I'd genuinely like to hear it.