DEV Community

Cover image for Would you merge a PR a bot opened? The flaky-or-real check we run first
Versyka
Versyka

Posted on Edited on AI-assisted

Would you merge a PR a bot opened? The flaky-or-real check we run first

"I'd never merge a pull request a bot opened." It's the first objection we hear about automated upgrades, and it's the right instinct. The answer isn't trust, it's evidence: you need to know whether a red test was caused by the change at all. Here's the routine, by hand first.

A test goes red on a pull request. The usual responses are "rerun the pipeline" or "start debugging". Both skip a cheaper question: is this failure caused by the change at all?

Step 1: rerun only the failed test

pytest tests/test_checkout.py::test_total -x
npx jest -t "total is correct"
Enter fullscreen mode Exit fullscreen mode
  • Fails, then passes: flaky. A timeout, a race, a test that depends on order, real network or the current time.
  • Fails twice: probably real. Keep going.

Step 2: run it on main

git stash && git checkout main
pytest tests/test_checkout.py::test_total
Enter fullscreen mode Exit fullscreen mode

If it fails on main too, your change didn't break it. It was already broken, and that's a separate ticket, not a reason to block this pull request.

Step 3: only now, debug

At this point you know the failure is real and new. That's the only kind worth blocking a merge on.

Why teams skip this

CI shows one colour per run. So the habits are "rerun until green", which hides real failures that happen to pass on a retry, or "block on red", which lets one flaky test hold up every pull request for months.

Baking it into CI

pytest can retry failures and report them separately:

pip install pytest-rerunfailures
pytest --reruns 1
Enter fullscreen mode Exit fullscreen mode

A test that only passes on the rerun is flagged in the output, so it can be tracked instead of forgotten.

The stronger version compares results before and after the change: the same tests run on the base commit and on the pull request, and only new failures count.

One catch: "fails, then passes" only means flaky if nothing else changed between the two runs. A different runner image or a different lockfile can make a test fail every time in one environment and pass every time in another, even at the same commit. So both runs, and the rerun, need the same environment, and the result should record it.

That's how Versyka decides whether an automated upgrade is safe. The run on the base commit, the run on the change and any rerun happen in the same sandbox. Tests that were already failing are listed in the pull request, and only a new, repeatable failure stops it from opening. Every pull request also shows how it was checked: the sandbox, the runtime versions, the sha256 of each lockfile before and after, and for a test that failed once and passed on its rerun, which tests failed the first time.

Top comments (2)

Collapse
 
raju_dandigam profile image
Raju Dandigam •

The base-commit comparison is the useful receipt for a bot PR. I'd be careful with "fails, then passes = flaky": a runner or dependency change can make outcomes deterministic per environment, even at the same SHA. Saving the runner image, lockfile, and retry results beside the base/PR test IDs would make that call reviewable. Does Versyka pin those conditions for the comparison?

Collapse
 
getversyka profile image
Versyka •

Agreed, "fails then passes" is only a fair call when the environment is held still. That's why Versyka runs the base commit, the change and the rerun in the same sandbox: a failure that's deterministic for that environment fails both times and never gets called flaky. If it failed on the base run too, it's reported as already failing, not blamed on the change.

And the PR records it under "How this was checked": the sandbox, the runtime versions, the sha256 of each lockfile before and after, and for a flaky call, the test IDs that failed the first time. So the call is reviewable rather than taken on trust.

Next on our list: when your own CI disagrees with the sandbox at the same SHA, the PR will show both results side by side, so the difference points straight at the environment.