Telling a Flaky Test Apart From a Real Regression

#ai #llm #machinelearning #programming

The build is red, the change looks unrelated, and somebody is about to say “that test is flaky, just rerun it”. They might be right. The four checks below take a few minutes and replace the argument with evidence, which matters because the cost of being wrong is asymmetric.

The question you are actually asking

“Is this flaky?” is not a well-formed question. The well-formed version is: does the failure rate of this test depend on my change? A flaky test fails at some rate that is a property of the test and the world, and your commit did not move it. A regression is a failure rate that your commit moved. Both can look identical in a single red build, and that is the entire difficulty.

Notice that this framing makes one attempt useless as evidence in either direction. A single failure is a draw from a distribution you have not characterised, and rerunning until it goes green is not a test, it is a search. Everything below is a way of drawing more samples cheaply and comparing two distributions rather than two outcomes.

The asymmetry is worth stating out loud, because it decides how much effort this deserves. Calling a regression “flaky” ships the defect and destroys the test’s credibility, so the next real failure is dismissed faster. Calling a flake a regression costs an hour of somebody’s afternoon. Bias the procedure towards the second error.

Check one: rerun this commit

Run the failing test alone, on the same commit, several times. Not the whole suite — the single test, in a loop, so you get a rate rather than an outcome.

# pytest: run one test 20 times and count outcomes
pytest tests/test_router.py::test_router_emits_a_tool_call \
  --count 20 -p no:randomly -q
# (--count comes from the pytest-repeat plugin)

# vitest: same idea, using the built-in repeats option
npx vitest run src/router.test.ts -t "emits a tool call" --repeats 20

Read the result as three cases. All twenty fail: this is a hard failure and almost certainly a regression, because a genuinely flaky test that fails twenty consecutive times has a per-run failure rate high enough that you would already know about it. All twenty pass: you have weak evidence for flakiness and no evidence about your change. A mix: you have a rate, and it is worth comparing to something.

The middle case is where every team stops too early, because twenty green runs feel conclusive. They are not. If your change took the failure rate from zero to one-in-thirty, twenty passing runs are the most likely outcome and prove nothing.

Two details make the loop trustworthy. Disable test-order randomisation for the loop, so you are sampling the model rather than also sampling the ordering — that is what -p no:randomly does above. And run the single test rather than the file: a neighbour that leaves a patched client or a populated cache behind will change the failure rate of the test you are measuring, and you will attribute the difference to your change.

Check two: rerun the last known-good commit

This is the check that answers the actual question and the one most often skipped, because it feels like extra work when the build is already red. Check out the last commit where this test was green, change nothing, and run the same loop with the same count.

git stash --include-untracked
git checkout <last-green-sha>
pytest tests/test_router.py::test_router_emits_a_tool_call --count 20 -q
git checkout -
git stash pop

If the old commit fails at a similar rate, your change is exonerated and something outside the repository moved — the model, the provider, a dependency you did not pin. If the old commit passes twenty out of twenty and the new one fails a third of the time, you have a regression, whatever the diff looks like. The diff looking unrelated is not evidence; prompt changes reach further than they appear to, which is the whole premise of bisecting a prompt change that introduced flakiness.

Run the two loops close together in time and with the same credentials, model version and concurrency. If you run the known-good commit an hour later, on a quieter shared key, you have varied three things at once and the comparison means nothing. Where the check is worth automating, this is exactly the probe that git bisect run repeats, so a script that runs one test N times and reports a pass count is reusable for both jobs.

Check three: evidence from outside your repo

Three sources settle a surprising fraction of these arguments in under a minute, and none of them requires running anything:

The provider’s status page and changelog. An incident window that brackets your failures is close to conclusive. So is a model deprecation or a silent point-release — see silent model updates for why an unpinned model alias is the most common version of this.
The response metadata you logged. If you record the model id actually served, and the system_fingerprint where the provider returns one, a change in either between the green run and the red one is the answer. OpenAI documents system_fingerprint as an identifier for the current combination of model weights and backend configuration, and describes seeded sampling as best-effort rather than guaranteed — so a fingerprint change is exactly the event that invalidates a reproduction.
Whether other tests moved at the same time. One test failing is a test problem. Eleven tests failing together, across unrelated features, is an infrastructure problem, and grouping them by cause rather than by name is what deduplicating flaky failures is for.

Whether a failure is yours or the provider’s is much easier to answer when the response metadata is recorded the same way for every provider. Multigrid logs the served model id, the upstream status and the latency for each request through one API, so “did the model change between the green run and the red one” is a query rather than an archaeology exercise across two vendors’ dashboards.

The decision table

Put the two loop results side by side. HEAD is the commit under review, GOOD is the last known-green commit, and each cell assumes the same number of repetitions on both.

HEAD        GOOD        conclusion
--------    --------    ----------------------------------------------
0/20 pass   20/20 pass  regression in your change. Do not merge.
0/20 pass   0/20 pass   broken outside the repo: provider, model, dep.
                        Pin the model and re-check before blaming code.
mixed       20/20 pass  regression that only shows sometimes. Treat as
                        a regression; the rate is your evidence.
mixed       mixed       pre-existing flake. Compare the rates; if HEAD
                        is clearly worse, treat as a regression.
20/20 pass  anything    unreproducible. Record it, do not quarantine on
                        a single failure, and check the dashboard next
                        week for a pattern.

The last row is the one that needs discipline. A single unreproducible failure is not enough to quarantine a test, because quarantining on one bad day is how a suite loses its coverage one test at a time. Record it and let the accumulated history decide — a published flake-rate tolerance exists so that this decision is made once, as policy, rather than re-argued by whoever is on call.