Once you have thirty unreliable tests, the question stops being “is this flaky” and becomes “which one do I fix on Thursday afternoon”. A score answers that, and the obvious score — failures per hundred runs — ranks badly for two specific reasons that are worth fixing.
What to count
The naive metric is failures per hundred runs. It has an immediate problem: a test that is simply broken fails a hundred times per hundred runs and tops the list, but it is not flaky, it is a regression, and it belongs to a completely different workflow. Any score that cannot tell those apart will spend your Thursday on the wrong thing.
So the raw input is per-attempt outcomes, and the aggregation is per run: for each CI invocation on each commit, a test either passed all attempts, failed all attempts, or did both. The third case is the flake. That definition, its window and the SQL that computes it are set out in setting an acceptable flake rate; this page assumes it and asks what to do with the resulting column.
The second problem with failures per hundred runs is subtler. It treats a test that failed ten times last Tuesday and has been green since as equivalent to one that fails twice a week, every week. Those are different situations with different actions — the first is a resolved incident, the second is a standing tax — and a metric that cannot separate them will keep putting the resolved one at the top of your list. Both problems have the same root: a rate discards the order of the observations, and the order is where most of the information is.
Flips beat failures
Order the runs of one test by time and look at the sequence of outcomes. Two tests can have identical failure rates and completely different characters:
test A: P P P P P P P P P P F F F F F F F F F F (10 failures / 20)
test B: P F P P F P F P F P P F P F P F P P F F (10 failures / 20)
failure rate: A = 0.50 B = 0.50
flips: A = 1 B = 13
flip rate: A = 1/19 = 0.05 B = 13/19 = 0.68
Test A is not flaky at all. It is a test that broke at a specific point in time and has been failing ever since — a regression, findable by bisect, with a first-bad-commit. Test B is genuinely unstable. The failure rate cannot distinguish them; the flip rate separates them completely.
flip_rate(test) = transitions(outcome_sequence) / (runs - 1)
where transitions counts positions i where outcome[i] != outcome[i-1],
over runs ordered by commit time, one outcome per run.
The flip rate has a natural ceiling worth knowing: a perfectly alternating sequence scores 1.0, and an independent coin-flip test with a 50% failure rate has an expected flip rate of about 0.5. Anything approaching 1.0 is not random noise but something alternating — test-order dependence, a shared resource being toggled, a two-node deployment serving different model versions. That is a different bug with a different fix, and the score points straight at it.
The small-sample problem
A test with four runs and one flake has a point estimate of 25%. A test with four hundred runs and forty flakes has a point estimate of 10%. Any ranking on the raw rate puts the first one above the second, which is backwards: you know almost nothing about the first test and a great deal about the second.
The standard fix is to rank on a lower confidence bound rather than on the estimate. The Wilson score interval’s lower bound is the usual choice because it behaves sensibly when the count is zero or the rate is near an endpoint, where the normal approximation does not.
from math import sqrt
def wilson_lower_bound(successes: int, n: int, z: float = 1.96) -> float:
"""Lower bound of the Wilson score interval. z=1.96 is ~95% one-sided 97.5%."""
if n == 0:
return 0.0
phat = successes / n
denom = 1 + z * z / n
centre = phat + z * z / (2 * n)
margin = z * sqrt((phat * (1 - phat) + z * z / (4 * n)) / n)
return (centre - margin) / denom
# ranking input: successes = flaky_runs, n = total_runs
wilson_lower_bound(1, 4) # a test seen 4 times, flaked once
wilson_lower_bound(40, 400) # a test seen 400 times, flaked 40 times
The effect is that evidence is required before a test can climb the list. A test with one flake in four runs is pushed down until it has accumulated enough history to justify a position; a test with a consistent moderate rate over hundreds of runs rises. This is the same adjustment used for ranking items by a proportion anywhere else, and it is the difference between a leaderboard of new tests and a leaderboard of real problems.
Apply the same bound to the flip rate rather than to the failure rate, since the flip rate is the quantity you actually want to rank on. The denominator is one less than the run count, which matters only for very small samples — and very small samples are precisely what the bound exists to demote, so the approximation is harmless. Pick one z and keep it: changing it between reports reshuffles the ranking for reasons unrelated to any test.
Ranking by damage, not by rate
A rate still is not a priority, because two tests with the same rate can cost wildly different amounts. Multiply by how much trouble each flake causes:
score(test) = wilson_lower_bound(flaky_runs, total_runs)
* runs_per_week(test)
* blast_radius(test)
blast_radius:
4 gates merges on the default branch
2 gates merges on a feature branch only
1 runs nightly, blocks nobody
Now the units mean something: roughly “expected blocked merges per week”. A test with a 30% flake rate that runs once a night is below a test with a 3% rate that runs on every push to a busy repository, which is the correct ordering and not the one a rate-sorted list gives you.
The blast-radius weights are deliberately coarse. Three levels are enough to get the ordering right and few enough that nobody spends an afternoon debating whether a test deserves a 2.5. Resist the urge to add a term for how important the covered feature is: everybody rates their own feature highest, and the resulting score stops being comparable across teams. Cost of the flake, not value of the feature, is what this ranking measures.
Add the failure signature as a grouping key before you rank — from deduplicating flaky failures — because twenty tests sharing one signature are one fix, and ranking them as twenty entries buries the single-test problems underneath a single cause you could resolve in an afternoon.
Ways this score gets misused
- As a target. A score that engineers are measured on is a score that goes down by deleting tests. Rank with it; never report it as a team metric.
- Across incompatible windows. Comparing a test that has existed for a year with one added last Tuesday requires the same window for both, or the confidence bound is doing arithmetic on different things.
- Without excluding known incidents. A four-hour provider outage will flip every test in the suite once and shuffle the whole ranking. Tag those runs by signature and exclude them; the outage is not a property of any test.
- On a suite where nothing is retried. With one attempt per run, a flake and a failure are the same event, and the entire score collapses to a failure rate with all the problems this page opened with.
Top comments (0)