“Some flakiness is acceptable” is true and useless. The version that changes behaviour is a written number, applied per test, with a defined statistic behind it and an agreed consequence when a test exceeds it. Here is how to derive that number for your suite rather than borrow one.
Define the statistic before arguing about it
Most disagreements about flakiness are disagreements about the denominator. Fix it first. A workable definition, and the one the rest of this page uses:
flake_rate(test, window)
= flaky_runs(test, window) / total_runs(test, window)
where a run is FLAKY if, within one CI invocation on one commit,
the test both failed and passed — that is, an attempt failed and a
retry of the same test on the same commit succeeded.
A run where every attempt failed is a FAILURE, not a flake.
A run where the first attempt passed is a PASS.
Three choices in there are worth defending. Counting per run rather than per attempt means a test with five retries does not look five times flakier than one with one. Requiring both outcomes on the same commit excludes genuine regressions, which are a different problem with a different owner. And keeping a window — thirty days is a reasonable default — means the statistic responds when somebody fixes something, instead of being dragged forever by a bad fortnight in March.
A suite that only ever runs each test once per commit cannot compute this at all, because a flake and a failure are indistinguishable without a second attempt. That is the practical argument for one retry even on suites that do not need retries to stay green: the retry is what makes the statistic measurable.
One more decision has to be made explicitly, because leaving it implicit is how two people end up quoting different numbers from the same data: whether a test that ran on a branch counts the same as one that ran on the default branch. Branch runs are noisier — the code under test is half-finished — and including them inflates every rate. Excluding them shrinks the sample by most of its volume. The workable answer is to compute the statistic over all runs but rank and enforce on default-branch runs only, and to say which one any number you quote came from.
Computing it from your own history
No published figure means anything for your suite, and there is no industry rate to quote — the number depends on your provider, your model, your temperature, your concurrency and what you assert on. Compute yours. If you are storing one row per attempt in the shape a minimal dashboard uses, it is one query:
-- one row per (run_id, test_id, attempt, status)
WITH per_run AS (
SELECT run_id,
test_id,
MAX(CASE WHEN status = 'pass' THEN 1 ELSE 0 END) AS any_pass,
MAX(CASE WHEN status = 'fail' THEN 1 ELSE 0 END) AS any_fail
FROM test_attempts
WHERE started_at >= NOW() - INTERVAL '30 days'
GROUP BY run_id, test_id
)
SELECT test_id,
COUNT(*) AS runs,
SUM(any_pass * any_fail) AS flaky_runs,
ROUND(100.0 * SUM(any_pass * any_fail) / COUNT(*), 2) AS flake_pct
FROM per_run
GROUP BY test_id
HAVING COUNT(*) >= 30
ORDER BY flake_pct DESC;
The HAVING COUNT(*) >= 30 is not decoration. A test with four runs and one flake has a point estimate of 25% and an interval so wide the estimate carries no information; ranking on it puts new tests at the top of your backlog forever. The small-sample problem and what to do about it belong to the flakiness score.
Why the per-test number has to be small
The instinct is that a few per cent per test sounds harmless. Suite size destroys that instinct, because independent per-test flake rates multiply. If a suite has n tests each flaking independently at rate p, the probability that a whole run is green is (1 - p)^n.
P(green suite) = (1 - p) ** n
n = 200 tests
p = 0.05 -> 0.95 ** 200 = 0.000035 (essentially never green)
p = 0.02 -> 0.98 ** 200 = 0.0176 (green ~1.8% of runs)
p = 0.005 -> 0.995 ** 200 = 0.3670 (green ~37% of runs)
p = 0.001 -> 0.999 ** 200 = 0.8186 (green ~82% of runs)
n = 40 tests
p = 0.005 -> 0.995 ** 40 = 0.8183
p = 0.002 -> 0.998 ** 40 = 0.9231
Independence is again an assumption, and here it is conservative in an unhelpful direction: LLM test failures are often correlated, because one provider incident hits every test at once. Correlation makes the green-build probability better than this formula predicts while making the red days much worse — you get long clean stretches punctuated by a run where forty tests fail together. That pattern is why grouping failures by cause matters more here than in an ordinary suite.
The practical reading of that table is that a per-test tolerance is not a number you can set independently of suite size. Doubling the number of model-calling tests roughly halves the tolerance each one is allowed, which means a policy written when the suite had forty tests silently becomes wrong at two hundred. Recompute it whenever the count changes materially, and record the suite size alongside the threshold so the next person can see which assumption it was derived under.
Turning a suite target into a per-test budget
Invert the formula. Decide what fraction of runs should be green without any human deciding anything — call it G — then the per-test budget is p = 1 - G^(1/n).
p = 1 - G ** (1 / n)
G = 0.95 (95% of runs green), n = 200 -> p = 1 - 0.95**0.005 = 0.000256
G = 0.90, n = 200 -> p = 1 - 0.90**0.005 = 0.000527
G = 0.90, n = 40 -> p = 1 - 0.90**0.025 = 0.00263
Those budgets are far tighter than the “under 2%” figure that circulates as folklore, and that is the useful finding: for a suite of any size, a per-test tolerance in the low single-digit per cent does not produce a green pipeline. It produces a pipeline that is almost always red and a team that has stopped reading it. If your budget comes out at a rate you cannot achieve for tests that call a model, the conclusion is not a laxer budget — it is that fewer of your tests should be calling a model. Push the majority onto recorded responses (testing without the model) and keep a small live set where the network is the point.
Writing it down as policy
The number is worth less than the consequence attached to it. A policy that changes behaviour has four clauses, and fits in a paragraph of your contributing guide:
- The statistic, defined as above, computed over a named window, from a named table. Ambiguity here is what restarts the argument.
- The threshold, derived from your own suite size and your own green-build target, with the derivation recorded so the next person can redo it when the suite doubles.
- The consequence. A test over threshold is quarantined automatically, not discussed — the point of the policy is to remove the discussion. Quarantine has its own wiring and exit criteria.
- The review cadence. Someone looks at the ranked list on a fixed schedule. Without this the policy is a threshold nobody evaluates.
Publish the derivation next to the number. A threshold with its working shown can be challenged on its assumptions, which is a productive argument; a bare threshold can only be challenged on authority, which is the argument the policy existed to end. In practice the sentence people need is the one that says what the number buys: at this tolerance and this suite size, roughly this fraction of runs will be green without anyone intervening.
One clause not to include: an exemption for “tests that call the model, because those are inherently flaky”. They are inherently nondeterministic, which is not the same thing. A test that asserts on a schema, a tool name or an invariant can be as stable as any other test, and the exemption removes the pressure that would have got it there.
Top comments (0)