Building FlakyDetector 0.2.2: Why Test Reliability Needs Evidence, Not Just a Score

#python #opensource #github #testing

Building FlakyDetector 0.2.2: Why Test Reliability Needs Evidence, Not Just a Score

By Artem Alimpiev — Python Backend Developer and creator of FlakyDetector

The most useful question I asked while working on FlakyDetector was not, “How many suspicious patterns can I detect?”

It was, “What does this result actually prove?”

That question shaped the work behind FlakyDetector 0.2.2: stronger evidence handling, a real execution-event pipeline, safer source handling, and a GitHub Action that can distinguish new risk from existing debt.

This is a follow-up to Catching Flaky Tests Before They Hit CI: Meet FlakyDetector. This time, the story is about making the tool's conclusions and delivery process more trustworthy.

Three questions that should never share one answer

Consider a test that contains a network call and a fixed sleep.

A static analyzer can identify those constructs. It cannot infer from their presence alone that the test has already failed intermittently. And even when runtime history contains both failures and passes, that does not establish the root cause.

In FlakyDetector, I now keep three categories explicit: static risk, observed execution behavior, and unverified explanation.

A finding can justify investigation. Comparable executions can establish an observed pattern. An explanation remains a hypothesis until the evidence supports it.

This distinction is central to the project's current contract. “No known risk” is not another way of saying “stable.”

For me, that is a product decision as much as an engineering decision. I would rather expose uncertainty than turn a plausible warning into an unsupported diagnosis.

Making findings usable in an existing repository

A new tool should not demand that a team fix every historical problem before it becomes useful.

I added source-free SARIF 2.1.0 reporting and a trusted-baseline policy. The analyzer keeps the full report, while the policy can focus on new or worsened findings. Existing debt stays visible instead of disappearing behind suppression.

The baseline comes from a trusted base revision, not a file that a pull request can quietly rewrite to approve itself. An incomplete analysis also remains an error; it does not become a successful scan because a severity threshold was relaxed.

The reusable GitHub Action connects this to job summaries, annotations, report artifacts, and optional Code Scanning upload. It analyzes source without installing the target application's dependencies or running its tests.

Runtime evidence needs context

I extended the pytest recorder rather than treating another retry as the whole solution.

The event model preserves test identity, invocation, attempt, phase, outcome, worker information, and execution provenance. Replayable monitors evaluate those events under explicit comparability rules.

A fail-then-pass sequence under comparable conditions supports an observation. A pass under one configuration and a failure under another may describe two different experiments.

The monitors therefore have an explicit insufficient_evidence state. They also distinguish elevated failure rate from intermittent behavior and suspected infrastructure incidents from confirmed causes. The monitor documentation describes those boundaries.

Quarantine follows the same principle: owner, issue, approval, expiry, and an audit trail. It changes a policy decision after execution. It does not rewrite a failure into a pass or make the test disappear.

Source code is not harmless diagnostic data

A helpful source preview can also disclose private implementation details or credentials.

I changed normal API responses and exports to omit source snapshots, snippets, and raw logs. Source viewing requires explicit configuration and authorization; source-bearing exports require additional confirmation. Sensitive responses use no-store.

Analysis runs in a killable Linux child process with resource limits. That is deliberately not advertised as a complete filesystem or network sandbox. Shared instances are restricted to one workspace, and redaction is documented as best-effort.

The security boundary matters more than an impressive “secure” badge.

The packaging check that caught my own mistake

During verification, the Python package built and installed successfully, but its independent smoke test rejected an API request.

The test had confused the request's destination with the client's peer address. The application's local-only access rule was doing its job; the test setup was wrong.

I corrected the test instead of weakening the rule.

That experience reinforced another decision: test the final archive, the independently installed wheel, and the container—not only imports from the working tree.

The successful main-branch CI run includes final-artifact checks on Python 3.12 and 3.13 and four dedicated runtime-plugin combinations. Its Python 3.12 artifact report records 390 passed, 3 skipped; frontend tests record 71 passed. The retry-related skips in the general suite are distinct from the dedicated plugin jobs that passed.

Those numbers demonstrate specific checks. They are not a real-world detection-accuracy benchmark.

What comes next

Version 0.2.2 is published on GitHub. Independent corpus evaluation, broader semantic coverage, and external consumer acceptance remain work to do. I am not attaching a calibrated flakiness probability to a result that has not earned one.

The engineering lesson I am taking forward is simple:

A reliability tool must make its uncertainty inspectable, not merely make its output look confident.

What would make you trust a new test-reliability tool enough to add it to your CI: explainable findings, runtime evidence, or a low-noise rollout policy?

Follow me for more practical Python backend engineering. Feedback and real counterexamples are especially welcome.

Repository · Documentation