Stop Rerunning Flaky Tests. Build an Evidence Log Instead.

You know the ritual. CI fails, you rerun the test, it passes. Human on-call duties vanish, but the CI badge glows green while the suite rots. Flaky tests are not random noise; they leave predictable fingerprints. The fix is not a retry button. It is a small evidence log plus a model that reads those logs quickly.

This article walks through a free, reproducible flaky-test triage workflow. You collect failure metadata locally, classify it with a tiny Python script, and then let a coding model generate hypotheses from the evidence. The whole loop runs on free model access and a free server option, both available in MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why rerunning a flaky test makes the problem worse

Rerunning hides the evidence. The moment you press the green retry button, you delete the exact output that tells you why the test flaked. Memory, timing, ordering, network, filesystem races: each one leaves a distinct trace. Without the trace, you are gambling, not debugging.

A better default: record every failed run, including the exit code, the failure message, the duration, and the test order. After ten reruns, you will see a pattern. Some flakes alternate between two failure messages; others always fail when a certain neighboring test runs. That pattern is the diagnosis.

The evidence collection script

Here is a minimal runner that executes one test file repeatedly and writes a structured report to JSON. It uses only the Python standard library, so you can drop it into any repo.

import json
import subprocess
import time
import sys
from pathlib import Path

TEST_FILE = sys.argv[1] if len(sys.argv) > 1 else "tests/test_flaky.py"
RUN_COUNT = int(sys.argv[2]) if len(sys.argv) > 2 else 15

results = []

for i in range(RUN_COUNT):
    start = time.perf_counter()
    proc = subprocess.run(
        ["pytest", TEST_FILE, "-x", "--no-header", "--tb=short"],
        capture_output=True,
        text=True
    )
    duration_ms = (time.perf_counter() - start) * 1000
    results.append({
        "run": i + 1,
        "exit_code": proc.returncode,
        "duration_ms": round(duration_ms, 2),
        "failure_snippet": proc.stdout[-500 if proc.returncode else 0:] if proc.stdout else "",
        "stderr_tail": proc.stderr[-300:] if proc.stderr else ""
    })

Path("flake_report.json").write_text(json.dumps(results, indent=2))

successes = sum(1 for r in results if r["exit_code"] == 0)
print(f"Passed {successes}/{RUN_COUNT} runs. Report written to flake_report.json")

Run it from the project root. The report gives you three columns that matter: exit_code, duration_ms, and failure_snippet. Now you have raw evidence, but raw evidence is still too verbose to skim. That is where classification comes in.

Classifying the flake by failure signature

The next step groups all runs that failed with the same message. A simple classifier can do this with a list of known patterns. Here is a starter set for common flakes.

import json

report = json.load(open("flake_report.json"))

SIGNATURES = {
    "TIMEOUT": ["Timeout", "timed out", "deadline"],
    "RACE": ["assert None is not None", "Cannot read property", "UnboundLocalError"],
    "ORDER_DEPENDENT": ["failed to import", "already imported", "fixture '...'"],
    "INFRA": ["ConnectionRefused", "ECONNRESET", "503", "database is locked"],
}

def classify(snippet):
    snippet = snippet or ""
    for label, patterns in SIGNATURES.items():
        if any(p.lower() in snippet.lower() for p in patterns):
            return label
    return "UNKNOWN"

for r in report:
    r["signature"] = classify(r["failure_snippet"])

json.dump(report, open("flake_report_classified.json", "w"), indent=2)

Now you have a decision surface. A pure TIMEOUT signature points at slow I/O or a tight deadline. A RACE signature points at thread or async ordering. An ORDER_DEPENDENT signature points at shared mutable app state. Ten reruns of a flaky test usually produce one dominant signature, and that signature tells you where to look before you touch any product code.

Turning the evidence into a decision table

A decision table makes the classification actionable. Save it inside your repo as FLAKE_TRIAGE.md.

Signature	Most likely cause	First patch to try
TIMEOUT	Too little wall-clock allowance	Increase timeout with a justified limit
RACE	Shared mutable state between async tasks	Serialize the shared resource for the test
ORDER_DEPENDENT	Test A leaks data into Test B	Reset fixtures at test start, not end
INFRA	Network or database flake	Mock the external service in this test
UNKNOWN	Nondeterministic algorithm or random seed	Capture the seed, reproduce offline

The table is not a law; it is a starting point. The point is that you do not open an issue that says "flaky test," because that is a statement of ignorance. Instead you open an issue that says "TIMEOUT flake in test_login_retry, 4/15 runs, median duration 2.3s." That issue can actually be fixed.

Where free models and a free server fit

You can run the rerun loop once, but flakes are seasonal. One week it appears, then it sleeps for two months. To catch it again you need a scheduled job on a machine you do not pay for. MonkeyCode's free server option lets you run the collection script on a cron schedule without provisioning a new instance. And when the report shows an UNKNOWN signature, you can paste the JSON snippet into the MonkeyCode chat; its free model access will suggest which statistical pattern to hunt next. The model is not magic, but it reads logs faster than you do.

A minimal cron line looks like this:

0 */4 * * * cd /opt/flake-sentry && python3 collect.py tests/test_login_retry.py 10 && python3 classify.py && ./archive_report.sh

Archive every report. After a month you will see which tests burn the most rerun budget, and you can retire them or quarantine them with a clear ticket.

What this workflow will not do

This approach does not fix the flake for you. It gives you a reproducible starting point, but if the root cause is a heisenbug deep inside a C extension or a kernel-level race, no log grouping will make it obvious. The free model suggestions are only as good as the captured output; if your failure snippet is empty because the process hung completely, you will need OS-level tracing instead. Also, free-tier models may rate-limit or change over time, so do not build your entire CI pipeline on a promise of zero cost.

Who should use this today

Use this workflow if you have at least one flaky test that has caused an unexplained CI failure in the past week. Skip it if you have no tests yet; write the happy-path coverage first. And if your team already pays for a flaky-test service, treat this as a simple fallback for repos where you do not want to wire another vendor into the build.

Start with the test that makes you sigh when you see it on the CI log. Run the collector, classify the signature, and put the report in your next PR. The retry button is a sedative; an evidence log is a diagnosis. If you try this, share which signature showed up in your report.

Comments

[Luis Cruz]

Your approach to handling flaky tests through evidence logging is a refreshing shift from the conventional retry strategy. By collecting and classifying failure metadata, you're not just documenting the issues but also paving the way for systematic debugging. I wonder if integrating a visualization of the failure patterns could further enhance the understanding of these flaky tests over time. If you need an extra hand in refining this workflow or developing additional features for the logging system, I’d be open to discussing a paid collaboration.