When Your Test Log Is 2,000 Lines Long: A Free-Server Flake Classifier

The most expensive part of a flaky test suite isn't the CI bill or the lost minutes. It's the slow, quiet habit of ignoring a red build because too many of those reds are false alarms. You start scanning the log for the same three test names, your eyes skip the middle, and eventually every failure email gets archived without being read. A month ago I was exactly that person, and the fix turned out to be a deliberately boring automation job running on a free server with a free model allowance.

The core conclusion comes first: the cheapest way to tame flaky tests is not a better retry plugin or a longer timeout. It's a scheduled agent that reads every failed test log once, stamps each failure with a verdict, and leaves the real human only the short list of hard failures.

The setup that worked for me was built on MonkeyCode, an open-source coding-agent toolkit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's currently published offer includes free model access with a 10-million-token allowance and a free server slot for scheduled agent jobs, both as of late August 2026. I treated that offer as a convenience layer, not a magic wand: the durable part is the small script you're about to see.

Why the Middle of the Log Is Where the Truth Dies

Pytest verbose output is honest but noisy. On one particularly bad day, my CI printed 2,000 lines, of which 14 were failures. Three of those failures were genuine regressions. The other eleven were the same two tests flipping between green and red depending on the runner's mood.

Reading 2,000 lines to find that pattern manually takes about twenty minutes, and you won't do it at 3 AM. So the real failure was not in the code. It was in the review loop: no one had time to separate signal from noise, so the signal started getting ignored.

That's the gap a flake classifier fills. It doesn't make the test suite perfect. It makes the suite readable again, which is arguably more valuable.

What the Free Server Slot Is Actually for

People hear "free server" and immediately imagine running a cluster of model containers. That's the wrong mental model for this job. A flake classifier needs almost no compute; it needs uptime, a cron daemon, and a network connection. It is precisely the kind of small, unattended workload that you don't want to babysit on a developer's laptop.

My earlier attempt ran on a local machine and died for the dumbest possible reason: the laptop went to sleep before the nightly job fired. The second attempt died when a coffee-shop wifi drop killed a forty-minute agent run halfway through the logs. The free server slot removes both failure modes. It acts as a night watchman, not as a GPU farm.

One warning before we go further: free tiers change. The 10-million-token allowance and the free server slot are what the project publishes today, but quotas and availability are not a contract. Verify the current numbers at the source before you build a permanent process around them. The script below is the transferable asset; the free tier is the convenience layer.

The Artifact: A Log-to-Report Preprocessor

Rather than sending all 2,000 lines to the model, I wanted a Markdown report that the agent could consume in chunks. Here is a small Python script that turns a raw pytest log into a concise, structured digest of every distinct failing test.

#!/usr/bin/env python3
"""flake_report.py — extract failing tests from a pytest log into a digest.

Usage: python3 flake_report.py path/to/test.log
Outputs the digest to stdout.
"""
import sys
import re
import collections

FAILED_RE = re.compile(r"FAILED (.+?)(?= - )")

def read_blocks(path):
    """Yield (test_name, context) for each failed test occurrence."""
    current = None
    lines = []
    for raw in open(path, errors="replace"):
        line = raw.rstrip("\n")
        m = FAILED_RE.search(line)
        if m:
            # flush previous failure block
            if current is not None:
                yield (current, lines[-8:])
            current = m.group(1).strip()
            lines = [line]
        elif current is not None:
            # keep small context around the failure marker
            if len(lines) > 12:
                lines.pop(0)
            lines.append(line)
    if current is not None:
        yield (current, lines[-8:])

def main():
    if len(sys.argv) != 2:
        sys.exit(f"Usage: {sys.argv[0]} test.log")
    counts = collections.Counter()
    blocks = list(read_blocks(sys.argv[1]))
    for name, _ in blocks:
        counts[name] += 1

    print("# Flake Digest\n")
    print("| Test | Occurrences |")
    print("| --- | --- |")
    for name, count in counts.most_common():
        print(f"| `{name}` | {count} |")
    print("\n## Failure Contexts\n")
    for name, lines in blocks:
        print(f"### `{name}`\n")
        print("```

")
        print("\n".join(lines))
        print("

```\n")

if __name__ == "__main__":
    main()

The script does not judge anything. It groups occurrences and preserves eight lines of context so the model can see the failure message without drowning in unrelated setup output. That context window matters more than you might think: a flaky test usually fails with the same five lines, and those lines are almost always buried in the last eight lines before the FAILED marker.

To run it, you do the ordinary pipeline:

pytest -q > /tmp/ci.log
python3 flake_report.py /tmp/ci.log > /tmp/flake_digest.md

Then you hand that digest to an agent runtime on the scheduled server and ask it to produce a JSON verdict per test:

{"test": "test_order_total_sync", "verdict": "flaky", "evidence": "TimeoutError in 4 of 6 runs; same input produces different result"}

On the free server slot, this whole dance runs on a cron line. Mine looks like this:

0 3 * * * cd /opt/triage && bash run_triage.sh >> /var/log/triage.log 2>&1

run_triage.sh is ten lines: it copies the latest CI artifact, runs the Python script, invokes the agent on the digest, and writes the verdicts to a channel that opens before standup.

The Verdict Table That Made It Usable

A model's judgment means nothing if the categories are vague. We used three verdicts and strict evidence rules.

Verdict	Evidence threshold	What the human does
flaky	Same test failed 3+ times in 7 runs with different stack traces	Ignore, add to quarantine list
hard	Same traceback across ≥2 runs, or new test name	Investigate today
unclear	One occurrence, ambiguous trace	Skim; assign a random human

That table is small, but it is the whole product. Without thresholds, the model will happily call a one-off network hiccup a regression. With them, the output becomes a triage report instead of a spiritual essay.

What Than Classifier Gets Wrong

This approach has sharp edges, and you should know them before copying it.

First, it only sees logs, not the code under test. A deep concurrency bug can look identical to a flaky timeout on every run, and no amount of log-digesting will teach the agent the difference. The verdicts are hypotheses, not diagnoses.

Second, the free model allowance encourages shorter prompts, which is good for cost but bad for nuance. If you need whole-file context, you will burn tokens fast and may still get a shallow verdict. Keep the digest small, or accept the tradeoff.

Third, three groups should not use this pattern at all:

• Teams under strict data-residency rules, because the log text leaves your CI runner and lands on a third-party server. Read the terms, then decide.
• Teams that need a deterministic audit trail. An LLM's flake verdict is not a repeatable proof; it is a heuristic with a probability attached.
• Teams already drowning in nightly jobs. Adding another scheduled agent before stabilizing the first one just moves the noise into a new pile.

The One Decision That Mattered

The biggest unlock was not the model. It was choosing to separate triage from diagnosis. The agent gets the first pass because it can read 2,000 lines in seconds. The human gets the short list because only a human can look at the flaky test, see the code it exercises, and remember the conversation from last sprint about that badly named helper function.

If your Monday starts with an unread CI email, try this: take last week's log, run the Python script, and see how many of the failures collapse into two or three recurring names. If the answer is more than zero, you have just found the first test to quarantine.

The free server slot and the free model allowance made the experiment cheap enough to try on a weekend. The habit it built — machine triages, human diagnoses — is what actually stuck.

One small, soft invitation: if you run this against your own logs and land on a different verdict table, I'd be curious which thresholds you chose. The script is the easy part; the thresholds are the real engineering.