Deduplicating Flaky Failures Across a Large Prompt Test Suite

#ai #machinelearning #tutorial #llm

When a suite that calls a model goes red, it usually goes red in clusters: one rate limit, one bad deploy, one model alias moving, and forty tests fail at once. Reading that as forty problems is the default behaviour of every test report, and it is wrong in a way that costs hours.

Why grouping by test name fails

A test report is organised by the thing the runner knows: file, class, test name. That organisation assumes failures are independent, which is a decent assumption for unit tests and a bad one for tests that all go through the same HTTP client to the same provider. When the shared dependency wobbles, every test that touches it fails, and the report presents them as unrelated because their names are unrelated.

The consequence is not just wasted reading. It distorts every downstream number: your flake counts spike for forty tests that are individually fine, your triage backlog fills with duplicates, and a genuine single-test regression that happened during the same run is buried in the noise. Grouping by cause is what makes the run legible — one incident, one row, forty affected tests, and any test that failed for a different reason standing out immediately.

What goes into a signature

A signature is a short string computed from a failure such that two failures with the same root cause produce the same string, and two with different causes do not. The fields worth including, in priority order:

The HTTP status, if there was one. A 429 and a 500 are different incidents even if the resulting exception text is identical. Capture this in the test harness — it is not in the traceback.
The provider error type or code. rate_limit_exceeded, overloaded_error, context_length_exceeded and their equivalents are the most discriminating field available, and they are stable strings.
The exception class name. AssertionError versus ValidationError versus a transport timeout is the first split most reports never make.
The assertion site. The last stack frame inside your own code — file and function, not line number, because line numbers move with every edit and would fragment the group across commits.
The served model id. The same assertion failing on two different models is two problems.

What must not go in: the test name, the full message, the request id, or any timestamp. Include the test name and every group has exactly one member, which is where you started.

The trade-off to keep in view is that a signature can be too coarse as easily as too fine. Hashing only the exception class collapses every AssertionError in the suite into one group, which is a single useless row. The test of a good signature is behavioural rather than aesthetic: when you look at the largest group, every member should have the same fix. If two members would be fixed by different people, add a field; if two groups would be fixed by the same change, remove one.

The normalisation rules

The message is where the useful discrimination lives and also where all the entropy is, so it needs normalising before it is hashed. Six substitutions do most of the work, and the order matters — do the specific ones before the general ones.

import re

SUBS = [
    (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-"
                r"[0-9a-f]{4}-[0-9a-f]{12}\b", re.I), "<uuid>"),
    (re.compile(r"\b(req|chatcmpl|msg)_[A-Za-z0-9]{6,}\b"), "<id>"),
    (re.compile(r"\b\d{4}-\d{2}-\d{2}[T ][\d:.]+Z?\b"), "<ts>"),
    (re.compile(r"0x[0-9a-f]+", re.I), "<addr>"),
    (re.compile(r"'[^']{40,}'"), "'<long>'"),   # inlined prompts and completions
    (re.compile(r"\b\d+\b"), "<n>"),          # do this LAST
]

def normalise(message: str) -> str:
    text = message.strip().split("\n")[0][:400]
    for pattern, replacement in SUBS:
        text = pattern.sub(replacement, text)
    return text

Two of those deserve a warning. Replacing every integer with <n> collapses “expected 1 tool call, got 0” and “expected 1 tool call, got 3” into one group, which is usually what you want and occasionally hides a real distinction; run it last so the more specific patterns claim their matches first. And truncating to the first line drops the model’s output from the message, which is the entire point — a completion embedded in an assertion message is unique every time and will defeat any grouping if you leave it in.

A script over your JUnit XML

Every runner worth using emits JUnit XML: pytest with --junitxml, Vitest with --reporter=junit --outputFile, Playwright with its junit reporter. That file is the input, and the standard library can parse it — no dependency required.

Make every job emit JUnit XML to a known path and upload it as an artifact. Do this even for jobs that pass; the pass rows are the denominator of every rate you will later want.
Parse each <testcase> and read its <failure> or <error> child. Where your runner emits the rerun dialect, also read <flakyFailure> and <rerunFailure>: those are the attempts that were retried, and they are the flake signal.
Compute the signature and group. Print the groups largest first.

import hashlib, sys, xml.etree.ElementTree as ET
from collections import defaultdict

FAILURE_TAGS = ("failure", "error", "flakyFailure", "rerunFailure")

def signature(case, node):
    parts = [
        node.tag,
        node.get("type") or "",
        normalise(node.get("message") or ""),
        case.get("classname") or "",
    ]
    raw = "|".join(parts)
    return hashlib.sha1(raw.encode()).hexdigest()[:10], raw

groups = defaultdict(list)
for path in sys.argv[1:]:
    for case in ET.parse(path).iter("testcase"):
        for node in case:
            if node.tag in FAILURE_TAGS:
                key, raw = signature(case, node)
                groups[key].append((case.get("name"), raw))

for key, members in sorted(groups.items(), key=lambda kv: -len(kv[1])):
    print(f"{len(members):4d}  {key}  {members[0][1][:110]}")
    for name, _ in members[:5]:
        print(f"        {name}")
    if len(members) > 5:
        print(f"        ... and {len(members) - 5} more")

The script deliberately reads several files at once, because the unit you want to group over is a CI run, not a job. A suite split across eight parallel shards writes eight XML files, and a rate limit that hit all eight is one incident; grouping per file would report it as eight. Pass the whole artifact directory.

Note that classname is in the signature but the test name is not. Including the class keeps failures in genuinely unrelated modules apart while still collapsing the twenty tests inside one module; drop it entirely if your suite groups everything into one class, since it then contributes nothing.

Using the groups

The output changes what a red build asks of you. A single group with forty members and a rate_limit_exceeded code is one action: reduce the suite’s concurrency, which the locally-green, CI-red page covers. A group of one, with an AssertionError at a specific function in your own code, is the thing you actually have to read.

Persist the signature alongside every attempt rather than computing it only in the moment. Once it is a column, the questions that were hard become trivial: which signature is new this week, which one accounts for most of the suite’s red time, whether the group that spiked on Tuesday has appeared before. That is one more field in the table a flake dashboard already needs, and it is the field that makes the dashboard worth building.

The rerun-related JUnit elements are a Surefire-derived extension, not part of any single standard, and support varies by runner and by CI product. Check what your runner actually emits before relying on <flakyFailure> being present.