TestFly Part 16: Smart Triage & Fuzzy Self-Healing — Zero-Token Flaky-Test Diagnosis

TL;DR: TestFly 1.0.6 adds a local, zero-LLM-cost heuristic layer that runs before any AI is ever consulted: SmartTriageEngine inspects exception hierarchies to classify a failure as SYSTEM_FLAKY, APPLICATION_BUG, or NEEDS_INVESTIGATION in microseconds, and FuzzyHealingEngine repairs a broken locator in under 5ms using plain Levenshtein distance against DOM candidates — no network call, no tokens, no API key. This is distinct from Part 4's LLM-based self-healing (testfly-mcp): that path is the last resort for truly ambiguous locators; the heuristic layer covered here is the first resort that silently handles the 80% of healing and triage cases that don't need a model at all. The PageKnowledge / Auto-POM system ties it together by letting TestFly learn a page's elements once and reuse that knowledge — locally — on every subsequent run.

Why Does Flaky-Test Triage Need a Heuristic Layer at All?

Every CI pipeline eventually accumulates a pile of red builds nobody trusts. The expensive part isn't re-running the test — it's figuring out why it failed: infrastructure hiccup, a real regression, or something that needs a human to look closer. Before TestFly 1.0.6, that triage was manual, or it meant shipping every failure's stack trace to an LLM and paying tokens for the privilege of hearing "this looks like a timeout."

SmartTriageEngine, wired into TestExecutionListener, does this triage synchronously, for free, as part of the failure-reporting path:

package io.testfly.flakiness;

public final class SmartTriageEngine {

    public enum TriageResult {
        APPLICATION_BUG,
        SYSTEM_FLAKY,
        NEEDS_INVESTIGATION
    }

    public static TriageResult triageFailure(Throwable exception, boolean wasRetriedAndPassed) {
        if (exception == null) return TriageResult.NEEDS_INVESTIGATION;

        // Rule 1: Immediate indicator of flakiness
        if (wasRetriedAndPassed) {
            return TriageResult.SYSTEM_FLAKY;
        }

        int flakyScore = 0;
        Throwable rootCause = getRootCause(exception);

        // Rule 2: Exception heuristics
        flakyScore += switch (rootCause) {
            case TimeoutException t -> 80;
            case StaleElementReferenceException s -> 80;
            case ConnectException c -> 90;
            case AssertionError a -> -50; // almost always an application bug
            case WebDriverException w -> 40;
            case null, default -> 0;
        };

        if (flakyScore >= 70) return TriageResult.SYSTEM_FLAKY;
        else if (flakyScore <= 0) return TriageResult.APPLICATION_BUG;
        else return TriageResult.NEEDS_INVESTIGATION;
    }
}

No model call, no prompt, no round trip — just Java 21 pattern-matching switch over the root cause of a (possibly deeply wrapped) exception chain. A ConnectException scores 90 and is immediately flagged SYSTEM_FLAKY; an AssertionError scores -50 and is immediately flagged APPLICATION_BUG, because an assertion failing is almost always a real product bug, not infrastructure noise. Anything that doesn't clear either threshold lands in NEEDS_INVESTIGATION — an honest "I don't know, a human should look" rather than a confident-sounding guess.

TestExecutionListener calls this on every failed test and writes the verdict straight into the report:

if (result.getThrowable() != null) {
    SmartTriageEngine.TriageResult triage =
        SmartTriageEngine.triageFailure(result.getThrowable(), false);

    String triageMsg = "[Triage] " + triage;
    if (triage == SmartTriageEngine.TriageResult.SYSTEM_FLAKY) {
        System.out.println("⚠️ " + triageMsg + " -> This is likely an infrastructure/flaky issue.");
    } else if (triage == SmartTriageEngine.TriageResult.APPLICATION_BUG) {
        System.out.println("🐞 " + triageMsg + " -> This is likely an application bug (Assertion).");
    }
    StepLogger.step(triageMsg, StepStatus.WARN);
}

A test that was retried and passed is flagged SYSTEM_FLAKY immediately — rule 1 short-circuits the whole exception-scoring path, because a pass-on-retry is the strongest possible signal of flakiness, stronger than any exception type.

How Fast Is Fuzzy Self-Healing?

FuzzyHealingEngine is the fallback that runs before TestFly ever asks an LLM to re-find an element. It's pure, allocation-light Java: compute Levenshtein distance between locator clues and attribute/text content of interactive DOM candidates, score them, and pick the best match above a threshold.

package io.testfly.healing;

public final class FuzzyHealingEngine {

    private static final String INTERACTIVE_ELEMENTS =
        "button, a, input, [role='button'], [role='link'], [tabindex='0']";
    private static final int THRESHOLD_SCORE = 5;

    public static WebElement tryFuzzyHeal(WebDriver driver, By originalLocator) {
        String desc = originalLocator.toString().toLowerCase();
        List<String> clues = extractClues(desc);
        if (clues.isEmpty()) return null;

        List<WebElement> candidates = driver.findElements(By.cssSelector(INTERACTIVE_ELEMENTS));

        WebElement bestMatch = null;
        int highestScore = 0;
        for (WebElement el : candidates) {
            int score = scoreElement(el, clues);
            if (score > highestScore) {
                highestScore = score;
                bestMatch = el;
            }
        }
        return highestScore >= THRESHOLD_SCORE ? bestMatch : null;
    }
}

Clue extraction strips locator syntax noise (cssselector, xpath, contains, text, data, testid) and keeps alphanumeric tokens of 3+ characters via a single regex pass. Scoring then weighs exact attribute containment higher than fuzzy text matches: an id match is worth 5 points, a placeholder or value match 4, a CSS class match 3. For element text that doesn't contain the clue outright, the engine allows one typo for short words (≤5 chars) or two for longer ones, using the same getLevenshteinDistance() implementation (classic single-array DP, O(n·m)) that backs the attribute scoring:

public static int getLevenshteinDistance(String a, String b) {
    int[] costs = new int[b.length() + 1];
    for (int j = 0; j < costs.length; j++) costs[j] = j;
    for (int i = 1; i <= a.length(); i++) {
        costs[0] = i;
        int nw = i - 1;
        for (int j = 1; j <= b.length(); j++) {
            int cj = Math.min(1 + Math.min(costs[j], costs[j - 1]),
                    a.charAt(i - 1) == b.charAt(j - 1) ? nw : nw + 1);
            nw = costs[j];
            costs[j] = cj;
        }
    }
    return costs[b.length()];
}

Only elements scoring at or above THRESHOLD_SCORE = 5 are eligible to replace the broken locator — a single weak signal (one 3-point class match, say) isn't enough on its own. This is what makes the pass sub-5ms on a typical DOM: no screenshot, no DOM-to-markup serialization, no network hop — just string comparisons over the already-fetched findElements() candidate list. Only when this engine returns null does TestFly escalate to the LLM-based self-healing covered in Part 4.

What Is Auto-POM / PageKnowledge?

The third piece closes the loop: PageKnowledgeStore persists a LearnedPageModel per URL path to .testfly/page-knowledge.json, so once TestFly (or a human, via recorded runs) discovers an element on a page, every later run — including on CI, as long as the file is restored — can reuse that knowledge instead of re-discovering or re-healing it.

public static Optional<LearnedPageModel> getPageModel(String url) {
    load();
    String key = normalizeUrl(url);
    return Optional.ofNullable(MODELS.get(key));
}

public static void recordElement(String url, LearnedElement element) {
    if (element == null || element.name().isBlank()) return;
    load();
    LearnedPageModel model = getOrCreate(url, null);
    model.putElement(element);
    save();
}

URLs are normalized to their path component (normalizeUrl) so /checkout?order=123 and /checkout?order=456 share the same learned model. On first load, if the store is empty, KnowledgeLearner.learnFromActionCache() bootstraps it from whatever action history already exists — so the cache isn't cold even on a fresh checkout.

LocalIntentResolver is the consumer: given a natural-language goal and the current URL, it tries to resolve the goal entirely from PageKnowledge, before considering an LLM call at all:

public static ActionPlan resolve(String url, String goal) {
    Optional<LearnedPageModel> modelOpt = PageKnowledgeStore.getPageModel(url);
    if (modelOpt.isEmpty()) return null;

    LearnedPageModel model = modelOpt.get();

    ActionPlan formPlan = tryResolveFormInput(model, url, goal);
    if (formPlan != null) return formPlan;

    return tryResolveSingleElement(model, url, goal);
}

It even handles a simple bilingual login pattern locally — keyword-matching "user"/"kullanıcı"/"email"/"posta" and "pass"/"şifre"/"parola" against the goal text plus quoted literal values, then emitting a TYPE → TYPE → CLICK action plan straight from the learned username/password/submit elements, no model inference required. Only a goal the local resolver can't match against the learned model falls through to the LLM agent path.

How Does the Zero-Token Layer Fit With Part 4's LLM Self-Healing?

Think of it as three concentric fallback rings, cheapest first:

1. PageKnowledge / LocalIntentResolver — if the page has been learned and the goal matches a known element or form pattern, resolve locally. No DOM re-scan beyond what's cached, no LLM.
2. FuzzyHealingEngine — if a known locator breaks (renamed class, shifted DOM), score live interactive elements by Levenshtein distance and attribute overlap. Sub-5ms, no LLM.
3. LLM-based self-healing (testfly-mcp, Part 4) — only reached when steps 1 and 2 both fail to produce a confident match. This is where token cost and network latency actually show up, and it's reserved for genuinely ambiguous cases.

SmartTriageEngine sits alongside this as a post-failure classifier rather than a pre-failure healer: it doesn't prevent the failure, it tells you fast — for free — whether the failure you already have is worth a retry, an investigation, or a bug ticket.

FAQ

Does SmartTriageEngine call an LLM to classify failures?

No. It's pure Java: it walks the exception's cause chain to the root, pattern-matches the root's type against a fixed switch expression, and adds/subtracts fixed point values. The only external signal it uses is whether the test passed on retry.

What score does FuzzyHealingEngine require before it heals a locator?

A cumulative score of at least THRESHOLD_SCORE = 5 across all extracted clues. Exact id matches contribute 5, placeholder/value matches 4, class matches 3, and fuzzy text matches (within 1–2 Levenshtein edits depending on clue length) contribute 4. Below 5, the engine returns null and lets the caller escalate.

Does Auto-POM require every page to be manually mapped first?

No — PageKnowledgeStore starts empty and grows via recordElement() as elements are discovered or healed during real runs; on an empty store, KnowledgeLearner.learnFromActionCache() also seeds it from prior recorded actions, so there's no separate "page object generation" step to run.

If FuzzyHealingEngine fails, does TestFly immediately fail the test?

No — a null return from tryFuzzyHeal is the signal to escalate to the LLM-based self-healing path described in Part 4, not a hard failure. The heuristic layer is explicitly a cost-saving first attempt, not a replacement for the AI fallback.