DEV Community: Hakan GÜL

TestFly Part 13: Actionability Engine — Auto-Wait and Verified Click/Fill/Clear Assertions

Hakan GÜL — Mon, 12 Oct 2026 18:53:03 +0000

TL;DR: TestFly 1.1.0 adds an actionability engine that every click, hover, type, clear and append call runs through before touching the DOM: visible, enabled where relevant, geometrically stable, not covered by another element, and scrolled into view if needed. On top of it, assertThat(x).click() / .fill(text) / .clear() turn the action itself into an assertion — they verify the click event actually reached the element and that typed text stayed in the field. A failure throws NotActionableException naming the element and the exact blocker.

How Does TestFly Decide an Element Is Ready for an Action?

Every interaction is tagged with an ActionKind, which decides which checks apply:

public enum ActionKind {
    CLICK(true, true),   // enabled, and not covered by another element
    HOVER(false, true),  // not covered; a disabled element can still be hovered
    EDIT(true, false);   // enabled; being covered doesn't matter for keyboard input
}

Every kind requires the element to be visible and stable regardless of its flags. WaitEngine.waitForActionable(element, kind) polls until all the required checks pass, re-scrolling the element into view if it drifts offscreen between polls:

public static WebElement waitForActionable(WebElement element, ActionKind kind) {
    TestFlyConfig.Actionability cfg = actionabilityConfig();
    if (!cfg.isEnabled()) {
        return kind == ActionKind.CLICK ? waitForClickable(element) : waitForVisible(element);
    }
    // visible, enabled (if required), stable bounding box, not covered (if required)...
}

A stale element bubbles up as StaleElementReferenceException so the caller (the Locator chain) can re-resolve it from scratch instead of failing on a detached node.

What Exactly Gets Checked, and in What Order?

Check	Applies to	Detail
Visible	all kinds	element has a non-empty bounding box and isn't `display:none`/`visibility:hidden`
Enabled	`CLICK`, `EDIT`	skipped for `HOVER` — a disabled button can still be hovered
Stable	all kinds	bounding box unchanged between two consecutive polls (default poll interval: 100ms)
Not covered	`CLICK`, `HOVER`	nothing else receives a hit-test at the element's centre; skipped for `EDIT`
Auto-scroll	all kinds	scrolls the element to the viewport centre if it's offscreen when checked

The stabilityIntervalMs config (default 100ms) is both the polling interval and the minimum time an element's bounding box must stay put — the same anti-animation idea Playwright uses, adapted to Selenium's WebElement.

What Happens When an Element Never Becomes Actionable?

public class NotActionableException extends TimeoutException {
    public NotActionableException(String message, Throwable cause) {
        super(message, cause);
    }
}

It extends TimeoutException, so any existing catch (TimeoutException e) handler keeps working. The message names the element and the specific blocker — for example which overlay is covering the centre point, or that the bounding box kept moving. Nothing silently clicks the wrong spot or types into a half-animated field.

How Do I Configure or Bypass Actionability Checks?

# testfly.yml
actionability:
  enabled: true           # false restores the pre-1.1.0 waits
  stabilityIntervalMs: 100
  hitTargetCheck: true
  autoScroll: true

public static final class Actionability {
    private boolean enabled = true;
    private int stabilityIntervalMs = 100;
    private boolean hitTargetCheck = true;
    private boolean autoScroll = true;
}

Setting actionability.enabled: false falls back to the legacy waits: waitForClickable for clicks, waitForVisible for everything else — useful during a migration, or if a page genuinely needs the old behaviour.

For a single locator, Locator.force() skips just the stability and not-covered checks (Playwright's force: true), while still requiring the element to be visible, and enabled where the action needs it:

find("#banner-close").force().click();

BasePage.doubleClick, rightClick, selectBy* and BaseConditions.click/type all route through the same waitForActionable call, so the behaviour is consistent across the whole API surface.

How Does `assertThat(x).click()` Verify a Click Actually Landed?

A plain Locator.click() performs the actionability wait and clicks — same as before. assertThat(x).click() goes one step further and turns the action into an assertion:

assertThat(loginButton).click();

Resolves the element the same way Locator.click() does, including self-healing, and waits until it's visible and enabled.
Scrolls it to the centre of the viewport and moves the mouse onto it with Actions, like a real user.
Waits until nothing else — a cookie banner, a modal, a sticky header — covers its centre point.
Clicks it, then asserts that the click event actually reached the element or one of its descendants.

On success it logs a passed Clicked: ... step. On failure it's a regular AssertionError (or a recorded soft failure in soft mode) naming which step failed. If the click navigates away or opens an alert before the event listener can read back the result, the click still counts as delivered — it doesn't fail just because the page moved on. Locator.click() itself is unchanged; this is an additive, stricter sibling on LocatorAssert.

How Do `assertThat(x).fill(text)` and `.clear()` Verify the Field Actually Changed?

assertThat(userNameField).fill("admin");
assertThat(passwordField).fill(password);
assertThat(searchField).clear();

Both share one path (fillAndVerify):

Resolve the element like Locator.fill(String) would (waiting, with self-healing), and wait until it's visible and enabled.
Wait until the field is not readonly, then scroll it to the viewport centre.
Clear it and type the text exactly as Locator.type(String) does.
Wait until the field's value equals the text and stays that way — catching cases where maxlength, an input mask, or framework state (React, Vue…) silently changed or reset what was typed.

clear() is the same path with an empty string. For type="password" fields, the value itself is never logged or put in a failure message — only its length is. Fields that reformat input as you type (phone masks, for example) will fail this stricter assertion; use Locator.fill(text) followed by assertThat(x).hasValue(formattedValue) for those instead.

FAQ

Does turning on actionability checks slow every test down?

Each action waits roughly one extra polling interval (100ms by default) compared to the pre-1.1.0 waits, since the engine needs at least two consecutive stable polls to confirm the bounding box stopped moving. Checks fail open when JavaScript is unavailable, so there's no hard dependency on a JS-capable session.

Does `assertThat(x).click()` replace `Locator.click()`?

No. Locator.click() is unchanged and still just performs the actionability wait and clicks. assertThat(x).click() is an additive, verified sibling that also asserts the click event reached the element, logs a Clicked: ... step, and supports as(...), within(...) and soft mode.

Can I combine `not()` with the verified `click()`/`fill()`/`clear()`?

No — these three methods call rejectNegation(...) and throw if not() was chained before them. They assert that an action happened and verifiably changed the page; there's no well-defined "negated click."

What's the difference between `force()` and `actionability.enabled: false`?

force() is per-locator and only skips the stability and not-covered checks — the element must still be visible, and enabled where the action requires it. actionability.enabled: false is global and reverts every action in the suite to the pre-1.1.0 waits (waitForClickable/waitForVisible), dropping all the new checks at once.

TestFly Part 14: Step Traces — Per-Step ARIA/DOM Snapshots for Visual Debugging

Hakan GÜL — Mon, 12 Oct 2026 18:50:44 +0000

TestFly Part 14: Step Traces — Per-Step ARIA/DOM Snapshots for Visual Debugging

TL;DR: TestFly 1.1.0 adds per-step snapshots to the trace: every step now records the
accessibility tree (or, with dom/both, the pruned page HTML) as it looked when the step
started, and a failed test also records the page after its last step with a before/after diff.
New config keys tracing.snapshots, tracing.snapshotMaxKb and tracing.snapshotOnFailureAfter
control what's captured, StepRecord carries the new fields, and TraceRecorder.saveIfEnabled
writes the trace — now from TestNG, JUnit 5, and Cucumber, with the HTML report linking to it
from every test.

What Is a Step Trace, and Why Snapshot Every Step?

A trace is one self-contained HTML file — target/traces/{ClassName}/{testMethod}-trace.html —
that replays a test's step timeline: step names, offsets, screenshots, the final failure
screenshot, the error and stack trace, and the video if recording is on. You open it in any
browser; it needs no server.

Screenshots tell you what the page looked like. They don't tell you what was actually in the
DOM — whether a button was disabled, whether an element had aria-hidden, whether the markup you
expected was even there. That's the gap per-step snapshots close: alongside each step's
screenshot, the trace now embeds a text snapshot of the page exactly as StepSnapshotCapture
saw it the moment the step began.

# testfly.yml
tracing:
  enabled: true

With tracing on, every step gets a snapshot by default — no extra code in the test.

What Does Each Step's Snapshot Contain?

Capture is driven by StepSnapshotCapture.captureForStep(), called once per step. It checks
tracing.enabled, reads the tracing.snapshots mode, and only touches the browser if a driver
session is actually alive — API and database tests, or steps logged before a browser exists,
simply get no snapshot:

// StepSnapshotCapture.java (simplified)
public static Snapshots captureForStep() {
    TestFlyConfig.Tracing tracing = tracing();
    if (tracing == null || !tracing.isEnabled()) {
        return Snapshots.NONE;
    }
    String mode = tracing.getSnapshots();
    boolean aria = mode.equals("aria") || mode.equals("both");
    boolean dom  = mode.equals("dom")  || mode.equals("both");
    WebDriver driver = liveDriver();
    if (driver == null) {
        return Snapshots.NONE;
    }
    int maxKb = tracing.getSnapshotMaxKb();
    return new Snapshots(aria ? captureAria(driver, maxKb) : null,
                          dom  ? captureDom(driver, maxKb)  : null);
}

StepRecord — the object that already held a step's name, offset, status and screenshot — grew
two new fields plus one that's filled in later, for a failed test's final state:

// StepRecord.java
public StepRecord(String name, long offsetMs, String status, String screenshotBase64,
                   String ariaSnapshot, String domSnapshot) { ... }

public String getAriaSnapshot()      { return ariaSnapshot; }
public String getDomSnapshot()       { return domSnapshot; }
public String getAfterAriaSnapshot() { return afterAriaSnapshot; } // set only on failure

aria captures the same YAML text ariaSnapshot() prints — small, readable, unaffected by CSS
class churn. dom captures the page HTML, pruned of scripts, styles and decoration by
DomPruner; it's bigger but shows markup the accessibility tree leaves out, like a CSS class
you're debugging.

How Do You Configure What Gets Captured?

# testfly.yml
tracing:
  enabled: true
  snapshots: both             # none | aria | dom | both — default: aria
  snapshotMaxKb: 256           # longest snapshot kept per step, in KB — default: 256
  snapshotOnFailureAfter: true # also snapshot the page after the last step of a failed test

TestFlyConfig validates tracing.snapshots eagerly — anything other than none, aria, dom
or both throws IllegalArgumentException at config-load time, not mid-run. A snapshot longer
than snapshotMaxKb is cut and ends with … truncated (N more characters), so one verbose page
never bloats the whole trace file.

What Does the Trace HTML Actually Show You?

Click a step in the timeline and the screenshot panel opens with tabs: ARIA (before step),
DOM (before step), and — only on the last step of a failed test — ARIA (after failure) and a
computed Diff tab. The diff is a plain line-based set difference between the before and after
YAML, rendered client-side in the trace's inlined JavaScript (lineDiff() in TraceRecorder):
lines only in the "before" snapshot are marked -, lines only in "after" are marked +. No
external diff library, no network call — the trace is one file you can email or drop in Slack.

Step blocks in the timeline also get a small 🌳 marker when they carry a snapshot, next to the
📷 marker for a screenshot, so you can see at a glance which steps have something to click into.

Does This Slow Down or Risk Failing My Tests?

Capturing a snapshot is one script call per step — a few milliseconds on a typical page — and it
only runs when tracing.enabled is on. StepSnapshotCapture wraps every capture in a
try/catch that returns null on failure rather than propagating: a page that can't be read, a
closed driver, or a flaky script call never fails the test, it just leaves that step without a
snapshot. TraceRecorder.saveIfEnabled itself is wrapped the same way:

// TraceRecorder.java
@TestFlyApi(since = "1.1.0")
public static void saveIfEnabled(String testId, String testName, boolean isPassing) {
    try {
        TestFlyConfig.Tracing tracing = TestFlyContext.getConfig().getTracing();
        if (tracing == null || !tracing.isEnabled()) return;
        if (isPassing && !tracing.isCaptureOnPass()) return;
        save(testId, testName);
    } catch (Exception ignored) {
    }
}

Does This Work With Cucumber?

Yes — as of 1.1.0, CucumberHooks calls the same trace machinery TestNG and JUnit 5 use, so
Gherkin scenarios get a trace file too, snapshots included, with no change to your feature files
or step definitions. The HTML test report also links directly to each test's trace from its
detail panel, so you go from a red row in the report to the exact step and snapshot that shows
what went wrong, in one click.

FAQ

Do I need to add code to my tests to get per-step snapshots?

No. With tracing.enabled: true, every step captures a snapshot automatically according to
tracing.snapshots (default aria). There's no per-step API to call.

What happens if a test has no browser, like an API test?

Nothing is captured. StepSnapshotCapture checks for a live WebDriver session before taking a
snapshot and returns no snapshot if none exists — it never creates or recreates a driver just to
capture one.

Can a snapshot capture failure cause my test to fail?

No. Every capture path is wrapped so an exception is swallowed and logged at FINE level, never
rethrown. A snapshot is best-effort; it never affects the pass/fail verdict.

Is the diff tab a real structural diff of the accessibility tree?

No — it's a line-based set difference between the "before" and "after" ARIA YAML text, computed
in the trace's own inline JavaScript. It's enough to spot what appeared or disappeared, not a
semantic tree diff.

TestFly Part 16: Smart Triage & Fuzzy Self-Healing — Zero-Token Flaky-Test Diagnosis

Hakan GÜL — Mon, 12 Oct 2026 18:50:22 +0000

TestFly Part 16: Smart Triage & Fuzzy Self-Healing — Zero-Token Flaky-Test Diagnosis

TL;DR: TestFly 1.0.6 adds a local, zero-LLM-cost heuristic layer that runs before any AI is ever consulted: SmartTriageEngine inspects exception hierarchies to classify a failure as SYSTEM_FLAKY, APPLICATION_BUG, or NEEDS_INVESTIGATION in microseconds, and FuzzyHealingEngine repairs a broken locator in under 5ms using plain Levenshtein distance against DOM candidates — no network call, no tokens, no API key. This is distinct from Part 4's LLM-based self-healing (testfly-mcp): that path is the last resort for truly ambiguous locators; the heuristic layer covered here is the first resort that silently handles the 80% of healing and triage cases that don't need a model at all. The PageKnowledge / Auto-POM system ties it together by letting TestFly learn a page's elements once and reuse that knowledge — locally — on every subsequent run.

Why Does Flaky-Test Triage Need a Heuristic Layer at All?

Every CI pipeline eventually accumulates a pile of red builds nobody trusts. The expensive part isn't re-running the test — it's figuring out why it failed: infrastructure hiccup, a real regression, or something that needs a human to look closer. Before TestFly 1.0.6, that triage was manual, or it meant shipping every failure's stack trace to an LLM and paying tokens for the privilege of hearing "this looks like a timeout."

SmartTriageEngine, wired into TestExecutionListener, does this triage synchronously, for free, as part of the failure-reporting path:

package io.testfly.flakiness;

public final class SmartTriageEngine {

    public enum TriageResult {
        APPLICATION_BUG,
        SYSTEM_FLAKY,
        NEEDS_INVESTIGATION
    }

    public static TriageResult triageFailure(Throwable exception, boolean wasRetriedAndPassed) {
        if (exception == null) return TriageResult.NEEDS_INVESTIGATION;

        // Rule 1: Immediate indicator of flakiness
        if (wasRetriedAndPassed) {
            return TriageResult.SYSTEM_FLAKY;
        }

        int flakyScore = 0;
        Throwable rootCause = getRootCause(exception);

        // Rule 2: Exception heuristics
        flakyScore += switch (rootCause) {
            case TimeoutException t -> 80;
            case StaleElementReferenceException s -> 80;
            case ConnectException c -> 90;
            case AssertionError a -> -50; // almost always an application bug
            case WebDriverException w -> 40;
            case null, default -> 0;
        };

        if (flakyScore >= 70) return TriageResult.SYSTEM_FLAKY;
        else if (flakyScore <= 0) return TriageResult.APPLICATION_BUG;
        else return TriageResult.NEEDS_INVESTIGATION;
    }
}

No model call, no prompt, no round trip — just Java 21 pattern-matching switch over the root cause of a (possibly deeply wrapped) exception chain. A ConnectException scores 90 and is immediately flagged SYSTEM_FLAKY; an AssertionError scores -50 and is immediately flagged APPLICATION_BUG, because an assertion failing is almost always a real product bug, not infrastructure noise. Anything that doesn't clear either threshold lands in NEEDS_INVESTIGATION — an honest "I don't know, a human should look" rather than a confident-sounding guess.

TestExecutionListener calls this on every failed test and writes the verdict straight into the report:

if (result.getThrowable() != null) {
    SmartTriageEngine.TriageResult triage =
        SmartTriageEngine.triageFailure(result.getThrowable(), false);

    String triageMsg = "[Triage] " + triage;
    if (triage == SmartTriageEngine.TriageResult.SYSTEM_FLAKY) {
        System.out.println("⚠️ " + triageMsg + " -> This is likely an infrastructure/flaky issue.");
    } else if (triage == SmartTriageEngine.TriageResult.APPLICATION_BUG) {
        System.out.println("🐞 " + triageMsg + " -> This is likely an application bug (Assertion).");
    }
    StepLogger.step(triageMsg, StepStatus.WARN);
}

A test that was retried and passed is flagged SYSTEM_FLAKY immediately — rule 1 short-circuits the whole exception-scoring path, because a pass-on-retry is the strongest possible signal of flakiness, stronger than any exception type.

How Fast Is Fuzzy Self-Healing?

FuzzyHealingEngine is the fallback that runs before TestFly ever asks an LLM to re-find an element. It's pure, allocation-light Java: compute Levenshtein distance between locator clues and attribute/text content of interactive DOM candidates, score them, and pick the best match above a threshold.

package io.testfly.healing;

public final class FuzzyHealingEngine {

    private static final String INTERACTIVE_ELEMENTS =
        "button, a, input, [role='button'], [role='link'], [tabindex='0']";
    private static final int THRESHOLD_SCORE = 5;

    public static WebElement tryFuzzyHeal(WebDriver driver, By originalLocator) {
        String desc = originalLocator.toString().toLowerCase();
        List<String> clues = extractClues(desc);
        if (clues.isEmpty()) return null;

        List<WebElement> candidates = driver.findElements(By.cssSelector(INTERACTIVE_ELEMENTS));

        WebElement bestMatch = null;
        int highestScore = 0;
        for (WebElement el : candidates) {
            int score = scoreElement(el, clues);
            if (score > highestScore) {
                highestScore = score;
                bestMatch = el;
            }
        }
        return highestScore >= THRESHOLD_SCORE ? bestMatch : null;
    }
}

Clue extraction strips locator syntax noise (cssselector, xpath, contains, text, data, testid) and keeps alphanumeric tokens of 3+ characters via a single regex pass. Scoring then weighs exact attribute containment higher than fuzzy text matches: an id match is worth 5 points, a placeholder or value match 4, a CSS class match 3. For element text that doesn't contain the clue outright, the engine allows one typo for short words (≤5 chars) or two for longer ones, using the same getLevenshteinDistance() implementation (classic single-array DP, O(n·m)) that backs the attribute scoring:

public static int getLevenshteinDistance(String a, String b) {
    int[] costs = new int[b.length() + 1];
    for (int j = 0; j < costs.length; j++) costs[j] = j;
    for (int i = 1; i <= a.length(); i++) {
        costs[0] = i;
        int nw = i - 1;
        for (int j = 1; j <= b.length(); j++) {
            int cj = Math.min(1 + Math.min(costs[j], costs[j - 1]),
                    a.charAt(i - 1) == b.charAt(j - 1) ? nw : nw + 1);
            nw = costs[j];
            costs[j] = cj;
        }
    }
    return costs[b.length()];
}

Only elements scoring at or above THRESHOLD_SCORE = 5 are eligible to replace the broken locator — a single weak signal (one 3-point class match, say) isn't enough on its own. This is what makes the pass sub-5ms on a typical DOM: no screenshot, no DOM-to-markup serialization, no network hop — just string comparisons over the already-fetched findElements() candidate list. Only when this engine returns null does TestFly escalate to the LLM-based self-healing covered in Part 4.

What Is Auto-POM / PageKnowledge?

The third piece closes the loop: PageKnowledgeStore persists a LearnedPageModel per URL path to .testfly/page-knowledge.json, so once TestFly (or a human, via recorded runs) discovers an element on a page, every later run — including on CI, as long as the file is restored — can reuse that knowledge instead of re-discovering or re-healing it.

public static Optional<LearnedPageModel> getPageModel(String url) {
    load();
    String key = normalizeUrl(url);
    return Optional.ofNullable(MODELS.get(key));
}

public static void recordElement(String url, LearnedElement element) {
    if (element == null || element.name().isBlank()) return;
    load();
    LearnedPageModel model = getOrCreate(url, null);
    model.putElement(element);
    save();
}

URLs are normalized to their path component (normalizeUrl) so /checkout?order=123 and /checkout?order=456 share the same learned model. On first load, if the store is empty, KnowledgeLearner.learnFromActionCache() bootstraps it from whatever action history already exists — so the cache isn't cold even on a fresh checkout.

LocalIntentResolver is the consumer: given a natural-language goal and the current URL, it tries to resolve the goal entirely from PageKnowledge, before considering an LLM call at all:

public static ActionPlan resolve(String url, String goal) {
    Optional<LearnedPageModel> modelOpt = PageKnowledgeStore.getPageModel(url);
    if (modelOpt.isEmpty()) return null;

    LearnedPageModel model = modelOpt.get();

    ActionPlan formPlan = tryResolveFormInput(model, url, goal);
    if (formPlan != null) return formPlan;

    return tryResolveSingleElement(model, url, goal);
}

It even handles a simple bilingual login pattern locally — keyword-matching "user"/"kullanıcı"/"email"/"posta" and "pass"/"şifre"/"parola" against the goal text plus quoted literal values, then emitting a TYPE → TYPE → CLICK action plan straight from the learned username/password/submit elements, no model inference required. Only a goal the local resolver can't match against the learned model falls through to the LLM agent path.

How Does the Zero-Token Layer Fit With Part 4's LLM Self-Healing?

Think of it as three concentric fallback rings, cheapest first:

PageKnowledge / LocalIntentResolver — if the page has been learned and the goal matches a known element or form pattern, resolve locally. No DOM re-scan beyond what's cached, no LLM.
FuzzyHealingEngine — if a known locator breaks (renamed class, shifted DOM), score live interactive elements by Levenshtein distance and attribute overlap. Sub-5ms, no LLM.
LLM-based self-healing (testfly-mcp, Part 4) — only reached when steps 1 and 2 both fail to produce a confident match. This is where token cost and network latency actually show up, and it's reserved for genuinely ambiguous cases.

SmartTriageEngine sits alongside this as a post-failure classifier rather than a pre-failure healer: it doesn't prevent the failure, it tells you fast — for free — whether the failure you already have is worth a retry, an investigation, or a bug ticket.

FAQ

Does SmartTriageEngine call an LLM to classify failures?

No. It's pure Java: it walks the exception's cause chain to the root, pattern-matches the root's type against a fixed switch expression, and adds/subtracts fixed point values. The only external signal it uses is whether the test passed on retry.

What score does FuzzyHealingEngine require before it heals a locator?

A cumulative score of at least THRESHOLD_SCORE = 5 across all extracted clues. Exact id matches contribute 5, placeholder/value matches 4, class matches 3, and fuzzy text matches (within 1–2 Levenshtein edits depending on clue length) contribute 4. Below 5, the engine returns null and lets the caller escalate.

Does Auto-POM require every page to be manually mapped first?

No — PageKnowledgeStore starts empty and grows via recordElement() as elements are discovered or healed during real runs; on an empty store, KnowledgeLearner.learnFromActionCache() also seeds it from prior recorded actions, so there's no separate "page object generation" step to run.

If FuzzyHealingEngine fails, does TestFly immediately fail the test?

No — a null return from tryFuzzyHeal is the signal to escalate to the LLM-based self-healing path described in Part 4, not a hard failure. The heuristic layer is explicitly a cost-saving first attempt, not a replacement for the AI fallback.

TestFly Part 15: Enterprise API Auth — RFC 7616 Digest, HMAC Signing & OAuth2 Refresh Tokens

Hakan GÜL — Mon, 12 Oct 2026 18:49:20 +0000

TL;DR: TestFly 1.1.0 replaces its old, non-functional "Digest" auth with a real RFC 7616 challenge/response implementation (MD5, SHA-256, SHA-512-256, -sess, qop=auth/auth-int), adds ApiAuth.hmacSigner() to sign the method, path+query, timestamp and body hash of every request, and adds ApiAuth.oauth2Refresh(...) for automatic refresh-token grants. A new ApiAuth.Intercepting interface lets these strategies run as the innermost interceptor so they see the exact request and response, and @UseAuth now accepts digest, hmac-signed and oauth2-refresh straight from testfly.yml.

Why Did TestFly's Digest Auth Need a Rewrite?

Before 1.1.0, ApiAuth.digest(username, password) did something that looked plausible but was simply wrong: it sent Authorization: Digest base64(user:password) on every request. No RFC 7616-compliant server accepts that — Digest auth isn't Basic auth with a different scheme name, it's a cryptographic challenge/response built from a nonce the server issues on a 401. Any test suite exercising a real Digest-protected API with the old code would fail at the auth layer, not the API layer.

The fix required building the actual protocol: parse the WWW-Authenticate header, compute HA1/HA2 per RFC 7616, track nonce counts, and know when a nonce has gone stale.

How Does TestFly Implement RFC 7616 Digest Authentication?

ApiAuth.digest(username, password) returns an ApiAuth.Intercepting implementation (DigestAuth) that is installed as the innermost interceptor — the one that actually sees the request going out and the response coming back, so it can resend a request after a challenge:

ApiClient.setGlobalAuth(ApiAuth.digest("qa-user", System.getenv("DIGEST_PASSWORD")));

apiClient().get("/api/secure/orders")
        .send()
        .assertStatus(200);

The flow on the wire:

The first request to an origin goes out without credentials.
A 401 WWW-Authenticate: Digest ... response is parsed into a Challenge (realm, nonce, opaque, algorithm, qop set, stale flag). When a server advertises several schemes, TestFly picks the strongest supported algorithm (SHA-512-256 > SHA-256 > MD5).
The request is resent once with a computed Authorization: Digest ... header.
The challenge is cached per origin for the current test (ApiClient.challengeCache()), shared across sendAsync calls, so later requests authenticate preemptively with an incrementing nonce count (nc) instead of round-tripping a 401 every time.
A stale nonce (server returns stale=true, or a new nonce on a later 401) triggers exactly one re-authentication with the fresh challenge. If the same nonce comes back unstaled, TestFly treats that as rejected credentials and returns the 401 to the caller instead of looping.

Supported algorithms are MD5, MD5-sess, SHA-256, SHA-256-sess, SHA-512-256 and SHA-512-256-sess, with qop=auth and qop=auth-int (which folds a hash of the request body into HA2). The core of the response hash, straight from the implementation:

String ha1 = hash(digestName, username + ":" + challenge.realm + ":" + password);
if (session) ha1 = hash(digestName, ha1 + ":" + challenge.nonce + ":" + cnonce);

String a2 = request.method() + ":" + digestUri;
if ("auth-int".equals(qop)) a2 += ":" + hash(digestName, request.body());
String ha2 = hash(digestName, a2);

String response = qop == null
        ? hash(digestName, ha1 + ":" + challenge.nonce + ":" + ha2)
        : hash(digestName, ha1 + ":" + challenge.nonce + ":" + nc + ":" + cnonce + ":" + qop + ":" + ha2);

-sess variants fold the client nonce into HA1 once per challenge, exactly as RFC 7616 specifies, instead of recomputing it on every request.

How Do You Sign Requests with HMAC?

Plenty of internal and partner APIs don't want Basic/Bearer at all — they want every request signed so a captured header can't be replayed against a different payload. TestFly 1.1.0 adds ApiAuth.hmacSigner(), a builder for ApiHmacSigner, another ApiAuth.Intercepting strategy:

ApiAuth signer = ApiAuth.hmacSigner()
        .apiKey(System.getenv("API_KEY"))
        .secret(System.getenv("API_SECRET"))
        .signatureHeader("X-Signature")
        .encoding(ApiHmacSigner.Encoding.HEX)
        .build();

ApiClient.setGlobalAuth(signer);

By default it signs a canonical string built from the method, path+query, a timestamp and a SHA-256 hash of the body — joined by newlines:

public static String defaultCanonical(SigningContext ctx) {
    return ctx.method() + "\n" + ctx.pathAndQuery() + "\n" + ctx.timestamp() + "\n" + ctx.bodySha256();
}

Everything about the signature is configurable through the builder: algorithm(...) (any JCA Mac algorithm, default HmacSHA256), keyHeader(...) / signatureHeader(...) / timestampHeader(...) (set any to null to omit it), encoding(...) (BASE64 or HEX), timestampFormat(...) (default epoch seconds), and canonical(...) to replace the whole signing string with a Function<SigningContext, String> for a server that expects, say, a different line order or an additional nonce. Because the signer runs as the innermost interceptor, it signs the exact request bytes about to hit the wire, including headers added by other ApiAuth strategies earlier in the chain — but not headers added afterward by legacy RequestInterceptor hooks, which run outside it.

The old ApiAuth.hmac(apiKey, secret, algorithm) is now deprecated: it only ever signed the API key string, never the request itself, so it can't match a server that verifies request signatures. It's kept for wire compatibility, but new code should use hmacSigner().

How Does OAuth2 Token Refresh Work?

ApiAuth.oauth2(...) already handled client-credentials tokens with caching; 1.1.0 adds ApiAuth.oauth2Refresh(tokenUrl, clientId, clientSecret, refreshToken) for flows where a long-lived refresh token exchanges for short-lived access tokens:

ApiClient.setGlobalAuth(ApiAuth.oauth2Refresh(
        "https://auth.example.com/token",
        System.getenv("CLIENT_ID"),
        System.getenv("CLIENT_SECRET"),
        System.getenv("REFRESH_TOKEN")));

The process-wide OAuth2TokenCache is shared by all three OAuth2 factories (oauth2, oauth2Password, oauth2Refresh) and is keyed so a client-credentials token, a password token and a refresh token for the same tokenUrl/clientId never collide. On expiry it first tries a refresh_token grant with whatever refresh token it has cached; if that grant fails outright, it falls back to re-running the original grant rather than failing the test:

if (cached != null && cached.refreshToken != null) {
    try {
        fresh = refresh(tokenUrl, clientId, clientSecret, cached.refreshToken);
    } catch (RuntimeException refreshFailed) {
        fresh = null; // fall back to the original grant
    }
}
if (fresh == null) fresh = grant.get();

Servers that rotate refresh tokens on every use are handled too: if the token response includes a new refresh_token, that's what gets cached for the next refresh; servers that omit it (no rotation) keep the one TestFly already has. ApiAuth.clearTokenCache() clears every cached token on demand, and the framework calls it automatically at suite end so tokens never leak between suites in the same JVM.

How Do You Wire This Up Declaratively with @UseAuth?

All three strategies are now first-class @UseAuth types, resolved from testfly.yml by the shared AuthStrategies class used by both the TestNG listener and the JUnit 5 extension:

# testfly.yml
api:
  auth:
    legacyPartner:
      type: digest
      username: ${DIGEST_USER}
      password: ${DIGEST_PASSWORD}
    signedService:
      type: hmac-signed
      apiKey: ${API_KEY}
      secret: ${API_SECRET}
      algorithm: HmacSHA256
      keyHeader: X-Api-Key
      signatureHeader: X-Signature
      timestampHeader: X-Timestamp
      encoding: base64
    refreshableAccount:
      type: oauth2-refresh
      tokenUrl: https://auth.example.com/token
      clientId: ${CLIENT_ID}
      clientSecret: ${CLIENT_SECRET}
      refreshToken: ${REFRESH_TOKEN}

@Test
@UseAuth("signedService")
public void createOrderIsSigned() {
    apiClient().post("/api/orders").body(order).send().assertStatus(201);
}

Setting keyHeader: none or timestampHeader: none (or leaving a header blank) omits that header entirely — useful when a server only wants the signature header and derives the key from an upstream gateway. The TestNG listener and JUnit 5 extension resolve @UseAuth through the exact same code path, so there's no divergence in behavior between the two runners.

What Else Changed Around API Auth in 1.1.0?

The ApiAuth.Intercepting interface itself is new infrastructure: any strategy that needs the final request bytes or the actual response — not just a header to attach before the request is built — implements it and ApiClient installs it as the innermost ApiInterceptor, closest to the transport. That's what lets DigestAuth resend a request after reading a 401, and lets ApiHmacSigner sign the request as it will actually leave the process. Both ship as package-private or final implementations; you reach them only through ApiAuth.digest(...) and ApiAuth.hmacSigner().

Two related fixes landed in the same release: ApiAuth.basicAuth now always encodes credentials as UTF-8 (previously platform-default), and the OAuth2 token cache is cleared at suite end instead of persisting for the life of the JVM — so a second suite in the same run can't silently reuse another suite's token.

FAQ

Does ApiAuth.digest(...) make two HTTP calls per request?
Only the first request to a given origin pays that cost. After the first 401/challenge round trip, the challenge is cached per origin for the current test, and later requests send a valid Authorization: Digest header preemptively with an incremented nonce count — no second request unless the nonce goes stale or rotates.

What happens if the Digest credentials are actually wrong?
TestFly resends once per fresh challenge. If the server returns the same non-stale nonce again, that's treated as a rejection of the credentials, not a recoverable challenge — the 401 is returned to the caller instead of retrying forever.

Can I use a custom HMAC canonical string instead of the default one?
Yes — ApiAuth.hmacSigner().canonical(ctx -> ...) takes a Function<SigningContext, String> with access to method, URI, timestamp, raw body bytes and headers, so you can match any server-side signing scheme, not just TestFly's default METHOD\npath?query\ntimestamp\nbodyHash.

Does oauth2Refresh break if the server never rotates refresh tokens?
No. If a refresh response omits refresh_token, TestFly keeps reusing the refresh token you supplied; it only swaps to a new one when the server actually returns one.

TestFly is a Java 21 Selenium/API test automation SDK. Previous entries in this series: Part 12 — Playwright-style assertions, Part 13 — actionability, Part 14 — step traces.

TestFly Part 17: Cross-Browser Reliability — BiDi Console Capture, Edge/Safari on Grid & Guaranteed Driver Cleanup

Hakan GÜL — Mon, 12 Oct 2026 18:49:15 +0000

TL;DR: TestFly 1.0.6 and 1.1.0 close three cross-browser reliability gaps: browser.bidi opens a real WebDriver BiDi connection to a locally launched Chrome, Edge or Firefox so ConsoleErrorCollector reads console.error calls and uncaught exceptions from BiDi log events instead of a JS shim — catching errors Firefox used to hide — and ConsoleErrorCollector.isBiDiActive() tells you which capture path ran. RemoteDriverProvider now resolves Edge and Safari on Selenium Grid and honors browser.matrix, where it previously only knew chrome/firefox and always read browser.name. And JVM shutdown hooks plus DriverManager.quitAllTrackedDrivers() guarantee every tracked browser — on every thread, not just the hook thread's own — is quit once when a run is aborted with Ctrl+C or System.exit.

How Does BiDi Catch Console Errors Firefox Used to Miss?

TestFly's captureConsoleErrors has always read browser console errors, but on Firefox — which exposes no browser logs over classic WebDriver — that meant injecting a JavaScript shim that only sees errors after it runs on the current page. An error thrown while the page is still loading, or on a page reached by following a link, was invisible to the shim.

browser.bidi (default false) fixes this at the protocol level: when a Chrome, Edge or Firefox session is launched locally, TestFly asks for a real WebDriver BiDi connection and subscribes to BiDi log events from the moment the test gets its browser:

# testfly.yml
browser:
  name: firefox
  bidi: true
  captureConsoleErrors: true

The capability request is one line, added only when BiDi is configured on:

// BiDiCapability.java
static void applyIfConfigured(MutableCapabilities options, TestFlyConfig config) {
    if (config != null && config.getBrowser() != null && config.getBrowser().isBidi()) {
        options.setCapability("webSocketUrl", true);
    }
}

ConsoleErrorCollector then opens a LogInspector over that connection per session, keyed in a WeakHashMap<WebDriver, BiDiCapture> so sessions are never pinned in memory after the driver is gone:

private static final Map<WebDriver, BiDiCapture> BIDI =
        Collections.synchronizedMap(new WeakHashMap<>());

Because the subscription starts at beginTest(driver), errors from a previous test are forgotten even on a reused, per-suite browser — no JS shim, no page-load race.

Safari and remote drivers (Selenium Grid, BrowserStack, Sauce Labs) have no BiDi connection in TestFly today. In that case one warning is logged per session and the framework falls back to the usual capture path — nothing throws.

To know which path actually ran, without reading logs:

if (ConsoleErrorCollector.isBiDiActive()) {
    // BiDi log events are driving capture for this session
}

// ConsoleErrorCollector.java
@TestFlyApi(since = "1.1.0")
public static boolean isBiDiActive() {
    try {
        return BIDI.containsKey(DriverManager.getDriver());
    } catch (RuntimeException e) {
        return false;
    }
}

How Do You Know Which Browser Features You're Trading Away?

A locally started Chrome or Edge gives TestFly the Chrome DevTools Protocol (CDP), which powers network mocking, device emulation, geolocation overrides, and the screencast-based video recorder. Firefox, Safari, Selenium Grid and cloud sessions don't expose CDP — even a remote Chrome doesn't — so those features degrade silently unless something says so.

BrowserFeatures (new in 1.1.0) names the four CDP-dependent features and what happens instead:

public enum Feature {
    NETWORK_MOCK("Network interception requires a local Chrome/Edge session... "
            + "routes are ignored and requests reach the real network."),
    CDP_RECORDING("Video recording uses periodic screenshots instead of the "
            + "DevTools screencast, so frames are less frequent."),
    DEVICE_EMULATION("Device emulation only resizes the window and overrides "
            + "navigator.userAgent in the current page..."),
    GEOLOCATION("The location is overridden through JavaScript in the current "
            + "page only and is lost on navigation.");
}

BrowserFeatures.supports(driver, feature) answers the question directly (driver instanceof ChromiumDriver for three of the four, HasDevTools for recording), and warnOnce(driver, feature) logs the fallback exactly once per browser session per feature, naming the browser:

BrowserFeatures.warnOnce(driver, BrowserFeatures.Feature.GEOLOCATION);

GeoLocation already calls this before falling back to its JS override on Firefox. The video recording fallback used to be silent; now every degraded feature says so.

How Do You Run Edge and Safari on Selenium Grid?

Before this release, RemoteDriverProvider only built ChromeOptions or FirefoxOptions and always read browser.name directly — so a browser.matrix run on a Grid used whatever browser.name happened to say, ignoring the matrix browser entirely, and Edge/Safari on Grid wasn't supported at all.

Browser resolution now goes through one shared method that both the local and remote paths call, BrowserContext (the matrix override) winning over YAML:

// DriverProviderFactory.java
public static String resolveBrowserName(TestFlyConfig config) {
    String contextBrowser = BrowserContext.get();
    if (contextBrowser != null && !contextBrowser.isBlank()) {
        return contextBrowser;
    }
    String name = config.getBrowser().getName();
    if (name == null || name.isBlank()) {
        throw new IllegalStateException(
                "No browser selected: set browser.name, or run through browser.matrix.");
    }
    return name;
}

RemoteDriverProvider.buildOptions(config, browser) is a public, independently testable method that now switches over all four Grid-supported browsers:

// RemoteDriverProvider.java
switch (browser.toLowerCase(Locale.ROOT)) {
    case "chrome" -> options = chromium(new ChromeOptions(), browserConfig.isHeadless(), arguments);
    case "edge" -> options = chromium(new EdgeOptions(), browserConfig.isHeadless(), arguments);
    case "firefox" -> { /* FirefoxOptions, -headless */ }
    case "safari" -> {
        if (browserConfig.isHeadless()) {
            System.err.println("[TestFly] Safari does not support headless mode via Selenium; ignoring headless=true.");
        }
        options = new SafariOptions();
    }
    default -> throw new IllegalStateException(
            "Unsupported browser for remote execution: " + browser + ". Supported: chrome, edge, firefox, safari.");
}
options.setCapability("unhandledPromptBehavior", "ignore");

A matrix config that runs the same suite across Chrome, Edge and Safari on a Grid now works unmodified:

# testfly.yml
execution:
  mode: remote
  gridUrl: http://grid.local:4444/wd/hub
browser:
  matrix: [chrome, edge, safari]

A companion fix (do not reuse a per-suite driver for another matrix browser) stops DriverManager from handing a Chrome per-suite driver to a thread that's supposed to be testing Edge — DRIVER_BROWSER_CONTEXT is now tracked per thread and checked before reuse. Another (quit the browser when post-launch session setup fails) makes every built-in provider quit the session before rethrowing when timeout configuration throws right after launch, so a setup failure no longer leaks a browser process.

How Does TestFly Guarantee Browsers Are Never Left Running?

DriverManager has registered a JVM shutdown hook since early on, but until this release it only quit per-suite drivers and whatever happened to be bound to the shutdown hook's own thread — which is never the thread that actually ran a per-test driver. Abort a parallel per-test run with Ctrl+C or System.exit, and worker-thread browsers kept running after the JVM exited.

The hook now calls quitAllTrackedDrivers(), which unions every driver that still holds a session permit with every registered per-suite driver, across all threads, and quits each exactly once:

static void quitAllTrackedDrivers() {
    java.util.Set<WebDriver> drivers =
            java.util.Collections.newSetFromMap(new java.util.IdentityHashMap<>());
    drivers.addAll(PERMIT_HOLDERS);
    drivers.addAll(SUITE_DRIVERS);
    for (WebDriver driver : drivers) {
        try {
            driver.quit();
        } catch (Throwable ignored) {
        } finally {
            SUITE_DRIVERS.remove(driver);
            releasePermitOf(driver);
        }
    }
}

PERMIT_HOLDERS is the set of drivers currently holding a session-slot permit — populated and cleared by exactly one code path per driver, which is also what makes execution.maxActiveSessions leak-proof. Normal suite teardown still calls DriverManager.quitAllSuiteDrivers() then quitDriver() from SuiteExecutionListener, TestFlyExtension (JUnit 5) and CucumberHooks; the shutdown hook is the safety net for abnormal exits, firing identical cleanup logic no matter which thread owned the browser.

This traces back to 1.0.6's original "Guaranteed Driver Cleanup" line (shutdown hooks plus @AfterAll listener execution): the hook existed from 1.0.6, this release makes it actually reach every thread's browser instead of only the suite-level ones.

FAQ

Does `browser.bidi` work with Selenium Grid or BrowserStack/Sauce Labs?

No. BiDi only applies to a locally launched Chrome, Edge or Firefox. On Safari or any remote driver, TestFly logs one warning and falls back to the previous capture path (CDP logs on Chrome, the JS shim elsewhere); nothing fails.

How do I check which console-error capture path is active for the current test?

Call ConsoleErrorCollector.isBiDiActive(). It returns true only when the current thread's driver has an open BiDi log subscription; it never throws, returning false if there's no driver or the check fails.

Can I run the same suite on Chrome, Edge and Safari against one Selenium Grid?

Yes, with execution.mode: remote, execution.gridUrl and browser.matrix: [chrome, edge, safari]. RemoteDriverProvider resolves the matrix browser ahead of browser.name and builds Grid-appropriate options for all four supported browsers, including Safari's headless restriction.

If I kill a parallel run with Ctrl+C, are all browser processes definitely gone?

Yes. The JVM shutdown hook now collects every driver that holds a session permit plus every per-suite driver, on every thread, and quits each one exactly once, regardless of which thread the hook itself runs on.

TestFly Part 12: Playwright-Style Assertions in Selenium — not(), ARIA Snapshots & Strict Mode

Hakan GÜL — Mon, 12 Oct 2026 18:40:51 +0000

TL;DR: TestFly 1.1.0 closes the gap with Playwright for Java's assertion API without leaving Selenium: LocatorAssert.not() negates the next check, hasRole() / hasAccessibleName() / hasAccessibleDescription() read the real accessibility tree instead of guessing from DOM attributes, matchesAriaSnapshot() compares a subtree against a Playwright-style YAML template, and opt-in locators.strict turns an ambiguous locator into a loud failure instead of a silent click on the wrong element.

Why Do Selenium Assertions Need a not() Method?

Negative assertions in classic Selenium/TestNG code usually mean one of two bad options: wrap the positive assertion in a try/catch and invert the result, or write a separate polling loop that waits for a condition to stay false. Both bury the real intent of the test in boilerplate.

TestFly's LocatorAssert now carries the same not() Playwright for Java has — it negates only the assertion directly after it in the chain:

import io.testfly.test.BaseTest;
import org.testng.annotations.Test;

import static io.testfly.assertion.Assertions.assertThat;

public class CheckoutErrorTest extends BaseTest {

    @Test
    public void testErrorBannerClearsAfterRetry() {
        open("/checkout");
        find("#pay-btn").click();

        assertThat(find("#error-banner")).hasText("Payment declined");

        find("#retry-btn").click();

        // Only the assertion right after not() is negated
        assertThat(find("#error-banner")).not().isVisible();
        assertThat(find("#spinner")).not().isAttached();
    }
}

As in Playwright, most negated assertions also pass when no matching element exists — not().isVisible() is satisfied whether the banner is hidden or gone from the DOM entirely. If the test needs the element to exist and just be in a different state, assert presence first: assertThat(status).isVisible().not().hasText("Error"). not() cannot be combined with the verified actions click(), fill() or clear().

What Are Deterministic Accessibility Assertions?

TestFly already shipped hasAriaLabel-style attribute checks. 1.1.0 adds assertions that read the computed accessibility tree the same way a screen reader would, through one bundled probe script — no axe-core injection, no LLM call:

@Test
public void testSubmitButtonIsAccessible() {
    open("/signup");

    assertThat(find("#submit"))
        .hasRole(Role.BUTTON)
        .hasAccessibleName("Create account");

    assertThat(find("#email"))
        .hasAccessibleDescription(Pattern.compile("(?i)required"));
}

hasRole(Role | String) checks the explicit role attribute or the implicit role of the HTML tag. hasAccessibleName and hasAccessibleDescription accept either an exact String (whitespace collapsed, same matching Locator.withName(...) uses) or a Pattern for a partial/case-insensitive match. Only the 36 standard ARIA Role values and visible text are reported; decorative images and hidden content are skipped, and Shadow DOM is not entered.

How Do ARIA Snapshots Work?

matchesAriaSnapshot() is TestFly's answer to Playwright's toMatchAriaSnapshot(): it captures the accessibility subtree under a locator and compares it against a YAML template — a partial, ordered constraint, not a full serialization diff.

@Test
public void testNavigationLandmarkStructure() {
    open("/dashboard");

    assertThat(By.tagName("nav")).matchesAriaSnapshot("""
        - link "Home"
        - link "Docs"
        - button "Account" [expanded=false]
        """);
}

Matching rules, in the style of Playwright:

Rule	Behavior
Role	must match exactly
Quoted name	case-insensitive substring match
`/regex/` name	matched with `find()`
`[attribute]` / `[level=2]`	every template attribute must be present on the page; extra page attributes are ignored

A malformed template fails immediately with IllegalArgumentException, before waiting on the timeout. On failure, the assertion error includes both the template and the captured snapshot, so the diff is visible without re-running anything. To write a template in the first place, print what TestFly currently sees:

System.out.println(find("nav").ariaSnapshot());
// or, for the whole page:
System.out.println(ariaSnapshot());

and paste the printed YAML into the test, trimmed to only the nodes that matter.

How Does Strict Mode Prevent Silent Multi-Element Bugs?

By default, when a locator resolves to several visible elements, TestFly — like classic Selenium — silently clicks or reads the first one. That is exactly how a renamed CSS class or an A/B-tested duplicate banner turns into a test that passes against the wrong element for weeks before anyone notices.

Opt in per project with:

# testfly.yml
locators:
  strict: true

With locators.strict: true, an action on a locator matching more than one visible element — and not narrowed with first(), nth() or last() — now throws immediately:

Strict mode violation: css=".card .btn-buy" resolved to 3 visible elements;
use first()/nth()/last() or narrow the locator. Candidates: <button.btn-buy#card-1>, <button.btn-buy#card-2>, <button.btn-buy#card-3>

isVisible() and isEnabled() surface the same violation instead of silently checking the first match; count(), elements() and assertThat(...) (for counting/enumerating) are unaffected, since ambiguity is the expected case there. Narrow the locator explicitly once strict mode flags it:

find(".card .btn-buy").nth(1).click();          // second card, explicit
find(".card").filter(withText("Pro Plan"))
             .find(".btn-buy").click();          // narrowed by content instead

Hidden duplicates — a closed mobile nav sitting next to the desktop one, a hidden modal clone — are filtered out before the check runs, so responsive layouts don't trigger false strict-mode violations.

FAQ

Does `not()` replace the existing Selenium assertion methods?

No. not() is a modifier on LocatorAssert — it negates whichever assertion comes right after it in the same chain. All existing positive assertions are unchanged.

Do the accessibility assertions require axe-core or an external library?

No. hasRole, hasAccessibleName, hasAccessibleDescription and matchesAriaSnapshot use one bundled probe script that reads the browser's computed accessibility tree directly — no axe-core injection and no LLM call.

Will enabling `locators.strict` break existing tests?

It can, by design — that is the point. Any locator that was silently matching more than one visible element before now fails explicitly. The fix is the same one Playwright recommends: narrow the locator with first(), nth(), last(), or a more specific selector.

Does strict mode affect `count()` or `elements()`?

No. Those methods exist specifically to work with multiple matches, so strict mode does not apply to them — only to actions (click, fill, …) and the boolean checks isVisible()/isEnabled().

Originally published on hakangul.lovable.app.

Scaling to 100,000 RPS: Deploying Distributed Load Testing on Kubernetes with Helm and Docker

Hakan GÜL — Sun, 30 Aug 2026 07:41:47 +0000

TL;DR: Single-machine load tests inevitably hit CPU and socket descriptor limits when pushing beyond a few thousand requests per second. In LocustPilot, we packaged the entire control plane and distributed worker architecture into Kubernetes Helm charts (helm/locust), allowing engineering teams to scale to 100,000+ RPS across cloud nodes in minutes with automated resource isolation.

When a Single Machine Hits the Wall

A single modern laptop or cloud VM running Locust can comfortably generate between 2,000 and 5,000 requests per second (RPS).

However, when preparing for enterprise-scale traffic events (e.g. streaming sports events, flash sales, or banking cutovers), you need tens of thousands of concurrent users. Attempting this on one machine leads to misleading test results:

CPU Saturation: Python processes hit 100% CPU on single cores, artificially increasing response times on the client side.
Socket Exhaustion (TIME_WAIT): The operating system runs out of ephemeral ports, causing false Connection Refused errors that have nothing to do with your backend.
Bandwidth Throttling: Cloud VM network interfaces bottleneck before your target servers are even stressed.

To scale reliably, you must distribute load generation across a cluster of worker nodes.

The Distributed Master-Worker Architecture

In distributed mode, Locust separates responsibilities across two roles:

Master Node: Runs the LocustPilot web control center, coordinates worker connections, aggregates incoming metrics, and streams telemetry to ReportPortal. It generates zero HTTP traffic itself.
Worker Nodes: Connect to the master, spawn virtual users, and bombard the target backend with requests.

       ┌───────────────────────────────┐
       │   LocustPilot Master (UI)     │
       │   (Streamlit + ReportPortal)  │
       └──────────────┬────────────────┘
                      │ Port 5557 (ZMQ)
         ┌────────────┼────────────┐
         │            │            │
   ┌─────▼────┐ ┌─────▼────┐ ┌─────▼────┐
   │ Worker 1 │ │ Worker 2 │ │ Worker N │
   └─────┬────┘ └─────┬────┘ └─────┬────┘
         │            │            │
         └────────────┼────────────┘
                      ▼
         [ Target Microservices ]

Packaging for Kubernetes with Helm (`helm/locust`)

LocustPilot provides a production-ready Helm chart located in helm/locust/.

1. The Optimized Container (`k8s.Dockerfile`)

FROM python:3.10-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    curl build-essential && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8501 5557
CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]

Photo by Fotis Fotopoulos on Unsplash

2. Deploying the Cluster with One Command

Using the pre-built Helm chart, deploying the entire distributed load testing cluster with ReportPortal telemetry requires a single command:

helm upgrade --install locust ./helm/locust \
  --namespace loadtest \
  --create-namespace \
  --set image.repository=my-registry.io/locust-pilot \
  --set image.tag=latest \
  --set replicaCount=10 \
  --set extraEnv[0].name=RP_TOKEN --set extraEnv[0].value=$RP_TOKEN \
  --set extraEnv[1].name=RP_ENDPOINT --set extraEnv[1].value=$RP_ENDPOINT \
  --set extraEnv[2].name=RP_PROJECT --set extraEnv[2].value=$RP_PROJECT

3. Tuning OS Limits for Massive Scale

When running high-volume workers inside Kubernetes, ensure your Pod security contexts allow high file descriptors:

# helm/locust/values.yaml
worker:
  resources:
    limits:
      cpu: 2000m
      memory: 2Gi
    requests:
      cpu: 1000m
      memory: 1Gi
  sysctls:
    - name: net.ipv4.tcp_tw_reuse
      value: "1"
    - name: net.core.somaxconn
      value: "65535"

What Is Next?

In the final installment (Part 5), we will connect all the pieces into your continuous delivery pipeline: Shift-Left Performance Testing with Automated Quality Gates in GitHub Actions and Bitbucket Pipelines.

👉 Explore the Helm Charts on GitHub

FAQ

How many workers are needed for 50,000 RPS?

Depending on endpoint payload size and SSL overhead, each CPU core running FastHttpUser can typically generate 1,500–3,000 RPS. A cluster of 20–25 worker pods can comfortably sustain 50,000+ RPS.

Can workers be dynamically auto-scaled?

Yes. Kubernetes Horizontal Pod Autoscaler (HPA) can scale worker pods based on CPU utilization or custom test duration parameters.

Enterprise Observability: Real-Time ReportPortal Integration with Gevent and Percentile Tracking

Hakan GÜL — Sun, 30 Aug 2026 07:41:12 +0000

TL;DR: High-throughput load tests generating thousands of requests per second can easily overwhelm reporting backends with log spam. In LocustPilot, we solved this by implementing an asynchronous gevent listener for ReportPortal that deduplicates error cascades, streams live endpoint metrics every 30 seconds, and calculates comprehensive response time percentiles (P50 to P99) with automated quality gate verdicts.

The Log Flood Dilemma in Load Testing

In standard functional UI or API tests, every failure is unique and warrants a detailed log entry.

In performance testing, the dynamic is completely inverted:

When an endpoint begins to degrade under 5,000 RPS, it does not fail once—it fails 50,000 times with the exact same 504 Gateway Timeout or Connection Refused error.
If your test framework attempts to log every single failure to a centralized telemetry platform like ReportPortal, you will DDoS your own reporting database, exhaust memory, and freeze the test execution.

To provide enterprise-grade observability without degrading load test performance, we built ReportPortalListener (app/core/rp_listener.py) in LocustPilot.

1. Background Telemetry with Gevent Greenlets

Locust relies on gevent for lightweight, non-blocking coroutines. Rather than blocking worker execution during network logging, LocustPilot spawns a background greenlet (_periodic_stats_logger) that flushes aggregated endpoint statistics every 30 seconds:

# app/core/rp_listener.py
import gevent
from reportportal_client.helpers import timestamp

def on_test_start(self, **kwargs):
    # Initialize ReportPortal Launch and Test Item
    self.launch_uuid = self.rp_client.start_launch(
        name=self.rp_launch_name,
        start_time=timestamp(),
        description=self.rp_launch_desc,
    )
    self.test_item_uuid = self.rp_client.start_test_item(
        name="Load Test Execution",
        start_time=timestamp(),
        item_type="STEP",
    )

    # Spawn background stats publisher
    self.running = True
    gevent.spawn(self._periodic_stats_logger)

def _periodic_stats_logger(self):
    """Log live endpoint stats every 30 seconds without blocking load workers"""
    while self.running:
        gevent.sleep(30)
        if self.running and self.rp_client and self.test_item_uuid:
            lines = ["📊 Live Endpoint Telemetry", "=" * 80]
            lines.append(f"{'Method':<8} {'Name':<40} {'Reqs':<8} {'Fails':<8} {'Avg(ms)':<10} {'RPS':<8}")
            lines.append("-" * 80)

            for stat in self.env.stats.entries.values():
                if stat.num_requests > 0:
                    lines.append(
                        f"{stat.method:<8} {stat.name[:40]:<40} "
                        f"{stat.num_requests:<8} {stat.num_failures:<8} "
                        f"{stat.avg_response_time:<10.0f} {stat.current_rps:<8.1f}"
                    )

            self.rp_client.log(
                time=timestamp(),
                message="\n".join(lines),
                level="INFO",
                item_id=self.test_item_uuid,
            )

This ensures team leads and developers watching the ReportPortal dashboard receive live health updates without injecting latency into the simulated users.

2. Smart Error Deduplication (Unique Error Logging)

When a microservice falters, LocustPilot captures the first instance of an error to preserve diagnostic context, while silently aggregating subsequent identical failures into a counter:

def on_request(self, request_type, name, response_time, response_length, exception=None, response=None, **kwargs):
    has_exception = exception is not None
    has_manual_failure = response is not None and getattr(response, "_manual_result", None) is False

    if has_exception or has_manual_failure:
        error_key = f"{request_type}:{name}"

        # Initialize counter on first occurrence
        if error_key not in self.last_stats:
            self.last_stats[error_key] = {"count": 0, "logged": False}

        self.last_stats[error_key]["count"] += 1

        # Only send the first failure payload to ReportPortal
        if not self.last_stats[error_key]["logged"] and self.rp_client:
            error_msg = str(exception) if has_exception else getattr(response, "_manual_result_msg", "Manual failure")
            self.rp_client.log(
                time=timestamp(),
                message=f"❌ First Failure Detected: {request_type} {name}\nDetails: {error_msg}",
                level="ERROR",
                item_id=self.test_item_uuid,
            )
            self.last_stats[error_key]["logged"] = True

Result: Zero log spam. If 100,000 requests fail with the same connection error, ReportPortal receives exactly 1 clean diagnostic entry with a summary count at the end of the test.

Photo by Mika Baumeister on Unsplash

3. Comprehensive Percentiles and Automated Quality Gates

Average response time is a dangerous vanity metric in performance engineering; a 200ms average can hide a 99th percentile spike of 12 seconds.

When a test completes (on_test_stop), LocustPilot computes full percentile distributions and enforces automated quality gates:

def on_test_stop(self, **kwargs):
    total = self.env.stats.total

    # 1. Percentile Distribution Table (50% to 100%)
    for stat in self.env.stats.entries.values():
        if stat.num_requests > 0:
            p50 = stat.get_response_time_percentile(0.50)
            p90 = stat.get_response_time_percentile(0.90)
            p95 = stat.get_response_time_percentile(0.95)
            p99 = stat.get_response_time_percentile(0.99)
            max_rt = stat.max_response_time

    # 2. Automated Quality Gate Rules
    has_issue = False
    if total.avg_response_time > 200:
        log_warning(f"⚠️ Average response time exceeded 200ms! (Actual: {total.avg_response_time:.2f} ms)")
        has_issue = True
    if total.fail_ratio > 0.05:
        log_warning(f"⚠️ Failed request ratio exceeded 5%! (Actual: {total.fail_ratio:.2%})")
        has_issue = True

    # 3. Final Build Verdict
    success_rate = (1 - total.fail_ratio) * 100
    status = "PASSED" if success_rate >= 75 and not has_issue else "FAILED"

    self.rp_client.finish_test_item(item_id=self.test_item_uuid, end_time=timestamp(), status=status)
    self.rp_client.finish_launch(end_time=timestamp())

What Is Next?

In Part 4, we move from single-instance tests to large-scale distributed load: deploying LocustPilot on Kubernetes using Helm charts and Docker worker nodes to simulate over 100,000 requests per second.

👉 Check out the ReportPortal Listener on GitHub

FAQ

Why is ReportPortal preferred over standard HTML reports?

HTML reports are static and isolated on developer machines. ReportPortal centralizes performance history across sprints, enables historical trend comparisons, and automatically flags regression anomalies.

Can custom quality gate thresholds be configured?

Yes. Quality thresholds (such as max average latency or acceptable failure ratio) can be configured via .env variables or custom BaseLocustUser validators.

Under the Hood: Building a Dynamic AST Test Runner and Subprocess Engine in Python

Hakan GÜL — Sun, 30 Aug 2026 07:40:38 +0000

TL;DR: Dynamically discovering test classes at runtime by executing importlib is dangerous because module-level code runs immediately. In LocustPilot, we solved this by building an Abstract Syntax Tree (AST) static scanner that identifies HttpUser and FastHttpUser classes without executing Python code, paired with an unbuffered subprocess.Popen engine for real-time log streaming.

The Hidden Trap of Dynamic Python Discovery

When building a web control plane for load testing, the first requirement is simple: automatically discover and list all test files in the UI dropdown.

Most developers jump straight to dynamic importing:

# ❌ The Naive Way: Dangerous runtime import
import importlib.util

spec = importlib.util.spec_from_file_location("module", file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module) # ⚠️ Executes all top-level code!

Why is this dangerous?

Module-level side effects: If a test file establishes database connections, reads external credentials, or executes setup logic at the module level, simply browsing the UI triggers those actions.
Crash propagation: A syntax error or missing optional dependency in one test file will crash the entire UI server.
High memory footprint: Importing dozens of heavy test modules pollutes the server process memory space.

We needed a way to inspect Python test files statically and safely before execution.

The Solution: Static Analysis via Python's AST

In LocustPilot (app/core/runner.py), we use Python's built-in ast (Abstract Syntax Tree) module. We parse the source code into a syntax tree and walk the nodes without executing a single line of code.

Here is the exact heuristic implemented in LocustPilot:

import ast
import re
from pathlib import Path

def _has_locust_user_class(path: Path) -> bool:
    """Heuristic: returns True if file defines a Locust User class.
    Detects subclasses of HttpUser/FastHttpUser via AST; falls back to regex.
    """
    try:
        src = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        return False

    try:
        tree = ast.parse(src)
        for node in ast.walk(tree):
            if isinstance(node, ast.ClassDef):
                for base in node.bases:
                    # Matches class MyTest(HttpUser)
                    if isinstance(base, ast.Name) and base.id in {"HttpUser", "FastHttpUser"}:
                        return True
                    # Matches class MyTest(locust.HttpUser)
                    if isinstance(base, ast.Attribute) and base.attr in {"HttpUser", "FastHttpUser"}:
                        return True
    except Exception:
        pass

    # Fallback regex check for custom decorators
    return (
        re.search(r"class\s+\w+\(.*?(HttpUser|FastHttpUser).*?\)", src) is not None
        or "@task" in src
    )

Why This Approach Wins:

⚡ Ultra-Fast: Parsing 100 Python files takes less than 15 milliseconds.
🛡️ Zero Side-Effects: No network sockets, database pools, or environment lookups are triggered.
🔍 Smart Filtering: Automatically excludes __pycache__, __init__.py, and helper utility files unless they contain runnable user classes.

Photo by Kevin Ku on Unsplash

Extracting Target Hosts Without Hardcoding

In real-world projects, different test files target different microservices or staging environments. LocustPilot uses an intelligent fallback chain to determine the default target host:

def extract_locustfile_host(path: Path) -> Optional[str]:
    # 1. Environment variable override takes highest precedence
    env_host = os.getenv("LOCUST_TARGET_HOST")
    if env_host:
        return env_host

    # 2. Extract host attribute directly from test file
    try:
        text = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        return None

    m = re.search(r"^\s*host\s*=\s*['\"]([^'\"]+)['\"]", text, flags=re.MULTILINE)
    if m:
        return m.group(1).strip()

    # 3. Fallback to centralized BaseUser target host
    if re.search(r"^\s*host\s*=\s*TARGET_HOST", text, flags=re.MULTILINE):
        base_user = BASE_DIR / "locustfiles" / "utils" / "base_user.py"
        if base_user.exists():
            base_text = base_user.read_text(encoding="utf-8", errors="ignore")
            m_target = re.search(r"TARGET_HOST\s*=\s*['\"]([^'\"]+)['\"]", base_text)
            if m_target:
                return m_target.group(1).strip()

    return None

This allows engineers to either set hosts per test file, use a shared BaseLocustUser, or dynamically override them via the Streamlit UI.

Subprocess Execution and Unbuffered Log Streaming

Running load tests inside the main web server process would freeze the UI. LocustPilot spawns an isolated subprocess.Popen running headless Locust while streaming logs in real time.

def run_locust(
    locustfile: Path,
    host: Optional[str],
    users: int,
    spawn_rate: float,
    run_time: str,
    run_dir: Path,
    csv_prefix: str = "stats",
    html_report: bool = True,
    csv_full_history: bool = True,
    stream_logs: bool = True,
) -> Tuple[subprocess.Popen, Path, Path, str, List[str]]:

    cmd = [
        "locust",
        "-f", str(locustfile),
        "--headless",
        "-u", str(users),
        "-r", str(spawn_rate),
        "--run-time", str(run_time),
        "--csv", str(run_dir / csv_prefix),
        "--logfile", str(run_dir / "locust.log"),
        "--only-summary",
    ]
    if host:
        cmd += ["--host", str(host)]
    if html_report:
        cmd += ["--html", str(run_dir / "report.html")]
    if csv_full_history:
        cmd += ["--csv-full-history"]

    # Dynamic PYTHONPATH injection so test libraries import cleanly
    env = os.environ.copy()
    py_paths = [str(LOCUSTFILES_DIR), str(LOCUSTFILES_DIR / "libs"), str(BASE_DIR)]
    env["PYTHONPATH"] = os.pathsep.join(py_paths)

    proc = subprocess.Popen(
        cmd,
        stdout=(subprocess.PIPE if stream_logs else subprocess.DEVNULL),
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1,  # Line-buffered output for instant streaming
        cwd=str(BASE_DIR),
        env=env,
    )
    return proc, logfile, html_path, start, cmd

Key Subprocess Design Choices:

bufsize=1: Enables line-buffered I/O. Standard Python output buffering waits for hundreds of bytes before flushing; bufsize=1 pushes log lines to Streamlit instantly.
PYTHONPATH Injection: Automatically adds libs/ and utils/ to Python path, allowing test scripts to use shared utilities without complex packaging.
Non-Blocking Polling: The Streamlit UI polls proc.stdout.readline() without blocking the event loop.

Architectural Summary

By decoupling test discovery (AST) from test execution (Subprocess), LocustPilot achieves:

Safety: Malicious or broken test scripts cannot crash the control plane.
Responsiveness: Heavy load generation runs in isolated processes without degrading web UI performance.
Clean Code: Test authors write standard Python without needing framework boilerplate.

What Is Next?

In Part 3, we will dive into Enterprise Observability: how LocustPilot leverages gevent greenlets to stream real-time P95/P99 latency percentiles and deduplicated error alerts to ReportPortal.

👉 Explore the LocustPilot Source Code on GitHub

FAQ

Why not use regex alone to find Locust classes?

Regex can produce false positives on comments or strings (e.g. inside docstrings). AST guarantees that we are inspecting actual Python class inheritance syntax.

Does `subprocess.Popen` limit load test throughput?

No. Headless Locust runs in its own OS process utilizing gevent coroutines. The Streamlit UI only reads lightweight summary lines, keeping CPU overhead below 1%.

Can I run tests with custom Python dependencies?

Yes. By configuring PYTHONPATH dynamically, any custom library inside locustfiles/libs/ or virtualenv is immediately accessible.

The Death of XML: Why We Built LocustPilot — A Modern Control Plane for Distributed Load Testing

Hakan GÜL — Sun, 30 Aug 2026 07:40:03 +0000

TL;DR: GUI-based load testing tools like JMeter create painful Git merge conflicts with massive XML files, while standard Locust lacks persistent run history and out-of-the-box team dashboards. LocustPilot bridges this gap by combining Python's code-as-test power with a modern Streamlit control center, dynamic test registries, ReportPortal observability, and instant Docker/Kubernetes deployment.

Why Is Traditional Performance Testing Broken?

Every test automation engineer who has prepared for a major traffic event (Black Friday, product launches, or high-volume campaigns) has faced the dreaded .jmx file conflict.

It is 6:00 PM. Two engineers update a JMeter test plan simultaneously. They commit their changes, and Git screams with a 4,000-line XML merge conflict. You cannot easily read it, you cannot safely merge it in a text editor, and one wrong closing tag corrupts the entire test suite.

For decades, performance testing was trapped in GUI-heavy, XML-driven tools. When Locust arrived, it revolutionized the industry by introducing Performance Testing as Code in pure Python.

Suddenly, your load tests were version-controlled, readable, and could leverage the entire Python ecosystem.

from locust import HttpUser, task, between

class QuickOrderUser(HttpUser):
    wait_time = between(1, 2)

    @task
    def checkout(self):
        self.client.post("/api/v1/orders", json={"item_id": 42, "quantity": 1})

What Was Missing in Vanilla Locust?

While Locust's core execution engine is lightweight and lightning-fast, scaling it across an enterprise engineering team reveals immediate operational friction:

No Out-of-the-Box Test Registry: You have to remember command-line paths (locust -f locustfiles/checkout_flow.py) or manage messy bash wrapper scripts.
Ephemeral Run History: The default Locust web UI is purely session-based. Once you stop the test or restart the container, all charts, response times, and failure logs vanish forever.
Disconnected Reporting: Sharing past test runs with backend developers requires manually downloading CSVs or hunting through local log files.
Lack of Enterprise Quality Gates: Teams need automated test thresholds (e.g., Fail if 99th percentile exceeds 250ms) and centralized observability dashboards like ReportPortal.

Photo by Glenn Carstens-Peters on Unsplash

Introducing LocustPilot: The Modern Control Plane

We built LocustPilot to turn standalone Locust scripts into a collaborative, production-grade load testing platform.

Instead of wrestling with CLI flags or losing past test metrics, LocustPilot provides an interactive Streamlit Command Center that automatically manages test discovery, execution, real-time log streaming, and persistent reporting.

Key Architectural Highlights:

🧪 Smart Test Discovery (AST Scanner): Automatically parses your test directory using Python's Abstract Syntax Tree (ast), detects all HttpUser / FastHttpUser classes, and exposes them in a clean dropdown menu.
📊 Persistent History & Plotly Charts: Every run is automatically archived with detailed latency percentiles (P50, P90, P95, P99), throughput graphs, and downloadable HTML/CSV bundles.
🔌 Native ReportPortal Integration: Streams live endpoint metrics and unique failure logs directly into ReportPortal with zero manual setup.
☁️ Cloud-Native Deployment: Ready for production with pre-configured Docker images and Kubernetes Helm charts (helm/locust).

Feature Matrix: How LocustPilot Compares

Capability	Apache JMeter	Vanilla Locust	LocustPilot
Test Definition	XML / Proprietary GUI	Pure Python Code	Pure Python Code
Version Control & Diffs	❌ Painful XML conflicts	✅ Clean Python Git diffs	✅ Clean Python Git diffs
Test Selection UI	❌ Manual file opening	❌ CLI arguments only	✅ Dropdown Test Registry
Run History	⚠️ Complex DB setups	❌ Session only (lost on stop)	✅ Persistent CSV/HTML/ZIP
Enterprise Telemetry	⚠️ Heavy plugins	❌ Basic web UI	✅ Built-in ReportPortal & Plotly
K8s & CI/CD Ready	❌ Heavy JVM overhead	⚠️ Manual worker setup	✅ Helm Chart & Docker Ready

How to Get Started in 30 Seconds

LocustPilot is completely open-source under the MIT license. You can launch the entire platform locally with a single Docker command:

# Clone the repository
git clone https://github.com/hakanngul/LocustPilot.git
cd LocustPilot

# Build and launch with Docker
docker build -t locust-pilot .
docker run -p 8501:8501 locust-pilot

Open your browser at http://localhost:8501, pick your target test from the dropdown, configure your user count, and hit Start Test.

What Is Next in This Series?

This is Part 1 of our 5-part LocustPilot deep dive. Over the next parts, we will explore:

Part 2: Under the hood of the dynamic AST Python test scanner and non-blocking log streamer.
Part 3: Enterprise observability: Streaming 99th percentile latencies to ReportPortal with Gevent.
Part 4: Scaling to 100,000 RPS on Kubernetes using Helm and distributed workers.
Part 5: Shift-Left performance testing: Enforcing automated build quality gates in CI/CD.

👉 Star and Explore LocustPilot on GitHub

FAQ

Why choose Python-based Locust over JMeter?

Python-based load testing allows developers and SDETs to treat performance tests as first-class software code. You can write reusable modules, use native Git diffs, debug directly in your IDE, and eliminate fragile XML files.

Does LocustPilot modify standard Locust scripts?

No. Any standard Locust script (HttpUser, FastHttpUser, @task) runs seamlessly in LocustPilot without requiring any code modifications or vendor lock-in.

Can I run LocustPilot in headless CI/CD pipelines?

Yes. LocustPilot supports both the interactive Streamlit UI for ad-hoc load runs and fully headless execution inside GitHub Actions, Bitbucket Pipelines, or Jenkins.

TestFly Part 8: Flakiness Quarantine Engine & Java SPI Extensibility

Hakan GÜL — Sun, 30 Aug 2026 07:39:29 +0000

Quarantine Engine & Java SPI Extensibility

The final pillar of TestFly is designed for long-term maintainability: Flakiness Quarantine and Java SPI Extensibility.

1. Flakiness Quarantine Engine (`testfly-quarantine.yml`)

Instead of deleting or commenting out flaky tests, quarantine them cleanly without blocking CI pipelines:

# testfly-quarantine.yml (Project Root)
quarantine:
  enabled: true
  tests:
    - className: io.testfly.examples.PaymentTest
      methodName: testStripeWebhook
      reason: "Waiting for third-party webhook fix (JIRA-4021)"
      owner: "@payment-team"

Quarantined tests are executed in CI, but their failures are flagged as QUARANTINED rather than failing the build.

2. Java SPI Extension Points

Extend TestFly without forking or modifying framework internals:

Extension Point	Interface	Registration
Custom Driver Provider	`NamedDriverProvider`	`DriverProviderRegistry.register()` or `META-INF/services`
Report Adapter	`ReportAdapter`	`ReportAdapterRegistry.register()` or SPI
Lifecycle Hook	`ExecutionHook`	`HookRegistry.register()` or SPI
Full Plugin	`TestFlyPlugin`	`PluginRegistry.register()` or SPI

Example: Custom Slack Report Adapter

package io.testfly.examples.plugins;

import io.testfly.reporting.ReportAdapter;
import io.testfly.metrics.ExecutionMetrics;

public class SlackReportAdapter implements ReportAdapter {

    @Override
    public String getName() {
        return "slack";
    }

    @Override
    public void onSuiteFinish(ExecutionMetrics metrics) {
        // Send Slack webhook with pass rate and failed test summaries
    }
}

Summary of the TestFly Masterclass

Across all 8 parts, we explored:

Part 1: Zero-boilerplate setup with BaseTest and testfly.yml
Part 2: Semantic locators (getByRole), BasePage, and WaitEngine
Part 3: CI/CD quality gates (BuildThresholdEnforcer) and parallel execution
Part 4: AI test generation and self-healing with testfly-mcp
Part 5: BDD with Cucumber 7, BaseCucumberSteps, and @retryable
Part 6: Built-in ApiClient, BaseApiTest, and hybrid workflows
Part 7: Bundled axe-core accessibility (accessibility()) and VisualAssert
Part 8: Flakiness quarantine engine and Java SPI extensibility

👉 Explore TestFly on GitHub and build reliable test suites today!

TestFly Part 7: Bundled axe-core Accessibility Assertions & VisualAssert Pixel Matching

Hakan GÜL — Sun, 30 Aug 2026 07:38:56 +0000

Accessibility & Visual Regression in TestFly

TestFly bundles axe-core 4.10.2 and a dedicated pixel-comparison engine VisualAssert directly into the core JAR.

1. Bundled axe-core Accessibility Scans (`accessibility()`)

In BaseTest, run a WCAG scan in a single line without external dependencies:

package io.testfly.examples.a11y;

import io.testfly.accessibility.Impact;
import io.testfly.test.BaseTest;
import org.testng.annotations.Test;

public class AccessibilityTest extends BaseTest {

    @Test
    public void checkoutPassesWcagAA() {
        open("/checkout");

        accessibility()
            .withTags("wcag2a", "wcag21aa", "wcag22aa")
            .withLevel(Impact.SERIOUS) // fails on SERIOUS or CRITICAL
            .excluding("#third-party-widget")
            .run();
    }
}

2. Pixel-Perfect Visual Regression (`VisualAssert`)

package io.testfly.examples.visual;

import io.testfly.test.BaseTest;
import io.testfly.visual.VisualAssert;
import io.testfly.visual.VisualTolerance;
import org.openqa.selenium.By;
import org.testng.annotations.Test;

public class VisualRegressionTest extends BaseTest {

    @Test
    public void homepageVisualMatchesBaseline() {
        open("/");

        // Full-page comparison with 1% tolerance
        VisualAssert.assertScreenshot("homepage", VisualTolerance.of(1));

        // Element-scoped comparison
        VisualAssert.assertScreenshot("login-box", By.id("login-card"));
    }
}

First Run: Baseline screenshot is generated automatically.
Subsequent Runs: Compares against baseline; diffs saved to target/visual-diffs/.
Update Baselines: Run with -DupdateBaselines=true.

DEV Community: Hakan GÜL

TestFly Part 13: Actionability Engine — Auto-Wait and Verified Click/Fill/Clear Assertions

How Does TestFly Decide an Element Is Ready for an Action?

What Exactly Gets Checked, and in What Order?

What Happens When an Element Never Becomes Actionable?

How Do I Configure or Bypass Actionability Checks?

How Does assertThat(x).click() Verify a Click Actually Landed?

How Do assertThat(x).fill(text) and .clear() Verify the Field Actually Changed?

FAQ

Does turning on actionability checks slow every test down?

Does assertThat(x).click() replace Locator.click()?

Can I combine not() with the verified click()/fill()/clear()?

What's the difference between force() and actionability.enabled: false?

TestFly Part 14: Step Traces — Per-Step ARIA/DOM Snapshots for Visual Debugging

TestFly Part 14: Step Traces — Per-Step ARIA/DOM Snapshots for Visual Debugging

What Is a Step Trace, and Why Snapshot Every Step?

What Does Each Step's Snapshot Contain?

How Do You Configure What Gets Captured?

What Does the Trace HTML Actually Show You?

Does This Slow Down or Risk Failing My Tests?

Does This Work With Cucumber?

FAQ

Do I need to add code to my tests to get per-step snapshots?

What happens if a test has no browser, like an API test?

Can a snapshot capture failure cause my test to fail?

Is the diff tab a real structural diff of the accessibility tree?

TestFly Part 16: Smart Triage & Fuzzy Self-Healing — Zero-Token Flaky-Test Diagnosis

TestFly Part 16: Smart Triage & Fuzzy Self-Healing — Zero-Token Flaky-Test Diagnosis

Why Does Flaky-Test Triage Need a Heuristic Layer at All?

How Fast Is Fuzzy Self-Healing?

What Is Auto-POM / PageKnowledge?

How Does the Zero-Token Layer Fit With Part 4's LLM Self-Healing?

FAQ

Does SmartTriageEngine call an LLM to classify failures?

What score does FuzzyHealingEngine require before it heals a locator?

Does Auto-POM require every page to be manually mapped first?

If FuzzyHealingEngine fails, does TestFly immediately fail the test?

TestFly Part 15: Enterprise API Auth — RFC 7616 Digest, HMAC Signing & OAuth2 Refresh Tokens

Why Did TestFly's Digest Auth Need a Rewrite?

How Does TestFly Implement RFC 7616 Digest Authentication?

How Do You Sign Requests with HMAC?

How Does OAuth2 Token Refresh Work?

How Do You Wire This Up Declaratively with @UseAuth?

What Else Changed Around API Auth in 1.1.0?

FAQ

TestFly Part 17: Cross-Browser Reliability — BiDi Console Capture, Edge/Safari on Grid & Guaranteed Driver Cleanup

How Does BiDi Catch Console Errors Firefox Used to Miss?

How Do You Know Which Browser Features You're Trading Away?

How Do You Run Edge and Safari on Selenium Grid?

How Does TestFly Guarantee Browsers Are Never Left Running?

FAQ

Does browser.bidi work with Selenium Grid or BrowserStack/Sauce Labs?

How do I check which console-error capture path is active for the current test?

Can I run the same suite on Chrome, Edge and Safari against one Selenium Grid?

If I kill a parallel run with Ctrl+C, are all browser processes definitely gone?

TestFly Part 12: Playwright-Style Assertions in Selenium — not(), ARIA Snapshots & Strict Mode

Why Do Selenium Assertions Need a not() Method?

What Are Deterministic Accessibility Assertions?

How Do ARIA Snapshots Work?

How Does Strict Mode Prevent Silent Multi-Element Bugs?

FAQ

Does not() replace the existing Selenium assertion methods?

Do the accessibility assertions require axe-core or an external library?

Will enabling locators.strict break existing tests?

Does strict mode affect count() or elements()?

Scaling to 100,000 RPS: Deploying Distributed Load Testing on Kubernetes with Helm and Docker

When a Single Machine Hits the Wall

The Distributed Master-Worker Architecture

Packaging for Kubernetes with Helm (helm/locust)

1. The Optimized Container (k8s.Dockerfile)

2. Deploying the Cluster with One Command

3. Tuning OS Limits for Massive Scale

What Is Next?

FAQ

How many workers are needed for 50,000 RPS?

Can workers be dynamically auto-scaled?

Enterprise Observability: Real-Time ReportPortal Integration with Gevent and Percentile Tracking

The Log Flood Dilemma in Load Testing

1. Background Telemetry with Gevent Greenlets

2. Smart Error Deduplication (Unique Error Logging)

How Does `assertThat(x).click()` Verify a Click Actually Landed?

How Do `assertThat(x).fill(text)` and `.clear()` Verify the Field Actually Changed?

Does `assertThat(x).click()` replace `Locator.click()`?

Can I combine `not()` with the verified `click()`/`fill()`/`clear()`?

What's the difference between `force()` and `actionability.enabled: false`?

Does `browser.bidi` work with Selenium Grid or BrowserStack/Sauce Labs?

Does `not()` replace the existing Selenium assertion methods?

Will enabling `locators.strict` break existing tests?

Does strict mode affect `count()` or `elements()`?

Packaging for Kubernetes with Helm (`helm/locust`)

1. The Optimized Container (`k8s.Dockerfile`)

Does `subprocess.Popen` limit load test throughput?

1. Flakiness Quarantine Engine (`testfly-quarantine.yml`)

1. Bundled axe-core Accessibility Scans (`accessibility()`)

2. Pixel-Perfect Visual Regression (`VisualAssert`)