DEV Community: nexus studio

Is this test flaky, or is it a real bug?

nexus studio — Sat, 17 Oct 2026 15:49:26 +0000

A flaky test failed and then passed on a retry without anyone fixing anything, and a real bug fails for a reason no retry changes. Nexus Studio, by Verdict System LLC, tells them apart by recording every retry: FLAKY for a failure that a retry passed, BREACHED for a real finding, and BREACHED_FLAKY for a failure that is itself inconsistent. A retry never turns a defect into a green run.

By the Nexus Studio team, Verdict System LLC. October 7, 2026.

What is a flaky test?

A flaky test is one whose attempts disagree: it fails once and passes on another try, with the code and the test unchanged. In Nexus Studio the word has one meaning. FLAKY is a test that failed, then passed on a retry, with nothing healed. Something is unstable, in the application or in the test, and nothing was fixed.

A hard failure that happens on every attempt is not flaky, even when the self-healing step tried and failed to repair a selector. An earlier version of the engine filed that case as FLAKY. Teams read FLAKY as "the test is bad" and move on, so a broken application shipped as noise. The engine no longer does that.

Do retries hide real bugs?

They can, and the reason is the number of states. With pass and fail only, a retry that passes turns a red run green, and an intermittent defect in the application becomes invisible. That is one of the most expensive kinds of bug to find. When red can also mean a blinked network or a dead browser, the team learns to rerun reds instead of reading them, and the suite loses trust.

Nexus Studio keeps the retry visible. A test that failed and then passed is not SECURE. It exits 0 in CI, so it does not break the build, but it shows in the report and in History as FLAKY, a pass with a reservation. See how the 14 verdicts are decided for the full order of checks.

What is the difference between FLAKY, BREACHED and BREACHED_FLAKY?

FLAKY: failed, then a retry passed it, and nothing was repaired. Read the failed attempt's raw-data.json, then the History trend. It exits 0.
BREACHED: the test failed on a single attempt, and no environment or harness pattern explains the error. An assertion failed. This is the bug report, and it exits 1.
BREACHED_FLAKY: the test failed on its last attempt after an earlier retry, and no environment or harness pattern explains it. The failure itself is inconsistent. Compare the attempts before you file a bug. It exits 1.

Two neighbors keep the picture honest. AUTO_RECOVER is a pass where self-healing replaced a selector, so the author's locator is dead and the test is not clean. A first attempt that failed only on time and then passed on retry is SECURE, because that shows latency, not instability.

Before a failure can be BREACHED or BREACHED_FLAKY, the classifier rules out the other explanations in a fixed order: a harness error, a crash, a block, an unreachable host and a timeout. Only a failure that none of those explains is a finding. That is why a timeout or a CAPTCHA does not arrive as a bug report.

How are retries counted?

The engine reads the attempt number of the run. A passing test with a retry number above zero is FLAKY, unless a selector was healed, in which case it is AUTO_RECOVER. A failing or timed-out test with a retry number above zero, and no environment explanation, is BREACHED_FLAKY. A clean pass needs retry zero and no healed selector.

FLAKY means the attempts disagreed, and nothing else. A failure on a single attempt, with zero retries, is never labeled flaky. Each History row also records retry_attempt and attempts, so the count stays with the run.

What evidence does each verdict leave?

FLAKY: screenshot.png, metadata.json and raw-data.json. The folder keeps the failed attempt's raw-data.json, so a retry that hid the failure still leaves its log. This follows the setting Keep evidence of a FLAKY / AUTO_RECOVER pass, which is on by default.
BREACHED_FLAKY: report.md, metadata.json, raw-data.json and screenshot.png, and its failure video is kept by default.
BREACHED: the same files, with the video kept by default.

The verdict is in the evidence folder's name, for example a name that ends in [BREACHED]. The Evidence page follows one failed test through every file.

How does History spot a test that flips?

History keeps one row per run for each test and browser or device, with the verdict, the retry attempt and the error signature. From those rows it shows trends by session, day or week: Regressed, Recovered, Still broken and Slower. A test that goes green, red and green again shows up there instead of disappearing into reruns.

The risk ranking, in the Evidence Locker and in Patrol, orders tests by how often they flip, break or cost time. It also shows a predicted risk from a logistic regression over your history, with an EWMA fallback when data is thin. Nothing is called flaky below five runs, because one or two runs cannot show a pattern.

Limits

Retries are what separate FLAKY from SECURE and BREACHED_FLAKY from BREACHED. With retries off, an intermittent failure is a BREACHED.
FLAKY says the attempts disagreed. It does not say why. Finding the cause in the application or in the test is still work for a person.
The classification reads error text, so a custom assertion message that contains words such as 403 or Forbidden can read as an environment problem. Open metadata.json when a verdict surprises you.
Below five runs, History and the risk ranking call nothing flaky.

Where to go next

The Evidence page shows every file a failed test leaves, and How Nexus Studio decides 14 test verdicts explains the order of the checks. The 30-day trial runs the same engine on your own tests.

Why do my tests fail when the application is fine?

nexus studio — Sat, 17 Oct 2026 15:48:28 +0000

An end-to-end test goes red when the network is slow, the DNS fails, the site answers with a 403 or a 429 or shows a bot challenge, or the browser dies, and none of that is a defect in your application. Nexus Studio, from Verdict System LLC, gives these failures their own verdicts: TIMEOUT, UNREACHABLE, BLOCKED and CRASHED. None of them is reported as a bug in the application.

By the Nexus Studio team, Verdict System LLC. October 7, 2026.

What makes a test fail without a bug?

A test only reports that something did not happen. The cause can sit in four places that have nothing to do with your code:

the page was slow and the test ran out of time;
the host could not be reached, for example because the name did not resolve;
the site defended itself with a 403, a 429, a CAPTCHA or a bot challenge, so the request never reached your product;
the browser or the process died.

When every one of these reads as a plain red, the team files them as bugs, or learns to rerun reds instead of reading them. Either way the real defect hides in the noise.

How does Nexus Studio tell a false failure from a bug?

When a test fails, the engine reads the error message and the network log of the run and tests them against groups of patterns in a fixed order. The first group that matches decides the verdict. The order is CONFIG_ERROR, then CRASHED, BLOCKED, UNREACHABLE and TIMEOUT, and only then BREACHED_FLAKY or BREACHED. So BREACHED, the verdict that means a bug report, is left for a failure that no environment or harness pattern explains. The patterns cover both Playwright and Selenium WebDriver wording.

What do TIMEOUT, UNREACHABLE, BLOCKED and CRASHED mean?

TIMEOUT: the test never got an answer in time. That is not proof the application is broken, and it is not proof it works: the run cannot tell. A navigation that ran out of time is a TIMEOUT, because it started and the page was slow.
UNREACHABLE: the host could not be reached at all. DNS or the connection failed before any page was observed, as with a name that did not resolve, a refused connection or an offline machine. A certificate error is not here: that one is a finding.
BLOCKED: the site defended itself with a 403, a 429, a block page or a bot challenge. When a defense was on screen, metadata.json names it. A real Android phone that fell asleep or is locked is BLOCKED too, as an unmet prerequisite.
CRASHED: the browser or the process died, for example a closed page, a DevTools protocol error or a dead WebDriver session. It is infrastructure: look at the machine and the driver.

The verdicts guide lists the exact error patterns behind each one.

What does my CI pipeline see?

Seven of the 14 verdicts exit 0: SECURE, AUTO_RECOVER, FLAKY, CHAOS_RESILIENT, PERFORMANCE_DEGRADED, A11Y_VIOLATION and SECURITY_VIOLATION. The other seven exit 1, and TIMEOUT, UNREACHABLE, BLOCKED and CRASHED are among them. So the build still goes red, because the test did not pass, but the report and History say which cause it was. The pytest and Selenium runners exit 2 when the run could not start at all.

What evidence does each false failure keep?

Each of the four keeps report.md, metadata.json, raw-data.json and a screenshot of the final frame. Video differs:

TIMEOUT and BLOCKED: video is off by default. Turn it on in Settings, under Keep failure video for.
UNREACHABLE: no video is offered, because no page loaded and the recording would be blank.
CRASHED: no video is offered, because the recording of a dead browser comes out broken.

Only Playwright web and emulated mobile runs record video. A real Android phone, a Selenium run and an API test do not.

With Jira switched on, a TIMEOUT files a ticket, and a block, a crash and an unreachable site never do.

What are the limits?

The classification reads error text. A message you wrote yourself that contains 403, Forbidden or Access Denied reads as BLOCKED. Keep custom assertion messages specific, and open metadata.json when a verdict surprises you.
A TIMEOUT is not proof that the application works. A test that times out on every run deserves a look.
A longer timeout does not hide a broken element: it fails later.
These verdicts name the cause of a failed run. They do not remove the cause: a blocked site is still blocked until you change how you reach it.

Where can I see it?

The Evidence page follows one failed test and lists every file each verdict leaves on your disk. The verdicts guide explains in what order the engine decides all 14. The 30-day trial runs the same engine on your own tests.