Your Cypress Retry Passed - But the Test Is Still Telling You Something

Attempt one fails. Attempt two passes. CI turns green.

Did the system recover from harmless noise, or did the retry hide an intermittent product defect?

Cypress retries are useful because browser tests meet networks, animations, databases, and asynchronous UI state. The mistake is treating a passing retry as if the first failure never happened.

A result has two dimensions

Store the final verdict and the attempt history separately:

Attempts	Final result	Reliability signal
Pass	Pass	Stable in this run
Fail, Pass	Pass	Flaky evidence
Fail, Fail, Fail	Fail	Consistent failure
Pass, Fail	Depends on strategy	Non-repeatable behavior

A pipeline may reasonably allow Fail, Pass while a team investigates. It should not classify it as equivalent to Pass.

The first failed attempt preserves evidence about the unstable boundary: an element appeared late, a request escaped the stub, state leaked from a previous test, or a backend was unavailable.

Choose retry semantics explicitly

Standard Cypress retries stop when a test passes:

import { defineConfig } from "cypress";

export default defineConfig({
  retries: {
    runMode: 2,
    openMode: 0,
  },
});

That is an availability-oriented policy: give the test more chances before failing the build.

For a high-signal suite, Cypress also provides experimental retry strategies. This configuration fails the final result if any attempt failed, while stopping as soon as a later pass proves flakiness:

export default defineConfig({
  retries: {
    experimentalStrategy: "detect-flake-but-always-fail",
    experimentalOptions: {
      maxRetries: 2,
      stopIfAnyPassed: true,
    },
    openMode: true,
    runMode: true,
  },
});

That policy is intentionally strict. Use it for release-critical journeys where intermittent behavior is itself a failure. Experimental retries are global, use boolean openMode and runMode, and can change in future Cypress releases, so pin and review the version.

Preserve the failed attempt

Cypress gives failed retry screenshots attempt-number suffixes. If you use Cypress Cloud, retry history, screenshots, video, and failure details are available in the test result history.

For local CI artifacts, retain evidence when any attempt failed—not only when the final verdict failed. In the Node event lifecycle, each test result contains attempts:

function hadFailedAttempt(results: CypressCommandLine.CypressRunResult) {
  if (results.status !== "finished") return true;

  return results.runs.some((run) =>
    run.tests.some((test) =>
      test.attempts.some((attempt) => attempt.state === "failed"),
    ),
  );
}

The exact reporting integration depends on your runner, but the policy should be clear: a flaky pass may unblock the job, yet it must still create a triage record with its failed-attempt artifacts.

Diagnose the class of flake

Do not begin by increasing the retry count. Classify the failure first.

Application race

The UI exposes an intermediate state as actionable. Fix the application state model or wait on a user-visible invariant.

Test race

The test uses arbitrary sleeps or reads before the page is ready. Replace time guesses with retryable queries and assertions.

State leakage

The outcome depends on order, cached data, or an uncleared session. Run the spec repeatedly in isolation and in shuffled suite order.

Dependency instability

A real network or service participates in a test intended to be deterministic. Stub it, or move the test to a separately classified integration suite.

Genuine environmental noise

Occasional browser or infrastructure failures do happen. Track their rate so “temporary” does not become permanent background failure.

Treat flake rate as owned work

Use a rolling window, not one alarming run:

flake rate = tests with mixed attempts / tests executed

Segment by spec, browser, operating system, and failure signature. Do not put unbounded error text into metric labels; keep detailed evidence in logs and artifacts.

Set an ownership rule: quarantine only with an issue, an owner, an expiry, and a reason. A quarantined test without an exit condition is a deleted test with extra steps.

Give flakiness its own budget

A release pipeline can separate product failure from suite instability without ignoring either:

type ReliabilityGate = {
  failedTests: number;
  flakyTests: number;
  maxFlakyTests: number;
};

function mayRelease(gate: ReliabilityGate) {
  return gate.failedTests === 0 && gate.flakyTests <= gate.maxFlakyTests;
}

That example is a policy shape, not a recommended threshold. A payment or authentication journey may allow zero flaky tests; a lower-risk suite may temporarily tolerate a small, owned backlog. Tighten the budget as causes are removed, and fail immediately when a known critical spec exhibits mixed attempts.

Trend the first-failure signature separately from the final verdict. If ten unrelated tests start failing first on the same network wait, the shared dependency—not ten selectors—is probably the better investigation target.

Retries answer a narrow question: can this test pass on another attempt? They do not answer whether the application is reliable. Preserve the first failure, choose the final-status policy deliberately, and make flaky green different from stable green.

References

• Cypress test retries
• Cypress experimental retry strategies