A flaky test gives you an unusually valuable experiment: the same test failed and then passed inside one recorded run.
Instead of rerunning it until the failure disappears, compare the attempts.
The Cypress Cloud CLI exposes Test Replay commands, network requests, and console logs as JSON. Its local replay viewer requires version 0.4.0 or newer, and the current CLI documentation requires Node.js 22.21.0 or later. Cloud CLI is currently Beta, so use the latest supported version for exploration and pin the validated version in repeatable automation.
I am a Cypress Ambassador. IDs and outputs below are synthetic, and the workflow was checked against the public Cloud CLI documentation.
Find a flaky test
npm install --global @cypress/cloud@latest
cy-cloud version
cy-cloud status
cy-cloud test list \
--projectId <projectId> \
--runNumber <runNumber> \
--status passed \
| jq '.tests[]
| select((.attempts | length) > 1)
| {testId, testName, attempts}'
A passing status with multiple attempts means at least one earlier attempt did not pass.
Pull both timelines
Replay attempts are one-indexed. If attempt one failed and attempt two passed:
cy-cloud replay timeline \
--testId <testId> \
--attempt 1 \
--commands --network --logs \
> failed-attempt.json
cy-cloud replay timeline \
--testId <testId> \
--attempt 2 \
--commands --network --logs \
> passing-attempt.json
Look for the first material divergence, not merely the final assertion.
| Evidence | Failed | Passed |
|---|---|---|
| Discount response | Arrived after assertion | Arrived before assertion |
| HTTP status | 200 | 200 |
| Console error | None | None |
| Displayed total | $50.00 |
$45.00 |
The API did not fail. The relative ordering changed. That points toward synchronization or a product race rather than a bad selector.
Use a consistent classification so the investigation does not become “flaky” versus “not flaky”:
| Class | Evidence pattern | Typical next step |
|---|---|---|
| Product race | DOM or state differs before the assertion | Fix application coordination |
| Test synchronization | Expected event occurs later, with no product defect | Wait on that observable event |
| Test data | Attempts receive different fixtures or records | Isolate and seed deterministically |
| Environment | Browser crash, unavailable dependency, resource pressure | Repair CI/runtime conditions |
| Unknown | Recording lacks the decisive boundary | Reproduce with more targeted evidence |
The label is a hypothesis backed by cited events, not a substitute for reproduction.
Narrow to the failure window
Large timelines produce noise. Ask for only the commands around the failed assertion plus nearby browser traffic:
cy-cloud replay timeline \
--testId <testId> \
--attempt 1 \
--commands \
--aroundFailure 5 \
--network \
--logs
Use cy-cloud replay info --testId <testId> first when you want a summary of what the replay contains.
Inspect recorded DOM state
replay view requires Cloud CLI 0.4.0+ and a Chromium-family browser:
cy-cloud replay view open --testId <testId>
cy-cloud replay view move --attempt 1 --time failure
cy-cloud replay view inspect \
--testId <testId> \
--selector '[data-cy="cart-total"]'
You can also pin before and after a specific command ID found in the timeline. This helps determine whether an earlier action changed the DOM, styles, or attributes before the assertion surfaced the problem.
Give an AI agent a bounded question
Do not ask, “Make the test pass.” Ask:
Compare failed-attempt.json with passing-attempt.json.
Identify the first material difference before the assertion.
Cite the event IDs supporting the conclusion.
Classify the cause as product, synchronization, test data,
environment, or unknown.
Propose a fix, but do not edit files.
Do not add fixed waits or weaken assertions.
That prompt makes unsupported guesses visible. If the evidence is insufficient, unknown is an acceptable answer.
Keep the evidence packet smaller than the entire replay. Include the failed and passing windows, run metadata, browser/version information, and the exact test source revision. Exclude unrelated request bodies and console content. A bounded packet reduces both privacy exposure and the chance that an AI assistant anchors on irrelevant details.
Verify locally
After reviewing the proposal:
- reproduce the scenario with controlled data;
- apply the smallest change;
- run the affected spec repeatedly;
- introduce a negative control that should fail;
- confirm the test still protects the business behavior.
This article deliberately covers recorded CI evidence, not a live local Cypress session. cypress tap is useful for the final local run, but it solves a different part of the workflow.
For CI-only failures, rerun the reviewed fix in a pinned Cypress Docker image before changing the test. A full image tag or digest holds Node and browser versions steady:
export CYPRESS_IMAGE='cypress/browsers@sha256:<approved-digest>'
docker compose run --rm cypress \
npx cypress run --spec cypress/e2e/checkout.cy.ts --browser chrome
Store the image digest and Cypress/browser versions with the failed and passing JSON. Docker cannot reproduce provider load or every CI resource constraint, but it can remove silent runner upgrades from the hypothesis set.
Security boundaries
Cloud CLI is read-only and follows the authenticated user's Cypress Cloud permissions. OAuth is recommended locally; CYPRESS_CLOUD_TOKEN is intended for non-interactive environments.
Still review:
- which projects the identity can read;
- whether the agent can access shell secrets;
- which request/response bodies Test Replay recorded;
- local replay-cache retention;
- whether generated JSON is safe to attach to an issue.
Structured evidence is better than a guess. It is not automatically safe evidence.
Cloud CLI requests are rate limited, so CI automation should fetch once, cache the resulting artifact within the job, and avoid letting several agents repeatedly query the same replay.
What would you learn by comparing your most recent flaky test's failed and passing attempts side by side?
Top comments (0)