Teaching an Agent to Triage Flaky Tests — and to Know When Not To

Teaching an Agent to Triage Flaky Tests — and to Know When Not To

Every team with a mature Cypress suite eventually meets the same quiet adversary: the flaky test. Not the broken test—the flaky one. It fails, then passes on retry, so nobody files it, so it lingers. Over time, "just re-run it" stops being a workaround and becomes a habit, and that habit costs you the one thing an end-to-end suite is supposed to give you: confidence.

We decided to stop treating flake as background noise and start treating it as a signal worth acting on every single morning. So we built an autonomous agent to do the triage a human never quite finds time for. What we learned building it had less to do with automation than we expected, and a lot to do with judgment.

Turning a Number Nobody Reads into a Decision

The data was already there. Cypress Cloud records a flaky_test_count on every CI run and can tell you precisely which tests flaked—the error name, the message, the failing line of the stack, attempt by attempt. That is a rich signal. It was also, for us, a number that scrolled past unread.

So we gave the agent a way to read it directly. We use the Cypress MCP server to dial into Cypress Cloud. It is the pipe through which the agent pulls recent runs and, for any run that flaked, the per-attempt failure detail behind each one. No scraping dashboards. No exporting CSVs. The agent simply asks Cypress Cloud what happened and gets back structured data, which is exactly what makes the rest of the loop possible.

Each weekday it scans the last fifty completed CI runs, tallies flakiness per test, and applies a deliberately blunt rule: a test is only a candidate if it flaked in three or more distinct runs. One bad night is not a pattern. A test that wobbles across three separate runs is telling you something. That threshold is the difference between a useful beacon and an alarm everyone learns to mute.

From there the loop is end to end, and this is where the second tool earns its place. We use Claude to review and fix the tests. It reads the failure detail surfaced through the Cypress MCP server, opens the actual spec file, groups failures into distinct modes, and writes a concrete, code-level diagnosis. Where a spec change is genuinely the right answer, it proposes the fix itself.

The agent then deduplicates against Jira so the same flake is never filed twice, creates one ticket per flaky test on our QE board, @mentions the appropriate team, posts a summary to Slack, and updates a running tally in Confluence so the trend is visible to anyone who cares to look.

The Most Important Thing It Does Is Refuse

Here is where the project stopped being about automation.

The obvious way to build a test-fixing agent is to let it fix everything. Point it at a red test and let it turn the test green. But a flaky test is often not a test problem at all—it is a product telling you the truth in an inconvenient way.

A backend might return a 500 during a read-after-write race. A page might render late because grading happens server-side. Increase the Cypress timeout and the test goes green, but you have not removed the flake. You have hidden a real bug and taught your suite to lie.

So the single most important rule in the agent is one of restraint.

It proposes a fix only when a spec change genuinely stabilizes the test. When the root cause is application-side or infrastructure-related, it files the ticket, classifies it clearly, and opens no pull request at all.

It is not there to make tests pass.

It is there to tell you why they don't.

When it does propose a fix, that fix stays inside a narrow lane:

Draft pull requests only
Cypress spec files only
Never application source code
Clearly labeled as auto-generated and unverified

The pull request's own CI run is the real verification. The agent's confidence is never mistaken for correctness.

Giving It a Memory—and a Rule About That Memory

Diagnosis is the part of this an agent is most likely to get wrong, so we gave it something to lean on: a committed file of confirmed root-cause patterns.

Each entry pairs a spec family and an error signature with:

The confirmed root cause
The correct classification
The band-aid fix to avoid

Before diagnosing anything, the agent reads this file and reuses what a human has already worked out rather than re-deriving a cause it has gotten wrong before.

The rule governing that file matters just as much as the file itself.

Confirmed causes only.

Never a fresh guess.

Codifying an unverified diagnosis would teach the agent to repeat exactly the mistake the file exists to prevent. A provisional hunch stays in the Jira ticket, where a human can weigh it. It never graduates to institutional memory until someone confirms it.

What Actually Changed

Flaky tests now get triaged within minutes instead of never.

The team wakes up to a Slack summary and a short stack of tickets, many with a draft fix already attached and the rest clearly marked as something a person needs to investigate.

We can finally see, month over month, whether the suite is getting healthier.

The outcome we did not anticipate is that a meaningful share of what we had lazily filed under "flaky" turned out to be genuine product race conditions—bugs that now receive a bug's attention instead of a timeout bump.

If I could leave other Cypress teams with one reflection, it is this:

The agent's most valuable trait is not how much it automates. It is that it knows the difference between a test problem and a product bug, and refuses to paper over the second.

AI can accelerate the triage none of us had time for. But the judgment about what deserves a fix—and what deserves the truth—still has to be built in on purpose.