You inherit a suite. It takes 40 minutes, a third of it is red, and nobody can tell you whether red means a bug. The first suggestion in the room is always the same: it's Cypress, we should move to Playwright.
That's a conclusion wearing the clothes of a diagnosis. There are three defensible verdicts for a suite you don't trust — refactor it, rebuild it, or delete parts of it — and flakiness has at least four independent causes, only one of which is the tool. On the projects our team has inherited, the failure has almost always sat in test design and test data, which no migration touches. On Cognota we went from roughly 120 failures per regression run to 0–1 without changing the framework at all. On QIMA we stayed on Cypress too, and upgraded it.
Here's the audit we run before anyone picks a verdict.
Audit before you decide — the four layers
Score four layers separately, because they fail independently and each has a different fix.
1. Test design and data. Does each test create the state it asserts on, and remove it afterwards? Are users generated per-run or is there one qa_user@… shared by 400 specs? Does a spec pass when you run it alone but fail in the suite? Do tests reach application state through the UI when an API endpoint exists?
2. Framework currency. What version are you on, how far behind is it, and is there an upgrade path? Are you calling interfaces the vendor deprecated two majors ago? Is the runner still maintained upstream at all? "Old" and "unmaintained" are different findings — on QIMA the diagnosis was an outdated Cypress version, and the fix was to move to the current one, not off it.
3. Execution infrastructure. Where do runs get triggered — a laptop, a nightly cron, a pipeline gate? Can the suite run in parallel? Is it pinned to one environment because URLs and credentials are hardcoded? Are results visible fast enough to act on? This layer can be the only broken one: on SoftNAS the published starting state was an improperly configured automation structure with no API testing, no performance testing, and automated run results "delivered very slowly." Four engineers over a year built the API suite from scratch, put a performance suite in CI, and got 45% faster delivery of automation test results, alongside 1,000+ scripts, 80% app coverage, and 200+ bugs reported, 30% of them critical. Nobody swapped the language or the runner.
4. Ownership. Who fixes a red test, within what SLA, and what happens if they don't? An unowned suite decays at a predictable rate regardless of how good the framework is.
The scoring rule: one failing layer points to refactor, two or more point to rebuild. With one caveat that outranks the count — a single architectural failure can outweigh three healthy layers. If the suite physically cannot run in parallel and your regression has to fit in a nightly window, that one finding decides it.
None of the teams below ran this exact audit; it's a generalization from what the engagements actually fixed.
Telling test-design failure from tool failure
Three probes, in this order:
# 1. Does the failing test pass on its own?
npx cypress run --spec cypress/e2e/billing/invoice.cy.ts
# 2. Does it survive repetition and concurrency?
npx playwright test billing/ --repeat-each=5 --workers=4
# 3. Does it survive a clean database instead of an inherited one?
APP_URL=$FRESH_ENV npm run test:e2e -- --grep @billing
Passes alone, fails in the suite: order dependency or shared state — test design. Fails identically in every mode: a real defect or a stale locator — test design again, or an actual bug you should be grateful for. Fails only above two workers: concurrency or data collision — still test design. Only after all three come back clean does the tool become a suspect.
The published research points the same way. The canonical root-cause study of flaky tests (Luo et al., FSE 2014 — old, but nothing since has displaced its taxonomy) classified 201 flaky-test-fixing commits across 51 Apache projects and found async wait at 45%, concurrency at 20%, and test order dependency at 12% — 77% of fixes in the top three categories, all of them defects in test code, none of them a framework you can migrate away from. A 2025 analysis of 10,000 test-suite runs across 24 Java projects found that 75% of flaky tests cluster with other flaky tests sharing a root cause, with a mean cluster size of 13.5 — which is why fixing one data-provisioning bug so often takes out dozens of failures at once.
And for calibration on what "normal" looks like: Google reported that about 1.5% of all test runs return a flaky result and almost 16% of their tests carry some level of flakiness. If your suite is at a third red, you are not looking at industry-baseline flake. You're looking at a defect.
Refactor — when the tests are wrong and the foundation isn't
If the framework is maintained, upgradable, and CI is fixable, rewrite the tests, not the stack.
Cognota is the clearest version of this we've published. Cypress, React/Express app, GitHub Actions, two automation QA engineers, one year. The starting state hit every layer of the audit: static uncleaned test data, static test users, deprecated interfaces, tests coupled to each other, application state reached by clicking through the UI, and a single supported environment. About 130 tests were refactored and around 30 bugs reported. The case study lists roughly 120 failures per regression run before, and 0–1 after; separately it reports a 90% reduction in flakiness and a 20% reduction in execution time. Those are two published figures that don't reconcile into one number, so treat them as two.
The mechanical change was ownership of data. Before:
// every spec logged in as the same seeded user and asserted on rows
// some earlier spec had created
beforeEach(() => {
cy.login('qa_user@example.test', 'Password123');
cy.visit('/courses');
});
it('archives a course', () => {
cy.contains('[data-test=course-row]', 'Onboarding 101')
.find('[data-test=archive]')
.click();
cy.contains('Archived').should('be.visible');
});
That test asserts on Onboarding 101 existing, unarchived, at the moment it runs. It is a bet on execution order. After:
// each test owns its user and its data, reaches state via API, and cleans up
beforeEach(function () {
cy.task('createUser').then((user) => {
this.user = user;
return cy.request('POST', '/api/courses', {
title: course-${crypto.randomUUID()},
ownerId: user.id,
});
}).then(({ body }) => {
this.course = body;
cy.session(this.user.email, () => cy.login(this.user.email, this.user.password));
cy.visit(/courses/${this.course.id});
});
});
afterEach(function () {
cy.request({ method: 'DELETE', url: /api/courses/${this.course.id}, failOnStatusCode: false });
cy.task('deleteUser', this.user.id);
});
Note what this is not: it isn't a longer wait, a cy.wait(2000), or a retry. Cypress documents that tests "should always be able to be run independently" and clears cookies and storage between them by default — but it cannot clear your database, and IndexedDB persists too. Isolation you don't build yourself, you don't have.
The other two refactor moves are cheaper and get skipped more often. Environment coupling:
// cypress.config.ts — one suite, N environments, zero hardcoded URLs
export default defineConfig({
e2e: {
baseUrl: process.env.APP_URL ?? 'http://localhost:3000',
env: { apiUrl: process.env.API_URL ?? 'http://localhost:4000' },
retries: { runMode: 2, openMode: 0 },
},
});
Cognota went from one supported environment to local plus staging on exactly this kind of config change. And retries: two in run mode is a diagnostic budget, not a mask. Record which tests only ever pass on attempt two and treat that list as a defect backlog, not as green.
QIMA is the same verdict at a larger scale — three engineers since 2021, on an outdated Cypress version with no performance autotests and release testing taking 3–4 days. The refactor moved to the current Cypress, restructured main classes around Page Object, made autotests environment-independent, generated about 90% of test data by API — which the case study ties to a 2.5x reduction in execution time — and put k6 autotests into the release flow. Autotests went from around 2,000 to around 3,000, coverage above 95%, release testing down to 1–2 days.
The limitation on refactoring is not technical. It's that a refactor decays back to the starting state if the team keeps writing tests the old way. Both engagements changed structural and configuration rules — Page Object boundaries, single-responsibility page classes, config-driven environments, API-first setup — not just the test bodies. Without that, you're paying refactor cost on a schedule.
Rebuild — what actually justifies starting over
Rebuild when three things are true at once: little in the existing suite is worth preserving, the suite is unmaintained rather than broken, and the target has to support something the current setup structurally never did — parallel execution, CI gating, cross-machine sharding.
Agorapulse met all three. The published before-state: a Cypress solution whose test scripts "were not maintained, they were unstable and unreliable," CI pipelines not configured for the automation framework at all, only 15% of tests automated, and one AQA engineer who couldn't cover QA activities. The rebuild landed on Playwright with TypeScript, GitHub Actions, Testmo for test management, and Slack failure alerts, reaching 1.6k+ automated test scripts, 90–95% of tests automated, and a nightly regression that finishes in about four hours; the team went from two automation engineers to one by the end.
Do not read that as "Playwright fixed it." The case describes a phased program: test management introduced, CI built, flaky and outdated tests eliminated, then coverage grown. The runner was one input.
Sprinklr is the scale version — five years, seven QA engineers, Java and Selenium. The published problems are a rebuild checklist in themselves: tests weren't reliable, took too long, "couldn't be integrated with other testing and DevOps tools," weren't scalable or maintainable, and "each run had a different number of randomly failed tests." The rebuilt framework runs 16 threads across multiple machines on 10 virtual machines with 100+ devices, and populates prerequisite test data directly into the database for speed. Reported outcomes: ~2,000 automated tests, 10,000+ test cases written, 90%+ coverage, 50% reduction in testing time, and roughly 12,000 bugs reported with 45% at major status.
Parallelism is where a rebuild earns its cost, and it forces the data question immediately, because sixteen threads sharing one account is just a race condition with better hardware. Playwright's docs note that workers are independent OS processes with isolated browser contexts, and that you can key data off testInfo.workerIndex to keep them from colliding:
export const test = base.extend<{ account: Account }>({
account: async ({ request }, use, testInfo) => {
const res = await request.post('/api/accounts', {
data: { email: w${testInfo.workerIndex}-${testInfo.testId}@example.test },
});
const account = await res.json();
await use(account);
await request.delete(/api/accounts/${account.id});
},
});
The honest limitation on both cases: they ran long, with dedicated engineers — Agorapulse since 2025, Sprinklr for five years — and neither publishes how the old suite gated releases while the new one was being built. That interim is the part of a rebuild proposal teams underestimate, and it's the strongest argument for refactoring instead when refactoring is defensible.
Delete — tests that cost more than they catch
Deletion is a legitimate verdict, and it's the one teams skip because removing tests looks like moving backwards.
Three deletion criteria:
- It fails unpredictably and nobody triages it. A test whose red is routinely ignored has already stopped being a test; it's just consuming pipeline minutes and eroding trust in every other result.
- It asserts on incidental UI. Exact copy, class names, element ordering that no requirement pins down. These fail on every design tweak and catch nothing.
- It duplicates a sibling. Three E2E specs covering one validation rule that a single API test covers deterministically.
Do this during stabilization, before you report a coverage number — a coverage figure computed over untrusted tests is a number about nothing. Agorapulse's rebuild explicitly eliminated flaky and outdated tests as a phase, before scaling to 1.6k+ scripts. Sprinklr's "different number of randomly failed tests" per run is the same signal from the other end: a suite where deletion never happened.
Quarantine is the humane version, as long as quarantine expires:
// flaky-quarantine.json — every entry has an owner and a death date
[
{ "spec": "e2e/billing/dunning.spec.ts", "owner": "@ivan", "expires": "2026-10-01" },
{ "spec": "e2e/reports/export.spec.ts", "owner": "@dana", "expires": "2026-09-20" }
]
CI fails on expired quarantine — the entry is fixed or the test is deleted
node -e '
const now = Date.now();
const stale = require("./flaky-quarantine.json").filter(e => Date.parse(e.expires) < now);
if (stale.length) { console.error("expired quarantine:", stale); process.exit(1); }
'
The limitation is real: deleting without a coverage map hides risk, and you won't notice until an escaped defect tells you. Pair removals with documented scenarios. On The Mortgage Office — since 2025, one automation QA engineer plus four foundational QA engineers on Playwright, k6, and Azure Pipelines — the documentation was the deliverable alongside the automation: 10+ detailed documents for major features, 70+ E2E cases with descriptions and steps, 40+ test cases for new features, a 100+ scenario automation suite, and 10+ load scenarios in CI. Automated smoke went from 1 hour to 20 minutes. Regression went from 2 days to 3 hours — both of those figures are for regression run manually, so don't read the second one as an automation result.
Most "migrations" are test-data projects in disguise
Line up what actually changed across these engagements and the pattern is hard to miss. Cognota: static uncleaned data to on-the-fly generation with cleanup, static to dynamic users. QIMA: ~90% of test data generated by API. Sprinklr: prerequisite data written straight into the database. The Mortgage Office: API-based test data generation across five distinct data patterns. SoftNAS: an API automation suite built from scratch.
Shared, static, uncleaned state is the dominant source of the failures teams attribute to their framework — and it is exactly the defect a migration carries across untouched, because you port the specs and the fixtures port with them. Two of the five teams above are still on Cypress. One of them upgraded it.
The assumption that makes API-first setup work is worth stating, because it's load-bearing: it needs endpoints that are stable, documented, and permissioned for a test principal. Where those don't exist, you're building test-only endpoints or writing to the database directly, and both are real engineering work with their own maintenance cost.
And the cost, plainly: these engagements ran between one and five years with dedicated engineers — Cognota one year with two, SoftNAS one year with four, Sprinklr five years with seven. None of them is a quarter. None of the published cases reports defect-escape rates or release-frequency changes, so nothing here supports a claim that coverage bought either.
The asymmetry is what should drive the decision. Refactoring is cheaper, reversible, and usually correct, because the defect is usually in the tests. Rebuilding is defensible only when little is worth preserving and the architecture genuinely can't carry the coverage you need — and it costs you a period where the old suite is gating releases badly and the new one isn't gating them yet. Deleting is free, immediate, and skipped anyway because the metric it moves looks like a regression.
Whichever verdict you land on, the artifact worth keeping is the audit. Four layers, scored, with evidence — that's the document that stops the next engineer from proposing a migration on layer two when the failure is on layer one.
Top comments (0)