DEV Community: Mykhailo Krasnovskyi

When Should You Refactor Automated Tests? 7 Warning Signs

Mykhailo Krasnovskyi — Mon, 12 Oct 2026 11:24:28 +0000

Only refactor your automated tests when they are still testing the correct things, but the code underlying them makes each change slow, fragile, or risky, and only rewrite them when they are testing the wrong things or when the framework can no longer keep up with where your product is heading.

The majority of suites should be refactored, while only a small number need to be rewritten. The expensive error is to confuse the two.

The 7 Warning Signs

One product change breaks dozens of tests
Every test starts with 40 lines of setup
Sleeps and retries hold the suite together
Tests only pass in one order
Nobody can tell what a test checks
UI tests check logic that doesn't need a UI
Only one person can fix a broken test

Entries 1 through 5 and 7 generally require a refactor; a rewrite is justified only when the framework, technology stack, or testing strategy is no longer suitable.

Why This Matters

A designer changes the name of one button, renaming "Checkout" to "Continue to payment," and the following morning, 63 automated tests fail.

The product functions properly, but the pipeline shows up as red, and one locator, which has been copied into 63 files, occupies the entire day of two automation engineers.

The true issue isn't whether we should fix these tests; it's the cost this suite's structure imposes on you each time the product changes.

Why Test Code Rots Faster Than Product Code

The product code receives a code review, has architecture discussions, and is supervised by a tech lead who cares about structure; test code, on the other hand, usually gets none of these things, is written under a deadline, is merged as soon as it passes, and is then ignored until it stops working.

3 habits speed up the decay:

The act of copying and pasting is safe. It's better than creating a helper when you have 200 copies of the same setup.
There is no monitoring of test maintenance, and the time spent on fixing tests is absorbed into "QA work."
The tests persist after the people who created them have left, and subsequent engineers continue using the framework even if they do not fully understand it.

1. One Product Change Breaks Dozens of Tests

The issue is that a single item, for example, a locator, URL, or label, is stored in numerous locations.

javascript
// test_checkout_guest.js
await page.click('#btn-checkout-v2');

// test_checkout_promo.js
await page.click('#btn-checkout-v2');

// ...and 61 more files

The fix: **use the **Page Object pattern. Each screen gets a class that holds its locators and actions. Tests call those actions instead of touching elements directly.

javascript
// pages/CartPage.js
export class CartPage {
constructor(page) { this.page = page; }
async proceedToCheckout() {
await this.page.getByTestId('checkout-button').click();
}
}

The result is a single product change and a single test change.

2. Every Test Starts With 40 Lines of Setup

You have to scroll past half a screen worth of login screens, data entry, and navigation before getting to the actual check, by which time the setup process has taken control of the suite.

It hurts in 3 ways:

-Slow runs.Every test logs in through the UI, even when it only checks a report page.
-Buried intent.The real assertion drowns in boilerplate, and reviewers stop reading.
-Painful updates.Change the login flow, and every copied setup breaks.

The solution: is to move the setup process into the fixtures and helpers. Log in once via the API and then reuse the session. Use factory functions to create the test data rather than filling in the forms.

The result: is that the tests are shorter, the runs take less time, and new engineers are able to read the suite on their first day.

3. Sleeps and Retries Hold the Suite Together

Search your code for sleep, wait(5000), Thread.sleep, and retry annotations. Each one marks a failure someone patched instead of fixing.

The patches cost you:

Slower runs with every hard-coded pause
Flakiness that is hidden - the flaky tests (that is, tests which pass and fail when the same code is run without any changes) - remain flaky even after retries.
Bugs in the products are masked, while the real timing problems are accompanied by testing issues.

The solution is to substitute the use of fixed sleeps with waits for an actual condition - such as an element appearing, a request finishing, or an API returning the expected status - and to eliminate each retry in turn while addressing the underlying cause.

The result is that feedback becomes faster, and the occurrence of red builds up trust within the team.

4. Tests Only Pass in One Order

Run the suite in a random order. If it then fails, it is because the tests depend on one another.

Common symptoms:

Test B expects a user that Test A created
Tests share one account, and one test changes its settings for all
The suite passes locally but fails in parallel CI runs

The importance lies in the fact that order-dependent tests prevent parallel execution and assign fault to the wrong test; test B fails even though it was caused by test A.

The fix: make every test independent. Each one creates its own data, cleans up after itself, and assumes nothing about what ran before.

The result is parallel runs, shorter pipelines, and failures that indicate the correct area.

5. Nobody Can Tell What a Test Checks

If there's a test your team hasn't written recently and you can't explain in one sentence what bug it would detect, then the test has lost its value, even if it passes.

Warning signs:

Vague names like test_checkout_3 or testFlowNew
No assertions. The test clicks through a flow and passes as long as nothing crashes.
Giant assertions. Thirty fields checked at once, so nobody knows which one mattered.
Commented-out checks disabled "temporarily"

The fix: name tests after the behavior they protect, for example guest_user_can_apply_promo_code_at_checkout. Verify one clear outcome per test and delete tests that can't justify themselves.

The payoff: a failing test's name tells your team exactly what broke.

6. UI Tests Check Logic That Doesn't Need a UI

The tax calculations, the discount rules, the permission checks, and the data validation constitute business logic and can be checked at the API level in a matter of milliseconds without being affected by any layout changes.

When most checks run through the UI, you pay the highest price for each one:

Longest execution time
Most flakiness from rendering and timing
Most maintenance whenever the design changes

Solution: transfer the logic checks to the API or service layer and continue to carry out UI tests for those situations in which only a UI can provide proof, such as users being able to complete key processes like signing up and checking out.

The result is a set of tools that are both faster and cheaper while still protecting critical flows.

7. Only One Person Can Fix a Broken Test

When the same engineer is the one people refer to in the case of a failed payment test, the issue is structural rather than staffing-related.

It looks like this:

New automation engineers need weeks before touching the framework
Custom utilities have no documentation
Nobody remembers why key framework decisions were made

Why it matters: when that person is out or leaves, the suite stalls. Broken tests pile up, and quarantined tests (tests pulled from runs "for now") become permanent.

The fix: simplify custom layers, follow patterns your team already knows, and keep a short README next to the code explaining how the framework fits together.

The payoff: any engineer can maintain the suite, and it survives team changes.

The Bigger Picture: Test Automation Debt

Such signs seldom occur in isolation; taken together, they constitute what is known as test automation debt, that is, the increasing cost of maintaining a suite relative to the value it provides.

Refactoring clears some of that debt. Deciding which fixes to make first and proving they have worked requires a more comprehensive plan. The guide on test automation debt explains how to evaluate the debt and prioritize the work to clean it up.

Refactor vs. Rewrite: How to Decide

An unnecessary rewrite takes months. The time required to refactor a suite that needed a rewrite is used up at a slower rate.

Clear signals you need a rewrite

You can't upgrade the framework without breaking most of the suite
The tool doesn't fit your stack, such as a desktop-era tool testing a modern single-page or mobile app
The suite never matched your real risks. Low-value screens are heavily covered, while payments and data integrity get little.
A refactor would touch more than half the tests anyway

Still unsure? Run a POC

Build one module within the new structure as a Proof of Concept and run it alongside the existing one for a few sprints before comparing the execution time, flakiness, and maintenance hours.

Real numbers settle the debate faster than meetings.

The Mistake Teams Keep Making

The most common mistake is to halt all testing activity to "fix the suite" in a single major effort. This approach almost never succeeds:

Feature work continues, so the product changes under the refactor
No new tests get written for weeks, leaving gaps in fresh code
The project overruns and gets cut halfway

The other mistake is not arranging the work and then hoping that the engineers will deal with it "when they have time" - deadlines always prevail.

If you want to find out more about what should be fixed, rewritten, or deleted, see our test automation debt guide. That document explains how to prioritize the cleanup without stopping new work.

Automated Test Refactoring Best Practices

Refactor in slices. Fix one pattern, like locators or setup, across one module, then move on.
Keep the suite green. Refactoring changes structure, not behavior. If a result changes, stop and investigate.
Separate refactoring from new tests. Mixed pull requests hide mistakes.
Touch it, clean it. Whoever edits a test leaves it slightly better.
Review test code like product code. Same standards, same reviewers.
Measure before and after. Track maintenance hours per sprint, execution time and flakiness to show what the work delivered.
Reserve capacity. A fixed share of every sprint beats a quarterly cleanup that never happens.

Quick Gut Check

One product change breaks many tests? Refactor locators and page objects.
Engineers rerun red builds without reading them? Fix waits, retries and test independence.
A new engineer can't understand a test in two minutes? Improve names, setup and assertions.
UI tests check business logic? Move those checks to the API layer.
Framework upgrades cause chaos? Consider a rewrite, starting with a POC.
Tests don't guard the flows that cost money? Rethinking your test strategy matters more than cleaner code.

Where to Get Help

Most in-house teams know their suite needs work. They don't have the hours. Engineers keep today's pipeline alive, developers ship features, and the refactor keeps sliding.

Our automated testing services team is 100% Middle and Senior QA Engineers. They assess your suite, decide with you whether it needs a refactor or a rewrite, and do the work alongside your releases, with no constant supervision from your side.

The result: faster runs, fewer false alarms and a suite that keeps pace with your product.

Want a straight answer on what your suite needs? Request a QA Assessment, and we'll show you where to start.

API Contract Testing vs Integration Testing: What's the Difference?

Mykhailo Krasnovskyi — Fri, 25 Sep 2026 15:02:31 +0000

A team we worked with ran 400 integration tests before every release. Full green suite. Deploy on Friday.

Payments broke in production two hours later.

The cause? A field got renamed in the response body. Every integration test ran on a shared staging environment where both services happened to be on compatible versions that week. The contract had already drifted underneath them.

Nothing local caught it. Nothing local was actually checking the contract. It was checking whether two specific deployed versions got along that particular day.

This mix-up costs teams real money. It comes from treating contract testing and integration testing like the same thing, wearing two different names. They're not. They catch different bugs, run at different speeds, and belong in different parts of your pipeline.

The One-Sentence Version

Integration testing checks whether real, running services work together correctly, end-to-end.

Contract testing verifies that two services agree on the shape of their communication without either running the other's actual code.

That's the whole split. Everything else is detail.

What Integration Testing Actually Tests

Integration tests spin up real dependencies. Real database. Real message queue. Real downstream API, or at least a faithful stand-in for one.

Then they run an actual scenario through the stack and check what comes out the other end.

A typical integration test for an order flow:

Create an order through the API
Confirm it lands in the database with the right status
Confirm the payment service gets called and returns a charge ID
Confirm the notification service sends a confirmation email
Confirm the order status updates to "confirmed"

This is valuable work. It's the only way to catch bugs living in the interaction itself, not just the interface.

Race conditions. Transaction rollbacks. Timing issues. Real business logic tangled across services. If your discount logic breaks when a coupon code and a loyalty tier stack in a specific order, only a real integration test finds that. A contract doesn't know what a coupon is.

But integration tests carry a cost that gets uglier as your architecture grows:

They need everything running at once. Every dependent service, healthy, in sync.
They're slow. Minutes, not seconds, once you're past a trivial suite. -** They're environment-fragile.** A test passes locally, then fails in CI because staging runs a different service version. -** Failures stay vague.** Something broke. Rarely what, or where, without real digging.

At three services, none of these bite hard. At fifteen, it starts eating whole afternoons.

What Contract Testing Actually Tests

Contract testing skips the "does this whole thing work" question. It asks a narrower one instead: Does this API still look the way its consumers expect it to look?

No live dependency chain. No shared staging environment.

The consumer service — say, checkout — writes a test against a mock of the provider, in this case, payments. It states exactly what request it sends and what response shape it expects in return. That expectation gets saved as a contract, usually a JSON file.

The provider team then runs that contract against their real, running service, in isolation, with zero consumer code involved.

If the provider's real response doesn't match the contract, verification fails. Right there. In the provider's own build. Before anything ships anywhere.

Here's what a Pact consumer test looks like in practice:

provider
.given('account 123 exists')
.uponReceiving('a request for account details')
.withRequest({
method: 'GET',
path: '/accounts/123',
})
.willRespondWith({
status: 200,
body: {
id: '123',
balance: like(500.00),
},
});

Notice what's missing. No real payment service. No database. No network call to anything live. Just a statement of expectations, turned into a check that actually runs.

Contract tests are fast — seconds, because there's nothing to spin up. They're precise about failure, because a broken contract points straight at the field or endpoint that changed.

And they scale in a way integration tests never do. Add a sixteenth service, and you get one more contract. Not a combinatorial mess of environments to keep in sync.

What they can't do: tell you the business logic is correct. A contract test confirms that balance is a number. It has zero opinion on whether that number is the right one.

Side-by-Side Comparison

Integration Testing

What runs: Real services, real dependencies
What it catches: Business logic bugs, timing issues, real end-to-end failures
Speed: Slow — minutes per run
Environment needs: Every dependent service running and healthy
Failure clarity: Vague — something broke somewhere in the chain
Scales with service count: Badly — more services means more coordination
Coverage of user journeys: Yes, full flows
Coverage of API shape drift: Only if the test happens to touch the changed field

Contract Testing

What runs: Mocked provider on the consumer side, real provider in isolation on the verification side
What it catches: Shape mismatches, breaking API changes, drift between what's expected and what's shipped
Speed: Fast — seconds
Environment needs: None — no shared environment at all
Failure clarity: Precise — this field, this endpoint, this contract
Scales with service count: Well — more services means more contracts, not more coordination
Coverage of user journeys: No — single interactions only
Coverage of API shape drift: Yes, by design

Where Contract Testing Wins Outright

Independent deployments. Payments wants to ship three times a day. Checkout ships weekly. Integration tests force coordination, or they run against stale versions and lie to you. Contract tests let payments verify against every active consumer contract on every build. Checkout verifies its own expectations without waiting for payments to deploy anywhere.

Fast feedback in CI. A contract test suite for one service boundary runs in seconds. You get an answer before your coffee's done. An integration suite covering the same boundary, with real dependencies, easily runs ten or twenty times longer.

Deployment gating. Most teams miss this part entirely. Tools like Pact support a can-i-deploy check. Before a service ships, it asks a broker whether its current version has been verified against every contract still in force. No means the deploy stops. Integration testing doesn't give you anything like this — it doesn't produce a portable, queryable compatibility record. Just a pass or fail for one run, in one environment, at one moment.

Debugging speed. A contract test fails, you know exactly what changed. An integration test fails, you're digging through logs across four services trying to figure out which one lied.

Where Integration Testing Wins Outright

Real bugs in real interactions. Contract testing checks shape. It has no idea whether your refund logic correctly triggers a retention workflow for enterprise annual accounts, but skips it for monthly ones. That's business logic playing out across services. Only a real integration test, hitting real code with a real scenario, catches that.

Full user journeys. Checkout isn't one API call. It's a cart, inventory check, payment, notification, order confirmation — in sequence, with state carried through the whole thing. Contract tests check each boundary on its own. They say nothing about whether the full journey holds together.

Data consistency across writes. Need to confirm a write to one service correctly triggers a downstream read in another, with real data flowing through? Contract testing can't help you there. That's an integration test's job. Full stop.

Catching timing and race conditions. Contract tests are stateless snapshots of expected shape. They don't run concurrently against a shared state. They can't surface the kind of race condition that only shows up when two requests hit a database in the wrong order.

The Mistake Teams Keep Making

The common failure isn't picking the wrong tool. It's picking one tool and dropping the other entirely.

Teams that go all-in on integration testing end up with slow pipelines, flaky environments, and breaking changes slipping through. The test that would've caught it wasn't run that day because it required a service that was down for maintenance.

Teams that go all-in on contract testing get fast, precise pipelines that miss real bugs in business logic. Nobody's running a full order through the system anymore. Everyone's happy their contracts are green while checkout quietly breaks in a way no contract could ever catch.

Neither approach alone is a real strategy. They're complementary layers, not competitors. That's the part that gets lost whenever someone tries to crown a winner.

How to Actually Split the Work

A reasonable line most teams land on, after enough pain:

Contract tests for every service-to-service API boundary. Every consumer states what it needs. Every provider verifies it on every build.
A smaller set of integration tests for the handful of user journeys that actually matter to the business. Checkout, signup — the flows that lose you money or customers if they break.
Deployment gates built on contract verification, not on waiting for a full integration suite to go green across every environment.
Integration tests running less often — nightly, or before major releases — instead of on every single commit. They're expensive, and running them 40 times a day buys you almost nothing extra.

This won't fit every team exactly. A payments-heavy fintech product probably needs more integration coverage on money-moving flows than a content platform does. But the shape of the split — contracts for boundaries, integration for journeys — holds up across most architectures we've seen.

A Quick Gut Check

Not sure which one you need for a given test? Ask:

Am I checking whether two services agree on a shape? Contract test.
Am I checking whether** a real scenario produces the right outcome? **Integration test.
Do I need this to run in seconds, on every commit? Contract test.
Do I need this to catch a bug that only shows up when real code runs against real code? Integration test.
Am I trying to stop a bad deployment before it ships? Contract test, with a gate.
Am I trying to prove the checkout flow works end to end? Integration test.

Most teams don't need to pick one. They need both, sized right, each doing the job it's actually good at.

Where to Go Deeper

Running microservices and breaking changes keep slipping through to production? The fix usually isn't more integration tests. It's contract testing at the boundaries, wired into your pipeline as a deployment gate.

Our guide on API contract testing for microservices walks through exactly how to set that up — including the Pact broker workflow and the can-i-deploy check that stops a broken contract before it ever reaches a real user.

How to Reduce Flaky Tests in CI/CD Without Rewriting Your Whole Test Suite

Mykhailo Krasnovskyi — Fri, 07 Aug 2026 21:26:37 +0000

Flaky tests rarely spread evenly across a suite. In most pipelines we audit, a small group of specs produces the majority of unreliable failures. So you can usually fix the pipeline without a rewrite.

This is a practical guide to stabilizing automation you already own. No definitions of flakiness, no "rewrite it in Playwright" advice. Just triage, root cause work, quarantine rules, and the metrics that tell you whether it worked.

Examples cover Playwright, Cypress, Selenium, GitHub Actions, and GitLab CI.

How can I reduce flaky tests in CI/CD without rewriting the entire test suite?

Find the small set of specs causing most failures, classify why each one fails, fix by category instead of one test at a time, and quarantine the rest so the pipeline stays honest.

A rewrite is tempting because the suite feels untrustworthy. But a rewrite moves the same design mistakes into new syntax. Hard-coded sleeps and shared test data will reappear in any framework. Fix the causes first. Then decide if you still need new tooling.

What causes flaky tests in CI/CD pipelines?

Almost every flaky test traces back to a small number of root causes. Classify before you fix, because the remedy differs by category.

The Selenium documentation is blunt about the first row. Race conditions between browser state and driver commands are "one of the primary causes of flaky tests." That is where most teams should start.

Why do tests fail in CI but pass locally?

This question comes up in every audit, and the answer is usually one of 6 things.

Speed. CI machines are often slower and more loaded than your laptop, so implicit timing assumptions break.
Parallelism. Local runs are frequently single-threaded. CI runs four or eight workers that fight over the same test data.
Headless rendering. Different viewport, no GPU, fonts missing, elements positioned differently.
Clean state. Your local browser has cookies, cached auth, and a warm database. CI starts empty.
Locale and timezone. Runners default to UTC. Date assertions written in your timezone drift.
Network shape. Local hits a dev server on localhost. CI crosses a network boundary with real latency.

As a fast diagnostic, run your suite locally with the same worker count and a fresh browser profile. If it fails there too, the problem is your tests, not the runner.

How should QA teams prioritize flaky tests in an existing automation suite?

Do not fix flaky tests in the order you find them. Rank them. Start by calculating a flake rate per spec over the last 30 days:

flake rate = (runs that failed then passed on retry) / total runs

Then score each test on three factors: flake rate, how often the spec runs, and whether it blocks merges. A test that fails 4% of the time on every pull request costs far more engineering hours than one failing 30% of the time in a nightly job.

Flaky test triage checklist

Run this on each candidate before writing a fix:

Does it fail in isolation, or only in a full parallel run?
Does it fail on a specific worker, browser, or shard?
Is there a hard-coded sleep or fixed timeout in the failure path?
Does it create or reuse data that another test touches?
Does it depend on a previous test, leaving the app in a certain state?
Does it call a third-party service that is not stubbed?
Do the failure timestamps cluster around deploys, backups, or cron jobs?
Which root cause row from the table above matches?
Fix now, quarantine, or delete?

Keep the last question open. Some flaky tests cover logic already tested at a lower level, so deleting them is a legitimate outcome.

How do you fix flaky automated tests in CI/CD?

Fix by category. The next 3 categories cover most of the work.

1. Replace sleeps with condition-based waits

A fixed sleep is a guess about timing. It fails when the app is slower and wastes time when it is faster.

Playwright assertions retry until they pass or time out, so this is usually enough:

// Fragile: assumes the row renders within 3 seconds

await page.waitForTimeout(3000);

await expect(page.locator('.order-row')).toHaveCount(1);

// Stable: retries the condition itself

await expect(page.locator('.order-row')).toHaveCount(1, { timeout: 10_000 });

In Cypress, wait on the request rather than the clock:

cy.intercept('POST', '/api/orders').as('createOrder');

cy.get('[data-cy=submit]').click();

cy.wait('@createOrder').its('response.statusCode').should('eq', 201);

cy.get('[data-cy=order-row]').should('have.length', 1);

For Selenium, use explicit waits and avoid one specific trap. The Selenium waits documentation warns against mixing implicit and explicit waits, because the combination produces unpredictable timeouts:

from selenium.common import NoSuchElementException, ElementNotInteractableException

from selenium.webdriver.support.wait import WebDriverWait

errors = [NoSuchElementException, ElementNotInteractableException]

wait = WebDriverWait(driver, timeout=10, poll_frequency=0.2, ignored_exceptions=errors)

wait.until(lambda d: d.find_element(By.CSS_SELECTOR, ".order-row").is_displayed())

Pick one strategy per project. Explicit waits give you control per interaction.

2. Give every test its own data

Shared fixtures are the most common cause of parallel-only failures. Generate data per test and seed it through the API instead of the UI:

test('user can cancel an order', async ({ page, request }) => {

const email = qa+${crypto.randomUUID()}@example.com;

const { id } = await (await request.post('/api/test/orders', {

data: { email, status: 'pending' },

})).json();

await page.goto(/orders/${id});

await page.getByRole('button', { name: 'Cancel order' }).click();

await expect(page.getByText('Order cancelled')).toBeVisible();

});

API setup removes a long UI path from the test and cuts the number of steps that can fail for reasons unrelated to what you are testing.

3. Control the network

Stub third-party calls in functional tests. Keep a separate, small set of contract tests that hit the real service on a schedule, so an outage at a payment provider does not block merges.

How can I stabilize Playwright, Cypress, or Selenium tests?

Framework-specific moves that pay off quickly.

Playwright. Use role and label locators over CSS chains. Enable trace: 'on-first-retry' so you get a full timeline for the failure without storing traces for every run. Keep tests isolated rather than reaching for serial mode, which the docs recommend for the same reason.

// playwright.config.ts

export default defineConfig({

retries: process.env.CI ? 2 : 0,

use: { trace: 'on-first-retry', video: 'retain-on-failure' },

});

Cypress. Turn on test isolation, use cy.session() for cached login, and replace cy.wait(ms) with aliased intercepts. Set retries only in run mode:

// cypress.config.js

retries: { runMode: 2, openMode: 0 }

Selenium. Centralize waits in one helper so timeouts are consistent, pin browser and driver versions in CI, and run in a container so local and pipeline environments match.

Across all three, disable CSS animations in your test build. It removes an entire class of timing failures in a single change.

Should engineering teams use retries for flaky tests?

Retries are a detection tool, not a cure. Used well, they keep the pipeline moving while you fix causes. Used badly, they hide real bugs.

The rule we apply: retries are allowed, but a passing-on-retry result must be recorded as flaky, not green. Playwright does this by default, sorting results into passed, flaky, and failed. Its Playwright test retries docs also expose testInfo.retry, which is useful for clearing server state before a second attempt:

test('checkout completes', async ({ page }, testInfo) => {

if (testInfo.retry) await resetCartState();

// ...

});

Two limits to set. Cap retries at two, since a test needing three attempts is broken rather than flaky. And never retry the whole pipeline job to get a green build, because that erases the signal you need.

In GitLab CI, scope job-level retries to infrastructure problems only:

e2e:

script: npx playwright test

retry:

max: 2

when:

  - runner_system_failure

  - stuck_or_timeout_failure

artifacts:

when: always

paths: [playwright-report/]

reports:

  junit: results.xml

That way runner crashes get retried and genuine test failures do not.

When should flaky tests be quarantined?

Quarantine when a test is unreliable enough to erode trust but too valuable to delete, and you cannot fix it this sprint. It keeps the main pipeline meaningful while the test stays visible.

Quarantine works only with an expiry date. Without one, the quarantine list becomes a graveyard.

Quarantine policy example

1. Eligibility: flake rate above 2% over 30 days, or 3+ false failures in a week.

2. Action: tag the test @quarantine and move it out of the blocking job.

**3. Ownership: **the owning team is assigned within 24 hours. No owner, no quarantine.

4. Time limit: 14 days. Fixed, deleted, or escalated at expiry.

**5. Cap: **quarantine holds no more than 2% of the suite. At the cap, stabilization work takes priority over new test development.

6. Visibility: quarantined tests run nightly and report to the team channel.

In GitHub Actions, run quarantined tests in a separate non-blocking job:

jobs:

e2e:

runs-on: ubuntu-latest

steps:

  - uses: actions/checkout@v4

  - run: npx playwright test --grep-invert @quarantine

  - uses: actions/upload-artifact@v4

    if: always()

    with:

      name: playwright-report

      path: playwright-report/

quarantined:

runs-on: ubuntu-latest

continue-on-error: true

steps:

  - uses: actions/checkout@v4

  - run: npx playwright test --grep @quarantine

The continue-on-error: true flag keeps these results reported without blocking the merge.

Key metrics teams should monitor

Stabilization work needs numbers, or you cannot tell progress from luck.

Track flake rate per spec alongside the suite-wide number. Averages hide the handful of tests doing the damage.

Do and Don't

Do:

Classify root cause before writing a fix
Rank by flake rate multiplied by run frequency
Seed data through the API, unique per test
Store traces and video on first retry
Report retry passes as flaky, never as green
Put an expiry date on every quarantined test
Pin browser and driver versions in CI
Review the flakiest five specs in your weekly QA sync

Don't:

Rewrite the suite before diagnosing it
Add sleeps to "make it stable"
Raise global timeouts as a blanket fix
Retry entire pipeline jobs to get green
Mix implicit and explicit waits in Selenium
Share user accounts or records across parallel tests
Skip tests silently with no owner or date
Judge progress on suite-wide averages

Where to start on Monday

Pull the last 30 days of CI results. Rank specs by flake rate times run frequency. Take the top five, classify each against the root cause table, and fix by category. Quarantine anything you cannot fix in two weeks, with an owner and a date.

In the audits we run, pipeline trust usually recovers from this alone, before anyone touches framework choice. If you would rather have an outside read on which tests to fix, quarantine, or delete, that is what test automation consulting services cover. In this case, experts audit the existing framework, stabilize what is worth keeping, and hand back the flake metrics and quarantine policy your team runs afterwards.

By the way, if you have a stabilization tactic that works on your pipeline, drop it in the comments, it would be nice to read your experience.