DEV Community

Anushka Srivastava
Anushka Srivastava

Posted on

I made my test suite flaky on purpose

#playwright #qa #typescript #devops

I am building a dashboard for observing test suite health. It shows how often tests pass, which ones are unstable, and what's slowing the suite down. Purpose to design this is to help make quick and reliable go/no-go decisions before a release.

A dashboard designed to confirm the stability of the system needs some instability to give a more accurate picture. A relative metric to decide "How reliable is the system?". I thought, why not seed some flaky tests in the suite.

_Playwright reports a test that fails and then passes as **flaky** a status derived from the retry counter, not recorded on any single attempt._

Three tests to seed instability

  1. Random chance: A full checkout flow - add to cart -> fill the form -> finish the order -> assert on the confirmation, and then a coin flip. This test will roughly fail 20% of the times. An independent random event on each retry. I know the expected fail rate, which makes it a useful baseline for me to check whether the pipeline is reporting everything properly. 
expect(Math.random()).toBeGreaterThan(0.2);
Enter fullscreen mode Exit fullscreen mode
  1. A timing dependency: A real assertion with a timeout that's too short on a medium-slow connection but adequate for a fast one. This is the closest test I could think of for "flakiness by accident" scenario. 
await expect(inventory.itemNames.first()).toBeVisible({ timeout: 150 });
Enter fullscreen mode Exit fullscreen mode
  1. A slow user: The site I'm testing against, saucedemo.com, has a deliberately glitched account called performance_glitch_user. Logging in as that user against a tight timeout gave me a test that is genuinely slower on every run, which also adds to the duration-trend chart rather than only to the stability data.

Two of them failed every single time

In my first few runs, two tests failed 100% of the time, which cannot be flagged as flakiness, it is a broken test.

One was a typo in an assertion. In the checkout flow, the confirmation header reads Thank you for your order! and I had written it without the exclamation mark. toHaveText is an exact match. So, the test died on that line every run and never reached the Math.random(). The 20% random chance flaky test never ran.

The other was a slip: It should have been inventory instead of InventoryPage to match the object created. Due to which itemNames was undefined threw a TypeError on every run before any timing was involved.

const inventory = new InventoryPage(page);
await expect(InventoryPage.itemNames.first()).toBeVisible({ timeout: 150 });
Enter fullscreen mode Exit fullscreen mode

Very silly mistakes which I would have missed if I hadn't read the actual error message.

The third one passed every single time

Once the first two were fixed, the slow-user test still wasn't flaking. I'd given it a 3000ms timeout on the assertion after login. The glitched user account should have exceeded it but it passed every time.
To find out why, I wrapped each phase in test.step() which records step durations, and ran the file six times. Five runs passed while one failed because the total time elapsed for it to reach the inventory page came out to be more than 6000ms. The table below shows where the time went:

Step Median Range
Navigate to login 337ms 270-727
Login 5204ms 5184-5219
Assert inventory reached 21ms 15-57

The assertion itself was taking only 21ms, it was never going to fail against a 3000ms timeout. The variance came from navigation: 270-727ms. So the login(~5.2s) sets the floor and navigation variance decides whether the total clears the ceiling.

The fix was to stop timing one step and time the whole thing:

test.setTimeout(6000);
Enter fullscreen mode Exit fullscreen mode

Now the timeout covers the navigation, the throttled login and the assertions together.

The defect and the asset

The fact remains, a flaky test is a defect. It burns CI time on retries and engineer time on failures that aren't real.

The flaky test runs give us historical data to measure the stability of the suite against every change in a release. It makes it easy to spot a problem that otherwise might go unnoticed.

  • It separates test defects from system defects. A race in a test is often a race in the application: async completion order, shared state, connection pool exhaustion. When such a test gets flaky status, it forces the tester to look into it more deeply to figure out the root cause which otherwise could have been ignored if the test kept getting passed on retries.
  • It turns a red build into a decision. "Six tests failed" is an hour of triage. "Six failed; five are known flaky at ~15%, one has never flaked in ninety days" is a go/no-go call in ten seconds.
  • It makes change visible. A test that flaked at 2% for six months and jumps to 30% after a deploy says something changed in the system, not the test. That pattern doesn't exist in any single run.

The same history supports prioritization: rank what to fix by flake rate rather than by whichever test broke the build most recently.

The seeded tests stay, but fenced. The file declares what it is, each test names the failure mode it models, and every other spec in the suite is expected to be deterministic. So a failure outside flaky.spec.ts is real. Without that boundary I wouldn't have an instrumented suite.

One thing I still don't know: whether the 270–727ms navigation variance is my connection, saucedemo, or Playwright's own startup. For seeding flakiness it doesn't matter. In a real suite that question is the whole investigation.

This is part of a project taking Playwright output through a Python ETL into DuckDB, dbt models and a test-health dashboard. Repo:

github.com/Anushka-Srivastava159/test-analytics-platform

Top comments (0)