Your Test Isn't Flaky. It Ate Its Own Test Data.

A test passes. You merge it. Next run it fails. The run after that, it fails again. And again.

Someone files it as flaky, bumps the timeout, adds a retry. It keeps failing.

It was never flaky.

Flaky is random. This has a direction.

That distinction sounds pedantic until it costs you a week.

A flaky test fails and passes without a pattern — a race, a timing window, a slow paint. You fix it by waiting for the right condition instead of the wrong one.

A test that passes once and then fails consistently is telling you something else. The first run changed the world. Either it consumed the data it needed, or it left state behind that makes its own precondition false on the next run.

The fixes are opposite. Flakiness is fixed by waiting. This is fixed by cleaning up after yourself. Treat one as the other and you buy a longer timeout and keep the bug.

Three shapes it takes

All three of these happened on one suite, and all three were first filed as "the environment is short of data".

1. The test consumed something that cannot be restored.

A test moved an item into a state the product offers no way back from. The product has an "add" control and no "remove". So the first run worked, and every run after arrived to find nothing left to add.

2. The test left state behind.

A test set a configuration value and never cleared it. Its own precondition was "no value is set". Run one: true. Run two: false, forever.

This one is nastier because the test is correct in isolation and the cleanup gap is invisible in review — nobody reads a spec asking "and what does this leave behind?"

3. The target aged out.

A search-index test created a record, waited for it to be indexed, asserted on it. Fine. Minutes later the same query returned nothing, because the thing had aged out of the index the test depended on.

Why it survives for months

Here is the part that makes this expensive rather than merely annoying.

From the outside, a self-drained precondition and a genuinely empty environment are indistinguishable.

The failure message is the same: no suitable record found. The obvious reading is "the environment is flaky / the data is gone / QA needs better test data". The reading nobody reaches for is "this test ate it".

So it gets filed under environment, and environment problems are somebody else's. The test stays broken and the suite learns to ignore it.

The diagnosis, in four rows

Run the spec twice. What the second run does tells you which one you have:

Second run	Cause	Fix
Precondition finds no candidate	The first run consumed the data	Create fresh data per run, or find a new target each time
A precondition assertion fails	The first run left state behind	Undo it in teardown — and prove the undo, don't assume it
Passes if you wait a few minutes	Eventual consistency — an index, a cache, a queue	Poll the condition; never a fixed sleep
Passes alone, fails in parallel	Two workers took the same shared object	A lock per resource

The gate: one extra run

npx playwright test tests/your.spec.ts --repeat-each=2

That is the whole acceptance criterion. A test must survive its own second run.

One extra run before the pull request, and it catches the entire class. I have yet to find a cheaper check with a better hit rate.

Three ways to own your data

In order of preference:

1. Create and clean up. The test makes its own data and removes it in teardown. The default, and the only one that scales.

2. Borrow and restore. The test uses existing data and puts it back exactly as it found it. Restore through the same API that changed it, and prove the restore rather than assuming it.

3. Borrow and rotate. For state the product offers no way back from. The target is found per run, never pinned, so repeated runs spread across the environment instead of draining one object.

And when a test genuinely cannot be idempotent, say so where the next person will look:

// NOT restored afterwards, deliberately. There is no reverse control: an item moved
// into this state offers no way back. Mitigated by finding the target per run rather
// than pinning one, so repeated runs spread across the environment instead of
// draining a single record.

A silent non-idempotent test is a defect. A documented one with a rotation strategy is a decision.

Now the quiet version

Everything above is the loud case: the test goes red, someone eventually looks.

There is a second version of this bug that never goes red.

When a test drains the data it needs, it often does not fail. It finds no candidate and skips.

Think about what that does to your numbers:

• failed stays at zero
• flaky rate stays flat
• and the pass rate goes up, because the denominator shrinks

You have a suite quietly running fewer tests every week, reporting a better number for it. It is a rare thing: a metric that improves as the thing it measures gets worse.

A team can run that way for a very long time.

So measure it

Totals cannot see one test changing its mind. You need per-test history — one row per run, recording each test's outcome — and then two patterns over it:

• regressed — was passing, now failing 3+ runs in a row. Red. Someone sees it.
• starved — was passing, now skipping 3+ runs in a row. Green. Nobody does.

Three runs in a row, not two: with retries enabled a merely unlucky test has already had several
attempts per run, so at three, bad luck stops being the cheaper explanation.

Getting a detector like that to be quiet enough that people read it is its own problem — the streak
has to end at the latest run, a filtered run must not count as evidence, and there has to be a
pass to point at. That is a different post.

The one thing to take away

When a test starts failing and the message says the data is missing, ask one question before you file it against the environment:

Did this test eat it?

That question costs ten seconds. Not asking it cost us three separate investigations.