DEV Community

Ankit Patel
Ankit Patel

Posted on

Your CI Runs Tests in Parallel. Your Test Data Doesn't Know That.

There's a specific kind of test failure that survives every fix you throw at it.

It passes locally. It passes when you re-run the single spec in CI. It passes when you
retry the job. Then it fails again two days later in a different spec entirely, one
nobody touched. Somebody adds retries: 2, the dashboard goes green, and the team
moves on.

That test isn't flaky. It's colliding.

The difference that matters

Flakiness and collision look identical on a dashboard and have nothing in common
underneath.

A flaky test is non-deterministic in isolation. Race condition, animation timing,
a network call without a proper wait. Run it alone a hundred times and it will
eventually fail on its own.

A colliding test is perfectly deterministic. Run it alone a thousand times, it passes a
thousand times. It only fails when another test is touching the same data at the same
moment.

The tell is in which test fails. Real flakiness fails the same spec repeatedly.
Collision moves. Today it's checkout, next week it's profile settings, and the only
thing they share is a database row.

If your failures wander, stop tuning waits. You have a state problem.

Where it comes from

Almost always the same place: a fixture written when the suite ran serially, that
nobody revisited after fullyParallel: true got switched on.

The classic is one seeded account.

// seeded once, used everywhere
const TEST_USER = { email: 'qa@example.com', password: 'Test123!' };
Enter fullscreen mode Exit fullscreen mode

Serially, that's fine. In parallel it's four workers logging into the same account,
invalidating each other's sessions, mutating the same cart, racing the same
notification list. Nothing is non-deterministic. Everything is contended.

The same shape shows up as a global-setup record every spec mutates, a hardcoded
order_id two suites both update, or a cleanup step that truncates a table while
another worker is mid-assertion.

Why the usual fixes plateau

Retries convert a real signal into a slower green. The collision still happens.
You've just agreed to stop hearing about it. Worse, retries hide the failure that would
have told you which shared record is the problem.

test.describe.serial works, and it's the fix most teams reach for. It also
surrenders the parallelism you turned on for a reason. Do it in enough places and your
20-minute suite is a 50-minute suite again, which is usually the moment somebody starts
arguing for deleting tests.

A bigger CI machine does nothing. Contention is logical, not about cores.

Each of these treats the symptom because the symptom is what the dashboard shows you.

Give every worker its own data

The fix is that no two workers should ever be able to see each other's records.
Playwright gives you the seam for this directly: worker-scoped fixtures.

// fixtures.ts
import { test as base } from '@playwright/test';

type WorkerFixtures = {
  account: { id: string; email: string; password: string };
};

export const test = base.extend<{}, WorkerFixtures>({
  account: [
    async ({}, use, workerInfo) => {
      // unique per worker AND per run
      const email = `qa+w${workerInfo.workerIndex}-${Date.now()}@example.com`;
      const account = await createAccount({ email, password: 'Test123!' });

      await use(account);

      await deleteAccount(account.id);
    },
    { scope: 'worker' },
  ],
});
Enter fullscreen mode Exit fullscreen mode

Two things are doing the work. scope: 'worker' means the account is created once per
worker process, not once per test, so you get isolation without paying signup cost on
every spec. And workerInfo.workerIndex plus a timestamp means worker 0 and worker 3
cannot collide, and neither can this run and the one still finishing from the last
merge.

Specs then say nothing about which user they're using:

import { test } from './fixtures';

test('cart survives a page reload', async ({ page, account }) => {
  await login(page, account);
  await addToCart(page, 'SKU-1041');
  await page.reload();
  await expect(page.getByTestId('cart-count')).toHaveText('1');
});
Enter fullscreen mode Exit fullscreen mode

For data that genuinely must be per-test rather than per-worker (an order that gets
mutated, say), use a test-scoped fixture and namespace it the same way, with
testInfo.parallelIndex or a run id in the key.

Then clean up by prefix, never globally:

// delete only what this worker created
await db.deleteWhere({ emailLike: `qa+w${workerInfo.workerIndex}-%` });
Enter fullscreen mode Exit fullscreen mode

A TRUNCATE in an afterAll is the single most reliable way to break a suite that was
otherwise fine.

The objection worth taking seriously

Creating real accounts per worker costs setup time, and on a large suite that is a
genuine tradeoff. Somebody will point out that one seeded user is faster.

They're right about the arithmetic and wrong about the comparison. Worker-scoped means
the cost is paid once per worker, not per test. On a 400-test suite across 4 workers,
that's 4 signups, not 400. Weigh those 4 against the retries you're currently running
and the hours spent re-investigating a wandering failure every sprint.

The other fair objection: some systems can't create accounts on demand: a legacy
provisioning flow, a paid third-party seat, a partner sandbox with fixed credentials.
That's real. There you want a pool: a fixed set of N accounts, one checked out per
worker, released at the end. Slower to set up, same isolation property. What doesn't
work is one account and hope.

What to do on Monday

  1. Pull your last 30 CI failures. Group them by spec. If the failures are spread across many specs rather than concentrated in a few, you have collisions, not flakiness.
  2. Grep the suite for hardcoded credentials and ids: qa@, test@, any literal id in a fixture. Each one is a shared mutable resource.
  3. Take the single most contended one and move it to a worker-scoped fixture with a worker-indexed key.
  4. Find every describe.serial and note why it was added. Some are protecting real sequences. Most are scar tissue from a collision nobody diagnosed.
  5. Check your cleanup for a global delete or truncate. Scope it by prefix.

Suites that stay fast under parallelism aren't the ones with clever waits. They're the
ones where no two tests can reach the same row.


I'm Ankit Patel, Director of Test Automation at QAble.
I spend most of my time inside other teams' test suites. Currently building Testbo-X, a
Playwright framework focused on intelligent test selection and parallel CI execution.

Top comments (0)