The suite is green when it runs serially. At eight workers it fails maybe one run in four, in a different spec each time. At sixteen it fails more often and each individual test takes longer than it did at eight. The obvious read is that the test code is bad and needs waits and retries. In most of the suites I've worked on, that read was wrong.
Two numbers make the point faster than an argument. On GoodShape we run 15 threads against roughly 3,800 scripts, and regression got about 2x faster. On a childcare management system we run 14 threads across 1,600+ web and 150+ mobile tests, and regression went from over 3 hours to about 30 minutes — roughly 6x. One thread apart, three times the difference in payoff.
Those aren't comparable engagements. Different baselines, different suite sizes, different scopes, different starting infrastructure. That is exactly why the comparison is useful: if the worker count predicted the speedup, 14 and 15 would land in the same place. They don't, because thread count is not the variable that determines what you get.
What "parallel" actually means in your runner
Before diagnosing anything, get the vocabulary straight, because "workers," "threads," and "shards" describe three different isolation guarantees.
Process vs. thread isolation
A worker in Playwright or Jest is an OS process with its own module registry, its own memory and its own browser context. A thread in a JVM or Ruby runner shares process memory, so anything global — a class-level cache, a singleton HTTP client, a mutable config object — is shared by construction. A shard is neither: it's a slice of the test list handed to a separate machine, with no shared memory at all but the same shared backend.
Turning it on is a one-liner in every runner:
// playwright.config.ts
export default defineConfig({
workers: process.env.CI ? 12 : 4,
fullyParallel: true,
});
Robot Framework
pabot --processes 15 --testlevelsplit tests/
Both give you N isolated runner contexts. Neither gives you N isolated databases. That gap is where parallel-only failures live.
This is not runner-specific. Across six suites we run 10 to 20 threads on Robot Framework (Abbott, GoodShape), WebdriverIO (NEXSYS-ONE, childcare), Cypress and Playwright (ChargeAfter, childcare), and Capybara/Selenium (CipherHealth). The runner changes what gets reset between tests. It never changes what happens in the database rows, the message queues, the feature-flag service or the third-party sandbox account, and that's where the bleed is.
The isolation differences are real, though, so don't port a config across runners and expect the same behavior. Cypress historically parallelizes at spec-file granularity — a spec is the atom. Playwright's fullyParallel splits inside a file, so two tests in the same describe block can run at once. A Selenium grid hands you sessions and takes no position at all on your data setup. "12 workers" means something different in each.
Why the same thread count produces different speedups
Three of our suites sit within one thread of each other and produced very different outcomes.
GoodShape: 15 threads, ~3,800 scripts, ~95% coverage, ~99% of smoke automated, regression 2x faster. The childcare system: 14 threads, moving from single-threaded to parallel, regression from >3 hours to ~30 minutes, smoke at 10 minutes. NEXSYS-ONE: 15 threads, 1,300+ scenarios across two suites, regression at 1.5 hours, built from a starting point of no regression testing and no CI at all.
None of these were controlled experiments, and the case studies don't isolate a cause — every one of them changed framework, coverage, environment count and process at the same time. Which is the honest conclusion: the thread number by itself explains nothing about the result.
The arithmetic behind that is boring and unavoidable. If 20% of your suite's wall time is a serial phase — a shared fixture build, a global login, a migration, a single-file setup every worker waits on — your ceiling is 5x regardless of how many workers you buy. If that serial fraction is 40%, your ceiling is 2.5x, and you hit it around 6 workers. Everything past that is spend with no return. Before touching the worker count, measure how much of the run is genuinely parallelizable. Most teams that "get nothing out of parallelism" have a serial fraction they've never measured.
The four things your tests fight over
Nearly every parallel-only failure I've chased traces back to one of four shared resources.
Data records
This is the big one, and it's measurable. In a study of 22,352 Python projects and 876,186 tests, 0.86% of tests were flaky — and of those, 59% were order-dependent (3,168 victims, 738 brittles), with another 28% attributed to infrastructure. Order dependency is a shared-state bug that serial execution hides by always producing the same order. Parallelism randomizes the order. You didn't create the bug; you removed the thing that was concealing it.
The ecosystem matters for the diagnosis, though. A 2022 study of 40 top-starred JavaScript repositories categorized 358 flaky-test commits and found order dependency rare, with concurrency (20.7%) and async wait (19.6%) dominant instead. An older analysis of 201 fixed flaky tests across 51 Apache projects put async wait at 45%, concurrency at 20% and test order dependency at 12%. So if you're on a JS stack, expect timing races before ordering races; if you're on Python or JVM, look at ordering first.
Accounts and sessions
test_user_1 logged in by two workers at once. What breaks depends on the app: server-side session invalidation on new login, rate limits on the auth endpoint, a single cart or draft per account, MFA codes consumed by whichever worker polls first. This is the failure that looks most like a "flaky login step" and is actually a capacity problem with one row in a users table.
Environment capacity
On CipherHealth, the before state was single-threaded execution against one local environment. The end state is 20 threads across 4 environments including production, with 1,400+ web scenarios, 100+ mobile scenarios, 250+ API tests, regression at 5 hours and smoke at 10 minutes. Environment count moved together with thread count. On ChargeAfter there was no dedicated automation environment at all; we built one plus per-environment pipelines alongside running 12 threads.
Google's 2017 analysis of their CI is older data but still the clearest public evidence of the mechanism: flakiness correlated with resource footprint at r² = 0.76 for RAM used, large tests were 14% flaky against 0.5% for small ones, and predicted flakiness rose from 1.5% at the smallest memory bucket to 9.2% at the 95th percentile. Adding workers is a direct increase in concurrent resource pressure on the same axis that correlates with flakiness.
Global config and feature flags
One flag service, one tenant, one system clock. Worker A flips a feature on to test the enabled path; workers B through L are now running against a different application than the one their assertions describe. Same for locale, currency, seeded admin settings and anything that writes to a shared settings table.
Making tests independent is a data problem
Here's the reflex worth questioning: when a test fails only in parallel, retry it. It passes on attempt two, the pipeline goes green, everyone moves on.
That retry told you the collision didn't happen the second time. It did not tell you which resource collided, and it converts a correctness signal into a recurring cost line. GitLab publishes theirs: as of March 2024, 5,200+ of 260,040 tests were classified flaky (about 2%), flaky tests accounted for at least 30% of master pipeline failures, and the waste came to 31,395 CI minutes a month, roughly $2,653. Quarantine and retry are containment. They are not a fix, and at scale they aren't cheap either.
The structural fix is that each test provisions its own data through the API and never looks anything up:
const uid = () =>
${process.env.TEST_WORKER_INDEX ?? 0}-${Date.now()}-${randomUUID().slice(0, 8)};
export async function createTenant(api: ApiClient) {
const slug = qa-${uid()};
const tenant = await api.post('/admin/tenants', { slug, plan: 'pro' });
const user = await api.post(/admin/tenants/${tenant.id}/users, {
email: ${slug}@example-qa.test,
password: process.env.SEED_PASSWORD,
});
return { tenant, user, cleanup: () => api.delete(/admin/tenants/${tenant.id}) };
}
The thing to look at is that nothing is fetched. There is no "find the test tenant," no shared fixture file, no test_user_1. The worker index in the identifier means two workers cannot generate the same key even within the same millisecond. Teardown deletes, but correctness doesn't depend on teardown running — a failed cleanup leaves garbage, not a broken next run.
Set up through the API rather than the UI wherever the UI isn't the thing under test: it's faster, it doesn't consume a browser session, and it fails loudly with a status code instead of a timeout. Page objects belong in the same conversation, because selector duplication across 1,300 scenarios is its own coupling problem. Both NEXSYS-ONE and the childcare system list API data creation and test independence as changes made alongside parallelization, and NEXSYS-ONE also adopted the page object pattern — the case studies list them as concurrent changes, not as a proven causal chain.
The limitation is real: API-created data drifts from what real user flows produce. No onboarding state, missing derived rows, no audit trail, none of the side effects a real signup fires. Some flows have to go through the UI because the UI is the system under test. For those, you accept either a serial lane or a pre-provisioned pool of accounts sized to your worker count.
Finding the point where more workers stop helping
The signals that you've passed the useful ceiling:
- Suite wall time stops dropping while per-test wall time rises. This is the definitive one, and it requires measuring per-test duration, not just total runtime.
- Failure rate rises as worker count rises, monotonically.
- Failures cluster by resource rather than by feature — everything touching one table, one queue, one third-party sandbox.
- Reruns pass. In a parallel context that's evidence of contention, not of a defect.
Finding the number is an experiment, not a guess, and it costs one night of CI:
for w in 4 8 12 16 20; do
for run in 1 2 3; do
npx playwright test --workers="$w" --reporter=json \
"results-w${w}-r${run}.json"
done
done
Then plot suite wall time, per-test p95 and failure count against w. The useful number is where wall time flattens, not where it stops improving entirely.
For what it's worth as a prior: the configurations we've actually landed on cluster narrowly — 10 on Abbott, 12 on ChargeAfter, 14 on childcare, 15 on NEXSYS-ONE and GoodShape, 20 on CipherHealth. That's an observed range across six engagements, not a recommendation, and none of them came from a documented tuning experiment. But it's consistent with a practical ceiling well below whatever your CI plan will sell you.
What parallelism costs you
Concurrency buys wall-clock time and pays for it in three currencies.
Debuggability. Interleaved logs are useless; you need per-worker log files or structured logs carrying a worker ID, plus traces and video retained per test. Budget the storage.
Reproducibility. A local single-worker run is a different system from a 16-worker CI run. Failures that only exist under contention won't reproduce on your laptop, which means your debugging loop is now "push and wait" unless you can run the parallel config locally.
Breadth. The matrix multiplies everything. On Abbott we went from 5 devices to 20+ across 27 localizations with 1,500+ scripts, taking regression from 2.5 weeks to 1 day and smoke from 7 days to 1 day, with coverage from 50% to 90%. ChargeAfter covers 3 Chrome and 2 Safari versions across 4,000+ UI and API scripts. Every added combination is maintenance, not just runtime.
And all of it has an owner. Abbott ran with 8 automation and 11 manual QA engineers; CipherHealth and ChargeAfter with 4 each; childcare with 6; GoodShape with 3; NEXSYS-ONE with 2. Parallel infrastructure without a maintainer degrades into a quarantine list.
When a test genuinely can't run concurrently
Some tests are legitimately serial: global config changes, destructive data operations, migrations, anything asserting on a singleton queue, anything driving a third-party sandbox that issues one session. A serial lane is a design decision, not an admission of failure.
test.describe.configure({ mode: 'serial' });
npx playwright test --grep-invert @serial # parallel lane
npx playwright test --grep @serial --workers=1 # serial lane
Keep that lane off the critical path — after the parallel lane, or nightly. A 40-minute serial stage gating every PR has moved the bottleneck, not removed it.
Segmentation by cadence tends to carry as much of the gain as raw concurrency. ChargeAfter runs smoke every 2 hours at 15–20 minutes, a post-PR API suite at 10 minutes, and full regression at 6 hours, down from 2 weeks of manual regression. The childcare system runs smoke at 10 minutes and regression at ~30 minutes; over the same period its production deploys went from 1 per 2 weeks to 10+, though the segmentation was one of several changes shipped at once. CipherHealth runs smoke at 10 minutes, regression at 5 hours, across 50+ automated jobs. Nobody in any of these setups waits for the full suite on a commit. The cost is pipeline sprawl: more lanes, more environment variables, more config drift, and one pipeline nobody remembers to update.
Before you raise the worker count
Three checks, in order.
- Does every test provision its own data through the API, with identifiers unique per run and per worker? If not, more workers means more polluter/victim pairs, and the study data says ordering is where the majority of them live.
- Can the environment absorb N concurrent sessions — connection pool, rate limits, grid slots, sandbox quota? If N workers point at one shared environment, that environment is your ceiling, and no config change moves it.
- Is the serial lane tagged, isolated and off the critical path?
Top comments (0)