Test Isolation

Test Isolation: A Lesson I Learned While Migrating Playwright Tests

During my software engineering internship, I helped optimize our CI pipeline by identifying which E2E tests could safely run in parallel. That work quickly taught me that the biggest obstacle wasn't Playwright or Python, it was test isolation.

This article is about that lesson.

What is test isolation?

A simple rule I now use is this: if a test can't run by itself with the same outcome, it probably isn't truly isolated.

A well-isolated test should produce the same result whether it:

• runs by itself
• runs first or last
• runs after another test
• runs in parallel with hundreds of other tests

To understand test isolation, it also helps to understand what state means.

State isn't limited to database rows. During the migration, I found tests interacting with many different kinds of state.

• database records
• global configuration
• filesystem resources
• application caches

If any of these are shared between tests, they become potential sources of hidden dependencies.

How tests lose isolation

As I started reading the existing test suite, I noticed a recurring pattern.

Many tests assumed something about the environment instead of creating it themselves. Some expected specific data to already exist. Others modified global settings without restoring them afterward. Some searched for rows based on their position in a table instead of using a stable identifier like a name or ID.

None of these looked particularly problematic when reading a single test.

The problems only appeared once the entire suite started running together.

One test would leave behind data another test didn't expect. A shared configuration would silently affect unrelated tests. A UI assertion would suddenly fail because another test inserted an extra row into the same table.

Individually, the tests appeared independent.

Together, they formed hidden dependencies.

Not all shared state is equally difficult to isolate

One realization that helped me reason about these failures was that not all shared state is equally difficult to isolate.

I found it useful to think of shared state in roughly four levels of complexity.

Level 1 — Test-owned entities

Projects, tasks, resources, or other data created exclusively for a single test.

These are usually straightforward to isolate because the test owns their entire lifecycle.

Level 2 — Shared database entities

Some database records are shared across multiple tests.

Simply creating unique data isn't always enough. Tests also need to restore the database to a predictable state after execution.

Level 3 — Global configuration

Feature flags and application settings are more difficult because deleting data is no longer the right solution.

Instead, tests need to restore whatever value existed before they modified it.

Level 4 — Hidden system state

This was the most subtle thing I encountered.

One test enabled a global configuration through a direct database update. Querying the database confirmed that the value had changed, yet the application continued reading the old value from an in-memory cache.

The test only failed if another test had recently modified the same flag.

At first glance it looked like a synchronization problem.

It wasn't.

It was hidden shared state.

That example completely changed how I thought about test isolation. Sometimes changing the database doesn't change what the application actually observes.

Test isolation is more than cleanup

Initially, I thought test isolation was mostly about deleting whatever a test created.

That turned out to be only one small part of the picture.

Over time, I found four principles that consistently produced more reliable tests.

1. Independent setup

Every test should prepare its own data.

If a test depends on another test creating data first, the tests are already coupled.

2. Restoration

Deleting created entities isn't the same as restoring modified state.

If a test changes a global setting, it should restore the previous value rather than simply leaving the new value behind.

3. Idempotent setup and cleanup

Setup and cleanup should be safe to execute multiple times.

Running them twice should still leave the system in a usable, predictable state.

This makes tests significantly more resilient to partial failures.

4. Pre-cleanup

This was one technique I learned from an engineer on my team.

Instead of assuming the environment is already clean, a test first cleans the state it depends on, executes, and then performs its normal cleanup afterward.

At first, this felt redundant.

Then I encountered a nightly CI run where a worker crashed halfway through execution. The test's finally block never ran.

That exposed something I hadn't considered before: a finally block only executes if the process survives long enough to reach it. CI cancellations or worker crashes can bypass it entirely, leaving behind partially modified state for the next execution.

Pre-cleanup solves exactly that problem. Instead of trusting inherited state, every test starts from a known state.

I saw the value of this when I began parallelizing parts of our test suite. Several legacy tests in our regression pipeline started failing intermittently.

At first, it looked like parallelization had introduced regressions. After a look, I realized it had simply exposed hidden shared-state assumptions that already existed in those tests.

Rather than fixing each failure individually, I refactored those tests to adopt the same pre-cleanup strategy. Each test now established its own known starting state instead of inheriting whatever the previous execution left behind.

Pre-cleanup isn't free, though.

It adds extra API calls or database operations before each test, increasing execution time. If setup requires UI interactions, i don't want to think about it.

For long-running E2E tests, I found that tradeoff worthwhile. A few extra seconds of setup were far less expensive than debugging intermittent failures.

Make isolation the framework's responsibility

As the migration progressed, I noticed another pattern.

Many tests were reimplementing the exact same lifecycle:

• backend provisioning
• setup
• teardown
• cleanup

Every duplicated implementation was another opportunity for cleanup to be forgotten or implemented slightly differently.

Instead of expecting every engineer to remember every cleanup rule, I moved those responsibilities into shared infrastructure during my migration. I introduced a reusable base test class that centralized setup, teardown, and cleanup logic, along with a registry that automatically tracked resources created during a test and cleaned them up in reverse order. As a result, individual tests only needed to describe what they wanted to create rather than how to clean up every intermediate resource.

The goal wasn't to make engineers remember more rules.
The goal was to make violating isolation harder in the first place.

The less isolation logic individual tests contain, the more consistent the entire suite becomes.

Why it matters

The biggest benefit of test isolation isn't faster CI.

It's confidence.

When tests are isolated:

• failures become reproducible
• debugging becomes much easier
• tests can run in any order
• engineers spend less time chasing flaky failures
• parallel execution becomes much safer

I also learned that not every test should be parallelized.

Tests that intentionally modify global application state or rely on shared mutable resources sometimes need to run sequentially. I believe understanding those boundaries is just as important as identifying tests that can run concurrently.

Final thoughts

Before this internship, I assumed most flaky E2E tests were caused by synchronization problems or timing issues.

Now that I'm halfway through the internship, I've realized that many of them were actually caused by isolation problems.

Hidden dependencies, shared mutable state, and assumptions about the execution environment often mattered far more than the automation framework itself.

Good test isolation isn't about writing more cleanup code.

It's about making every dependency explicit, owning the state each test touches, and designing tests that behave the same regardless of execution order.

Once that foundation exists, improvements like reliable parallel execution become a natural outcome instead of a risky optimization.