There’s a point in almost every end-to-end test suite where the team starts blaming the tests.
The login test fails once every 40 runs.
The onboarding flow occasionally opens on step three instead of step two.
A notification appears before the assertion is ready.
Someone adds a two-second wait.
Problem solved.
Until next Tuesday.
What we call “flaky tests” are often something more interesting: state problems that the test suite happens to expose.
Modern web applications have a lot more state than they did ten years ago. There’s browser state, server state, authentication state, feature-flag state, WebSocket state, local storage, cookies, background jobs, third-party services, extensions, and whatever your frontend framework is currently keeping alive in memory.
The test runner is often the least mysterious thing in the system.
The hardest bugs happen between steps
Consider a five-step onboarding process.
Most teams test it like this:
- Fill out step one.
- Continue.
- Fill out step two.
- Continue.
- Finish.
That’s useful.
But customers don’t behave like a test script.
They click Back.
They reload halfway through.
They open another tab.
They submit twice.
Their session expires.
They abandon the flow and return tomorrow.
That’s why I liked this breakdown of testing multi-step onboarding flows without missing state leaks, validation problems, and back-button bugs.
The important thing isn’t adding 30 more assertions.
It’s testing the transitions.
Stateful applications tend to break in the gaps between the happy-path steps.
This applies well beyond onboarding. If your application contains long-lived sessions, dynamic forms, approval processes, or workflows that stretch across multiple screens, this guide on evaluating automation platforms for multi-step workflows and session-heavy flows covers many of the same issues.
Real-time applications make this worse
Then you add WebSockets.
Now the browser can change without the browser doing anything.
A notification arrives.
A status becomes “Approved.”
A dashboard updates.
A chat message appears.
The naive approach is to sprinkle sleeps everywhere:
wait 2 seconds
check status
But if the event normally arrives in 200 milliseconds and occasionally takes 2.2 seconds, you haven’t fixed the timing problem.
You’ve created a slower flaky test.
A better model is to wait for a meaningful application condition.
There’s a useful discussion of this in how to test WebSocket and real-time UI updates without creating flaky browser suites.
The distinction matters:
Time passing is not an application state.
“Wait until the order becomes processed” is meaningful.
“Wait three seconds” is guessing.
That sounds obvious when you write it down, yet huge automation suites are held together with exactly those guesses.
Mock less than you think — but more than zero
Another place teams get stuck is deciding whether E2E tests should hit real services.
I don’t think there’s a universal answer.
If every browser test calls seven third-party APIs, your suite can become a distributed systems monitoring tool instead of a product test.
But if you mock everything, you can build a beautiful green test suite for an application that doesn’t actually work.
A good compromise is to ask what the test is supposed to prove.
If you’re testing:
- how your UI handles an error response, stub it;
- a complicated edge case that is expensive to reproduce, mock it;
- the actual integration between two systems, use the real service.
This article on when to mock, stub, or hit real services in Playwright E2E tests goes deeper into that tradeoff.
The mistake is turning this into ideology.
“E2E tests must always use real services” sounds principled until Stripe has an incident and your deployment pipeline becomes unusable.
Browser state is part of your application
Cookies are another great example.
A consent banner looks trivial until you realize that accepting it might affect:
- analytics initialization,
- advertising scripts,
- local storage,
- accessibility,
- page layout,
- tracking events,
- subsequent sessions.
Testing whether the button disappears is not enough.
The more useful question is whether the application enters the correct state afterward.
There’s a good practical checklist in how to test cookie consent banners without breaking analytics, tracking, or accessibility.
Browser extensions can create an even stranger version of the same problem.
Password managers, ad blockers, developer extensions, privacy extensions, or your own product extension can inject DOM nodes, scripts, styles, and event handlers.
Suddenly a test behaves differently on one machine and everyone blames Selenium.
If that sounds familiar, debugging Chrome extension side effects without polluting your main test suite is worth reading.
Don’t page people because a selector blinked
There’s one last consequence of all this.
If your test suite has uncertain state, be very careful before connecting every failure to Slack, Teams, or PagerDuty.
Alerts create an implicit promise:
This deserves your attention.
Break that promise often enough and people stop paying attention.
Then the one important failure gets ignored along with the 43 harmless ones.
The better approach is to classify failures first.
Is this:
- a confirmed product regression?
- an infrastructure problem?
- an environment outage?
- a test maintenance issue?
- a transient retry?
- an unknown failure worth investigating?
Only some of those deserve an interruption.
If you’re wiring browser automation into incident channels, this guide on what to check before adding browser test alerts to Slack, Teams, or PagerDuty is a useful sanity check.
Flakiness is information
A flaky test is annoying.
But it’s also telling you something.
Sometimes the selector is bad.
Sometimes the test is badly designed.
And sometimes the test is exposing the fact that your application has five sources of state nobody has clearly modeled.
Deleting the test fixes only one of those problems.
The more productive question isn’t:
Why did the automation fail?
It’s:
What assumption did this test make that wasn’t true this time?
That question tends to lead somewhere useful.
Top comments (0)