DEV Community

Cover image for The Hardest E2E Tests Live at the Boundaries
Simon Gerber
Simon Gerber

Posted on

The Hardest E2E Tests Live at the Boundaries

The easiest automated test in the world looks something like this:

  1. Open page.
  2. Click button.
  3. Assert text.

Unfortunately, that’s not where expensive bugs usually live.

They live at boundaries.

The moment your application has to interact with another system, another browser context, another identity provider, another clock, or another human team, automation gets much more interesting.

Authentication Is Basically a Collection of Edge Cases

“Test login” sounds simple.

Then you ship a real enterprise application.

Now you have:

  • SSO
  • MFA
  • OTP codes
  • expiring sessions
  • idle logout
  • refresh tokens
  • re-authentication
  • multiple tabs
  • redirects to identity providers

Your clean little login test becomes a distributed systems test wearing a browser costume.

Session timeout is a perfect example.

You could literally wait 30 minutes during every test run.

Please don’t.

The better approaches in testing session timeout, idle logout, and re-authentication are about controlling the state instead of making CI experience time at human speed.

The same goes for evaluating a platform that needs to handle MFA, SSO, OTP, and expiring sessions.

These aren’t exotic requirements anymore.

They’re normal SaaS requirements.

Payments Are Even Worse

Checkout combines nearly every annoying browser automation problem.

You have:

  • payment iframes
  • cross-origin content
  • redirects
  • 3DS authentication
  • external payment providers
  • popups
  • asynchronous callbacks

And sometimes the browser leaves your domain entirely.

That makes comparisons like Endtest vs Playwright for payment iframes, 3DS, and redirect checkout more meaningful than comparing how each tool handles a simple button click.

The difficult part of automation is rarely syntax.

The difficult part is surviving reality.

AI-Driven Interfaces Add Another Boundary

Now add AI.

Traditional applications are approximately deterministic.

Input A usually produces Output B.

AI applications introduce probability.

The interface may be deterministic while the output isn’t.

So your test strategy has to change.

An assertion like:

response === expected_response

may become almost useless.

Instead you care about:

  • intent
  • required facts
  • prohibited output
  • schema
  • confidence
  • behavioral drift

That’s why monitoring AI output drift in production test gates is becoming an interesting problem.

The product might not “break” in the traditional sense.

It may slowly become worse.

That’s harder to catch.

Tools Have Different Costs at These Boundaries

This is also where automation tool comparisons get more nuanced.

If your application changes aggressively and AI is involved in the UI itself, the question isn’t:

“Which testing tool has more features?”

It’s:

Which tool creates less operational overhead for the way my team actually ships software?

That’s the interesting part of comparisons such as Endtest vs Functionize for AI-driven UI changes.

Two products can theoretically automate the same workflow while creating very different amounts of maintenance.

And maintenance compounds.

Saving ten minutes once is irrelevant.

Saving ten minutes on something your team does 500 times is a product feature.

People Are Part of the System Too

There’s one boundary engineers sometimes forget:

Humans.

You can have an excellent test suite and still ship slowly because nobody knows who investigates failures.

Or the offshore QA team discovers problems eight hours after engineering finishes for the day.

Or the test team can execute a release but doesn’t understand what changed.

That’s why evaluating offshore QA partners based on overlap hours and release handoffs isn’t simply procurement trivia.

Coordination latency is part of your QA system.

A bug found at 9:05 AM by someone sitting beside the developer is very different from the same bug reported at 1:00 AM through a spreadsheet.

This Is Where Automation ROI Is Won

Basic tests are important.

You should absolutely test that users can log in and buttons work.

But once those tests exist, I’d spend disproportionate effort on the boundaries:

Authentication.

Payments.

External services.

Session state.

AI behavior.

Redirects.

Asynchronous state.

Human handoffs.

Those are the places where assumptions collide.

And they’re the places where a good automated test can save you from the kind of bug that doesn’t produce a cute little red unit test.

It produces a Slack message from a customer.

Those are considerably more expensive.

Top comments (0)