Every team I talk to says the same thing about their E2E suite, in roughly this order:
- "We should have more tests."
- "Actually, we have a lot of tests."
- "Most of them are skipped."
That third line is the real story. The suite isn't thin. It's rotting. Somebody wrote 340 tests over two years, 90 of them now fail intermittently, and rather than debug each one, the team wrapped them in .skip() and moved on. The coverage number in the dashboard still says 78%. The coverage that actually runs in CI is closer to 50%.
Nobody plans this. It happens one Friday afternoon at a time.
Why tests rot faster than code
Application code has a forcing function: if it breaks, users complain. Test code has the opposite. If a test breaks, the cheapest available action is to make it stop running. There's no user on the other end of a skipped test.
So the decay is asymmetric. Every refactor that touches the DOM breaks selectors. Every new async boundary introduces a race. Every API version bump invalidates a fixture. And each individual break is small enough that skipping is rational in the moment, even though the aggregate is a suite nobody trusts.
The thing that makes this worse than it sounds: a suite you don't trust is worse than no suite at all. If the pipeline is red 30% of the time for reasons unrelated to the change, engineers stop reading the pipeline. Then a real regression goes out, and the postmortem says "the tests caught it, but nobody looked."
The taxonomy nobody writes down
When a test fails, it's one of five things. Teams treat them identically, which is the root error, because only two of them mean anything.
1. Real regression. The code broke. This is the test doing its job. Rare, and valuable.
2. Intentional change. The behavior changed on purpose and the test encodes the old behavior. The test isn't broken — it's out of date. Needs updating, not fixing.
3. Brittle selector. div > div:nth-child(3) > button stopped resolving because someone added a wrapper. The behavior under test is fine. The test's grip on the DOM slipped.
4. Timing. The test asserts before the app finished rendering. Passes locally, fails in CI, passes on rerun. The classic flake.
5. Environment. Stale fixture, expired token, a seeded database that drifted, a third-party sandbox that went down.
Categories 3, 4, and 5 are roughly 70–80% of failures in a mature E2E suite, and none of them tell you anything about your product. They're the suite failing to describe itself accurately. That is the maintenance tax, and it's the thing that eats the QA week.
What "auto-fixing a test" actually requires
The obvious move is to point an LLM at a failing test and ask it to fix the test. This mostly works and is mostly dangerous, because the model's easiest path to a passing test is to weaken the assertion. Ask a model to make a failing test pass and you'll get expect(result).toBeDefined() where you used to have a real check. Green pipeline, zero coverage. You've automated the .skip().
So the hard part isn't generation. It's the constraint system around it. A few things that matter:
Classify before you repair. The system has to decide which of the five categories it's in before touching anything. If it's a real regression, the correct action is to report it and change nothing. Repairing a test that correctly caught a bug is the worst possible outcome, and a naive fix loop does it constantly.
Diff against intent, not just against failure. A test knows what it was trying to verify — that's in its name, its assertions, its setup. A repair is only valid if the repaired test still verifies the same thing. Comparing the pre- and post-repair assertion semantics catches most of the weakening problem.
Use the commit as evidence. If the selector broke in the same PR that renamed a component, that's category 3 with high confidence. If the test broke with no adjacent code change, that's category 4 or 5. The version control history is a strong prior and it's free.
Never repair silently. Every fix should arrive as a diff a human approves, with the classification and the reasoning attached. The moment the system writes to the suite without review, you've traded a maintenance problem for a trust problem.
Prove the repair. Run the repaired test against the last known-good commit as well as HEAD. If it passes on both, the fix is probably sound. If it passes on HEAD and fails on the known-good commit, the repair changed what the test means.
That last one is the check most implementations skip, and it's the cheapest high-signal validation available.
Where this leaves the human
Not out of a job — the failure classification is where judgment lives, and category 2 (intentional change) is genuinely undecidable without product context. A machine can tell you the assertion no longer matches the behavior. Only a person knows whether that's a bug or a feature.
What changes is the ratio. If 75% of your failures are selectors, timing, and environment, and those can be classified and repaired under review, then the QA engineer's week stops being triage and starts being the thing they were hired for: deciding what's worth testing at all.
That's the bet behind what I'm building. Testera is agentic test management — agents write your tests, and fix them. A few companies are running it now, including my own.
The part I'd push back on
If you're evaluating anything in this space, including mine, the question to ask is not "can it generate tests." Everything can generate tests. The questions that separate the tools:
- What does it do when it can't tell a regression from a selector break? (Correct answer: stops and asks.)
- Does it verify the repair preserved the assertion's meaning, or just that it's green?
- Can you audit every change it made and why?
If a tool can't answer those, it's not maintaining your suite. It's automating the .skip() with extra steps.
Building Testera IO. If your suite has a skipped-test graveyard, I'd genuinely like to hear what put it there — the failure patterns are more varied than the five above and I'm still collecting them.
Top comments (0)