The question I get from QA leads with a mostly-manual suite is almost always framed as volume: how much should we automate, and how fast.
That framing is what produces suites nobody trusts a year later. The useful question is order — which tests move first, which move third, and which stay manual on purpose.
The filter we use has four factors: business-critical, repeated, stable, and expensive to run by hand. The first three are gates. The fourth only breaks ties. Coverage percentage is what you report afterwards, not what you optimize for — it is a lagging indicator that says nothing about whether the automated 40% is the 40% that matters.
Why "automate everything eventually" isn't a plan
It isn't a plan because it almost never happens. In PractiTest's 2025 State of Testing report, 1% of respondents said automation had replaced all their manual testing, 20% said it replaced 75% or more, and 29% said it replaced a quarter or less. The share reporting no automation impact at all did fall meaningfully — from 26% in 2023 to 14% in 2025 — so adoption is real. Full replacement is not. If the realistic end state is a partial suite, then composition matters more than size, and composition is a sequencing decision.
When we picked up Agorapulse, the starting point was 15% automation coverage on an unmaintained Cypress suite, one automation engineer, and a manual regression pass before every release. The suite now sits at 90–95% regression coverage with 1,600+ Playwright/TypeScript scripts and a roughly 4-hour regression run.
The instructive part is not the delta. It's that the original 15% was mostly unusable — outdated tests over flows that had moved on — so it was rebuilt rather than extended. Low coverage was not the failure. Coverage of the wrong things, unmaintained, was.
The four-factor filter: critical, repeated, stable, expensive
- Business-critical means a break costs revenue, data integrity, or a compliance obligation. This one is not a QA judgment call. Pull the top drivers from support tickets and ask product which flows they'd roll back a release for; the list you get back rarely matches the one QA would have written alone.
- Repeated means the test runs every release, or several times within one. A scenario executed twice a year has a payback horizon measured in years.
- Stable means the flow and its selectors are unlikely to change in the next one to two releases. Not "well built" — just not scheduled for redesign.
- Expensive to run manually is time × frequency × people. It is the tiebreaker between candidates that already passed the first three gates.
type Candidate = {
name: string
critical: boolean // product + support agree a break is severe
runsPerRelease: number // 0 if it isn't in the regression pass
stable: boolean // no planned flow/UI change in 1-2 releases
manualMinutes: number // one full pass, one person
peopleNeeded: number
}
// Gate first, rank second.
const eligible = (c: Candidate) =>
c.critical && c.runsPerRelease >= 1 && c.stable
const manualCost = (c: Candidate, releasesPerQuarter: number) =>
c.manualMinutes * c.peopleNeeded * c.runsPerRelease * releasesPerQuarter
const order = candidates.filter(eligible)
.sort((a, b) => manualCost(b, 6) - manualCost(a, 6))
The thing to look at is that the gate is boolean and the cost is only a sort key. The common mistake is collapsing all four into one weighted score, which lets a big manual-cost number outvote instability — and that is exactly the trade that produces a flaky suite.
Why stability outranks complexity
Google's testing team published the largest breakdown of this I know of. Across roughly 4.2 million tests, flakiness tracked the surface, not the cleverness of the test: 0.5% of small tests were flaky, 1.6% of medium tests, and 14% of large ones. By tool, Java WebDriver came in at 10.45%, Python WebDriver at 18.72%, and the Android emulator at 25.46%. Their conclusion was that size predicts flakiness better than tool choice does. That post is from 2017 and the tooling has moved on, but the shape of the finding has not.
A more recent measurement points the same way. A 2025 study of 24 Java projects ran 10,000 test suite executions, identified 810 flaky tests, and found that 75% of them belonged to a cluster of co-failing tests — with networking and external dependencies as the predominant causes. Flakiness is mostly a property of the environment and the surface you chose to test through, not of one badly written test.
The practical implication: your first automated tests should sit on the most stable surface that still covers the critical flow. On WoundTech — which had no effective automation at all before — one engineer over six months got smoke testing to 100% automation coverage and regression to over 80%, with 130+ scripts and 30+ bugs logged along the way. Note the order those two numbers arrived in. Smoke was finished before regression was near done, because the smoke set was the small, stable, obviously-critical core.
What to deliberately leave manual, at least for now
Three categories fail the stability gate almost by definition: UI that is actively being redesigned, one-off and exploratory scenarios, and rare edge cases that run once a quarter. Automating those costs more in maintenance than running them by hand costs in labour.
RevenueHero is the clean example of getting this wrong before anyone got it right. Before our team came in there was no formal QA process and no test documentation — but there were automation scripts. They just didn't check the required functionality. Effort had gone into automating a surface that wasn't the one that mattered. The rebuild put in TestRail, Percy, and a Page Object structure, and now runs 200+ end-to-end scripts plus 70+ API scenarios at over 90% coverage, with regression down to 1.5 hours across three threads. Over the engagement, 400+ bugs were reported and 80% of them were high priority — blocker, critical, or major.
That case also shows where "leave it manual" often turns into something better: automate one layer down. When the UI is churning but the behaviour underneath is not, API scenarios give you the critical-flow coverage without inheriting the DOM's instability. Two engineers have run this since 2022, so this is a steady-state result, not a launch number.
"Leave it manual" is a decision with an expiry date. Revisit the list every planning cycle; a flow that was being redesigned last quarter may now be the most stable thing you own.
Sequencing in practice: from smoke tests to full regression
The Agorapulse rollout ran in four phases, and the shape generalizes better than the numbers do:
- Rebuild the foundation — Playwright/TypeScript architecture, testing standards, nothing shipped for coverage's sake.
- Stabilize — automate the critical user flows, and actively eliminate flaky tests rather than retrying them.
- Scale — 630+ additional tests in this phase alone, CI integration through GitHub Actions, automated failure alerts into Slack.
- Optimize — pre-release testing automated, focus shifts to monitoring and maintenance.
Coverage expansion is phase three, not phase one. In practice the mechanism for that is boring tagging plus a tiered CI run:
test('checkout with a saved card', { tag: ['@smoke', '@critical'] }, async ({ page }) => {
// ...
})
PR gate: the trusted core only, must be green to merge
npx playwright test --grep "@smoke"
nightly: everything that has earned CI time
npx playwright test --grep "@critical|@regression" --workers=4
quarantine: runs and reports, never blocks a merge
npx playwright test --grep "@quarantine" || true
The quarantine tag is the part people skip. Without it, a flaky test gets deleted, forgotten, and then rewritten from scratch six months later by someone who doesn't know it was flaky the first time.
BookThatApp shows the widening in full. Before: regression took 4 days, under 200 automated tests, one browser, one device, UI-only checks, a single execution thread, no CI/CD, no security testing. After: regression at roughly 2 hours, 1,000+ automated scripts, over 80% coverage, 4 browsers, 5 devices, UI and API tests, 4 parallel threads on GitHub Actions, and security testing running regularly. That was one full-stack QA engineer across an 8-year engagement — which is the timeline you should anchor on before quoting the 4-days-to-2-hours line anywhere. The case page separately reports a 40% overall testing time reduction; that is a different metric from the regression duration, and the two shouldn't be added together or reconciled.
Expand into edge cases once the core is trusted
WeHeartIt went from roughly two weeks of manual regression to about two hours, with ~2,000 scripts and 95% application coverage across six browsers and versions, built by three engineers over a year.
Buried in that case is the number that argues against "more automation is always better": multithreading alone cut a full run from 12 hours to 2. The suite had grown large enough to take 12 hours before the parallel infrastructure existed to run it. Adding tests did not make the regression pass faster; for a while it made it slower, and it only paid off once the execution capacity caught up. If you widen coverage ahead of your ability to run it, the intermediate state is worse than what you started with.
Where this framework breaks down
- It assumes you have something to prioritize. Motion came in with no documented test cases, no structured QA process, no CI/CD integration, no cross-browser validation, and zero test coverage. There was no manual regression suite to sort — the first job was building the process, which one engineer did over five months, ending with 80+ test cases, 75+ automated tests, 90% coverage of critical features, and 40+ bugs logged including 7 critical. That is a different problem than the one this filter solves. Scope it to suites that already exist.
- It assumes "stable" and "business-critical" can be agreed on. In a team where product, support, and QA disagree about which flows are severe, the gate produces a shortlist nobody signs off on. Get that argument resolved before writing scripts, not after.
- It doesn't make maintenance free. WeHeartIt's parallel execution ran on 15 virtual machines on a single 64GB server, 40–45 browsers at once, three per VM. That is a standing infrastructure and upkeep bill, and it grows with the suite. Prioritization changes the slope of that cost; it doesn't remove it.
One more honesty note: none of these case pages document an internal four-factor checklist. They record outcomes. The filter is how I'd describe the sequencing in retrospect, and it's a working heuristic — not a validated model with a control group behind it.
A rollout that held up over time
The Agorapulse engagement moved from two automation engineers down to one from April 2026, while holding 90–95% coverage and the ~4-hour regression run. I would not present headcount reduction as an expected outcome — it depends on the product's rate of change as much as on the suite — but it's the clearest evidence I have that sequencing pays off on a delay. A suite assembled stable-surface-first gets cheaper to keep. A suite assembled by manual-cost-first gets more expensive, because the tests that cost the most to run by hand are frequently the ones over the least stable UI.
That upkeep gap is where prioritization actually cashes out. In PractiTest's 2024 report, only 40% of respondents said their test cases were well written and maintained; 27% said well written but not maintained, 15% reported many duplications, and 18% had no organized cases at all. Roughly six in ten teams are carrying a maintenance problem, and the timelines above — six months to eight years — are long enough that any suite you build now will spend most of its life in maintenance rather than construction.
Reach for this filter when you have an existing manual regression suite, a release cadence you can count, and someone in product or support who will tell you which flows actually hurt when they break. Skip it when you have no documented test cases yet — build the process first — or when the product is being rewritten next quarter, in which case nothing passes the stability gate and the honest answer is to wait a release and re-run the list.
Top comments (2)
Your emphasis on the order of automation over sheer volume is a crucial insight that many teams overlook. By prioritizing tests based on their business impact, frequency, and stability, you not only enhance the reliability of the regression suite but also align testing efforts with real-world needs. I particularly appreciate your point about avoiding the pitfalls of a weighted scoring system that might misrepresent test priorities; that’s a lesson I’ve learned the hard way in previous projects. If you’re considering further enhancements or scaling this approach, I’d love to explore a paid collaboration to help reinforce your implementation. What strategies do you find most effective for maintaining test stability as your suite evolves?
Thank you! For me, maintaining stability comes down to clear test ownership, reliable test data and environments, resilient selectors, and treating flaky tests as defects rather than simply rerunning them. It’s also important to review the suite regularly and remove tests that no longer protect meaningful risks. As the suite grows, what not to automate—or what to retire—becomes just as important as what to add.