DEV Community: Surendranath Reddy Jillella

The End of the Testing Pyramid: What Replaces It in the AI Era

Surendranath Reddy Jillella — Thu, 06 Aug 2026 08:01:20 +0000

We have discussed the cost of no automation and reasoned "when to test".
This doesn't automatically solve the test coverage question - the QA teams are swamped enough, and if you ask them to test across the SDLC, that can only lead to testing what is easy or whatever is quick.

Hence the next question - What tests to prioritise?

The testing pyramid was and is gospel, still, and every sprint planning meeting with a QA team included some version of: "we need more unit tests at the base, fewer E2E tests at the top."

The pyramid made complete sense - for its time. I'm not sure it makes sense anymore.

What the Pyramid Was Actually Solving

Mike Cohn introduced the testing pyramid in Succeeding with Agile (2009), though the concept was sketched out earlier around 2003–04. His original framing had three layers: unit tests at the base, service/integration tests in the middle, UI tests at the top. Martin Fowler popularised it further on his blog, and it became a de facto standard across the industry.

The underlying logic was about cost:

Unit tests: cheap to write, fast to run, easy to pinpoint failures
Integration tests: moderate cost, slower, harder to isolate failures
E2E/UI tests: expensive to write, slow to run, brittle - and when they fail, you often spend an hour figuring out what actually broke

So the pyramid was really a cost-optimisation heuristic: do more of the cheap thing, less of the expensive thing - sensible advice when humans are writing every test by hand.

The AI Problem with This Logic

Here's what changes when AI can generate test cases: the marginal cost of writing the 100th test approaches zero.

Generating 10 unit tests costs me roughly the same as generating 100. The input cost - my time writing a prompt, reviewing output - is largely fixed, not variable.

Assumption: This assumes LLM output quality is "good enough to review and merge" without significant rework. Current research (MDPI, 2025) shows LLMs are effective for test generation on well-scoped, lower-complexity functions, but accuracy drops for complex, stateful logic. So this cost collapse is real for breadth of coverage, less so for depth in complex systems.

If writing isn't the bottleneck anymore, the pyramid's rationale starts to crack. But there's a catch.

The Cost That Didn't Collapse: Maintenance

Writing tests and maintaining tests are two different problems. AI hasn't fully solved the second one yet.

Industry data suggests test maintenance consumes anywhere from 30% to 50% of a QA team's automation effort.

That's not a pyramid problem; it's a maintenance problem. And it compounds fast when you generate tests faster than you can maintain them.

If AI test generation significantly outpaces AI test maintenance and healing capabilities, teams could end up with larger, flakier suites than they started with. This is a genuine risk in 2026 - generation tools have matured faster than maintenance/healing tools. The balance matters.

So the new model can't just be "generate everything." It has to be smarter about what gets generated.

What Should Replace the Pyramid

I've been thinking about this as risk-weighted coverage rather than a structural pyramid. The question isn't "how many tests at each layer?" - it's "where does a failure hurt most, and am I covered there?"

A few principles that seem to hold up:

1. Coverage by business risk, not code structure

A payment-processing function with 60% unit coverage is riskier than a string formatter with 95% coverage. The pyramid doesn't capture this. A risk-weighted model does. The first question before generating any test should be: what breaks the business if this fails?

2. The E2E tax is shrinking, not gone

E2E tests are still slow. They're still brittle when they fail. But AI-assisted generation makes writing them dramatically cheaper. The remaining cost is infrastructure and maintenance.

Assumption: As AI-based self-healing for selectors and UI state matures, the E2E maintenance cost will fall meaningfully. This is directionally happening but isn't yet the default reality for most teams in 2026.

3. A "testing mesh" over a rigid pyramid

Rather than a strict hierarchy, think of tests as a mesh - overlapping coverage at multiple levels, weighted toward high-risk areas. Some paths get unit + integration + E2E coverage. Low-risk utility code gets unit only. The shape is determined by risk analysis, not by a fixed ratio rule.

What This Means Practically

If you're maintaining a mature test suite:

Audit which E2E tests cover high-risk paths vs. low-risk happy paths. The latter are usually candidates for deletion or downgrade.
Use AI to generate edge-case unit tests for your critical business logic - that's where current LLM-based generation adds the most value with the least maintenance overhead.
Don't generate tests uniformly. Generate them where failure actually matters.

If you're greenfielding:

Start with risk mapping before writing a single test. What breaks the business if it fails?
Let AI fill coverage breadth; invest human effort in the tests for high-risk paths that require business context to assert correctly.

The Real Question

The pyramid isn't dead - it's become one valid shape among many. The right test distribution depends on your system's risk profile, not a universal rule.

The harder question the AI era forces is: who decides what's risky? And that's still a deeply human judgment. AI can generate tests for you, but it can't tell you what matters to your business unless you've told it.

P.S. - I am Surendranath from QApilot and this post is part the series Test Debt

Shift-Left is Dead. Long Live Shift-Everywhere

Surendranath Reddy Jillella — Thu, 30 Jul 2026 07:53:13 +0000

Last week, I presented the cost of no testing - Hidden cost of no test automation
It is now clear that testing is essential, if not earlier. The next question that comes to mind is When to test? - Shift Left is an answer that many leadership mandates suggest.
The practical interpretation of Shift-Left is simple - QA was going to write fewer tests, and developers were going to write more. The phrase was presented as a philosophy. It felt more like a headcount justification.

That's not what Larry Smith meant when he coined the term.

What Shift-Left Actually Said

Smith's article "Shift-Left Testing" was published in Dr. Dobb's Journal in September 2001. The observation was straightforward: on any project timeline, testing was always concentrated at the right side - near delivery. He proposed moving it left - closer to inception - to catch defects earlier and cheaper.

The economics behind it were sound. IBM Systems Sciences Institute data (originally from 1981 internal training materials) showed bugs found after release cost 60–100x more to fix than bugs found during design. Even if those exact multipliers are contested - The Register ran a worthwhile piece in 2021 questioning whether the original IBM study was ever a peer-reviewed paper - the directional finding has been consistently confirmed by NIST research, Capers Jones' analysis of 12,000+ projects, and anyone who has sat in a production incident war room at 2 am.

The idea was right. The execution became something else.

What Shift-Left Became

In practice, shift-left was operationalised as:

Developers writing unit tests they didn't always have the domain context to write well
CI gates nobody trusted because flaky tests made failure signals into noise
QA "embedded" in scrum teams without changing when or how testing actually happened
Static analysis tools added to pipelines that nobody read the output of

The mandate to shift left didn't come with a corresponding shift in tools, time allocation, or context. It was process vocabulary layered on top of unchanged behavior.

Assumption: This characterization reflects the most common pattern of shift-left adoption - process-driven, without sufficient tooling change. It won't apply to every team. Organizations that genuinely restructured roles, tooling, and feedback loops saw real results. But the pattern of "shift-left = devs write more tests with no additional support" is widespread enough to be a recurring frustration in engineering retrospectives. If your team executed shift-left well, you're in the minority.

The original promise - catch bugs earlier - was only partially delivered, for most teams.

What's Actually Different Now

The 2024 Developer Survey found developers spend only 24% of their time writing code. The rest goes to debugging (roughly 41%), meetings, reviews, documentation, and operational work.

That ratio is persistent and damning. A huge chunk of debugging time is spent on issues that better earlier feedback - not just earlier testing, but earlier signals - would have caught before they compounded.

What AI tooling is changing, when it's actually embedded rather than bolted on, is making quality checks ambient rather than phased:

In the IDE: Copilot-style tools flag issues as you type. Some generate test stubs inline for the function you just wrote.
In the PR: AI reviewers analyze diffs for missing edge cases, untested branches, and logic gaps - not just style and formatting.
In CI: AI-based test generation triggered on diff, targeting new and changed code specifically rather than running the full regression suite.
In production: Error monitoring that surfaces coverage gaps by tracing prod failures back to untested paths.

This isn't shift-left. It's shift-everywhere. Quality isn't a phase you move earlier - it's a layer that runs in parallel with development at every stage.

The Distinction That Matters

Shift-left was about when in the SDLC to apply testing.

Shift-everywhere is about continuous presence of quality signals regardless of SDLC phase.

A developer gets immediate feedback in their IDE. The same PR gets an AI review pass. The CI pipeline runs targeted tests on exactly the changed code. An alert the next day surfaces a production error and maps it back to a coverage gap.

The SDLC doesn't stop being sequential - code still gets designed, written, reviewed, deployed. But quality signals are no longer tied to a single phase of that sequence.

Assumption: "Shift-everywhere" as described assumes a mature AI tooling stack. Most teams in 2026 have 1–2 of these layers, not all four. The full ambient quality model is where the industry is heading, but it's not the default state yet. I'd estimate fewer than 25% of teams have more than two of these layers genuinely integrated and trusted, based on the tooling adoption curves I've seen.

What Hasn't Changed

AI doesn't fix everything. A few things remain stubbornly manual:

Understanding what to assert. Generating a test is easy. Knowing what the correct behavior should be for a non-obvious edge case - one that depends on business rules, not just code logic - still requires a human who understands the domain.

Flaky tests. AI can reduce new flakiness by generating more deterministic tests, but it doesn't automatically rehabilitate an existing flaky suite. That's still cleanup work.

Cross-service test strategy. What to test across service boundaries, under real load, in failure scenarios - that's architectural thinking that LLMs don't consistently reason well about yet.

The Practical Takeaway

If you're a developer
Shift-left was never the enemy. The enemy was treating it as a process change without a tooling change. The AI-era version of the same idea is more actionable because the tooling is actually there.

If you're running an engineering or QA team
Stop thinking about where in the pipeline quality lives. Start thinking about how many stages have real-time quality feedback embedded. Count the stages where a developer gets an automated signal about quality within minutes of writing code. If the answer is one - probably CI - you have room to move.

P.S.: This article is part of the Test Debt series.
About me - I lead the AI team at QApilot - an AI-native mobile test automation company, and we build products to help you achieve the quality standards that your users demand.

The Hidden Cost of No Test Automation: A Back-of-Napkin Calculation

Surendranath Reddy Jillella — Thu, 23 Jul 2026 12:48:07 +0000

Nobody cancels a sprint to do a testing ROI analysis. The cost of skipping test automation doesn't have a line item in your budget. It doesn't show up on a dashboard. It accumulates in the background - slower releases, developer burnout, customer-reported bugs - until something catastrophic makes it visible.

I want to make it visible before that happens. Here's a calculation you can actually run for your team.

The Math That Got Buried

The most widely cited figure in software testing economics: bugs found in production cost 60–100x more to fix than bugs found during design.

It traces back to IBM Systems Sciences Institute research, originally from 1981 internal training materials. The Register ran a 2021 piece questioning whether this ever existed as a peer-reviewed paper - a fair challenge. But the directional finding - that bugs get exponentially more expensive as they age through the SDLC - has been confirmed independently by:

NIST's 2002 study on software quality infrastructure, which pegged software defects at $59.5 billion in annual cost to the US economy

Capers Jones' research across 12,000+ projects, which consistently found similar multipliers

BetterQA's analysis of SDLC-stage fix costs, which found production bugs run approximately 30x the cost of catching the same bug in development

The exact multiplier is debatable. The direction is not.

For this calculation, I'm using the conservative end.

Bug Cost by SDLC Stage

Stage	Relative Cost	Example: $200 fix during development
Design / requirements	0.5x	$100
Development (caught by dev)	1x	$200
QA / testing phase	5x	$1,000
Staging / UAT	10x	$2,000
Production	30x	$6,000

Assumption: These multipliers use the conservative end of the cited range (IBM goes to 100x; I'm using 30x for production). Actual cost per bug varies enormously by system criticality, customer impact, and bug type. A cosmetic UI bug in production costs far less than a data corruption bug. These numbers are directional tools for making the case, not inputs for precise budgeting. Adjust the production multiplier up for customer-facing, data-sensitive, or regulated systems.

The $6,000 production cost is not just engineer debug hours. It includes: incident coordination, customer communication, potential data cleanup, postmortem time, and possible SLA penalties. Reputational cost is not included, because it's hard to quantify and easy to dismiss.

The Manual Regression Tax

If you have no test automation, you have manual regression. Here's what that actually costs.

Assumptions for this calculation:

Mid-size team: 15 engineers
2-week sprint cycle (26 releases per year)
1 QA engineer (or developer in QA rotation) running regression each cycle
Loaded engineer cost: $120/hour

Assumption: $120/hr is a US-market estimate for a mid-level engineer including salary, benefits, and overhead. Adjust for your market and seniority. European teams might run $80–$100/hr; senior engineers in high-cost markets might run $180–$200/hr.

A realistic 200-test manual suite:

Activity	Time per cycle	Annual cost
Test execution	4 hours	$12,480
Triage of environment failures (not real bugs)	1 hour	$3,120
Re-test after bug fixes	3 hours	$9,360
Total	8 hours	$24,960

That's approximately $25,000/year in labour, just to manually regression-test a 200-case suite on a 2-week cycle.

This doesn't count:

Releases delayed because testing wasn't finished in time
Bugs that escaped because the manual suite didn't cover edge cases
The opportunity cost of that QA engineer's time not going toward exploratory or risk-based testing

The Release Confidence Tax

This one is harder to put a number on, but arguably more expensive.

When your team doesn't trust the test suite - or doesn't have one - release decisions turn conservative. Teams:

Ship less frequently ("let's batch this with next sprint to reduce risk")
Batch changes, which paradoxically increases risk per release
Proliferate feature flags as a crutch, adding operational complexity
Hold informal war rooms before every major release Developers already spend 41% of their time debugging, according to 2024 survey data. A non-trivial chunk of that is diagnosing issues that better test coverage would have caught before they merged.

Conservative framing: if better test automation saves even 2 hours/week per developer by reducing the time spent debugging regressions and environment issues, that's:

2 hrs × 15 engineers × 50 weeks × $120/hr = $180,000/year

That's a large number. Use it carefully - it's the upper bound of what's plausible, not a guarantee.

Run This for Your Team

Here's the back-of-napkin formula:

Step 1: Manual regression labour
= (test cases) × (avg minutes per test / 60) × (releases/year) × (hourly rate)

Step 2: Bug escape cost
= (production bugs per quarter × 4) × (avg cost per production incident)
  [Estimate your cost per incident: engineering hours + customer impact time]

Step 3: Release velocity tax
= (avg days delayed per release) × (releases/year) × (daily cost of delay)
  [Daily cost of delay = revenue at risk + opportunity cost of unshipped features]

Total annual cost of no automation = Step 1 + Step 2 + Step 3

For most teams, Step 2 alone - the production bug escape cost - exceeds the full annual cost of building and maintaining a test automation suite.

What Automation Actually Costs

The counterargument: "automation is expensive too." Fair. Here's the honest breakdown.

Initial setup (realistic, not optimistic):

Item	Estimate
Framework selection and setup	2–3 days
Writing 200 automated tests	3–4 engineer-weeks
CI integration and pipeline config	2–3 days
Total initial investment	~4–5 engineer-weeks

At $120/hr loaded cost: ~$19,200–$24,000 initial investment.

Annual maintenance:

Test maintenance consumes 30–50% of the initial build effort per year for a well-maintained suite.

Assumption: Industry data shows maintenance consuming 30–80% of automation budget. The lower end (30–50%) applies to suites that are actively managed, not allowed to grow without pruning, and built on stable APIs rather than brittle UI selectors. If you're running a Selenium-heavy UI suite on a frequently changing frontend, you're likely on the higher end.

Annual maintenance ≈ 30% × $22,000 = ~$6,600/year

5-year total cost of ownership:

Initial build:     $22,000
5 years maintain:  $33,000
Total:             $55,000

Compare that to the $25,000/year manual regression cost, compounded over 5 years: $125,000 - and that's before counting escaped bugs and delayed releases.

The ROI case is straightforward. The real objection isn't cost. It's time: teams without automation are usually under delivery pressure and can't find the runway to invest. That's a legitimate constraint - but it's worth naming clearly, because "we can't afford it" and "we don't have time right now" have different solutions.

Where AI Changes the Math

AI-based test generation shifts the cost curve at the margins. Here's honestly what it changes and what it doesn't.

What it reduces:

Initial authoring time. Studies on LLM-assisted test generation show 50–70% reduction in test authoring time for well-scoped code. The "4–5 engineer-weeks to write 200 tests" shrinks, potentially to 1.5–2 weeks.
Edge case coverage. LLMs generate negative cases, boundary values, and permutations that humans commonly skip under time pressure. The coverage you get for free is worth more than the authoring time saved.
Maintenance, partially. Self-healing test features (adaptive locators, selector repair) reduce churn caused by UI changes. This is real but not total - tests that encode business logic assumptions still break when business logic changes.

What it doesn't change:

You still need to understand what you're testing and why. AI generates tests; it doesn't understand your business rules.
Flaky tests still require cleanup. Generating more tests on top of a flaky foundation makes things worse.
The strategic decisions - what to test, at what level, with what priority - still require a human with context. The fundamental ROI argument doesn't change with AI. It gets better: the initial investment drops, which makes the case even cleaner. But the math works either way.

P.S.: It's worth mentioning that I lead the AI team at QApilot - an AI-native mobile test automation company. I might have a bias for more automation with AI, but when that bias is based on data, it's ok I guess.