DEV Community: Surendranath Reddy Jillella

Flaky Tests Persist Because Everyone Is Ignoring Them Rationally

Surendranath Reddy Jillella — Thu, 27 Aug 2026 12:40:13 +0000

You have done everything right.

You made the economic case for automation and got the investment approved. You distributed quality checks across the SDLC instead of piling them at the end. You replaced pyramid thinking with risk-weighted coverage. You stopped reporting a coverage percentage that was lying to you.

Six months later, your engineers have started ignoring test failures.

Not because they are careless. Because ignoring test failures became the rational choice. This article is about how that happens, why it happens to teams that know better, and why it is the final form of Test Debt.

What is flakiness?

A flaky test is a test that fails intermittently without any change to the code it covers. It sometimes passes and sometimes fails, with no consistent pattern. The most common root causes are timing issues in async operations, test-order dependencies, shared mutable state, and coupling to external services.

All of these are fixable. The fixable nature of the problem is not what makes it interesting. What makes it interesting is that teams fix very little of it, and teams with strong engineers who care about quality fix very little of it.

The reason is not the technical difficulty.

The scale

The numbers are worth stating clearly, because they establish what is actually at stake here:

At Google, approximately 16% of tests show some form of flakiness, and 84% of transitions from passing to failing involve a flaky test rather than a genuine regression.

At Microsoft, roughly 25% of test failures in large-scale CI systems are caused by flakiness, not actual code defects. The average time a developer spends per flaky test investigation: 30 minutes, before determining it was not a real failure.

Atlassian estimated 150,000 developer hours per year consumed by flaky test investigation before they built automated detection tooling.

Slack's mobile test failure rate reached 56.76% before they intervened. More than half of all test failures were noise.

These are not teams with poor engineering culture. They are among the best-resourced and most technically sophisticated organisations in the industry.

The rational inaction problem

Here is the core insight this article is built around: flaky tests are the only common engineering quality problem where every person in the chain responds correctly given their local information, and the outcome is still catastrophic.

The developer who sees a test failure and hits retry is not being careless. In their experience, 85% of unexplained failures are environment issues, not regressions. Investigating would take 30 minutes and probably turn up a race condition in test infrastructure that is not their code. Retry is the rational choice.

The QA engineer who logs the failure as "likely flaky, monitoring for pattern" is not dropping the ball. Without detection tooling, they cannot quickly distinguish a flaky failure from a real regression. Flagging and watching is the rational choice.

The infrastructure team that says "it is a test quality issue, not infrastructure" is not deflecting. Flaky tests usually are a test quality issue. Declining scope is the rational choice.

The engineering manager who does not put flakiness remediation on the sprint roadmap is not mismanaging. There are always user-facing features with higher business visibility. Flakiness does not appear in the product backlog with an owner or a deadline. Deferring is the rational choice.

Nobody made a bad decision, but nobody fixed anything as well.

This is the organisational failure mode this series has been describing in different forms since Article 1: costs that are invisible and diffuse; don't get addressed, because the people experiencing the cost in small increments are never the people with authority to fix it at the root.

The rational inaction framing applies most clearly to mid-to-large engineering organisations with specialisation between roles. In small teams of fewer than 10 engineers, the same person often occupies multiple positions in the chain described above, which creates different incentive dynamics. The argument is strongest for teams of 20 or more where there is clear separation between developer, QA, infrastructure, and management roles.

Why retry is the most expensive button in your CI interface

When a team normalises retrying flaky tests, they make a subtle but load-bearing decision: test failures no longer reliably indicate real problems.

That decision has a half-life. Once a team accepts that some failures are noise, the threshold for investigating failures drifts upward. A suite with 5% flakiness earns mild skepticism. A suite with 20% flakiness is treated as broken infrastructure. A suite with 50% flakiness is effectively ignored.

Slack documented their mobile test failure rate reaching 56.76% before they intervened. At that point, the pipeline provides negative value. It adds latency to every deployment while producing no useful signal about regressions. You have paid to build, maintain, and run a suite that has become pure noise.

The connection to Article 1 in this series is direct: Article 1 described the cost of having no test automation. A flaky, untrusted test suite is a different path to the same destination. You are making deployment decisions without a reliable test signal. The investment from Articles 1 through 4 has been eroded to zero. You are still paying 100% of the maintenance cost and receiving close to 0% of the coverage value.

The accountability structure that actually works

The research on teams that have genuinely reduced flakiness shows a consistent pattern. The technical fixes vary by codebase and language. The organisational fix does not vary.

At Microsoft, flaky tests are automatically assigned to the developer who owns the affected code. Not a team. A named individual. Developers with more than 10 open flaky test bugs have their PR merges blocked until they close them.

At Atlassian, the approach combined three elements: automated detection (so no manual effort required to identify flaky tests), immediate quarantine (the test is removed from the blocking suite while investigation is open, keeping the pipeline trustworthy), and a named owner assigned within 24 hours of identification.

Neither of these is primarily a technical solution. Both are accountability structures. They answer the question "whose problem is this?" with a specific name and a specific timeline, rather than leaving it as a shared responsibility that falls on no one.

The practical framework, regardless of team size:

Make the number visible first. Pull your CI data for the last 30 days. Find every test that failed and then passed on retry. Express that as a percentage of total runs. Most teams have never seen this number. It is the most important signal in their pipeline, and it is not surfaced by default in any major CI tool.

Quarantine, do not delete. A flaky test usually covers real behaviour. Removing it from the blocking suite preserves pipeline signal while investigation is open. Deleting it removes coverage. These are different decisions with different consequences.

Assign by name, not by team. "The QA team owns flakiness remediation" produces the same outcome as no ownership. "Jason owns these four tests, resolution expected by end of sprint" produces a different outcome.

Set a flakiness budget. Define a maximum acceptable rate; 1% is a reasonable starting threshold, and treat it the same way you treat error rate or latency: a metric with a threshold, a dashboard, and a named owner who is accountable when it is breached.

Assumption: The 1% flakiness budget threshold is based on commonly cited industry practice rather than a formally derived number. Google and Microsoft operate internal flakiness thresholds but do not publicly document the specific percentages. Teams with very large test suites (10,000 or more tests) may need to calibrate differently based on acceptable investigation volume.

Why AI detection changes the equation but not the root cause

Automated flaky test detection, including AI-based pattern recognition across CI runs, removes the manual work that was the excuse for not addressing flakiness earlier.

Without detection tooling, asking a team to track flakiness is asking them to do manual bookkeeping across every CI run. Under any reasonable delivery pressure, this does not happen.

With detection tooling, the identification is automatic. The question of whether to address it becomes purely organisational: who owns it and what happens if they do not.

This is why detection tools are necessary but not sufficient. Teams that deploy automated flaky test detection without the accountability structure around it often end up with a dashboard showing 12% flakiness that nobody acts on. The bottleneck was never the detection. It was the ownership.

Closing the series

This series started with a single argument: the cost of not investing in testing is real, measurable, and invisible until it is catastrophic.

Each article has described a different mechanism through which testing investment either fails to materialise or silently loses value:

Article 1 - The investment does not happen because nobody has made the cost of not having it visible (Article 1).
Article 2 - The investment happens too late in the cycle because the tooling for real-time quality feedback was not in place when the policy changed.
Article 3 - The investment covers the wrong things because the pyramid model was a cost heuristic that got mistaken for a structural law.
Article 4 - The investment measures the wrong things because a coverage gate creates the incentive to pass the gate rather than catch the bugs.
And then in this article, we spoke about how the investment erodes silently because flakiness is everyone's problem in a diffused sense and nobody's problem in an accountable sense.

The through-line is not tooling. The through-line is that each of these problems persists because the cost is invisible and the accountability is diffused. The moment you make the cost visible and assign the accountability to a specific person, the problem becomes tractable. Not easy, not fast, but tractable.

Article 3 ended with the observation that deciding what is risky is still a human judgment. Every article in this series has made a version of that same point. The investment decision is human. The timing decision is human. The prioritisation decision is human. The measurement decision is human. The accountability decision is human.

The tools have never been the bottleneck.

What we are trying to do at QApilot is collapse the gap between 'the cost exists' and 'someone sees it and can act on it.' That is true for coverage gaps, for flakiness, for untested risk paths. The tooling is there. The visibility is what has been missing.

Why Hitting Your Coverage Target Is Making Your Tests Worse

Surendranath Reddy Jillella — Fri, 21 Aug 2026 06:04:26 +0000

I had 87% coverage, and we still broke the billing flow on launch day.

Not because of a gap in the percentage. Because 87% was covering the wrong things. The tests were written to pass a gate, not to catch a failure.

That is a more common story than most teams admit. And the reason it keeps happening is not that engineers are careless. It is that the incentive structure you created made it the rational outcome.

The series checkpoint

The first three articles in this series built the investment case for testing and then dismantled the received wisdom about how to execute it. We've made the economic argument for automation. We've restructured when quality checks happen across the SDLC. We've replaced the pyramid model with something shaped by risk rather than by code hierarchy.

Now, when someone asks: how do you know if it is working?

The answer most teams give is their coverage percentage. This article is about why that answer is structurally broken, and why fixing it is a management decision before it is a tooling decision.

What coverage percentage actually measures

Coverage percentage tracks which lines of your code were executed during a test run. If a line ran, it counts as covered.

That is the complete definition.

It does not measure whether the test asserted anything meaningful about that line. It does not measure whether both branches of a conditional were exercised. It does not measure whether the specific inputs that cause failures were ever tried. A test that calls a payment function and checks assert response is not None covers the same lines as a test that validates the transaction ID, amount, currency, error code, and retry behaviour. The coverage tool treats them identically.

The research on this is unambiguous. A 2017 study by Kochhar et al. examined the correlation between code coverage and actual bug rates across 100 large open-source Java projects. The finding: the coverage of existing test suites has an insignificant correlation with the number of bugs found after release. Inozemtseva and Holmes found separately that line coverage is the metric that correlates least with real defect detection compared to other available options.

You are using the most popular quality metric in software engineering. It is also the least predictive of actual quality.

Goodhart's Law in your CI pipeline

Goodhart's Law comes from economics: when a measure becomes a target, it ceases to be a good measure. The principle describes what happens when you attach consequences to a proxy metric instead of to the underlying goal it was meant to represent.

Coverage percentage is a proxy metric. The underlying goal is: tests that catch bugs before production. When you make the proxy metric a gate, teams optimise for the proxy, not the goal. The badge turns green. The risk does not go down.

This is not a character flaw. It is a predictable response to an incentive you designed. Under delivery pressure, engineers do the rational thing: they hit the number using the fastest available path.

How the gaming plays out

The fastest path to coverage percentage is testing happy paths. User submits form successfully. API returns expected response. Data saves to the database. These scenarios are easy to construct, execute quickly, and each one covers a high line count because they touch the main execution path.

What does not get covered:

Error handling paths run on exception conditions that are tedious to construct in tests. Retry logic, fallback behaviour, and timeout handling live in branches that require specific setup. These are also, as a direct consequence, the places where the most consequential bugs hide.
Boundary conditions require generating test inputs at the edges of valid ranges, null states, empty strings, Unicode characters, and maximum lengths. Each of these covers almost the same lines as a single typical input but requires separate test construction. Under time pressure, one test covers the line.
Conditional branches show as "covered" once either branch executes. A function with a condition that checks user role, applied to admin and guest differently, can show as fully covered if only one role ever runs the test. The other branch remains untested with no coverage signal.

The outcome is a suite at 85% coverage where the well-tested parts are tested three times over, and the failure-prone parts have one shallow test each. Teams with high line coverage but low integration validation have been observed producing more than double the production incidents of teams with lower but more balanced coverage.

Assumption: The claim that coverage-gamed suites produce higher production incident rates than lower but risk-balanced suites is based on observed patterns in industry case studies, not a controlled experiment. The "double production incidents" figure comes from analysis of specific teams and may not generalize uniformly. It should be read as directional evidence of the problem rather than a precise multiplier.

What this costs you

In the first article in this series, the framing was: the cost of not investing in testing is invisible until it is catastrophic.

Coverage gaming creates a specific variant of that problem. You are paying the full maintenance cost of a test suite that is not delivering the risk reduction you paid for. Every test written to hit a percentage rather than catch a bug still needs to be maintained when the code changes, still runs in every CI cycle, and still occupies space in the mental model of everyone who reads the suite.

You have made the right investment decision. A portion of that investment is funding theatre rather than protection. The cost shows up in the budget. The corresponding value does not show up in production stability.

A better measurement

The honest version of coverage is mutation score. Mutation testing introduces deliberate small bugs into your code: flipping a comparison operator, removing a conditional check, changing a return value. Your test suite runs against each mutated version. If the tests catch the mutation (they fail), the mutation is killed. If the tests pass despite the code being wrong, the mutation survives.

A mutation score of 80% means your tests caught 80% of introduced bugs. That is a direct claim about defect detection, not about line execution. These are different claims.

A real case documented on dev.to reported 93% line coverage that turned out to be a 34% mutation score. The team had been reporting a green badge on a suite where two-thirds of the introduced bugs would have survived.

Mutation testing is computationally expensive, which is a real constraint. The practical approach: do not run it across the entire codebase. Run it on the paths where failure is most costly, specifically the risk-weighted paths identified in the work from Article 3. Use line coverage as a floor (it tells you about completely untested code) and mutation score as the signal for whether the tests you have are actually doing anything on the critical paths.

Branch coverage is a useful intermediate step. It is stricter than statement coverage, it surfaces the conditional gaps that line coverage misses, and most existing coverage tools already support it. Switching from statement to branch coverage as your CI gate raises the bar without requiring new tooling.

Assumption: Mutation testing adoption is growing but not mainstream as of mid-2026. Frameworks exist for most major languages (Stryker for JavaScript and TypeScript, PITest for Java, mutmut for Python) but are not a default feature in major CI platforms. The claim that most teams do not use mutation testing is based on tooling adoption patterns, not a formal survey.

The behavioural fix

The tool change matters less than the conversation change.

If coverage percentage is a pass/fail gate in your pipeline, you have created the incentive to game it. The alternative is not removing coverage tracking. It is removing the gate behaviour.

Report coverage as a trend, not a threshold. A team whose coverage has been at 78% for six months is in a fundamentally different position than one whose coverage dropped from 88% to 68% after a feature push. The trend carries information. The snapshot at a point in time does not.

Pair that with the question you actually want answered: are the highest-risk paths in this codebase meaningfully tested? That question does not have a single percentage answer. It requires looking at branch coverage on critical paths and, where the stakes are highest, mutation score. Neither of these is hard to implement. They are just not the defaults.

PS: I am Surendranath, leading the AI team at QApilot, and we are solving problems specific to mobile app testing through our AI-native product. Reach out if you are interested to try it out!

The End of the Testing Pyramid: What Replaces It in the AI Era

Surendranath Reddy Jillella — Thu, 06 Aug 2026 08:01:20 +0000

We have discussed the cost of no automation and reasoned "when to test".
This doesn't automatically solve the test coverage question - the QA teams are swamped enough, and if you ask them to test across the SDLC, that can only lead to testing what is easy or whatever is quick.

Hence the next question - What tests to prioritise?

The testing pyramid was and is gospel, still, and every sprint planning meeting with a QA team included some version of: "we need more unit tests at the base, fewer E2E tests at the top."

The pyramid made complete sense - for its time. I'm not sure it makes sense anymore.

What the Pyramid Was Actually Solving

Mike Cohn introduced the testing pyramid in Succeeding with Agile (2009), though the concept was sketched out earlier around 2003–04. His original framing had three layers: unit tests at the base, service/integration tests in the middle, UI tests at the top. Martin Fowler popularised it further on his blog, and it became a de facto standard across the industry.

The underlying logic was about cost:

Unit tests: cheap to write, fast to run, easy to pinpoint failures
Integration tests: moderate cost, slower, harder to isolate failures
E2E/UI tests: expensive to write, slow to run, brittle - and when they fail, you often spend an hour figuring out what actually broke

So the pyramid was really a cost-optimisation heuristic: do more of the cheap thing, less of the expensive thing - sensible advice when humans are writing every test by hand.

The AI Problem with This Logic

Here's what changes when AI can generate test cases: the marginal cost of writing the 100th test approaches zero.

Generating 10 unit tests costs me roughly the same as generating 100. The input cost - my time writing a prompt, reviewing output - is largely fixed, not variable.

Assumption: This assumes LLM output quality is "good enough to review and merge" without significant rework. Current research (MDPI, 2025) shows LLMs are effective for test generation on well-scoped, lower-complexity functions, but accuracy drops for complex, stateful logic. So this cost collapse is real for breadth of coverage, less so for depth in complex systems.

If writing isn't the bottleneck anymore, the pyramid's rationale starts to crack. But there's a catch.

The Cost That Didn't Collapse: Maintenance

Writing tests and maintaining tests are two different problems. AI hasn't fully solved the second one yet.

Industry data suggests test maintenance consumes anywhere from 30% to 50% of a QA team's automation effort.

That's not a pyramid problem; it's a maintenance problem. And it compounds fast when you generate tests faster than you can maintain them.

If AI test generation significantly outpaces AI test maintenance and healing capabilities, teams could end up with larger, flakier suites than they started with. This is a genuine risk in 2026 - generation tools have matured faster than maintenance/healing tools. The balance matters.

So the new model can't just be "generate everything." It has to be smarter about what gets generated.

What Should Replace the Pyramid

I've been thinking about this as risk-weighted coverage rather than a structural pyramid. The question isn't "how many tests at each layer?" - it's "where does a failure hurt most, and am I covered there?"

A few principles that seem to hold up:

1. Coverage by business risk, not code structure

A payment-processing function with 60% unit coverage is riskier than a string formatter with 95% coverage. The pyramid doesn't capture this. A risk-weighted model does. The first question before generating any test should be: what breaks the business if this fails?

2. The E2E tax is shrinking, not gone

E2E tests are still slow. They're still brittle when they fail. But AI-assisted generation makes writing them dramatically cheaper. The remaining cost is infrastructure and maintenance.

Assumption: As AI-based self-healing for selectors and UI state matures, the E2E maintenance cost will fall meaningfully. This is directionally happening but isn't yet the default reality for most teams in 2026.

3. A "testing mesh" over a rigid pyramid

Rather than a strict hierarchy, think of tests as a mesh - overlapping coverage at multiple levels, weighted toward high-risk areas. Some paths get unit + integration + E2E coverage. Low-risk utility code gets unit only. The shape is determined by risk analysis, not by a fixed ratio rule.

What This Means Practically

If you're maintaining a mature test suite:

Audit which E2E tests cover high-risk paths vs. low-risk happy paths. The latter are usually candidates for deletion or downgrade.
Use AI to generate edge-case unit tests for your critical business logic - that's where current LLM-based generation adds the most value with the least maintenance overhead.
Don't generate tests uniformly. Generate them where failure actually matters.

If you're greenfielding:

Start with risk mapping before writing a single test. What breaks the business if it fails?
Let AI fill coverage breadth; invest human effort in the tests for high-risk paths that require business context to assert correctly.

The Real Question

The pyramid isn't dead - it's become one valid shape among many. The right test distribution depends on your system's risk profile, not a universal rule.

The harder question the AI era forces is: who decides what's risky? And that's still a deeply human judgment. AI can generate tests for you, but it can't tell you what matters to your business unless you've told it.

P.S. - I am Surendranath from QApilot and this post is part the series Test Debt

Shift-Left is Dead. Long Live Shift-Everywhere

Surendranath Reddy Jillella — Thu, 30 Jul 2026 07:53:13 +0000

Last week, I presented the cost of no testing - Hidden cost of no test automation
It is now clear that testing is essential, if not earlier. The next question that comes to mind is When to test? - Shift Left is an answer that many leadership mandates suggest.
The practical interpretation of Shift-Left is simple - QA was going to write fewer tests, and developers were going to write more. The phrase was presented as a philosophy. It felt more like a headcount justification.

That's not what Larry Smith meant when he coined the term.

What Shift-Left Actually Said

Smith's article "Shift-Left Testing" was published in Dr. Dobb's Journal in September 2001. The observation was straightforward: on any project timeline, testing was always concentrated at the right side - near delivery. He proposed moving it left - closer to inception - to catch defects earlier and cheaper.

The economics behind it were sound. IBM Systems Sciences Institute data (originally from 1981 internal training materials) showed bugs found after release cost 60–100x more to fix than bugs found during design. Even if those exact multipliers are contested - The Register ran a worthwhile piece in 2021 questioning whether the original IBM study was ever a peer-reviewed paper - the directional finding has been consistently confirmed by NIST research, Capers Jones' analysis of 12,000+ projects, and anyone who has sat in a production incident war room at 2 am.

The idea was right. The execution became something else.

What Shift-Left Became

In practice, shift-left was operationalised as:

Developers writing unit tests they didn't always have the domain context to write well
CI gates nobody trusted because flaky tests made failure signals into noise
QA "embedded" in scrum teams without changing when or how testing actually happened
Static analysis tools added to pipelines that nobody read the output of

The mandate to shift left didn't come with a corresponding shift in tools, time allocation, or context. It was process vocabulary layered on top of unchanged behavior.

Assumption: This characterization reflects the most common pattern of shift-left adoption - process-driven, without sufficient tooling change. It won't apply to every team. Organizations that genuinely restructured roles, tooling, and feedback loops saw real results. But the pattern of "shift-left = devs write more tests with no additional support" is widespread enough to be a recurring frustration in engineering retrospectives. If your team executed shift-left well, you're in the minority.

The original promise - catch bugs earlier - was only partially delivered, for most teams.

What's Actually Different Now

The 2024 Developer Survey found developers spend only 24% of their time writing code. The rest goes to debugging (roughly 41%), meetings, reviews, documentation, and operational work.

That ratio is persistent and damning. A huge chunk of debugging time is spent on issues that better earlier feedback - not just earlier testing, but earlier signals - would have caught before they compounded.

What AI tooling is changing, when it's actually embedded rather than bolted on, is making quality checks ambient rather than phased:

In the IDE: Copilot-style tools flag issues as you type. Some generate test stubs inline for the function you just wrote.
In the PR: AI reviewers analyze diffs for missing edge cases, untested branches, and logic gaps - not just style and formatting.
In CI: AI-based test generation triggered on diff, targeting new and changed code specifically rather than running the full regression suite.
In production: Error monitoring that surfaces coverage gaps by tracing prod failures back to untested paths.

This isn't shift-left. It's shift-everywhere. Quality isn't a phase you move earlier - it's a layer that runs in parallel with development at every stage.

The Distinction That Matters

Shift-left was about when in the SDLC to apply testing.

Shift-everywhere is about continuous presence of quality signals regardless of SDLC phase.

A developer gets immediate feedback in their IDE. The same PR gets an AI review pass. The CI pipeline runs targeted tests on exactly the changed code. An alert the next day surfaces a production error and maps it back to a coverage gap.

The SDLC doesn't stop being sequential - code still gets designed, written, reviewed, deployed. But quality signals are no longer tied to a single phase of that sequence.

Assumption: "Shift-everywhere" as described assumes a mature AI tooling stack. Most teams in 2026 have 1–2 of these layers, not all four. The full ambient quality model is where the industry is heading, but it's not the default state yet. I'd estimate fewer than 25% of teams have more than two of these layers genuinely integrated and trusted, based on the tooling adoption curves I've seen.

What Hasn't Changed

AI doesn't fix everything. A few things remain stubbornly manual:

Understanding what to assert. Generating a test is easy. Knowing what the correct behavior should be for a non-obvious edge case - one that depends on business rules, not just code logic - still requires a human who understands the domain.

Flaky tests. AI can reduce new flakiness by generating more deterministic tests, but it doesn't automatically rehabilitate an existing flaky suite. That's still cleanup work.

Cross-service test strategy. What to test across service boundaries, under real load, in failure scenarios - that's architectural thinking that LLMs don't consistently reason well about yet.

The Practical Takeaway

If you're a developer
Shift-left was never the enemy. The enemy was treating it as a process change without a tooling change. The AI-era version of the same idea is more actionable because the tooling is actually there.

If you're running an engineering or QA team
Stop thinking about where in the pipeline quality lives. Start thinking about how many stages have real-time quality feedback embedded. Count the stages where a developer gets an automated signal about quality within minutes of writing code. If the answer is one - probably CI - you have room to move.

P.S.: This article is part of the Test Debt series.
About me - I lead the AI team at QApilot - an AI-native mobile test automation company, and we build products to help you achieve the quality standards that your users demand.

The Hidden Cost of No Test Automation: A Back-of-Napkin Calculation

Surendranath Reddy Jillella — Thu, 23 Jul 2026 12:48:07 +0000

Nobody cancels a sprint to do a testing ROI analysis. The cost of skipping test automation doesn't have a line item in your budget. It doesn't show up on a dashboard. It accumulates in the background - slower releases, developer burnout, customer-reported bugs - until something catastrophic makes it visible.

I want to make it visible before that happens. Here's a calculation you can actually run for your team.

The Math That Got Buried

The most widely cited figure in software testing economics: bugs found in production cost 60–100x more to fix than bugs found during design.

It traces back to IBM Systems Sciences Institute research, originally from 1981 internal training materials. The Register ran a 2021 piece questioning whether this ever existed as a peer-reviewed paper - a fair challenge. But the directional finding - that bugs get exponentially more expensive as they age through the SDLC - has been confirmed independently by:

NIST's 2002 study on software quality infrastructure, which pegged software defects at $59.5 billion in annual cost to the US economy

Capers Jones' research across 12,000+ projects, which consistently found similar multipliers

BetterQA's analysis of SDLC-stage fix costs, which found production bugs run approximately 30x the cost of catching the same bug in development

The exact multiplier is debatable. The direction is not.

For this calculation, I'm using the conservative end.

Bug Cost by SDLC Stage

Stage	Relative Cost	Example: $200 fix during development
Design / requirements	0.5x	$100
Development (caught by dev)	1x	$200
QA / testing phase	5x	$1,000
Staging / UAT	10x	$2,000
Production	30x	$6,000

Assumption: These multipliers use the conservative end of the cited range (IBM goes to 100x; I'm using 30x for production). Actual cost per bug varies enormously by system criticality, customer impact, and bug type. A cosmetic UI bug in production costs far less than a data corruption bug. These numbers are directional tools for making the case, not inputs for precise budgeting. Adjust the production multiplier up for customer-facing, data-sensitive, or regulated systems.

The $6,000 production cost is not just engineer debug hours. It includes: incident coordination, customer communication, potential data cleanup, postmortem time, and possible SLA penalties. Reputational cost is not included, because it's hard to quantify and easy to dismiss.

The Manual Regression Tax

If you have no test automation, you have manual regression. Here's what that actually costs.

Assumptions for this calculation:

Mid-size team: 15 engineers
2-week sprint cycle (26 releases per year)
1 QA engineer (or developer in QA rotation) running regression each cycle
Loaded engineer cost: $120/hour

Assumption: $120/hr is a US-market estimate for a mid-level engineer including salary, benefits, and overhead. Adjust for your market and seniority. European teams might run $80–$100/hr; senior engineers in high-cost markets might run $180–$200/hr.

A realistic 200-test manual suite:

Activity	Time per cycle	Annual cost
Test execution	4 hours	$12,480
Triage of environment failures (not real bugs)	1 hour	$3,120
Re-test after bug fixes	3 hours	$9,360
Total	8 hours	$24,960

That's approximately $25,000/year in labour, just to manually regression-test a 200-case suite on a 2-week cycle.

This doesn't count:

Releases delayed because testing wasn't finished in time
Bugs that escaped because the manual suite didn't cover edge cases
The opportunity cost of that QA engineer's time not going toward exploratory or risk-based testing

The Release Confidence Tax

This one is harder to put a number on, but arguably more expensive.

When your team doesn't trust the test suite - or doesn't have one - release decisions turn conservative. Teams:

Ship less frequently ("let's batch this with next sprint to reduce risk")
Batch changes, which paradoxically increases risk per release
Proliferate feature flags as a crutch, adding operational complexity
Hold informal war rooms before every major release Developers already spend 41% of their time debugging, according to 2024 survey data. A non-trivial chunk of that is diagnosing issues that better test coverage would have caught before they merged.

Conservative framing: if better test automation saves even 2 hours/week per developer by reducing the time spent debugging regressions and environment issues, that's:

2 hrs × 15 engineers × 50 weeks × $120/hr = $180,000/year

That's a large number. Use it carefully - it's the upper bound of what's plausible, not a guarantee.

Run This for Your Team

Here's the back-of-napkin formula:

Step 1: Manual regression labour
= (test cases) × (avg minutes per test / 60) × (releases/year) × (hourly rate)

Step 2: Bug escape cost
= (production bugs per quarter × 4) × (avg cost per production incident)
  [Estimate your cost per incident: engineering hours + customer impact time]

Step 3: Release velocity tax
= (avg days delayed per release) × (releases/year) × (daily cost of delay)
  [Daily cost of delay = revenue at risk + opportunity cost of unshipped features]

Total annual cost of no automation = Step 1 + Step 2 + Step 3

For most teams, Step 2 alone - the production bug escape cost - exceeds the full annual cost of building and maintaining a test automation suite.

What Automation Actually Costs

The counterargument: "automation is expensive too." Fair. Here's the honest breakdown.

Initial setup (realistic, not optimistic):

Item	Estimate
Framework selection and setup	2–3 days
Writing 200 automated tests	3–4 engineer-weeks
CI integration and pipeline config	2–3 days
Total initial investment	~4–5 engineer-weeks

At $120/hr loaded cost: ~$19,200–$24,000 initial investment.

Annual maintenance:

Test maintenance consumes 30–50% of the initial build effort per year for a well-maintained suite.

Assumption: Industry data shows maintenance consuming 30–80% of automation budget. The lower end (30–50%) applies to suites that are actively managed, not allowed to grow without pruning, and built on stable APIs rather than brittle UI selectors. If you're running a Selenium-heavy UI suite on a frequently changing frontend, you're likely on the higher end.

Annual maintenance ≈ 30% × $22,000 = ~$6,600/year

5-year total cost of ownership:

Initial build:     $22,000
5 years maintain:  $33,000
Total:             $55,000

Compare that to the $25,000/year manual regression cost, compounded over 5 years: $125,000 - and that's before counting escaped bugs and delayed releases.

The ROI case is straightforward. The real objection isn't cost. It's time: teams without automation are usually under delivery pressure and can't find the runway to invest. That's a legitimate constraint - but it's worth naming clearly, because "we can't afford it" and "we don't have time right now" have different solutions.

Where AI Changes the Math

AI-based test generation shifts the cost curve at the margins. Here's honestly what it changes and what it doesn't.

What it reduces:

Initial authoring time. Studies on LLM-assisted test generation show 50–70% reduction in test authoring time for well-scoped code. The "4–5 engineer-weeks to write 200 tests" shrinks, potentially to 1.5–2 weeks.
Edge case coverage. LLMs generate negative cases, boundary values, and permutations that humans commonly skip under time pressure. The coverage you get for free is worth more than the authoring time saved.
Maintenance, partially. Self-healing test features (adaptive locators, selector repair) reduce churn caused by UI changes. This is real but not total - tests that encode business logic assumptions still break when business logic changes.

What it doesn't change:

You still need to understand what you're testing and why. AI generates tests; it doesn't understand your business rules.
Flaky tests still require cleanup. Generating more tests on top of a flaky foundation makes things worse.
The strategic decisions - what to test, at what level, with what priority - still require a human with context. The fundamental ROI argument doesn't change with AI. It gets better: the initial investment drops, which makes the case even cleaner. But the math works either way.

P.S.: It's worth mentioning that I lead the AI team at QApilot - an AI-native mobile test automation company. I might have a bias for more automation with AI, but when that bias is based on data, it's ok I guess.