DEV Community: QApilot

App Intents, AppFunctions, and the Interface Your Test Suite Has Never Driven

Surya Polavarapu — Thu, 17 Sep 2026 11:45:32 +0000

iOS 27 reached every compatible iPhone on 14 September, and most of the coverage has understandably been about Siri - the new architecture, the conversational mode, the access to personal context. Underneath that consumer story is a change in who can reach into your app, and most development and testing teams haven't caught up with it.

The side doors are not new

The UI is the storefront: browse, add to cart, check out, in that order. Apple has been opening side doors into that shop for a decade.

SiriKit (2016) was a voice side door, but only for a few Apple-approved jobs: send a message, request a ride, start a workout, pay someone. Uber did not have to invent a general “Siri API.” It implemented “book a ride,” and Siri knew how to ask for pickup and destination. A person could complete a core action without opening the app. That is already a second interface.

Shortcuts (2018) was a recipe side door. A user, or your team, could save a sequence: “log this workout,” “order my usual,” “set Do Not Disturb and open Maps.” Tapping the shortcut, or saying “Hey Siri, run Morning,” ran those steps without walking the screens. QA rarely treated Shortcuts as a first-class surface, but customers were already using the app that way.

App Intents (2022) was Apple replacing those one-off side doors with a catalog of actions the system can call: Siri, Spotlight search, the Action Button, widgets, Shortcuts. “Create event,” “play this playlist,” “show my order.” Same idea as publishing a short list of APIs, except the callers are Apple’s system features instead of a website.

So the surface is old. What arrived this year is a far more capable visitor using it.

What changed is the visitor

Siri AI doesn't need the ten phrases you published. It reads the catalogue, looks at what's on screen and at personal context, forms a plan from a sentence, and calls actions in whatever order that plan requires.

Apple has been promising this visitor since WWDC 2024, where personal context, onscreen awareness and App Intents-powered in-app actions were announced together. The features were officially delayed in March 2025, slipped through that year and into early 2026, and landed on phones in September. The long gap matters here, because it means many teams filed App Intents under "did that in 2022" and stopped thinking about it. Three things are true now that weren't then.

Discovery flipped. Previously a user had to know a Shortcut existed or say a phrase you'd published. Now the assistant finds the action. Anything in the catalogue is in play — including intents someone added years ago for a Shortcuts integration and nobody has looked at since.

Sequence is no longer your design. A Shortcut runs the order you wrote. A screen flow only offers the next legal step. An assistant composing "move lunch, tell the attendees, book a car" calls whatever functions it decides it needs, in combinations no screen ever permitted. Every invariant a team relies on because "the user can't get there from here" needs rechecking.

Absence became a product problem. In 2022, skipping App Intents made you weaker in Shortcuts and the app still worked. Apple's documentation now describes SiriKit, Intents and IntentsUI as providing legacy support for existing interactions and points developers to App Intents for modern integration. SiriKit still covers its old domains; it does not carry you onto the new surface. Users will ask Siri to do the thing, Siri will do it in a competitor's app, and nothing will appear in your crash reports.

Google made the same move this year. Android 17 expanded AppFunctions, described in the release announcement as contributing an app's capabilities as orchestratable tools for Android MCP, the on-device equivalent of the Model Context Protocol. MCP is how agents call tools on servers; Google has pointed the same idea at the apps installed on a phone. Gemini integration is still in private preview with trusted testers, so Android teams have more runway — but two platforms converged on one pattern inside a single cycle. That's why 2026 is the story and 2022 wasn't.

Why this lands on QA

An App Intent or an AppFunction is a function signature: typed parameters, entity types it accepts and returns, discoverability settings, and a policy governing what authentication is required before it runs. Nothing a UI test depends on is present on that path — no screen to read, no element to tap, no navigation sequence quietly establishing state along the way.

Beyond the ordering problem above, three properties make this hard to verify.

Parameter values arrive from an interpreter. A UI constrains its own input: a date picker only emits a valid date, a dropdown only emits a member of a known set. An intent parameter is populated by a model reading a sentence, which makes "next Friday," spoken by someone in a timezone the app was never tested in, a live input to a function whose date picker never had to handle it.

Authentication and confirmation moved off the screen. When the action executes outside your UI, so does the moment where the user agrees to it. Apple exposes settings that govern this, and getting them wrong doesn't fail loudly — it means something happens the user never clearly consented to, a defect that surfaces in a review rather than in CI.

The caller changes without your code changing. This one has no precedent. A UI test pins a contract between your code and a human whose behaviour is stable. An intent's caller is a model updated on the platform's schedule. The same function, with the same tests green, can start being invoked differently after an OS update you didn't ship.

The tooling exists. Its scope is the issue.

Worth being precise, because the gap isn't an absence of tools. Apple shipped App Intents Testing alongside iOS 27 — an XCTest framework that runs intents, entities, enums and queries out of process, the way Siri and Shortcuts perform them, with type-erased APIs letting a test reference an intent by name and set its parameters without linking against the app target. It covers onscreen context through view annotations. On Android, adb shell cmd app_function list-app-functions enumerates what a package registers, execute-app-function invokes one with a JSON parameter string, and the AppFunctions Testing Agent goes further, letting you observe behaviour in a conversational flow.

These are good tools and anyone publishing intents should be using them. What they verify is that a function does what it claims when called. That layer is well served.

What remains unverified is the system the function now sits inside: whether a model read an ambiguous utterance the way you expected, whether an agent composing across three apps called your functions in an order your data model survives, whether any of that held after the platform updated the assistant underneath you. Those are properties of an interaction rather than of a function, and the discipline for testing them hasn't formed yet. Security teams recognised the shape early — tooling that inventories published intents and audits what they expose appeared quickly, because a new caller with access to an app's data and actions reads immediately as a threat model worth mapping. The quality equivalent is further behind.

Where a team can start

Inventory first. List every intent or function the app publishes, and record its parameters, its authentication policy, and what data it can reach. That list is usually longer than expected, for the reason above: intents accumulate, and the ones added for Shortcuts in 2022 are now exposed to a caller nobody anticipated.

Then use the platform tools for what they're good at, and be explicit about what sits outside them. Boundary cases on every parameter. Invocation from states the UI cannot produce. Confirmation behaviour when the app isn't in the foreground. Behaviour when the underlying data has changed between the agent reading it and acting on it. Past that, a team is making judgement calls without a safety net — and knowing that is better than assuming a green suite covers it.

The doors have been there for years. What's new is that something on the other side can now read the whole catalogue and decide for itself which ones to open.

We've spent years at QApilot building tests that describe a goal rather than a sequence of taps, because the surface underneath keeps moving. An assistant calling your app is a similar problem arriving from a new direction and needs to be addressed.

AI Test Generation for Mobile Apps Looks Great in Demos. Here Is Why It Decays in Production.

Surendranath Reddy Jillella — Thu, 10 Sep 2026 11:58:33 +0000

I have been in developer tooling long enough to have watched the same cycle repeat: a new capability arrives, it looks genuinely impressive in controlled conditions, teams adopt it at scale, and then six months later the results are underwhelming, and nobody quite knows why.

AI test generation for mobile apps is currently in the impressive-demo phase. The question worth asking now, before the disappointment phase, is: what breaks when this hits a real production codebase?

The answer has a name. It is called constraint decay. And it is not being talked about enough in the mobile testing conversation.

AI Is Changing Mobile Test Generation

The shift is real and measurable. Mobile teams are increasingly using large language models to generate test cases directly from source code, user stories, or UI recordings. A bunch of tools now offer AI-assisted generation as a core feature.

The promise is straightforward: point an AI at your codebase or your screen recordings, and it generates test cases. Teams that used to spend three weeks writing an Appium test suite can now get a first draft in hours.

For greenfield screens with simple logic, this works well. For a login screen with a text field and a submit button, any capable LLM will generate reasonable tests quickly. The output looks comprehensive. Coverage numbers climb. Teams ship more tests than they wrote by hand.

This is where the problem starts.

What the Current Conversation Is Missing

Most discussions about AI test generation for mobile focus on two things: how fast you can generate tests, and how well self-healing handles UI changes. Both matter. But there is a third problem that almost nobody in the mobile testing space is naming directly.

AI test generation tools are evaluated and marketed under conditions that do not resemble your actual app.

The screens used in demos are simple. The apps used in case studies are well-documented, architecturally clean, and purpose-built for the demonstration. Your payment flow that involves three third-party SDKs, a backend session token, biometric authentication on iOS, back-stack handling differences on Android, and a compliance requirement from your legal team three months ago is a fundamentally different target.

When real structural constraints are present, AI performance on code generation tasks drops substantially. That is not a hypothesis. It is the finding of a 2025 research paper that should be required reading for anyone building or buying AI testing tooling.

Constraint Decay: What the Research Actually Says

A May 2026 paper published on arXiv (arXiv:2605.06445) studied how LLM coding agents perform across increasing levels of structural constraint. The researchers ran evaluations across 80 greenfield generation tasks and 20 feature-implementation tasks spanning eight web frameworks, measuring performance at four constraint levels.

L0 (unconstrained): The agent has full architectural freedom. It can use any framework, any structure, any pattern it chooses.

L1 (basic framework specified): A framework is specified. The agent still has broad flexibility within it.

L2 (API contracts added): Data contracts and API shapes are specified. The agent must conform to existing interfaces.

L3 (fully constrained): Framework, API contracts, ORM, and data layer are all specified. The agent must operate within a fully defined structural environment.

At L0, top models exceeded 85% assertion pass rate. By L3, average performance had dropped 30 percentage points. The relative loss was 40% of baseline performance. The worst-performing configuration dropped 45 percentage points, losing 62% of its L0 score.

The researchers named this phenomenon constraint decay: as structural requirements accumulate, agent performance declines substantially.

Assumption: The constraint decay paper measures coding agent performance on backend generation tasks, not test generation specifically. The extrapolation to test generation is logical given that test generation under real constraints (business rules, architecture, platform specifics) faces the same structural complexity problem, but this has not been formally studied under the same experimental design. The directional argument holds; the exact percentages would differ.

What This Looks Like in Mobile Testing

The L0 to L3 mapping translates directly to mobile test generation:

L0 (demo conditions): Generate tests for a login screen with an email field and a password field. The AI knows nothing about your architecture and has full freedom. It produces reasonable coverage of basic flows.

L3 (your actual app):

Generate tests for your checkout flow that must account for:

Session tokens from your auth service that expire after 15 minutes of inactivity
Biometric authentication on iOS 17 and above, with a PIN fallback below that version
Android back-stack behaviour that differs between the checkout and payment confirmation screens
A third-party payment SDK that behaves differently when the user has saved cards vs. first-time entry
A promotional discount engine that applies different rules based on user segment
A compliance requirement that prevents storing card details in local state

An AI given the codebase alone cannot infer most of this from source files. The session timeout is a backend configuration. The iOS version cutoff is a product decision. The Android back-stack behavior is a known edge case from a production incident six months ago. The promotional rules live in a database, not in the app code. The compliance requirement is a document in your legal team's folder.

At L3, the AI is not generating the wrong tests because it is a bad model. It is generating the wrong tests because it does not have the context to know what the right tests are. The constraints are real, and they are not in the codebase.

How to Manage Constraint Decay in Practice

Constraint decay is not a reason to avoid AI test generation. It is a reason to be deliberate about how you apply it.

Give AI bounded context, not the full codebase.

Dumping an entire mobile codebase into a prompt and asking for test generation is the L0 approach applied to an L3 problem. The model will produce tests that reflect the structure of the code it can see, not the business logic that lives outside it.

Instead: scope each generation request to a specific module, a specific user flow, or a specific change in a pull request. The tighter the scope, the closer the model gets to the actual constraints.

Refresh tests with the context that triggered the code change.

A test suite is only as fresh as its understanding of why the code is the way it is. When a developer changes the checkout flow because of a compliance update, the AI generating tests for that diff needs to know the compliance context, not just the code diff.

Without that context, the regenerated tests will cover the code change correctly but miss the intent behind it. Six months of small changes like this without context refresh is how a test suite becomes structurally correct and functionally wrong.

Use mutation testing to detect when generated tests have decayed.

AI-generated tests look syntactically correct even when they have stopped catching real regressions. Mutation testing, which introduces deliberate small bugs and checks whether your tests catch them, surfaces this decay without requiring a production incident.

If your mutation score is falling while your test count is rising, constraint decay has already arrived.

The Part AI Cannot Do: Deciding What to Test

Here is the argument that the research points to but does not state directly.

AI does not know your app's priority hierarchy. It does not know that the payment confirmation screen is ten times more important than the profile settings screen. It does not know that the offline mode failure on Android was your most-reported bug last quarter. It does not know that your highest-revenue user segment uses an older device running a specific OS version that has a known rendering quirk.

None of this is in the codebase. All of it should be in the test strategy.

When teams hand test generation entirely to AI without a priority layer, they get tests that reflect code structure rather than business risk. The checkout flow and the account deletion flow may receive the same test coverage weight, because from the code's perspective they are similar in complexity. From the business's perspective, they are not remotely comparable.

This decay compounds over time. As the app evolves, AI-generated tests without human priority context drift toward testing implementation details rather than user outcomes. The tests stay syntactically valid. They pass. They just stop catching the failures that matter.

The role of the human in AI-assisted mobile test generation is not to review every generated test line by line. It is to supply the priority layer that the AI cannot infer from source code:

Which flows are revenue-critical
Which user segments have platform-specific behaviour that must be covered
Which failures have caused production incidents and therefore warrant regression tests specifically written to prevent recurrence
Which upcoming feature changes are high-risk enough to warrant hand-written scenario tests rather than AI-generated coverage

With that priority layer in place, AI generation becomes significantly more useful. The model operates within a bounded, well-specified context rather than trying to infer what matters from code structure alone.

Tools Worth Knowing

Several tools have made genuine progress on bringing human context into AI test generation for mobile:

QApilot takes a codeless approach that keeps a human in the authoring loop by design. Test creation is still human-directed, with AI assisting rather than replacing.

Maestro has moved toward declarative, intent-driven test authoring that integrates with AI generation through its MCP interface. The declarative format makes it easier to specify what a test is trying to validate, not just what steps to execute.

Kobiton generates Appium scripts from recordings and supports cross-platform porting. More useful for teams with existing manual test flows they want to automate than for greenfield AI generation.

testRigor explicitly supports human-in-the-loop review on AI-generated tests and is designed around natural language test authoring that keeps the intent visible and reviewable.

None of these fully solve the constraint decay problem. They each address parts of it. The gap that remains across all of them is the priority and business context layer, which currently requires the human to supply explicitly rather than inferring from the codebase.

Assumption: The tool descriptions above are based on publicly available information as of mid-2026. Tool capabilities in this space change rapidly. Verify current feature sets before making purchasing decisions.

The Practical Starting Point

If you are using AI test generation for a mobile app today, three things are worth doing before you generate another test:

Map your high-risk flows before prompting. Write down the ten user journeys where a failure would hurt your users or your business the most. Feed that list into your generation prompts as explicit priority context. This is the difference between L0 and L3 generation.

Scope generation to change, not to coverage. AI test generation produces the most value when it is scoped to what just changed in the codebase, with the context of why that change was made. Full-suite generation from a complete codebase is the least constrained and least accurate application of the technology.

Treat the first run as a draft, not a suite. AI-generated tests are a starting point. Someone who understands the app needs to review whether the generated tests are covering what the business actually cares about, not just what the code happens to implement.

The goal is not to remove humans from mobile test generation. The goal is to remove humans from the parts of mobile test generation that do not require human judgment, and preserve them for the parts that do.

Deciding what to test requires human judgment. Writing the test for a decision already made is increasingly something AI can do well.

P.S. - I am Surendranath, Head of AI at QApilot.

Liquid Glass Is Here: Testing Considerations for Developers and Quality Engineers

Surya Polavarapu — Thu, 03 Sep 2026 11:12:57 +0000

Liquid Glass is Apple's new design language announced at WWDC 2025 & shipped with iOS 26. Apple calls it their "broadest software design update ever" - the most significant visual overhaul since iOS 7 moved from skeuomorphism to flat design in 2013.

All UI elements like tab bars, buttons, switches, sliders - are now translucent and dynamic. They mimic real glass: they refract light, reflect their surroundings, and change appearance based on what's underneath them. Scroll a page and the navigation bar shifts in real time based on the content beneath it. The same button can look completely different against a dark wallpaper vs a light one.

What happened when it reached real devices?

In beta builds, users reported a lot of functional rather than purely aesthetic problems: 

Notifications on the lock screen were barely visible against transparent background. The Control Centre blended into whatever was behind it, making toggles hard to find
Navigation became unpredictable: Safari's forward button appears and disappears depending on context, search bars moved, and back buttons sometimes vanished entirely.
Touch targets got smaller and more crowded - tab bars were squeezed, spacing reduced
Contrast was insufficient for anyone with even mild vision difficulties - many users immediately turned on "Reduce Transparency" and "Increase Contrast" in accessibility settings just to make their phones usable

The Nielsen Norman Group published a detailed analysis in March 2026 that was unusually pointed for a publication that typically stays measured: the interface is restless, less predictable, less legible, and constantly pulling focus rather than supporting seamless access to content. They specifically called out that Apple has crowded and shrunk touch targets, seemingly tossing out the long-standing guideline of at least 0.4cm between targets and 1cm × 1cm tap areas.

Apple eventually responded in iOS 26.1 beta 4 by introducing a "Tinted" setting that tones down the transparency - a soft reversal of one of its most disputed visual decisions. The community largely received this as an acknowledgement that the initial implementation went too far.

The Mandate

Apple understood that a change this large needed a runway. So when iOS 26 shipped, they built an opt-out into Xcode, but requires all iOS apps to fully support the Liquid Glass design system by September 2026. 
This isn't optional. The iOS 27 beta, released at WWDC June 2026, confirms that UIDesignRequiresCompatibility , the flag teams used to opt out of Liquid Glass - is ignored when building with the iOS 27 SDK. Teams that used this flag to defer the migration now have regressions hidden in their code that QA never considered. Those regressions will all surface simultaneously when they adopt the new SDK.

Critical Considerations

The most important action for any team still using the compatibility flag is to remove the flag, build against the iOS 26 SDK, run the app on a physical device, and do a full regression pass now - before Xcode 27 is made mandatory, before a bug fix release forces the issue. 
Visual testing breaks down. Tools comparing pixel to pixel will flag everything during regression and the actual failures would get buried in the noise. Liquid Glass elements render differently depending on what's behind them - wallpaper, content, dark vs light mode, even time of day in some cases. There is no stable baseline to compare against. The same screen, in the same device can produce a different visual output on every run. Pixel-diffing tools flag everything as a regression.

Touch target testing becomes necessary. The NN/g piece documents that Apple went against its own guidelines on tap target sizing. For apps adapting to Liquid Glass, elements that previously passed accessibility and tap-size checks may no longer. Every interactive element in a migrated app needs re-verification - not just against design specs but against real device behaviour.

The hidden regression problem. Teams that deferred Liquid Glass via the compatibility flag now have untested code paths. The single most important action for these teams: remove the compatibility flag, build with Xcode 26 against a physical iOS 26 device, and run a full regression pass - discover the regressions at your own pace rather than under Xcode 27 deadline pressure. Most haven't done this yet.

What testing against Liquid Glass actually requires

When Liquid Glass migration shifts the position of a tab bar item, reformats a navigation element, or resizes a button, test steps that relied on the previous element location or identifier fail. Not because the feature broke but because the anchor the test was using no longer sits in the same place. Many elements shifting simultaneously, across many screens, where the alternative is manually triaging and rewriting every broken test step by hand.

At QApilot, our AI native system works through a resolution hierarchy - accessibility identifiers, relative positioning, visible text, and a vision-based fallback that identifies the element the way a human tester would: by what it looks like and where it sits in the flow. The healed step is flagged for review before the fix persists in the system - the team sees what changed and approves it.

Our Auto Bug Finder surfaces accessibility gaps per screen during Crawler execution - contrast issues, missing labels, elements that fail WCAG checks - precisely the category of failure Liquid Glass migration is most likely to introduce.

Liquid Glass shipped with real performance issues on older devices - early builds showed sluggish rendering and elevated GPU load that Apple progressively resolved through subsequent updates. An app migrating to Liquid Glass today still carries that risk, particularly on devices outside the latest hardware generation. QApilot captures CPU, memory, battery, and rendering metrics during test execution - surfacing performance regressions as part of the same run that validates functional behaviour.

The September 2026 deadline and the Xcode 27 SDK mandate are arriving in the same window. For any team planning a release in the next twelve months, the question of whether their app has been tested against Liquid Glass is worth asking now.

Flaky Tests Persist Because Everyone Is Ignoring Them Rationally

Surendranath Reddy Jillella — Thu, 27 Aug 2026 12:40:13 +0000

You have done everything right.

You made the economic case for automation and got the investment approved. You distributed quality checks across the SDLC instead of piling them at the end. You replaced pyramid thinking with risk-weighted coverage. You stopped reporting a coverage percentage that was lying to you.

Six months later, your engineers have started ignoring test failures.

Not because they are careless. Because ignoring test failures became the rational choice. This article is about how that happens, why it happens to teams that know better, and why it is the final form of Test Debt.

What is flakiness?

A flaky test is a test that fails intermittently without any change to the code it covers. It sometimes passes and sometimes fails, with no consistent pattern. The most common root causes are timing issues in async operations, test-order dependencies, shared mutable state, and coupling to external services.

All of these are fixable. The fixable nature of the problem is not what makes it interesting. What makes it interesting is that teams fix very little of it, and teams with strong engineers who care about quality fix very little of it.

The reason is not the technical difficulty.

The scale

The numbers are worth stating clearly, because they establish what is actually at stake here:

At Google, approximately 16% of tests show some form of flakiness, and 84% of transitions from passing to failing involve a flaky test rather than a genuine regression.

At Microsoft, roughly 25% of test failures in large-scale CI systems are caused by flakiness, not actual code defects. The average time a developer spends per flaky test investigation: 30 minutes, before determining it was not a real failure.

Atlassian estimated 150,000 developer hours per year consumed by flaky test investigation before they built automated detection tooling.

Slack's mobile test failure rate reached 56.76% before they intervened. More than half of all test failures were noise.

These are not teams with poor engineering culture. They are among the best-resourced and most technically sophisticated organisations in the industry.

The rational inaction problem

Here is the core insight this article is built around: flaky tests are the only common engineering quality problem where every person in the chain responds correctly given their local information, and the outcome is still catastrophic.

The developer who sees a test failure and hits retry is not being careless. In their experience, 85% of unexplained failures are environment issues, not regressions. Investigating would take 30 minutes and probably turn up a race condition in test infrastructure that is not their code. Retry is the rational choice.

The QA engineer who logs the failure as "likely flaky, monitoring for pattern" is not dropping the ball. Without detection tooling, they cannot quickly distinguish a flaky failure from a real regression. Flagging and watching is the rational choice.

The infrastructure team that says "it is a test quality issue, not infrastructure" is not deflecting. Flaky tests usually are a test quality issue. Declining scope is the rational choice.

The engineering manager who does not put flakiness remediation on the sprint roadmap is not mismanaging. There are always user-facing features with higher business visibility. Flakiness does not appear in the product backlog with an owner or a deadline. Deferring is the rational choice.

Nobody made a bad decision, but nobody fixed anything as well.

This is the organisational failure mode this series has been describing in different forms since Article 1: costs that are invisible and diffuse; don't get addressed, because the people experiencing the cost in small increments are never the people with authority to fix it at the root.

The rational inaction framing applies most clearly to mid-to-large engineering organisations with specialisation between roles. In small teams of fewer than 10 engineers, the same person often occupies multiple positions in the chain described above, which creates different incentive dynamics. The argument is strongest for teams of 20 or more where there is clear separation between developer, QA, infrastructure, and management roles.

Why retry is the most expensive button in your CI interface

When a team normalises retrying flaky tests, they make a subtle but load-bearing decision: test failures no longer reliably indicate real problems.

That decision has a half-life. Once a team accepts that some failures are noise, the threshold for investigating failures drifts upward. A suite with 5% flakiness earns mild skepticism. A suite with 20% flakiness is treated as broken infrastructure. A suite with 50% flakiness is effectively ignored.

Slack documented their mobile test failure rate reaching 56.76% before they intervened. At that point, the pipeline provides negative value. It adds latency to every deployment while producing no useful signal about regressions. You have paid to build, maintain, and run a suite that has become pure noise.

The connection to Article 1 in this series is direct: Article 1 described the cost of having no test automation. A flaky, untrusted test suite is a different path to the same destination. You are making deployment decisions without a reliable test signal. The investment from Articles 1 through 4 has been eroded to zero. You are still paying 100% of the maintenance cost and receiving close to 0% of the coverage value.

The accountability structure that actually works

The research on teams that have genuinely reduced flakiness shows a consistent pattern. The technical fixes vary by codebase and language. The organisational fix does not vary.

At Microsoft, flaky tests are automatically assigned to the developer who owns the affected code. Not a team. A named individual. Developers with more than 10 open flaky test bugs have their PR merges blocked until they close them.

At Atlassian, the approach combined three elements: automated detection (so no manual effort required to identify flaky tests), immediate quarantine (the test is removed from the blocking suite while investigation is open, keeping the pipeline trustworthy), and a named owner assigned within 24 hours of identification.

Neither of these is primarily a technical solution. Both are accountability structures. They answer the question "whose problem is this?" with a specific name and a specific timeline, rather than leaving it as a shared responsibility that falls on no one.

The practical framework, regardless of team size:

Make the number visible first. Pull your CI data for the last 30 days. Find every test that failed and then passed on retry. Express that as a percentage of total runs. Most teams have never seen this number. It is the most important signal in their pipeline, and it is not surfaced by default in any major CI tool.

Quarantine, do not delete. A flaky test usually covers real behaviour. Removing it from the blocking suite preserves pipeline signal while investigation is open. Deleting it removes coverage. These are different decisions with different consequences.

Assign by name, not by team. "The QA team owns flakiness remediation" produces the same outcome as no ownership. "Jason owns these four tests, resolution expected by end of sprint" produces a different outcome.

Set a flakiness budget. Define a maximum acceptable rate; 1% is a reasonable starting threshold, and treat it the same way you treat error rate or latency: a metric with a threshold, a dashboard, and a named owner who is accountable when it is breached.

Assumption: The 1% flakiness budget threshold is based on commonly cited industry practice rather than a formally derived number. Google and Microsoft operate internal flakiness thresholds but do not publicly document the specific percentages. Teams with very large test suites (10,000 or more tests) may need to calibrate differently based on acceptable investigation volume.

Why AI detection changes the equation but not the root cause

Automated flaky test detection, including AI-based pattern recognition across CI runs, removes the manual work that was the excuse for not addressing flakiness earlier.

Without detection tooling, asking a team to track flakiness is asking them to do manual bookkeeping across every CI run. Under any reasonable delivery pressure, this does not happen.

With detection tooling, the identification is automatic. The question of whether to address it becomes purely organisational: who owns it and what happens if they do not.

This is why detection tools are necessary but not sufficient. Teams that deploy automated flaky test detection without the accountability structure around it often end up with a dashboard showing 12% flakiness that nobody acts on. The bottleneck was never the detection. It was the ownership.

Closing the series

This series started with a single argument: the cost of not investing in testing is real, measurable, and invisible until it is catastrophic.

Each article has described a different mechanism through which testing investment either fails to materialise or silently loses value:

Article 1 - The investment does not happen because nobody has made the cost of not having it visible (Article 1).
Article 2 - The investment happens too late in the cycle because the tooling for real-time quality feedback was not in place when the policy changed.
Article 3 - The investment covers the wrong things because the pyramid model was a cost heuristic that got mistaken for a structural law.
Article 4 - The investment measures the wrong things because a coverage gate creates the incentive to pass the gate rather than catch the bugs.
And then in this article, we spoke about how the investment erodes silently because flakiness is everyone's problem in a diffused sense and nobody's problem in an accountable sense.

The through-line is not tooling. The through-line is that each of these problems persists because the cost is invisible and the accountability is diffused. The moment you make the cost visible and assign the accountability to a specific person, the problem becomes tractable. Not easy, not fast, but tractable.

Article 3 ended with the observation that deciding what is risky is still a human judgment. Every article in this series has made a version of that same point. The investment decision is human. The timing decision is human. The prioritisation decision is human. The measurement decision is human. The accountability decision is human.

The tools have never been the bottleneck.

What we are trying to do at QApilot is collapse the gap between 'the cost exists' and 'someone sees it and can act on it.' That is true for coverage gaps, for flakiness, for untested risk paths. The tooling is there. The visibility is what has been missing.

Why Hitting Your Coverage Target Is Making Your Tests Worse

Surendranath Reddy Jillella — Fri, 21 Aug 2026 06:04:26 +0000

I had 87% coverage, and we still broke the billing flow on launch day.

Not because of a gap in the percentage. Because 87% was covering the wrong things. The tests were written to pass a gate, not to catch a failure.

That is a more common story than most teams admit. And the reason it keeps happening is not that engineers are careless. It is that the incentive structure you created made it the rational outcome.

The series checkpoint

The first three articles in this series built the investment case for testing and then dismantled the received wisdom about how to execute it. We've made the economic argument for automation. We've restructured when quality checks happen across the SDLC. We've replaced the pyramid model with something shaped by risk rather than by code hierarchy.

Now, when someone asks: how do you know if it is working?

The answer most teams give is their coverage percentage. This article is about why that answer is structurally broken, and why fixing it is a management decision before it is a tooling decision.

What coverage percentage actually measures

Coverage percentage tracks which lines of your code were executed during a test run. If a line ran, it counts as covered.

That is the complete definition.

It does not measure whether the test asserted anything meaningful about that line. It does not measure whether both branches of a conditional were exercised. It does not measure whether the specific inputs that cause failures were ever tried. A test that calls a payment function and checks assert response is not None covers the same lines as a test that validates the transaction ID, amount, currency, error code, and retry behaviour. The coverage tool treats them identically.

The research on this is unambiguous. A 2017 study by Kochhar et al. examined the correlation between code coverage and actual bug rates across 100 large open-source Java projects. The finding: the coverage of existing test suites has an insignificant correlation with the number of bugs found after release. Inozemtseva and Holmes found separately that line coverage is the metric that correlates least with real defect detection compared to other available options.

You are using the most popular quality metric in software engineering. It is also the least predictive of actual quality.

Goodhart's Law in your CI pipeline

Goodhart's Law comes from economics: when a measure becomes a target, it ceases to be a good measure. The principle describes what happens when you attach consequences to a proxy metric instead of to the underlying goal it was meant to represent.

Coverage percentage is a proxy metric. The underlying goal is: tests that catch bugs before production. When you make the proxy metric a gate, teams optimise for the proxy, not the goal. The badge turns green. The risk does not go down.

This is not a character flaw. It is a predictable response to an incentive you designed. Under delivery pressure, engineers do the rational thing: they hit the number using the fastest available path.

How the gaming plays out

The fastest path to coverage percentage is testing happy paths. User submits form successfully. API returns expected response. Data saves to the database. These scenarios are easy to construct, execute quickly, and each one covers a high line count because they touch the main execution path.

What does not get covered:

Error handling paths run on exception conditions that are tedious to construct in tests. Retry logic, fallback behaviour, and timeout handling live in branches that require specific setup. These are also, as a direct consequence, the places where the most consequential bugs hide.
Boundary conditions require generating test inputs at the edges of valid ranges, null states, empty strings, Unicode characters, and maximum lengths. Each of these covers almost the same lines as a single typical input but requires separate test construction. Under time pressure, one test covers the line.
Conditional branches show as "covered" once either branch executes. A function with a condition that checks user role, applied to admin and guest differently, can show as fully covered if only one role ever runs the test. The other branch remains untested with no coverage signal.

The outcome is a suite at 85% coverage where the well-tested parts are tested three times over, and the failure-prone parts have one shallow test each. Teams with high line coverage but low integration validation have been observed producing more than double the production incidents of teams with lower but more balanced coverage.

Assumption: The claim that coverage-gamed suites produce higher production incident rates than lower but risk-balanced suites is based on observed patterns in industry case studies, not a controlled experiment. The "double production incidents" figure comes from analysis of specific teams and may not generalize uniformly. It should be read as directional evidence of the problem rather than a precise multiplier.

What this costs you

In the first article in this series, the framing was: the cost of not investing in testing is invisible until it is catastrophic.

Coverage gaming creates a specific variant of that problem. You are paying the full maintenance cost of a test suite that is not delivering the risk reduction you paid for. Every test written to hit a percentage rather than catch a bug still needs to be maintained when the code changes, still runs in every CI cycle, and still occupies space in the mental model of everyone who reads the suite.

You have made the right investment decision. A portion of that investment is funding theatre rather than protection. The cost shows up in the budget. The corresponding value does not show up in production stability.

A better measurement

The honest version of coverage is mutation score. Mutation testing introduces deliberate small bugs into your code: flipping a comparison operator, removing a conditional check, changing a return value. Your test suite runs against each mutated version. If the tests catch the mutation (they fail), the mutation is killed. If the tests pass despite the code being wrong, the mutation survives.

A mutation score of 80% means your tests caught 80% of introduced bugs. That is a direct claim about defect detection, not about line execution. These are different claims.

A real case documented on dev.to reported 93% line coverage that turned out to be a 34% mutation score. The team had been reporting a green badge on a suite where two-thirds of the introduced bugs would have survived.

Mutation testing is computationally expensive, which is a real constraint. The practical approach: do not run it across the entire codebase. Run it on the paths where failure is most costly, specifically the risk-weighted paths identified in the work from Article 3. Use line coverage as a floor (it tells you about completely untested code) and mutation score as the signal for whether the tests you have are actually doing anything on the critical paths.

Branch coverage is a useful intermediate step. It is stricter than statement coverage, it surfaces the conditional gaps that line coverage misses, and most existing coverage tools already support it. Switching from statement to branch coverage as your CI gate raises the bar without requiring new tooling.

Assumption: Mutation testing adoption is growing but not mainstream as of mid-2026. Frameworks exist for most major languages (Stryker for JavaScript and TypeScript, PITest for Java, mutmut for Python) but are not a default feature in major CI platforms. The claim that most teams do not use mutation testing is based on tooling adoption patterns, not a formal survey.

The behavioural fix

The tool change matters less than the conversation change.

If coverage percentage is a pass/fail gate in your pipeline, you have created the incentive to game it. The alternative is not removing coverage tracking. It is removing the gate behaviour.

Report coverage as a trend, not a threshold. A team whose coverage has been at 78% for six months is in a fundamentally different position than one whose coverage dropped from 88% to 68% after a feature push. The trend carries information. The snapshot at a point in time does not.

Pair that with the question you actually want answered: are the highest-risk paths in this codebase meaningfully tested? That question does not have a single percentage answer. It requires looking at branch coverage on critical paths and, where the stakes are highest, mutation score. Neither of these is hard to implement. They are just not the defaults.

PS: I am Surendranath, leading the AI team at QApilot, and we are solving problems specific to mobile app testing through our AI-native product. Reach out if you are interested to try it out!

Google upgraded Android's UI language, can test automation keep up?

Surya Polavarapu — Thu, 13 Aug 2026 10:48:42 +0000

On May 19, Google made it official: Jetpack Compose is now the only way forward for Android UI. The older approach of writing screens in XML, the markup language that defined app development for over a decade, is now placed in maintenance mode. No new feature launches, bug fixes only. 

For most developers, this landed as good news. Compose is genuinely better to build with: less code, faster iteration, cleaner architecture. So most of the attention since has been on what developers gained and barely anything on what happens to the test suites!

What actually changed under the hood

To understand why this matters for quality teams, it helps to know what automation tools have always relied on.

Android testing frameworks typically found elements on screen using identifiers: resource IDs, class names, positions in a structure called the View hierarchy. Every button, text field, and screen element in an XML-based app lives in that hierarchy. Your test says "find the element with ID btn_checkout" and the framework finds it.

Compose doesn't produce Views. It produces a semantics tree - a structure closer to how a screen reader navigates an app than to how traditional layouts worked. These are two different models.

Appium can interact with Compose, but not in the way most teams currently have it set up. It requires switching to the UIAutomator2 driver specifically - not the Espresso driver - and it only works if developers have explicitly added testTag modifiers to Compose elements and enabled testTagsAsResourceId in the app configuration. Those are deliberate, manual steps that someone on the engineering side has to take for every element that needs to be reachable by a test.

Most teams migrating existing apps to Compose haven't done this. The result is a test written to find btn_checkout by its resource ID will silently fail on a Compose screen - not because the button disappeared, but because the element it was looking for no longer exists in that form. Tests stop finding elements and the suite starts failing. So when teams are quickly rewriting UI code, they are quietly invalidating years of recorded test steps, locators, and automation logic

The same problem we solved for Flutter is now arriving for the Android native ecosystem

This isn't a new class of problem. Flutter introduced a version of it years ago, and it was worse. Flutter renders entirely on its own canvas, bypassing the native UI framework completely. UIAutomator2 sees nothing. XCUITest sees nothing. There is no View tree, and no semantics tree accessible through standard drivers. The only reliable paths are Flutter Driver - which only works on debug builds, not release ones.

At QApilot, we built our proprietary layer using advanced methods like OCR & computer vision precisely because standard locators find nothing. No custom framework setup or coordinate scripts - the test finds the element visually, the way a human would.

Compose is a step along the same spectrum. The accessibility layer is still reachable if the app is configured correctly. The problem is that "configured correctly" now requires deliberate choices from the development team that most aren't making yet.

The abstraction layer we've built normalises Native, Flutter, React Native, and WebView apps into a single locator and action interface. Compose First is another reason why that kind of abstraction matters. The specific UI framework your team chose to build in shouldn't be a variable that determines whether your test suite survives a platform update. 

It's worth watching how quickly Android teams migrate. Compose adoption has been steady but slower than Google would prefer - partly because legacy codebases are large, and partly because migration comes with invisible costs that don't show up in a developer experience comparison.

The End of the Testing Pyramid: What Replaces It in the AI Era

Surendranath Reddy Jillella — Thu, 06 Aug 2026 08:01:20 +0000

We have discussed the cost of no automation and reasoned "when to test".
This doesn't automatically solve the test coverage question - the QA teams are swamped enough, and if you ask them to test across the SDLC, that can only lead to testing what is easy or whatever is quick.

Hence the next question - What tests to prioritise?

The testing pyramid was and is gospel, still, and every sprint planning meeting with a QA team included some version of: "we need more unit tests at the base, fewer E2E tests at the top."

The pyramid made complete sense - for its time. I'm not sure it makes sense anymore.

What the Pyramid Was Actually Solving

Mike Cohn introduced the testing pyramid in Succeeding with Agile (2009), though the concept was sketched out earlier around 2003–04. His original framing had three layers: unit tests at the base, service/integration tests in the middle, UI tests at the top. Martin Fowler popularised it further on his blog, and it became a de facto standard across the industry.

The underlying logic was about cost:

Unit tests: cheap to write, fast to run, easy to pinpoint failures
Integration tests: moderate cost, slower, harder to isolate failures
E2E/UI tests: expensive to write, slow to run, brittle - and when they fail, you often spend an hour figuring out what actually broke

So the pyramid was really a cost-optimisation heuristic: do more of the cheap thing, less of the expensive thing - sensible advice when humans are writing every test by hand.

The AI Problem with This Logic

Here's what changes when AI can generate test cases: the marginal cost of writing the 100th test approaches zero.

Generating 10 unit tests costs me roughly the same as generating 100. The input cost - my time writing a prompt, reviewing output - is largely fixed, not variable.

Assumption: This assumes LLM output quality is "good enough to review and merge" without significant rework. Current research (MDPI, 2025) shows LLMs are effective for test generation on well-scoped, lower-complexity functions, but accuracy drops for complex, stateful logic. So this cost collapse is real for breadth of coverage, less so for depth in complex systems.

If writing isn't the bottleneck anymore, the pyramid's rationale starts to crack. But there's a catch.

The Cost That Didn't Collapse: Maintenance

Writing tests and maintaining tests are two different problems. AI hasn't fully solved the second one yet.

Industry data suggests test maintenance consumes anywhere from 30% to 50% of a QA team's automation effort.

That's not a pyramid problem; it's a maintenance problem. And it compounds fast when you generate tests faster than you can maintain them.

If AI test generation significantly outpaces AI test maintenance and healing capabilities, teams could end up with larger, flakier suites than they started with. This is a genuine risk in 2026 - generation tools have matured faster than maintenance/healing tools. The balance matters.

So the new model can't just be "generate everything." It has to be smarter about what gets generated.

What Should Replace the Pyramid

I've been thinking about this as risk-weighted coverage rather than a structural pyramid. The question isn't "how many tests at each layer?" - it's "where does a failure hurt most, and am I covered there?"

A few principles that seem to hold up:

1. Coverage by business risk, not code structure

A payment-processing function with 60% unit coverage is riskier than a string formatter with 95% coverage. The pyramid doesn't capture this. A risk-weighted model does. The first question before generating any test should be: what breaks the business if this fails?

2. The E2E tax is shrinking, not gone

E2E tests are still slow. They're still brittle when they fail. But AI-assisted generation makes writing them dramatically cheaper. The remaining cost is infrastructure and maintenance.

Assumption: As AI-based self-healing for selectors and UI state matures, the E2E maintenance cost will fall meaningfully. This is directionally happening but isn't yet the default reality for most teams in 2026.

3. A "testing mesh" over a rigid pyramid

Rather than a strict hierarchy, think of tests as a mesh - overlapping coverage at multiple levels, weighted toward high-risk areas. Some paths get unit + integration + E2E coverage. Low-risk utility code gets unit only. The shape is determined by risk analysis, not by a fixed ratio rule.

What This Means Practically

If you're maintaining a mature test suite:

Audit which E2E tests cover high-risk paths vs. low-risk happy paths. The latter are usually candidates for deletion or downgrade.
Use AI to generate edge-case unit tests for your critical business logic - that's where current LLM-based generation adds the most value with the least maintenance overhead.
Don't generate tests uniformly. Generate them where failure actually matters.

If you're greenfielding:

Start with risk mapping before writing a single test. What breaks the business if it fails?
Let AI fill coverage breadth; invest human effort in the tests for high-risk paths that require business context to assert correctly.

The Real Question

The pyramid isn't dead - it's become one valid shape among many. The right test distribution depends on your system's risk profile, not a universal rule.

The harder question the AI era forces is: who decides what's risky? And that's still a deeply human judgment. AI can generate tests for you, but it can't tell you what matters to your business unless you've told it.

P.S. - I am Surendranath from QApilot and this post is part the series Test Debt

Shift-Left is Dead. Long Live Shift-Everywhere

Surendranath Reddy Jillella — Thu, 30 Jul 2026 07:53:13 +0000

Last week, I presented the cost of no testing - Hidden cost of no test automation
It is now clear that testing is essential, if not earlier. The next question that comes to mind is When to test? - Shift Left is an answer that many leadership mandates suggest.
The practical interpretation of Shift-Left is simple - QA was going to write fewer tests, and developers were going to write more. The phrase was presented as a philosophy. It felt more like a headcount justification.

That's not what Larry Smith meant when he coined the term.

What Shift-Left Actually Said

Smith's article "Shift-Left Testing" was published in Dr. Dobb's Journal in September 2001. The observation was straightforward: on any project timeline, testing was always concentrated at the right side - near delivery. He proposed moving it left - closer to inception - to catch defects earlier and cheaper.

The economics behind it were sound. IBM Systems Sciences Institute data (originally from 1981 internal training materials) showed bugs found after release cost 60–100x more to fix than bugs found during design. Even if those exact multipliers are contested - The Register ran a worthwhile piece in 2021 questioning whether the original IBM study was ever a peer-reviewed paper - the directional finding has been consistently confirmed by NIST research, Capers Jones' analysis of 12,000+ projects, and anyone who has sat in a production incident war room at 2 am.

The idea was right. The execution became something else.

What Shift-Left Became

In practice, shift-left was operationalised as:

Developers writing unit tests they didn't always have the domain context to write well
CI gates nobody trusted because flaky tests made failure signals into noise
QA "embedded" in scrum teams without changing when or how testing actually happened
Static analysis tools added to pipelines that nobody read the output of

The mandate to shift left didn't come with a corresponding shift in tools, time allocation, or context. It was process vocabulary layered on top of unchanged behavior.

Assumption: This characterization reflects the most common pattern of shift-left adoption - process-driven, without sufficient tooling change. It won't apply to every team. Organizations that genuinely restructured roles, tooling, and feedback loops saw real results. But the pattern of "shift-left = devs write more tests with no additional support" is widespread enough to be a recurring frustration in engineering retrospectives. If your team executed shift-left well, you're in the minority.

The original promise - catch bugs earlier - was only partially delivered, for most teams.

What's Actually Different Now

The 2024 Developer Survey found developers spend only 24% of their time writing code. The rest goes to debugging (roughly 41%), meetings, reviews, documentation, and operational work.

That ratio is persistent and damning. A huge chunk of debugging time is spent on issues that better earlier feedback - not just earlier testing, but earlier signals - would have caught before they compounded.

What AI tooling is changing, when it's actually embedded rather than bolted on, is making quality checks ambient rather than phased:

In the IDE: Copilot-style tools flag issues as you type. Some generate test stubs inline for the function you just wrote.
In the PR: AI reviewers analyze diffs for missing edge cases, untested branches, and logic gaps - not just style and formatting.
In CI: AI-based test generation triggered on diff, targeting new and changed code specifically rather than running the full regression suite.
In production: Error monitoring that surfaces coverage gaps by tracing prod failures back to untested paths.

This isn't shift-left. It's shift-everywhere. Quality isn't a phase you move earlier - it's a layer that runs in parallel with development at every stage.

The Distinction That Matters

Shift-left was about when in the SDLC to apply testing.

Shift-everywhere is about continuous presence of quality signals regardless of SDLC phase.

A developer gets immediate feedback in their IDE. The same PR gets an AI review pass. The CI pipeline runs targeted tests on exactly the changed code. An alert the next day surfaces a production error and maps it back to a coverage gap.

The SDLC doesn't stop being sequential - code still gets designed, written, reviewed, deployed. But quality signals are no longer tied to a single phase of that sequence.

Assumption: "Shift-everywhere" as described assumes a mature AI tooling stack. Most teams in 2026 have 1–2 of these layers, not all four. The full ambient quality model is where the industry is heading, but it's not the default state yet. I'd estimate fewer than 25% of teams have more than two of these layers genuinely integrated and trusted, based on the tooling adoption curves I've seen.

What Hasn't Changed

AI doesn't fix everything. A few things remain stubbornly manual:

Understanding what to assert. Generating a test is easy. Knowing what the correct behavior should be for a non-obvious edge case - one that depends on business rules, not just code logic - still requires a human who understands the domain.

Flaky tests. AI can reduce new flakiness by generating more deterministic tests, but it doesn't automatically rehabilitate an existing flaky suite. That's still cleanup work.

Cross-service test strategy. What to test across service boundaries, under real load, in failure scenarios - that's architectural thinking that LLMs don't consistently reason well about yet.

The Practical Takeaway

If you're a developer
Shift-left was never the enemy. The enemy was treating it as a process change without a tooling change. The AI-era version of the same idea is more actionable because the tooling is actually there.

If you're running an engineering or QA team
Stop thinking about where in the pipeline quality lives. Start thinking about how many stages have real-time quality feedback embedded. Count the stages where a developer gets an automated signal about quality within minutes of writing code. If the answer is one - probably CI - you have room to move.

P.S.: This article is part of the Test Debt series.
About me - I lead the AI team at QApilot - an AI-native mobile test automation company, and we build products to help you achieve the quality standards that your users demand.

I Used to Hate App Updates. Then I Saw What Happens Behind the Screen

Harini Mukesh — Mon, 27 Jul 2026 10:30:00 +0000

From "Ugh, Another Update?" to "Wait... There's So Much Behind This"

A few months ago, if my phone showed "Update Available" while I was in the middle of using an app, my first reaction was always the same.

"Seriously? Right now?"

Sometimes I'd postpone it. Other times I'd update it just because the notification wouldn't stop bothering me. Either way, I never thought much about it. As long as the app opened, did what I wanted, and didn't crash, I was happy.

Like most people, I only noticed an app when something went wrong.

If it was a free app and it kept crashing, I'd uninstall it and look for another alternative. There are plenty of apps that solve the same problem anyway.

But paid apps were different.

The moment I pay for a subscription, my expectations go up. During the free trial, I suddenly become a tester without realizing it. I click every button, explore every feature, and try to decide if it's worth my money. If something feels broken, I'm probably not renewing.

Looking back, I realized I only cared about the final experience. I never wondered how many people worked on the app, how many times it was tested, or how much effort went into making everything feel smooth.

That changed when I got the chance to explore a mobile testing platform.

Until then, I genuinely believed testing meant opening the app, clicking a few buttons, making sure nothing crashed, and calling it a day.

I couldn't have been more wrong.

The deeper I explored, the more I realized that every screen, every button, every animation, every permission popup, and every update notification has an incredible amount of planning and testing behind it.

It felt like discovering an invisible world that had always existed behind every app I use but one I had never noticed.

I Thought Testing Was Just Clicking Buttons... I Couldn't Have Been More Wrong

If someone had asked me what software testing meant before this experience, I would've probably said,

"Open the app, click a few buttons, make sure everything works, and you're done."

Turns out, that's probably the easiest part.

What surprised me most was that testing isn't about checking whether an app works when everything goes according to plan. It's about making sure it still works when things don't.

What happens if someone denies a permission request?

What if the internet disconnects during a payment?

What if the user rotates the phone halfway through filling a form?

What if a notification interrupts the app?

What if everything works perfectly on one device but breaks on another?

These aren't rare situations, they're everyday user behaviour. And someone has to think about every one of them before the app reaches us.

That's when I came across terms like functional testing, where every feature is verified to work as expected, and exploratory testing, where testers intentionally explore an app the way real users would, looking for unexpected issues instead of following a fixed script.

I also learned about cross-platform testing making sure the same app behaves consistently across Android, iOS, different screen sizes, and different OS versions. Something as simple as a button can behave differently depending on the device.

Then there are edge cases those unusual situations that don't happen often but can completely break the user experience if they're ignored.

The more I learned, the more I understood one thing.

Good testing is invisible.

When everything works perfectly, nobody thinks about the hundreds of scenarios that were tested beforehand. We simply assume the app is supposed to work that way.

One line stayed with me throughout this journey:

Testing isn't about proving that an app works. It's about trying to discover where it doesn't.

Because it's far better for a tester to find those problems than for thousands of users to discover them after launch.

One Bug Doesn't Mean One Person Fixes It

Another thing I completely misunderstood was what happens after a bug is found.

I always assumed the issue simply went back to "the developer."

A button isn't working?

The developer fixes it.

The app crashes?

The developer fixes it.

Simple.

But that's not how software teams work. Imagine a tester finds multiple issues on a single login screen. The button colour doesn't match the design. The spacing is inconsistent. The login API returns the wrong error message. The keyboard hides the login button on a particular Android device. To me, that looked like one screen with four bugs.

In reality, those issues belong to different teams.

The UI or UX team handles visual inconsistencies like colours, spacing, typography, and layouts. Mobile developers focus on how the screen behaves. Backend engineers investigate API responses and business logic. Sometimes Android and iOS teams even work separately because the same feature can behave differently across platforms.

That completely changed how I looked at testing.

I realized it isn't just about finding bugs, it's about communicating them clearly so the right people can fix the right problems.

One thing I found particularly interesting while exploring QApilot was how findings can be organized instead of being dumped into one long report. Visual issues can go to the design team, functional issues to developers, and backend-related findings to the engineers responsible for those services.

That might sound like a small detail, but when multiple teams are working on the same product, structured reporting saves a lot of time and confusion.

Before this experience, I thought testing ended once someone discovered a bug.

Now I think that's where collaboration really begins.

Behind every app we use are designers, developers, testers, product managers, and engineers, all working together to make something feel effortless for the rest of us.

And honestly, that's something I had never appreciated until I looked behind the screen.

Exploring QApilot Made Me Think Differently About Testing

By this point, I had stopped looking at testing as just another step before releasing an app. Instead, I started seeing it as something that builds confidence not just for the team creating the app, but for the people using it.

That's when I spent more time exploring QApilot itself.

What I liked immediately was that I didn't need to be an experienced QA engineer to understand the platform. The interface felt simple enough to explore without constantly referring to documentation, which made learning much easier.

The first feature that caught my attention was the Crawler.

Initially, I thought it would simply click random buttons. But I soon realized its purpose was much more practical. It explores an app the way a curious user might, moving through different screens and uncovering user journeys you may not think of manually.

Then came Record & Play.

It made me think about how repetitive testing can become. Every new release often means repeating the same login flow, navigation, or validation steps. Instead of doing those tasks from scratch every single time, Record & Play allows testers to automate the repetitive parts and spend more time exploring new features and unexpected scenarios.

The feature that stood out to me the most, though, was CoWork.

Whenever people talk about AI, the conversation usually becomes, "Will it replace people?"

CoWork gave me a different perspective.

It felt less like AI replacing testers and more like AI working alongside them. It helps generate and organize test cases, while the tester still reviews, guides, and makes the final decisions. That balance made much more sense to me than expecting AI to handle everything on its own.

Exploring these features made me realize that modern testing isn't just about finding bugs anymore. It's about making the entire testing process smarter, faster, and easier without taking people out of the equation.

If It Can Help a Vibe Coder, Imagine What It Can Do for QA Teams

One thought kept coming back to me while I was learning all this.

Building software has become easier than ever.

Today, someone with an idea can use AI coding tools or vibe coding platforms to build a working app in days instead of months. That's incredible because it allows more people to bring their ideas to life.

But building an app is only half the journey.

Making sure it's reliable is something entirely different.

Users don't care whether an app was written by an experienced engineer, a solo founder, or an AI coding assistant. They only care about one thing, it should work.

If the app crashes during a payment, freezes during onboarding, or breaks on a particular device, most users won't wait for an explanation. They'll leave a poor review or move on to another app.

That's where I started seeing the bigger value of testing.

If I were building my first app today, I'd want to know whether the important user journeys worked before anyone downloaded it. I'd want someone or something to explore the app the way a real user would and point out issues I might have missed.

That's exactly why tools like this make so much sense for startup founders and indie developers. They provide confidence before launch.

And if they can simplify testing for someone with little experience, I can only imagine how much more useful they become for professional QA teams managing hundreds of test cases, multiple releases, and different devices every day.

For me, that was the biggest takeaway.

AI isn't replacing testing.

It's helping people spend less time repeating the same work and more time solving the problems that still need human thinking.

The Next Time My Phone Asks Me to Update...

It's funny how quickly a perspective can change.

A few months ago, an app update was just another interruption. I'd tap "Update Later" and continue with whatever I was doing.

Now, whenever I see that notification, I think about everything that probably happened before it reached my phone.

Someone may have discovered a bug that only appeared on a particular device.

Someone may have found an issue while testing a user flow that nobody expected.

A designer, developer, tester, and product team may have worked together to fix it before millions of users ever noticed. Most of that work is completely invisible. And maybe that's the whole point. When an app works smoothly, we don't stop to appreciate the effort behind it. We simply expect it to work. This experience didn't turn me into a QA engineer. It did, however, turn me into a much more curious user.

Now, whenever I open an app, I find myself wondering about the work happening behind the screen. How many scenarios were tested before this feature reached me? How many conversations happened before this button behaved exactly the way it should? How many problems were solved before I even had the chance to experience them? Those are questions I never would've asked a few months ago.

The next time my phone asks me to install an update, I'll probably still wish it had chosen a better time.

But instead of thinking,

"Why another update?"

I'll probably think,

"Someone found a problem before I did and they're making sure I never have to experience it."

As users, we only see the finished product.

Behind every smooth experience is an enormous amount of invisible work.

And after getting a glimpse of that world, I don't think I'll ever look at app updates or mobile apps the same way again.

The Hidden Cost of No Test Automation: A Back-of-Napkin Calculation

Surendranath Reddy Jillella — Thu, 23 Jul 2026 12:48:07 +0000

Nobody cancels a sprint to do a testing ROI analysis. The cost of skipping test automation doesn't have a line item in your budget. It doesn't show up on a dashboard. It accumulates in the background - slower releases, developer burnout, customer-reported bugs - until something catastrophic makes it visible.

I want to make it visible before that happens. Here's a calculation you can actually run for your team.

The Math That Got Buried

The most widely cited figure in software testing economics: bugs found in production cost 60–100x more to fix than bugs found during design.

It traces back to IBM Systems Sciences Institute research, originally from 1981 internal training materials. The Register ran a 2021 piece questioning whether this ever existed as a peer-reviewed paper - a fair challenge. But the directional finding - that bugs get exponentially more expensive as they age through the SDLC - has been confirmed independently by:

NIST's 2002 study on software quality infrastructure, which pegged software defects at $59.5 billion in annual cost to the US economy

Capers Jones' research across 12,000+ projects, which consistently found similar multipliers

BetterQA's analysis of SDLC-stage fix costs, which found production bugs run approximately 30x the cost of catching the same bug in development

The exact multiplier is debatable. The direction is not.

For this calculation, I'm using the conservative end.

Bug Cost by SDLC Stage

Stage	Relative Cost	Example: $200 fix during development
Design / requirements	0.5x	$100
Development (caught by dev)	1x	$200
QA / testing phase	5x	$1,000
Staging / UAT	10x	$2,000
Production	30x	$6,000

Assumption: These multipliers use the conservative end of the cited range (IBM goes to 100x; I'm using 30x for production). Actual cost per bug varies enormously by system criticality, customer impact, and bug type. A cosmetic UI bug in production costs far less than a data corruption bug. These numbers are directional tools for making the case, not inputs for precise budgeting. Adjust the production multiplier up for customer-facing, data-sensitive, or regulated systems.

The $6,000 production cost is not just engineer debug hours. It includes: incident coordination, customer communication, potential data cleanup, postmortem time, and possible SLA penalties. Reputational cost is not included, because it's hard to quantify and easy to dismiss.

The Manual Regression Tax

If you have no test automation, you have manual regression. Here's what that actually costs.

Assumptions for this calculation:

Mid-size team: 15 engineers
2-week sprint cycle (26 releases per year)
1 QA engineer (or developer in QA rotation) running regression each cycle
Loaded engineer cost: $120/hour

Assumption: $120/hr is a US-market estimate for a mid-level engineer including salary, benefits, and overhead. Adjust for your market and seniority. European teams might run $80–$100/hr; senior engineers in high-cost markets might run $180–$200/hr.

A realistic 200-test manual suite:

Activity	Time per cycle	Annual cost
Test execution	4 hours	$12,480
Triage of environment failures (not real bugs)	1 hour	$3,120
Re-test after bug fixes	3 hours	$9,360
Total	8 hours	$24,960

That's approximately $25,000/year in labour, just to manually regression-test a 200-case suite on a 2-week cycle.

This doesn't count:

Releases delayed because testing wasn't finished in time
Bugs that escaped because the manual suite didn't cover edge cases
The opportunity cost of that QA engineer's time not going toward exploratory or risk-based testing

The Release Confidence Tax

This one is harder to put a number on, but arguably more expensive.

When your team doesn't trust the test suite - or doesn't have one - release decisions turn conservative. Teams:

Ship less frequently ("let's batch this with next sprint to reduce risk")
Batch changes, which paradoxically increases risk per release
Proliferate feature flags as a crutch, adding operational complexity
Hold informal war rooms before every major release Developers already spend 41% of their time debugging, according to 2024 survey data. A non-trivial chunk of that is diagnosing issues that better test coverage would have caught before they merged.

Conservative framing: if better test automation saves even 2 hours/week per developer by reducing the time spent debugging regressions and environment issues, that's:

2 hrs × 15 engineers × 50 weeks × $120/hr = $180,000/year

That's a large number. Use it carefully - it's the upper bound of what's plausible, not a guarantee.

Run This for Your Team

Here's the back-of-napkin formula:

Step 1: Manual regression labour
= (test cases) × (avg minutes per test / 60) × (releases/year) × (hourly rate)

Step 2: Bug escape cost
= (production bugs per quarter × 4) × (avg cost per production incident)
  [Estimate your cost per incident: engineering hours + customer impact time]

Step 3: Release velocity tax
= (avg days delayed per release) × (releases/year) × (daily cost of delay)
  [Daily cost of delay = revenue at risk + opportunity cost of unshipped features]

Total annual cost of no automation = Step 1 + Step 2 + Step 3

For most teams, Step 2 alone - the production bug escape cost - exceeds the full annual cost of building and maintaining a test automation suite.

What Automation Actually Costs

The counterargument: "automation is expensive too." Fair. Here's the honest breakdown.

Initial setup (realistic, not optimistic):

Item	Estimate
Framework selection and setup	2–3 days
Writing 200 automated tests	3–4 engineer-weeks
CI integration and pipeline config	2–3 days
Total initial investment	~4–5 engineer-weeks

At $120/hr loaded cost: ~$19,200–$24,000 initial investment.

Annual maintenance:

Test maintenance consumes 30–50% of the initial build effort per year for a well-maintained suite.

Assumption: Industry data shows maintenance consuming 30–80% of automation budget. The lower end (30–50%) applies to suites that are actively managed, not allowed to grow without pruning, and built on stable APIs rather than brittle UI selectors. If you're running a Selenium-heavy UI suite on a frequently changing frontend, you're likely on the higher end.

Annual maintenance ≈ 30% × $22,000 = ~$6,600/year

5-year total cost of ownership:

Initial build:     $22,000
5 years maintain:  $33,000
Total:             $55,000

Compare that to the $25,000/year manual regression cost, compounded over 5 years: $125,000 - and that's before counting escaped bugs and delayed releases.

The ROI case is straightforward. The real objection isn't cost. It's time: teams without automation are usually under delivery pressure and can't find the runway to invest. That's a legitimate constraint - but it's worth naming clearly, because "we can't afford it" and "we don't have time right now" have different solutions.

Where AI Changes the Math

AI-based test generation shifts the cost curve at the margins. Here's honestly what it changes and what it doesn't.

What it reduces:

Initial authoring time. Studies on LLM-assisted test generation show 50–70% reduction in test authoring time for well-scoped code. The "4–5 engineer-weeks to write 200 tests" shrinks, potentially to 1.5–2 weeks.
Edge case coverage. LLMs generate negative cases, boundary values, and permutations that humans commonly skip under time pressure. The coverage you get for free is worth more than the authoring time saved.
Maintenance, partially. Self-healing test features (adaptive locators, selector repair) reduce churn caused by UI changes. This is real but not total - tests that encode business logic assumptions still break when business logic changes.

What it doesn't change:

You still need to understand what you're testing and why. AI generates tests; it doesn't understand your business rules.
Flaky tests still require cleanup. Generating more tests on top of a flaky foundation makes things worse.
The strategic decisions - what to test, at what level, with what priority - still require a human with context. The fundamental ROI argument doesn't change with AI. It gets better: the initial investment drops, which makes the case even cleaner. But the math works either way.

P.S.: It's worth mentioning that I lead the AI team at QApilot - an AI-native mobile test automation company. I might have a bias for more automation with AI, but when that bias is based on data, it's ok I guess.

The Documentation Intern That Never Sleeps

Harsh Chandgotia — Fri, 12 Jun 2026 12:41:02 +0000

When I joined QAPilot, I noticed something interesting.

Some of the most experienced people on the team were spending hours every sprint on work that was important, but highly repetitive: tracking engineering changes ticket by ticket, and updating our GitBook pages to keep the user-facing documentation in sync.

That meant reading through closed Jira tickets, figuring out which doc pages were affected, rewriting those pages, and drafting customer-facing release notes, every single sprint. The information needed for all of this already existed across Jira, GitLab, and GitBook. It just needed to be gathered, connected, and acted on.

The more I looked at it, the more it felt like a workflow orchestration problem rather than an expertise problem. So I built an AI-powered pipeline to handle documentation impact analysis and regeneration, orchestrated through GitHub Actions, and designed around human review rather than blind automation,

The Shape of the Pipeline

Before getting into how each piece works, it's worth laying out the shape of the whole system, because everything below is really just a closer look at one part of this.

First, the pipeline gathers everything relevant to the sprint, tickets, code changes, screenshots, and the current state of the docs, into a knowledge base. Second, it works out what that knowledge base actually means for the documentation: which pages are affected, and why. A person reviews that before anything gets written. Third, and only after that review, it regenerates the affected pages and drafts release notes, which go through one more round of review before anything is published.

The same pattern repeats at every stage: plan first, act second, and put a person between the two.

Step 1: Building the Documentation Knowledge Base

Before the system can decide what's out of date, it needs to know two things: what changed, and what the docs currently say. So the pipeline starts each run by assembling a knowledge base for the sprint, drawn from four sources, each answering a different question.

From Jira, it pulls the sprint's tickets, what was supposed to change, in the team's own words. From GitLab, it optionally pulls the merge requests and commit diffs behind those tickets, what was actually built, which doesn't always match what was planned. From the tickets' attachments, it pulls screenshots and runs them through a vision-capable model to generate structured descriptions of what the feature actually looks like, which text alone often doesn't capture. And from GitBook, it pulls the entire existing documentation space, what's already written, so the system has something to compare against.

That last one turned out to be more involved than it sounds. GitBook doesn't store its content as markdown, it stores it as a proprietary JSON node tree, essentially a deeply nested structure of typed blocks (headings, paragraphs, lists, code blocks, images, links) that its editor uses internally. To remove unnecessary noise, I built a recursive converter that walks the tree and reconstructs it as clean markdown, preserving structure like nested lists and embedded images along the way.

It's also worth mentioning how the pipeline is able to access all these systems in the first place.

Our GitLab instance is self-hosted behind the company VPN, which means it isn't reachable from the public internet. GitHub-hosted runners execute in GitHub's infrastructure, so they have no network path to internal services such as GitLab. As a result, any workflow that needed to fetch merge requests, commit diffs, or repository metadata would simply fail because those systems were inaccessible from the runner.

To solve this, the entire workflow runs on a self-hosted EC2 runner deployed within the company's internal network. GitHub allows external machines to register themselves as self-hosted runners by installing the GitHub Actions runner agent and linking it to a repository or organization. Once registered, the EC2 instance appears as an available runner inside GitHub Actions and can receive workflow jobs just like GitHub-hosted runners.

Because the runner operates inside the same trusted environment as GitLab, and other internal services, it can securely communicate with them without requiring additional exposure to the public internet.

Step 2: The Mapping Layer

With the knowledge base in place, here's the part of the pipeline that does the real thinking. The most interesting part of this system isn't writing documentation, it's figuring out what needs to change in the first place.

Before any page gets rewritten, the pipeline runs an impact analysis. For every ticket in the sprint, it asks the model to reason through a few questions: which product feature did this change touch? Is the change visible to users, or purely internal? Which existing documentation pages describe that feature? And given that, should one of those pages be updated, or does this need a brand-new page?

Take a hypothetical example: a ticket adds a two-factor authentication step to the password reset flow. The model recognizes this as touching account security and being user-facing, finds that the "Resetting Your Password" page already describes the old flow and needs updating, and flags that a new "Setting Up Two-Factor Authentication" page might be needed if one doesn't already exist.

The output of this stage isn't documentation, it's a structured map: this ticket affects these pages, for these reasons. Separating this from generation, as its own explicit stage, made a bigger difference to output quality than any prompt tweak I tried. It gives the system a plan to inspect before it writes anything, and it gives reviewers something concrete to check: a proposed relationship between a change and a page, with reasoning attached, rather than a wall of regenerated text to proofread.

Human Review Before Generation

Once the mapping is ready, the pipeline opens a GitHub Issue listing every proposed ticket-to-page relationship, along with the model's reasoning for each. A reviewer, usually the PM who ran the sprint, reads through it. Most relationships are correct as-is. When one isn't, the reviewer doesn't need a special interface: they leave a comment with a small JSON snippet describing the correction.

The pipeline picks this up on its next run and folds it into the approved mapping.

No database. No custom review portal. No separate workflow engine. GitHub Issues became the system of record for the entire mapping step, which sounds almost too simple, but it meant reviewers were working in a tool they already used every day, and every decision and correction was automatically logged and auditable.

Step 3: Controlled Document Regeneration

With the mapping approved, the second workflow runs, and this is where the actual writing happens. For each page flagged as needing an update, the system pulls the current markdown and asks the model to revise only the sections relevant to the change, with explicit instructions to leave everything else untouched. This matters for a few reasons: it keeps the diff small and reviewable, it stops the model from quietly rewriting an unrelated paragraph in a slightly different voice, and it means a reviewer's job is "does this new section make sense" rather than "re-read the whole page for unintended changes."

For pages that don't exist yet, like our hypothetical "Setting Up Two-Factor Authentication" page, the model writes from scratch, but it's given a handful of existing pages from the same section as style references, so the new page reads like it belongs in the same documentation set rather than something a different author wrote.

Alongside the updated pages, the workflow also drafts customer-facing release notes for statuspage. These are deliberately a separate output from the documentation updates, because the audience is different: docs explain how a feature works in full, while release notes are a short, plain-language summary of what changed for someone using the product. Both the updated pages and the release notes are posted back to GitHub for one final round of review before anything goes live.

Keeping It Fast Enough to Run Every Sprint

One more piece is worth mentioning, because it's what makes running this every sprint practical rather than painful.

The mapping stage in Step 2 doesn't hand the model the full markdown of every GitBook page, for a documentation site of any real size, that would be an enormous amount of context. Instead, each page gets summarized first, and those summaries are what is fed into the mapping step. But summarizing the entire documentation space on every single run was expensive, in both time and tokens, for pages that hadn't changed at all since the last sprint.

The fix was a caching layer: GitBook automatically syncs its documentation content to a GitHub repository, allowing the pipeline to use repository SHAs as a lightweight change detection mechanism. Page summaries are persisted between runs as GitHub Actions artifacts, and each new run compares the latest repository state against the previous one to identify which pages have actually changed. Only those pages are re-summarized, while unchanged summaries are loaded directly from the cache. It's a relatively small architectural addition, but it's the difference between a pipeline that's practical to run every week and one that gradually becomes too expensive and slow to justify.

Engineering Lessons

The biggest lesson was that integration work is often harder than intelligence work. The LLM prompts were only one part of the system, most of the complexity came from stitching together Jira, GitLab, GitBook, GitHub Actions, VPN-restricted infrastructure, and multiple data formats into something reliable.

I also learned that building effective AI systems is less about finding the perfect prompt and more about designing the right architecture around the model. Planning stages, review gates, validation layers, and structured outputs had a far greater impact on quality than prompt tweaks ever did.

QA in 2030: What Changes, What Stays, and What Disappears

S.Pradyumna — Fri, 05 Jun 2026 12:30:00 +0000

Building software is getting cheap. But trusting it is not.

Says Mobin Thomas in his wonderful session at BrowserStack's Breakpoint. This stayed with me.

That is not a prediction. That is the structural reality of the next decade for everyone who works in software quality. One projection was particularly difficult to ignore. It showed that between 2026 and 2030, the cost of building software falls sharply while the cost of trust remains unchanged. That widening gap is where Quality Engineering will live.

The question is not whether AI will change QA. It already has. The real question is what exactly changes, what stays the same, and what quietly disappears.

What Is Already Changing

Three forces are compressing the cost of building software, and they do not add together. They multiply.

The first is silicon. Inference costs are falling roughly tenfold every year. AI is becoming economically viable in places where it was not before, and that changes everything downstream.

The second is the agentic stack. Code generation, code review, test generation, log triage, defect routing. All of these are collapsing to fractions of their former cost. What used to take teams multiple days is being compressed into hours.

The third is tooling proliferation. Every layer of the software development lifecycle now has agentic options. Most are average. Some are exceptional. By 2028, even the laggards are expected to close the gap. When that happens, differentiation moves away from which tools you use and toward the quality of judgment you bring to using them.

Right now, we are in a phase called Augmentation. AI sits alongside the human, who remains the decision maker. Test generation from requirements, self-healing locators, and log-triage assistants are already embedded in pipelines. Early adopters are already reporting meaningful productivity gains. The skill shift begins here - prompting, evaluation, and review become everyday disciplines. Tool specific expertise begins to depreciate.

By 2027, Delegation arrives. Agents own bounded slices of work end to end. An agent reads a ticket, generates tests, runs them, files a defect, proposes a fix, and validates it. This is a direction platforms such as QApilot are already beginning to explore. Humans become approvers, exception handlers, and stewards of the agent ecosystem. The hardest problem in this phase is the handoff - when does an agent escalate? To whom? With what context? That is real engineering work, and it is largely undone in most organisations today.

By 2029, we would be in a phase called Governance. Code self-heals, deployments become continuous and conditional on behavioural evidence, and pre-production increasingly gives way to simulation. QE no longer tests software. QE defines the conditions software must earn the right to exist.

Not every industry will reach the same phase of AI-driven QA at the same time, and that is completely fine. Companies building consumer apps, retail platforms, or internal business tools can afford to move fast. If something breaks, the damage might be mostly financial, a bad review, a lost customer, a quick fix. For these teams, the most advanced phase of AI-driven quality engineering arrives roughly when the technology is ready for it.

However, industries like defence, healthcare, and financial services operate differently and deliberately so. When software fails in those environments, the consequences go far beyond a bad review. A wrong calculation in a trading system, a security gap in critical infrastructure, an error in a medical context, these are not problems you recover from with a patch. So, these industries move at a pace that matches their regulations, not just their technical capability. Both timelines are valid. Neither one is the wrong way to approach the future.

What Stays: The Things AI Cannot Replicate

The projection was clear: execution scales with compute. Judgment does not.

Three things remain irreducibly human.

Judgment is the ability to understand what good means before an agent tries to build it. What does "good enough to ship" look like in this domain, for this customer, on this kind of Tuesday? Agents can produce outputs. They cannot answer that question reliably or consistently.

Imagination, in other words seeing the failures the agents will not see. Asking what a malicious user would do, what a confused user would try, what a regulator would look for. Imagining the person on the other end when the software breaks, the one whose claim is denied, whose trade slips. Adversarial imagination and empathy for failure remain deeply human capabilities. They are not qualities that can be prompted into existence.

Experience is pattern recognition that compute cannot synthesize. Domain depth means knowing the failure modes specific to your industry, not as broad abstractions but as concrete realities. One phrase captured this well: scar tissue. You have seen this pattern break before. You know exactly what is about to go wrong. That is the value of experience.

This raises an interesting question: does experience still matter after a few years? The answer depends entirely on which kind.

Procedural experience depreciates fast, a specific Selenium pattern from 2018, a particular Jira workflow, tool certifications, niche test-management interfaces. These are being commoditised. Judgment experience appreciates. Knowing that a certain kind of release at a certain time of quarter always breaks in a particular way. Knowing what "good enough" actually means in a specific domain. The instinct that flags a passing-but-wrong build. That kind of experience does not depreciate. The broader lesson is straightforward: tool experience is going away. Scar tissue is not.

All of these qualities point to a larger reality. As execution becomes cheaper and more abundant, the limiting factor shifts elsewhere. It shifts toward trust.

In many ways, this is the philosophy behind platforms such as QApilot. The goal is not to replace judgment, imagination, or experience, but to automate the repetitive work around them, so they can focus on the decisions that ultimately determine quality.

Why Trust Becomes the Scarce Resource

There is a common belief in software teams that speed is everything: ship the product, fix the problems as they come. It sounds practical and for a while, it works. But it has a ceiling, and most teams only discover that ceiling when they have already gone past it.

Here is the core issue. The cost of building software is falling fast. The cost of earning user trust is not. Every time a product ships with known gaps in quality, that trust takes a small hit and unlike a bug, trust does not get fixed in the next release. It has to be rebuilt slowly, over time, through consistent reliability. That is not something you can automate.

A simple example brings this distinction into focus. Think about fraud detection in 2030. One part of the system is a rules engine whose behaviour has been understood for years. Teams know how it responds, regulators understand its boundaries, and its failure modes are familiar.

Another part is an adaptive AI model that continuously updates how it scores transactions. There is no fixed version to certify once and forget. Trust comes not from static validation but from observing how the system behaves over time, especially under unusual conditions.

Both systems perform the same job, but they earn trust in fundamentally different ways. If judgment, imagination, and trust become the scarce resources, the next question is who will be responsible for institutionalising them. The answer is likely to reshape the structure of software teams themselves.

Who Is in the Standup in 2030?

If this trajectory holds, three roles become increasingly important.

The Quality Architect who is typically senior and often a former lead SDET, writes the behavioural specifications that agents conform to. This person owns the trust contracts for major systems and talks more to product than to developers. They are not writing test scripts. They are writing what trustworthy looks like for each system, codified and signed.

A new role may emerge: the Agent Conductor. Part SRE, part prompt engineer, and part team lead. This person operates the agent fleet day to day by tuning prompts, monitoring performance, retiring drifting agents, and maintaining the team's working relationship with autonomous agents.

The Domain Authority is the domain specialist whose expertise cannot easily be commoditised. This person knows healthcare claims, or trading mechanics, or telecom provisioning in much the way a master craftsperson knows their material. Agents can be trained on this judgment. But the judgment originates here.

How the Shift Looks from Different Seats

The implications of this shift differ depending on where you sit.

For the practitioner, the signal was this: The teams that go deep into a domain will hold their ground. Tools will commoditise. Domain pattern recognition will not. The tester who understands insurance claims, trading mechanics, or telecom provisioning carries knowledge that agents can be trained on but cannot originate. That is the portfolio worth building.

For leaders, it was a budget question. The projected shift over the next 18 months moves spend away from tool licenses and script maintenance and toward behavioural specification capability and relationships with risk and regulatory colleagues. The teams without governance capability when regulation fully arrives will be caught unprepared. For QE leaders, the signal was clear, if you are not already building relationships with your risk and compliance teams, you are already behind. Regulation is evolving alongside these changes. Historically, every major compliance framework has expanded the scope of quality engineering. The AI Act appears set to do the same by introducing new expectations around behavioural assurance, agent governance, and traceability. Teams that delay building governance capabilities may find themselves reacting rather than leading.

For executives, the signal was the simplest of the three: trust is the scarce input, not models, not compute. QE produces trust. In 2030, trust is what software is sold on. Fund it accordingly.

Already Walking That Path

The future of quality engineering will not be defined by who can write the most tests. It will be defined by who can build trust into increasingly autonomous systems. That shift is already underway.

Quality engineering is moving from verifying outputs to governing behaviour. As systems become more autonomous, the challenge is no longer simply whether software works, but whether it can be trusted to keep working as conditions change.

Platforms such as QApilot are already beginning to reflect that shift, treating trust as something that must be engineered continuously rather than verified at the end. The tools will evolve. The agents will become more capable. What will remain is the need for systems people can trust. That is the future quality engineering is moving toward, and it is the path QApilot is already walking.

QA Is Not Going Away. It Is Going Up.

AI is not replacing QA. It is transforming it into the most strategically important function in the software development lifecycle.

The profession is moving into the gap between how cheaply software can be built and how expensively trust must be earned. That gap is not closing. It is growing. The trend itself is difficult to ignore.

The tools are changing. The roles are changing. The work is changing.

What stays is the part that was never about the tools in the first place.