From Quantity to Quality: A Risk-Based Approach to AI-Assisted QA at Scale

Artemii Zakharov — Tue, 22 Sep 2026 09:18:00 +0000

We write more tests than ever, yet bugs still reach production. This post introduces a risk-based approach to QA. We'll show how AI-driven impact analysis can flag what needs testing, using one of Salmon's own releases — a rewritten transfer flow and a changed card issuance fee calculation — as the example. Instead of testing features, we focus on consequences: financial loss, user impact, and operational cost.

This is the kind of approach that allows Salmon's team to maintain quality while keeping up with development speed in the AI era, so read on.

The math stopped working

At some point, one of our products at Salmon had 2,000 test cases: about 500 automated, 1,500 manual. Split across eight QA engineers, on paper, that's roughly 57 hours of regression testing per release. In practice, every release took five days.

Fifty-seven hours assumes pure execution: no analysis, no retesting, no coordination. Real regression testing never runs that clean.

The product was also several connected products linked by cross-product flows. A defect in one small piece could ripple into services that never touched the original change. This might lead to lost money, eroded trust, a support queue full of the same complaint, and a funnel that leaks users.

Five days of full-team regression is the realistic cost of the old model. QA joins after a feature is built, writes test cases, expands coverage, runs a full regression pass, and everyone feels safe because the coverage number looks good.

But a coverage number can hide how much effort went into a project. Easy-to-test areas get tested thoroughly. Yet, important areas stay under-tested because nobody flagged them as different.

This flow manages to hold up when systems are smaller. That stops once a system gets more interconnected and the pace of change accelerates. AI-assisted development alone has pushed commit volume up by roughly a quarter year over year in a lot of orgs. QA capacity hasn't grown anywhere near that fast, and it isn't going to.

Test case count was never the measure of quality. What matters is where the attention goes, and deciding that before you're deep in a regression pass.

One screen, three bugs, three different problems

Take a form we ship at Salmon, where a client enters bank details, and imagine three bugs on it:

A typo in the page title
A confusing input mask on the account number field
A broken validation rule that lets the confirm button submit an empty or invalid form straight to the backend

Ask a room of engineers which matters most and the answer is likely unanimous: the broken validation. It causes real financial and operational damage, bad records in the database, and requires manual cleanup by an operations team.

This is worth formalizing:

Risk = (Impact × Blast Radius) ÷ Safety Net

Impact — the pain if it goes wrong, for users, trust, and the business.
Blast radius — how widely the underlying change actually reaches. A one-line fix to shared authentication code has a much bigger blast radius than it seems.
Safety net — whatever already protects you, such as automated tests, design review, or existing process.

Let's run those three bugs through the formula:

Typo — low impact, small blast radius, decent safety net (design review would likely catch it). Low risk, quick fix, move on.
Confusing mask — medium impact (confusion, support tickets, funnel drop-off), small blast radius, weak safety net (a test can confirm the field exists, not whether it's clear). Medium risk, worth a real conversation with the design team.
Broken validation — high impact (direct financial exposure), wide blast radius (this logic runs on nearly every form in the app), no safety net. Unquestionably high risk, worth expanded testing, and possibly holding the release.

The real problem is one level up

A real release involves dozens of tickets, hundreds of changed files, changes to underlying data models, all on top of a test suite that already exists. The issue is figuring out what to prioritize across a large diff. This necessitates something closer to a live map between what changed in the code and what that means for test coverage.

Where does AI impact analysis fit?

Here, AI's job is compressing the time it takes to get a clear picture of what a release touches. The risk decisions stay with people. This can mean fifteen to twenty minutes instead of days of manual tracing. Output quality is a direct function of input quality:

The full release branch diff, so the model reads everything that changed, not one file at a time.
Ticket descriptions, so it has business context, not just a diff. A diff shows what changed. A ticket tells us why.
The existing test case repository and its mapping to code (Allure TestOps, Zephyr Scale, TestRail — the tool doesn't matter), so the model knows what's already covered.

A useful report flags key risk areas in plain language and shows any unmapped new functionality that shipped without test coverage — exactly what the old model tends to miss, since the team is busy running the existing suite and feels productive doing it.

It also triages existing test cases into "must-test", "recommended", and "safe-to-skip", and highlights selective automation targets once your suite has grown into the hundreds and started costing real CI time.

A real release

Let's say one release comes with forty tickets, roughly seven hundred changed files, fifteen data models touched, four hundred existing test cases.

Under the old model we would open all four hundred, run them broadly, for about three days, with no real way to know in advance which ones mattered.

With AI impact analysis inserted into the pre-merge job, the same release produced a report in fifteen to twenty minutes flagging the two areas that actually mattered: a rewritten transfer flow and a changed card issuance fee calculation.

Roughly 200 of the four hundred cases (50%) came back must-test, given how wide the blast radius was. Another 83 (about 20%) were recommended, backed by a solid existing safety net. The rest were "safe to skip".

To illustrate, one skipped area had 25 associated test cases with zero code changes and zero dependencies, and running them would have been pure overhead. That saved time went straight into the parts of the release touching money and operations.

Full disclosure

This didn't work from day one. The first iteration was rough: regression time went up before it went down, extra hours went into tuning inputs, and perceived risk on releases got worse for a stretch.

Adding AI to a regression process can make things more expensive before it makes it cheaper. It took a few release cycles before the setup stabilized and paid back the early investment.

Budget for this ramp-up, rather than expecting immediate savings.

Where this approach still struggles

Four failure modes are worth naming.

A diff shows what changed, not the full blast radius, so hidden runtime dependencies that only surface at compile time, container build, or in-browser stay invisible.
Cross-cutting changes that look small but touch shared authentication, models, or storage still need a human sanity check at the end.
Weak or missing test mapping means the model doesn't have a reliable ground truth and will guess. A confident wrong answer is worse than no answer.
Brand-new features without history give the model nothing to calibrate against; it doesn't know what's critical or where the business boundaries are, and that context still has to come from a person.

What should still be human?

Users today are accustomed to well-built products and don't extend much patience for unpredictable UX. When something breaks, people rarely trace the failure back to which team owns it. They just leave.

A simple rule helps decide what belongs in automation: if two QA engineers testing the same thing independently would test it the same way every time, it's a strong automation candidate. Think login, registration, other repeatable critical paths, plus basic checks like whether a screen loads and expected elements are present.

On the other hand, novel, unstable scenarios with no history, and exploratory testing of ambiguous UX — like that confusing input mask — still need a human.

Quality over quantity

Test case count was never a good proxy for quality. Replacing "how much did we test" with impact, blast radius, and existing safety net forces a more useful question: given all three, where does this release need attention?

Getting that right, fast enough to keep up with real release velocity, matters more than adding another few hundred test cases to a suite nobody has time to fully run. That's the same read that flagged the transfer flow and fee calculation changes early enough to matter, and let the rest wait.

DEV Community: Artemii Zakharov