A pull request comes in. The function is tidy, the names are clear, and the unit tests pass. It also treats a $20 store credit as a 20% discount.
Nothing about the code looked wrong. That is the kind of bug worth planning for.
AI coding assistants write plenty of code that is fine. They also write code that compiles, reads well, passes the obvious test, and quietly makes an assumption nobody asked for. QA doesn’t need to distrust every generated line. It needs to look in different places.
Test behavior, not how convincing the code looks
Generated code usually looks finished. Consistent style, sensible names, maybe a docstring. Reviewers relax when code looks like that, and testers do too.
Here’s the discount bug:
The requirement said wholesale customers get a percentage off and loyalty members get a fixed credit. The prompt only mentioned percentages, so the code only handles percentages. A test with a wholesale customer passes. A loyalty member with a $20 credit on a $50 order gets $10 off.
The code is clean. The business behavior is wrong. Test against the requirement, not against what the code appears to intend.
Push harder on edge cases
AI tools don’t always miss edge cases. But a generated implementation often follows the most obvious pattern in the prompt or the surrounding code. If the requirement doesn’t spell out the exceptions, the code may not handle them.
So, the familiar list gets more attention: empty and null values, boundaries, negative numbers, very large inputs, duplicate records, odd formats, timeouts, partial responses, and users in states the happy path never visits.
One realistic case: a generated CSV importer passes every test with the sample file. Then a customer uploads a file exported from Excel. It starts with an invisible byte order mark, so the first header no longer matches, and every row fails validation. The sample file never had that character.
Test the assumptions hidden inside the implementation
This is where I’d spend the most time.
Every implementation assumes things. Generated code makes those assumptions silently, based on what seemed likely. Watch for assumptions about input shape, whether data is always present, authentication state, ordering, time zones, locale, permissions, uniqueness, API response structure, error handling, and storage.
One question is worth asking on every change:
What did this implementation assume that the requirement never promised?
Example: a generated “today’s orders” report uses new Date() on a server running in UTC. For users in Sydney, orders placed in the morning show up in yesterday’s report. Nothing in the requirement said “UTC.” Nothing said otherwise either, and the code picked one.
During review, write the assumptions down. The risky ones become test cases.
Where generated code meets the rest of the system
Some problems only appear once the code talks to real services, real packages, real users, and the rest of the codebase.
API and integration testing matter more
Generated code can work perfectly in isolation and fail against a real service. Unit tests usually mock the dependency, and the mock is often built on the same assumption as the code.
The unit test mocks a 200 with a populated items array. In production, the service returns 204 No Content when a customer has no invoices, and res.json() throws. It also paginates after 100 results, so large customers silently lose invoices. And there’s no status check, so a 500 error body gets parsed as data.
Test optional fields, status codes, retries, expired tokens, rate limits, pagination, timeouts, schema changes, and third-party failures. Contract tests against the real schema help because they don’t share the code’s assumptions.
Review dependencies the AI adds
Assistants sometimes suggest packages, and sometimes those packages don’t exist. A study by Spracklen and colleagues generated 576,000 code samples with 16 code generation models and analyzed them for references to packages that don’t exist. The risk is that an attacker publishes a malicious package under a commonly hallucinated name, and later users who get the same suggestion install it.
For each new dependency, check that it exists and is the package you think it is. Check that it’s maintained, that the version makes sense, that it doesn’t conflict with what you already have, and whether the license needs review. Also ask if it’s needed at all. A ten-line helper is sometimes better than a new package.
This isn’t a fringe concern. OWASP’s 2025 Top 10 added software supply chain failures as a new category, replacing the older one about vulnerable and outdated components.
Security-sensitive code needs a different bar
Not every generated change deserves the same scrutiny. Authentication, authorization, payments, personal data, file uploads, input validation, sessions, secrets, and admin features do.
There’s some evidence that AI help can make people overconfident here. In a Stanford user study by Perry and colleagues, participants with an AI assistant wrote less secure code than those without one, and were also more likely to believe their code was secure. That study used an older model, so treat it as a warning about confidence, not a current defect rate.
For these areas, add negative tests and permission boundary tests. Log in as user A and request user B’s invoice by changing the ID. Try the admin endpoint with a regular token. Pair that with manual review from someone who knows the domain. Broken access control is still the top category in OWASP’s 2025 list, and it’s a business logic problem that scanners and generic tests often miss. Automated tests alone aren’t enough here.
Look for duplicated logic
An assistant can write a correct solution without knowing that similar logic already exists elsewhere in the codebase. You end up with two validation rules, two date formatters, three API wrappers, and error handling that differs by feature.
A typical case: free shipping eligibility gets generated twice. The cart page checks the subtotal before discounts. Checkout checks it after discounts. A customer sees “free shipping” in the cart and gets charged at checkout.
Each feature passes its own tests. The bug only shows up when QA compares the two workflows side by side. That comparison is worth doing deliberately whenever similar features ship close together.
How to adjust your test strategy
Knowing where to look is half of it. The other half is deciding how much effort each change gets, and when QA gets involved.
Regression testing has to keep up with the change rate
The speed gain is real. In a GitHub-controlled experiment, developers with Copilot finished an HTTP server task in JavaScript 55.8% faster than the control group.
That doesn’t mean more bugs. It means that if a team ships more changes in the same period, the regression surface grows faster too. Protect critical user journeys with automated regression. Keep API and integration regression suites current. Use risk to decide what runs on each change, instead of trying to automate everything.
Not every AI-generated change needs the same test depth
This is a practical way to think about it, not an industry standard. Adjust it to your own system.
Low-risk changes, like updating button labels or styling, usually need nothing more than smoke tests and your existing regression suite. A light review is enough.
A refactor in a well-covered area, such as extracting a helper method, sits slightly higher. Unit and regression tests should catch most problems, and a normal developer review covers the rest.
New API endpoints or business logic are medium risk. New filtering logic, for example, needs unit, API, integration, and regression tests, plus review from a developer or QA engineer who knows what the feature is supposed to do.
New third-party packages fall between medium and high. If the assistant adds a document processing library, check the package itself before you test the code that uses it, then run integration tests against real inputs.
Anything touching authentication, permissions, or payments is high risk. Changing refund authorization should get unit, integration, security, regression, and end-to-end testing, and an expert should review it before it ships.
Database schema changes and data processing are also high risk. Splitting a customer table needs migration testing, integration and regression tests, and a tested rollback plan. Review is mandatory.
Legacy code with weak test coverage is the one teams most often underestimate. Before letting an assistant change old pricing logic, write characterization tests that capture what the code does today, then run the relevant regression tests after the change. Someone who knows that code should review it.
Get involved before the code exists
When an implementation takes an hour, testing only after it’s done means QA is always catching up.
Earlier involvement can be simple. Read acceptance criteria before work starts and point out the missing exceptions, like the loyalty credit. List the risky scenarios. Agree on how integrations should behave when the other side fails. Flag security-sensitive changes so they get the right review. A clear requirement also makes the generated code better, because the assistant has fewer gaps to fill with guesses.
What should stay human?
Nobody needs to review every generated line. Review effort should follow risk.
Some judgment still belongs to people: reading an ambiguous requirement, noticing missing business context, deciding whether odd behavior is a bug, judging user impact, exploratory testing, and reviewing security-critical code. The hardest call is spotting output that is technically correct and practically wrong. Tests rarely catch that on their own.
What to change in CI/CD
You don’t need a new pipeline. You need gates that match risk. Run fast tests on every change. Run broader regression when high risk modules change. Add API contract checks and dependency scanning. Require extra review for paths like auth and payments, and block deployment when critical tests fail.
The goal is not to test more of everything
Adding tests everywhere is the wrong response. Move scrutiny to where generated code tends to go wrong: assumptions, edge cases, integrations, security boundaries, dependencies, regression risk, and business behavior.
AI can make implementation faster. QA’s job is to make sure confidence doesn’t arrive faster than understanding.
Top comments (0)