AI helps most with the work around your tests: drafting, triage, maintenance and explaining results. The tests that decide whether you ship should stay as code you can version, review and rerun.
This post covers where each side earns its place, with the evidence, and how to wire them together.
Key takeaways
| Takeaway | What it means |
|---|---|
| AI accelerates testing, but doesn't replace verification | Use AI for test drafting, maintenance, failure triage, and analysis. Review its output before relying on it. |
| Generated tests need stronger validation | Check assertions, edge cases, and mutation scores. A passing test doesn't always catch real defects. |
| Keep performance tests deterministic | Store tests in version control and use explicit SLOs and assertions for pass/fail decisions. |
| Use AI to interpret load test results | Gatling Enterprise offers AI Run Summary, Run Comparison, and Trend Analysis to help identify performance changes. |
| Connect AI agents to testing workflows | Gatling's MCP Server lets compatible agents manage tests, inspect runs, and retrieve performance data. |
| Testing AI applications requires different metrics | Measure time to first token, generation latency, token throughput, and output validity, not just HTTP response codes. |
What is AI software testing?
AI software testing uses artificial intelligence to support activities such as generating test cases, maintaining automated tests, analyzing failures, and identifying potential defects.
It covers two related but different practices:
- AI for software testing: Using AI to improve how teams create, execute, maintain, and analyze tests.
- Testing AI-powered software: Evaluating applications that rely on AI models, such as chatbots, copilots, recommendation systems, and autonomous agents.
Both require reliable validation. AI can accelerate testing workflows, but its output still needs review, and the results must be assessed against clearly defined requirements.
Understanding the different testing approaches
| Approach | Main purpose | Example |
|---|---|---|
| Traditional test automation | Execute predefined, repeatable checks | Run API tests with explicit assertions |
| AI-assisted testing | Help generate, maintain, or analyze tests | Suggest test cases or summarize failed runs |
| Testing AI applications | Validate AI system behavior and reliability | Check response quality, latency, and robustness of an LLM endpoint |
Why AI matters for testing now
Teams ship more often, and manual test effort does not grow at the same rate. Coding agents have widened the gap by producing more code per engineer. Every change still needs verification.
The industry data points at verification, not generation, as the weak spot:
- In Capgemini's World Quality Report 2025-26, 89% of organizations were piloting or deploying generative AI in quality engineering. Only 15% had scaled it enterprise-wide. The top barriers were data privacy (67%), integration complexity (64%) and hallucination or reliability (60%).
- SmartBear's 2026 survey found that 46% of respondents had shipped AI-generated code that later failed in production. 73% worried that application quality was suffering. It is a vendor survey, so read it as directional.
- The DORA 2025 report found that AI adoption now correlates with higher throughput and with lower delivery stability.
Adoption is broad and maturity is thin. If code is cheap to produce, the checks have to get stronger, not just faster.
Where AI already helps
- Drafting tests from requirements or code: Agents turn acceptance criteria, API contracts and existing code into test plans and scripts. Playwright's Test Agents split the job into a planner, a generator and a healer. The planner writes Markdown scenarios you can review in a pull request before any code exists, which is the cheapest place to catch a wrong assumption.
- Reducing maintenance: When a button is renamed or a selector breaks, AI can propose the repair. It is the most mature use in commercial tools.
- Summarizing failures and grouping errors: This is where the production evidence is strongest. Uber's FlakyGuard repairs 47.6% of reproducible flaky tests, and developers accepted 51.8% of its fixes. ByteDance's LogSage analyzed more than a million CI executions at over 80% end-to-end precision. Neither tool is perfect. Both save engineers a lot of reading.
- Generating realistic test data: Synthetic data was the top generative AI use case in the World Quality Report, and its use rose from 14% to 25%. Privacy drives it. Ask the model for generators and edge-case rows, not for copies of production records. Gatling has a guide to generating data in a simulation.
- Explaining results in plain language: A product manager should not need a percentile table to learn that checkout slowed down after Tuesday's release.
Common AI software testing use cases
AI-assisted testing
| Use case | How AI helps | What engineers still need to verify |
|---|---|---|
| Test generation | Proposes scenarios from requirements and existing code | Assertions, edge cases, and coverage |
| Test maintenance | Suggests fixes for outdated test scripts | Whether the updated test checks the intended behavior |
| Failure analysis | Groups errors and summarizes likely causes | Root cause using logs and execution evidence |
| Regression testing | Helps identify potentially affected areas | Adequate coverage and reliable pass/fail checks |
| Performance testing | Summarizes load-test results and highlights trends | Latency, throughput, error thresholds, and repeatability |
| Testing AI applications | Helps evaluate model responses and behavior | Output quality, safety, consistency, and performance |
AI can reduce repetitive work across these activities, but its role should match the reliability requirements of each test.
Where AI falls short
For starters, the output is not deterministic: the same prompt can give you a different test tomorrow. That is fine for a draft and a problem for a regression gate.The bigger risk is the assertion. A generated test can compile, run, pass and cover the code while checking almost nothing. The research is consistent on this:
- A 2025 study by the International Conference on Automated Software Engineering (ASE) found LLM-generated oracles reached about a 43% mutation score. That is similar to programmer-written tests on average, but weak on complex behavior.
- On real-world functions, one benchmark measured LLM tests at roughly 40% mutation score after five attempts and about 20% after one. The same models scored far higher on a toy benchmark, so headline numbers flatter reality.
- An October 2026 study about evaluating ChatGPT performance for unit tests analyzed 14,320 LLM-written tests andfound incorrect assertions in 15.6% of failures, plus a strong happy-path bias. Coverage of null, negative and empty arguments reached only 34.8%, 23.8% and 38.8%.
Trust and accountability are the other half. In SmartBear's data, 47% of respondents could not explain how AI contributed to an incident, and 44% review 60% or less of agent output. When production breaks, accountability lands on the team, not on the model.
The documented fix is verification. Meta's ACH system generated targeted mutants and asked an LLM for tests that kill them. Engineers accepted 73% of the resulting tests. About half would have been discarded under a coverage-only rule.
Performance testing: AI on top of reliable load tests
A load test run produces a lot of data: percentiles over time, error breakdowns, injection profiles, per-request metrics. Finding the bottleneck means reading all of it. That is a good job for AI, because the AI interprets a result and does not decide what passes.
Gatling Enterprise Edition has three AI analysis features, each answering a different question:
- AI Run Summary explains what happened in one run, with insights on response times, injection profile, errors and assertions.
- AI Run Comparison takes 2 to 5 runs and reports differences in throughput, errors, latency percentiles and CPU. It returns a verdict of Similar, SomeDiscrepancies or Divergent.
- AI Trend Analysis reads the last 10 valid runs of a test and returns Stable, SomeIssues or Degrading, with a confidence level and recommendations.
All three are generated on demand, labeled as informational, and can be disabled by an admin. Run Summary can vary slightly between clicks on identical data. So do not let it make the pass/fail call. Failure Diagnosis serves a related purpose for broken runs: it points you at the root cause. For a hands-on walkthrough, see how to get started with AI analysis.
The test stays code. A Gatling simulation lives in your repository in Java, JavaScript, TypeScript, Kotlin or Scala. It is versioned, reviewed in a pull request and repeatable. That is the test-as-code approach, and it is why a result from last month can still be reproduced today.
SLOs add the contract. An SLO is a threshold, such as a p95 bound or an error-ratio bound, evaluated continuously during the run and reported as the percentage of time the condition held. If you are not sure what thresholds to set, the SLO Advisor asks four questions about your service and suggests p95, p99 and error-ratio targets with the reasoning behind each. The SLO guide explains the concepts.
The split is simple:
What stays consistent between test runs?
| Component | Defined by | Changes between runs? |
|---|---|---|
| Test definition | Code in your repo | Only through a reviewed commit |
| SLOs and assertions | Explicit thresholds | Only when you change them |
| AI analysis | A model reading the results | Can vary slightly |
Testing as code in an agent-driven workflow
Coding agents increasingly work with test tools through the Model Context Protocol (MCP). Instead of pasting output into a chat window, the agent calls tools directly. The pattern exists across the ecosystem, from Playwright MCP for browser tests to mcp-k6 for k6.
Gatling's MCP Server follows it. It is a local server that exposes your Gatling Enterprise account to Claude Code, Cursor, Windsurf, VS Code or any MCP-compatible client. Since the July 2026 update, the agent can both read and write across the test lifecycle:
- list teams, packages, locations, tests and runs
- register and delete source repositories a test can build from
- create, update and start tests
- read run logs and per-request or per-group reports, share a public link, and stop runs
Each tool declares the role it needs (Read, Configure or Start), so you can scope the API token to exactly what the agent is allowed to do. It ships as an NPM package and a Docker image, and needs Node.js 20 or later, or Docker. The Gatling blog post on the MCP Server shows it in practice.
The diagnosis workflow is the part I find most useful. When a run fails, the agent cross-references the execution logs against the performance report and tells you whether it was a build error, an injection crash or a stop-criteria breach. When a run succeeds, you can ask "what's the p95 for the login request?" and get it from the per-request data.
Gatling also ships agent skills for bootstrapping projects, configuration as code, and converting scripts. There are converters for JMeter and LoadRunner scripts. Treat converted output like any generated code: review it and run it before you trust it. The same goes for the AI assistants for VS Code and Cursor. The assistant stores your model API key in the IDE's secrets storage and does not send it to Gatling's servers. Gatling's own guidance is that AI-generated code can be incomplete or wrong, and should be verified through testing, peer review and performance validation.
To keep agent-written changes covered, put the load test in the pipeline. Whatever an agent changes, a simulation runs in CI and the SLOs decide whether the build passes. The agent can write the change, but it does not get to grade it. Gatling's guides on performance testing in CI/CD and detecting performance regressions cover the setup.
This is already how some regulated teams work. Nickel, a BNP Paribas subsidiary, treats performance tests as a mandatory gate in its release process, reviewed through its Change Advisory Board. Its team runs 20 to 30 tests a month. You can read the customer story. An agent writing code does not change that gate, it just gives the gate more to catch.
A practical adoption framework
- Start with low-risk, high-volume work: Failure summaries, log triage and flaky-test classification come first. A wrong answer costs a few minutes of review, not a bad release.
- Keep test definitions in version control and require human review: Never let an agent merge its own tests. Treat any diff that changes an expected value as a possible bug report. Run self-healing off the main gate and have it open pull requests.
- Verify AI-written tests before trusting them: Confirm they fail on a deliberately broken build. Run mutation testing on the changed code (PIT for Java, Stryker for JavaScript and TypeScript, mutmut for Python). Run them 10 to 20 times to catch flakiness. Have a different model or a person review them.
- Define success metrics before rollout: Test count and line coverage do not show value. Better measures are time to diagnose a regression, flaky-test rate, escaped defects, change-failure rate and the mutation score of AI-written tests.
- Keep outputs portable: Octomind shut down in April 2026. Customers who had exported Playwright code kept working suites. Code in your repo outlives a vendor.
If you are earlier on the road, Gatling's checklists for going from 0 to 1, from 1 to N and from N to continuous give a sequence to follow.
Load testing AI features is a separate problem
Testing with AI and testing AI are different jobs. If you ship an LLM feature, a load test that checks only status codes will mislead you. A request can return HTTP 200 while the model returns truncated or malformed output.
Latency also splits in two: time to first token (TTFT), which scales with prompt size, and total generation time, which scales with output length. Capacity should be measured in tokens per minute as well as requests per second, because providers meter tokens and those limits usually bite first. Model user think time with an open workload model, so queues can build the way they do in production. See Gatling's guides to load testing an LLM API, SSE testing, workload models and the AI and LLM use case page.
What comes next
Authoring is moving into the IDE and the command line, through agent skills and MCP servers. That makes the test artifact in your repository more important and the testing platform less of a moat.
Testing AI is also becoming part of testing. Analyst criteria now include testing RAG pipelines and agent tool calls, and OpenAI agreed to acquire Promptfoo in March 2026.
Several questions remain open:
- Evidence: In one review of 72 publications on AI and software testing, only three were industrial studies. Vendor results are not universal outcomes.
- Trust: In SmartBear's survey, 83% said autonomous testing would help them keep up with AI code volume. Trust in results was still the top barrier.
- Shakeout: Funded startups have closed and others have been acquired. Expect more.
- Return on investment: The World Quality Report found an average 19% productivity gain, with about a third of organizations seeing 1 to 9%. The gap appears to come from integration, not from the tool bought.
Conclusion
AI earns its place in testing by speeding up authoring, triage and analysis. Code earns its place by being the part you can trust: versioned, reviewable and repeatable, with explicit thresholds that decide pass or fail.
Look at your own process this week. Where does your team lose the most time between "a test failed" and "we know why"? Start there. Try Gatling Enterprise Edition for free, run one test with an SLO, and click AI Run Summary on the result.
If you want to see it with your own stack, book a demo. If you prefer to start with the open-source engine, download Community Edition. Not sure which fits? Compare the editions.
Top comments (0)