AI test automation architecture is the system that tells AI what to test.
It also defines how to run tests and prove the result.
I split it into three layers: orchestration, execution, and evidence.
Without all three, AI testing becomes prompt output with no production gate.
Why tool lists fail
Most AI testing content starts with tools.
That is backwards.
AI means software that predicts.
Predictions can help QA teams move faster.
But predictions do not prove quality.
The 3-layer model
| Layer | Plain meaning | Main question |
|---|---|---|
| Orchestration | test control plan | What risk should this cover? |
| Execution | actual test run | Did it run in the real pipeline? |
| Evidence | proof from runs | Can a human review it? |
The practical gate
Use this before AI-generated tests ship:
| Gate | Pass condition |
|---|---|
| Scope | The test maps to one named risk |
| Data | Test data setup is explicit |
| State | Browser state is controlled |
| Run | The test passes in CI |
| Evidence | Trace or equivalent proof exists |
| Review | A human can explain the failure mode |
CI means automated build server.
MCP means tool connection standard.
Playwright is a browser test tool.
Together, they can help AI agents run useful tests.
But the architecture must prove each run.
The rule
Never ask AI to expand test coverage first.
Build the proof system before that.
Generation is cheap.
Evidence is the architecture.
Read the canonical version:
https://www.anton.qa/blog/posts/ai-test-automation-architecture-3-layer-system
Anton Gulin is the AI QA Architect, the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.
Top comments (8)
I like the way you’ve separated execution from evidence. I’ve been exploring AI-assisted testing recently, and I’m finding that the evidence part is probably what makes the results easier to trust.
A test can pass, but that doesn’t always mean it tested the right thing. Having some context around what was actually validated and why the AI made that decision feels really important.
I’m curious how this would work in a larger test suite, where there could be hundreds of AI-driven decisions happening during a run.
At that scale, nobody reviews every decision live. Evidence like a trace lets you inspect any run without rerunning it. Failures get human eyes first, passes get audited on demand.
That makes sense. I especially like the idea of auditing passing tests when needed instead of trying to review everything.
It also makes me wonder how we identify a “suspicious pass” - a test can stay green while slowly drifting from its original intent.
Do you think AI could help catch those cases, or is that still something QA should mainly watch for?
AI can help with the suspect list. It can scan run history for tests that never fail for a real reason. The verdict stays human though. Break the behavior it guards. A drifted test stays green.
Yeah, that makes sense. The idea of intentionally breaking the feature is actually a pretty simple way to see if the healed test still does its job.
I think that kind of check could be especially useful for the important tests.
Agreed, the tests that matter most are the right place to start. Happy to keep exploring this with you in DMs.
I like the idea of letting AI narrow things down instead of trying to review everything manually. The evidence part is probably what makes that approach much more practical in a real test suite.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.