A Staff SDET's field guide to the architectural shift — for juniors who want to actually understand it, not just hear the buzzword.
Your test fails at 2 AM. The pipeline goes red. A human wakes up, opens the trace viewer, stares at a screenshot, greps the logs, guesses it's a selector change, fixes it, reruns, goes back to sleep.
Now imagine the pipeline doesn't wait for that human. It looks at the failure itself, forms a hypothesis, checks the DOM, checks the API response, decides whether the app broke or the test broke, patches the test if it's safe to, reruns it, and only wakes a human up if it's not sure.
That's not "AI will replace testers." That's a new layer sitting on top of the framework you already built. This article is about exactly what that layer is made of, what it can and can't be trusted with, and what you — as a QA engineer — need to learn next.
No hype. No "70% faster" stats pulled from nowhere. Just architecture, trade-offs, and one worked example you can map onto your own codebase.
1. The Shift in One Picture
Traditional automation loop:
flowchart LR
A[Write] --> B[Run]
B --> C[Fail]
C --> D["Human investigates"]
D --> E[Fix]
E --> F[Rerun]Every arrow after "Fail" requires a person. The framework can tell you that something broke. It cannot tell you why, and it definitely can't decide what to do about it.
Agentic QA loop:
flowchart LR
A[Goal] --> B[Plan]
B --> C[Act]
C --> D[Observe]
D --> E[Diagnose]
E --> F[Modify]
F --> G[Retest]
G --> H[Report]
G -.loop until confident or escalated.-> CThe difference isn't "more AI." It's that the system now has a goal instead of a script, and a loop instead of a dead end. A traditional test is a fixed sequence of steps someone wrote in advance. An agent is given an objective — "verify checkout completes for a logged-in user" — and decides, step by step, how to pursue it, using the actual state of the application in front of it.
Keep that distinction in your head for the rest of this article: script = predetermined steps. agent = goal + reasoning + tools + feedback loop.
2. What a Traditional Automation Framework Actually Does
Before touching agents, it's worth being precise about what you already have, because the agent doesn't replace any of this — it sits on top of it.
A mature framework is a stack of responsibilities:
| Layer | Responsibility | Example |
|---|---|---|
| Test code | Encodes expected behavior as assertions | expect(page.locator('#total')).toHaveText('$49.99') |
| Page Objects / Components | Abstract UI structure from test logic | CheckoutPage.submitOrder() |
| Utilities | Shared helpers (auth, data setup, waits) | loginAs(user) |
| Test data | Inputs and fixtures | JSON/DB seed data, factories |
| Assertions | Pass/fail decision logic | Playwright's web-first expect
|
| Configuration | Environments, browsers, retries | playwright.config.ts |
| Reporting | Human-readable results | HTML report, Allure, trace viewer |
| CI/CD | Trigger, execute, gate | GitHub Actions, Jenkins |
This stack is deterministic by design. That determinism is a feature — it's why automation is trustworthy. An agent doesn't remove any of these layers. It adds a reasoning layer that can operate these layers instead of a human doing it manually when things go wrong.
3. What Changes When an Agent Enters the Architecture
Here's the honest list of new capabilities — and new problems — that show up the moment you add an agent.
- Reasoning — the system forms a hypothesis ("this looks like a timing issue, not a locator change") instead of just reporting a stack trace.
- Tool use — the agent can call real tools: run a test, read a log file, query an API, inspect the DOM, open a git diff — not just generate text about them.
- Context — it needs to know the application's structure, the test's intent, recent commits, and prior failures, not just the current error.
- Observation — it reads back the result of its own actions (a screenshot, an accessibility snapshot, an HTTP status) and updates its plan.
- State — it tracks what it has already tried across a multi-step investigation, instead of starting fresh each time.
- Decision-making — it chooses between several next actions (retry / inspect further / modify code / escalate) based on confidence, not a fixed script.
- Controlled action — and this is the one that actually matters — it can change things: rerun a suite, edit a locator, open a PR, post to Slack. This is what separates "AI-assisted" from "agentic," and it's exactly where guardrails become non-negotiable.
That last point is the whole ballgame. An LLM that suggests a fix in a chat window is low risk. An agent that applies the fix, commits it, and reruns the pipeline is a different risk category entirely — even if the underlying model is identical.
4. Traditional vs. AI-Assisted vs. Agentic vs. Self-Healing — The Comparison Table
This is the table people usually get wrong, because "self-healing" and "agentic" get used interchangeably. They are not the same thing.
| Dimension | Traditional Automation | AI-Assisted Automation | Self-Healing Automation | Agentic QA |
|---|---|---|---|---|
| Autonomy | None — runs exactly what's written | None — suggests, human applies | Narrow — auto-fixes locators only | Broad — plans and executes multi-step investigation |
| Decision-making | Fixed pass/fail assertion | Human decides after AI suggestion | Rule-based similarity match (e.g., nearest DOM match) | Model reasons over observed state to choose next action |
| Tool usage | Test runner only | None (chat/IDE suggestion) | Internal locator-matching heuristic | Multiple tools: browser, runner, logs, API, source code, git |
| Debugging | Human reads logs/trace | AI explains logs, human interprets | N/A — doesn't debug, just adapts selectors | Agent inspects logs, DOM, network, diffs to form root cause |
| Retesting | Manual rerun | Manual rerun | Automatic (same test, healed locator) | Agent decides if/how to rerun, possibly with modified test |
| Risk profile | Low — deterministic, predictable | Low — advisory only | Medium — can mask a real UI regression as a "fix" | Medium–High — can take unreviewed action if unguarded |
| Human involvement | Full ownership of triage & fix | Full ownership, AI as assistant | Configuration + periodic audit | Approval gates, escalation review, strategy decisions |
| Best for | Stable, well-understood flows | Speeding up authoring/debugging | Cosmetic UI drift (renamed classes, reordered DOM) | Ambiguous failures needing multi-step investigation |
Trade-off to internalize: self-healing automation solves a narrow, well-bounded problem (locator drift) with a narrow, well-bounded mechanism (similarity matching). Agentic QA solves a much wider problem (arbitrary failure investigation) with a much wider mechanism (an LLM planning over tools) — which means wider power and wider blast radius. Don't reach for an agent to solve what self-healing already solves cheaply and safely.
5. One Failure, Three Ways to Handle It — Login Workflow
Scenario: your CI run just failed on should log in with valid credentials. Here's how each generation of automation handles it, with actual JS/Playwright code.
The test:
// login.spec.js
import { test, expect } from '@playwright/test';
test('should log in with valid credentials', async ({ page }) => {
await page.goto('/login');
await page.getByLabel('Email').fill('user@example.com');
await page.getByLabel('Password').fill('Secret123!');
await page.getByRole('button', { name: 'Sign in' }).click();
await expect(page.getByTestId('dashboard-welcome')).toBeVisible();
});
It fails with: TimeoutError: element not found — [data-testid="dashboard-welcome"]
A) Traditional Automation
The pipeline goes red. That's it. The framework's job ends at "assertion failed." A human opens the HTML report, downloads the trace, scrubs through screenshots, and manually decides: is this a real regression, a flaky wait, or a renamed test-id? Time to resolution depends entirely on who's on call and how fast they can context-switch.
B) AI-Assisted Automation
Same failure, but now an engineer pastes the stack trace and screenshot into a chat interface. The model reads the trace and says something like: "The welcome banner's data-testid may have changed — check the latest frontend diff for the dashboard component." Useful. Faster than manual grepping. But the human still opens the repo, checks the diff, decides, and edits the test by hand. The AI never touched the pipeline.
C) Agentic QA Workflow
The agent is wired into the pipeline with tool access. On failure it:
- Observes — pulls the Playwright trace, screenshot, and accessibility snapshot at the point of failure.
-
Diagnoses — queries the app's current DOM via a Playwright MCP-style tool call and compares it against the expected
data-testid. Findsdashboard-welcome-v2now exists wheredashboard-welcomeused to. - Checks intent — pulls the last few commits touching the dashboard component to see if this was an intentional rename (it was — part of a rebrand PR merged two hours ago).
- Decides — this is a cosmetic, intentional rename, not a functional regression. Confidence is high enough to act.
- Acts — updates the locator in the test file, opens a draft PR (does not merge it), and reruns the test against the fix.
- Reports — posts to the team channel: "login.spec.js failed due to a test-id rename in PR #482 (dashboard rebrand). Draft fix opened in PR #611, test now passing locally. Awaiting review."
Notice what the agent did not do: it did not merge its own fix, it did not silently loosen the assertion, and it did not decide on its own that the rebrand was "fine" from a product standpoint — it only judged that the locator mismatch was cosmetic. That distinction — technical judgment vs. product judgment — is exactly the line covered in Section 8.
6. Practical Architecture: How the Pieces Actually Fit Together
Why each box matters:
- Tools, not vision: In 2026, the dominant pattern for browser-based agents is reading a structured accessibility snapshot rather than a screenshot — Microsoft's open-source Playwright MCP server exposes dozens of browser-control tools this way, which makes agent actions deterministic and auditable instead of vision-model guesswork.
- Observe is not optional: an agent that acts without reading back the result of its action is just automation with extra steps and extra risk. The observe step is what turns "action" into a controlled feedback loop instead of a blind write.
- Evaluate is the guardrail chokepoint: this is where confidence thresholds, allow-lists of permitted actions, and human-approval gates live. Everything downstream of "Evaluate" should be scoped by policy, not by the model's own judgment alone.
- Escalate is a feature, not a failure: a good agent architecture treats "I'm not sure, ask a human" as a successful outcome, not a fallback of last resort. Teams that punish escalation (by treating it as "the AI didn't work") end up training the system, implicitly, to guess more confidently — which is the opposite of what you want.
7. What "Taking Action" Actually Means
"Agentic" gets thrown around loosely. Concretely, action means the agent can do some subset of the following, through real tool calls:
- Running tests — triggering a specific spec or suite, not just reading results.
- Inspecting failures — pulling traces, console logs, network waterfalls, accessibility snapshots.
- Navigating the application — clicking through the live app to check whether a flow still works, independent of the failing test's script.
- Collecting artifacts — screenshots, HAR files, DB state snapshots, timestamps — building an evidence trail, not just a verdict.
-
Modifying test code — updating a locator, adjusting a wait strategy, adding a missing
await— scoped to test code, ideally as a draft PR, never a direct merge. - Rerunning tests — after a change, to confirm the fix actually resolves the failure rather than just silencing it.
- Escalating uncertain failures — routing to a human with a structured summary when confidence is low or the change touches assertions/business logic.
A simplified version of a modify-and-retest tool call, using a Playwright MCP–style setup, looks like this conceptually:
// agent-tools.js — simplified illustrative tool wrapper
// (structure mirrors how an MCP-based agent would call Playwright tools)
async function inspectFailure(page, testId) {
const snapshot = await page.accessibility.snapshot();
const trace = await page.context().tracing.stop({ path: `traces/${testId}.zip` });
return { snapshot, trace };
}
async function proposeLocatorFix(oldLocator, domSnapshot) {
// agent reasoning step happens here (LLM call, not shown),
// returns a candidate replacement based on structural similarity
return findClosestMatch(oldLocator, domSnapshot);
}
async function applyFixAsDraftPR(filePath, oldLocator, newLocator) {
// scoped to test files only — never application source code
await git.checkoutBranch('agent/fix-login-locator');
await editFile(filePath, oldLocator, newLocator);
await git.commit('fix(test): update stale locator after rebrand (agent-proposed)');
await git.openDraftPR({ reviewers: ['qa-team'] });
}
async function rerunAndConfirm(specFile) {
const result = await runPlaywright(specFile);
return result.status === 'passed';
}
Notice the shape: every action is narrow, logged, and reversible. That's not incidental — it's the actual design requirement for any agent you let near your pipeline.
8. What Agents Still Cannot Reliably Determine
This is the section junior engineers should bookmark. An agent can be extremely good at technical diagnosis and still be the wrong entity to make these calls:
- Business intent — whether a behavior change is "expected" often depends on a roadmap conversation, a support ticket, or a product decision the agent has no visibility into.
- Acceptable risk — a flaky test on a marketing page and a flaky test on a payment confirmation step are not the same risk, even if the failure signature looks identical.
- Whether a test should exist at all — an agent can tell you a test is failing; it can't reliably tell you the test is testing the wrong thing, or that a whole flow is no longer worth covering.
-
Whether changing an assertion masks a real defect — this is the single most dangerous action an agent can take unsupervised. Loosening
toHaveText('$49.99')totoBeVisible()makes the test pass. It does not mean checkout still charges the right amount. - Whether a passing test provides meaningful coverage — a green suite full of tests that got quietly "healed" into shallow assertions is a worse state than a red suite, because it's a false signal that's actively hiding risk.
Staff-level framing: the job isn't shrinking, it's moving. The scarce skill is no longer writing automation steps — it's engineering reliable systems that can reason and act around automation, and knowing exactly where to put the fence around them. That's a systems-design skill, not a scripting skill, and it's a more senior skill than what most of us started our careers doing.
9. What SDETs Need to Learn Next
| Area | Why it matters | Where to start |
|---|---|---|
| Agent fundamentals | Understand planner/reasoning loops, not just prompts | Study ReAct-style agent loops, ADK/agent-framework docs |
| Tool calling | Agents are only as good as the tools they can invoke | Practice writing well-scoped, single-purpose tool functions |
| MCP (Model Context Protocol) | The emerging standard for exposing tools/context to LLMs | Read the MCP spec; try Microsoft's Playwright MCP server locally |
| Browser automation internals | You need to know what the agent is actually calling | Deepen Playwright fundamentals — accessibility tree, tracing, network interception |
| Context & state management | Agents fail silently when context is stale or too large | Learn context-window budgeting, memory/session design |
| Agent evaluation | "It worked once" isn't evidence | Learn eval harnesses — golden datasets of known failures, pass/fail scoring for agent decisions |
| Observability & tracing | You'll debug the agent, not just the app | OpenTelemetry basics, structured logging for multi-step agent runs |
| Guardrails & policy | This is the actual engineering discipline here | Allow-lists, approval gates, sandboxed branches, rollback design |
| CI/CD integration | Agents live in pipelines, not just chat windows | GitHub Actions/Jenkins integration patterns, draft-PR workflows |
| Test strategy | The highest-leverage skill, and the one AI can't do for you | Risk-based test design, coverage strategy, what not to automate |
10. What Not to Automate With an Agent
Concrete no-go list, not a vague warning:
- Direct writes to production data or production environments — an agent's "checkout works" investigation should never run against real customer accounts or real payment rails.
- Merging its own pull requests — draft only, human-reviewed, always.
-
Deleting or disabling failing tests to make CI green — this is the agentic equivalent of
.skip()-ing everything, and it's worse when it's automated because it happens silently and at scale. - Loosening assertions on regulated or financial flows (pricing, tax calculation, payment confirmation) without a named human sign-off.
- Security- and auth-sensitive test paths — session handling, permission boundaries, PII fields — where a "plausible fix" can quietly create a real vulnerability.
- Irreversible destructive actions — dropping test databases, rotating shared credentials, force-pushing to shared branches.
- Anything where "looks the same" isn't good enough — visual-only comparisons on financial documents, legal text, or compliance-critical copy, where subtle wording changes matter more than layout.
Rule of thumb: if a wrong decision here is expensive, irreversible, or invisible until a customer hits it — keep a human explicitly in the loop, not just "available."
11. Five-Minute Checklist: Is My Framework Ready for Agentic QA?
Score yourself honestly:
- Do my tests produce structured, machine-readable artifacts (traces, accessibility snapshots, JSON reports) — not just pass/fail in a console log?
- Is my test code organized well enough (Page Objects, clear naming, isolated specs) that an automated diff to one file wouldn't ripple unpredictably elsewhere?
- Do I have CI/CD hooks where an external process could safely trigger a rerun or open a draft PR without direct merge rights?
- Do I have a clear, written list of what should never be auto-modified (assertions on financial/security flows, production config)?
- Is there a human-review gate (required PR review, branch protection) that a bot account cannot bypass?
- Do I have observability into why a test failed beyond the assertion message (network logs, DOM state, recent commits)?
- Could I explain, in one sentence, the difference between what my self-healing tooling already handles and what would require full agentic reasoning?
Five or more checked: you're structurally ready to pilot agentic workflows on a low-risk test suite. Fewer than five: fix the framework fundamentals first — an agent layered on a fragile framework just automates the fragility faster.
12. The Maturity Path
flowchart TD
A["Traditional SDET"] --> B["AI-Assisted SDET<br/>adds AI as a debugging/authoring assistant"]
B --> C["AI-Augmented SDET<br/>integrates AI suggestions into daily workflow, still manual application"]
C --> D["Agentic Test Engineer<br/>builds/configures tool-using agents for bounded tasks, e.g. locator healing"]
D --> E["AI QA Agent Engineer<br/>designs multi-step agent workflows, guardrails, escalation policy"]
E --> F["Owns the architecture: planners, tool contracts, evals, observability, policy"]You don't skip stages by reading one article — but you can see, clearly, that every stage is still a quality engineering skill. The tools changed. The judgment requirement didn't.
The Actual Point
Agentic QA doesn't remove the need for a human who understands the product, the risk, and the business — it removes the tedium of manually operating the investigation loop a skilled QA engineer already runs in their head. The framework gets more capable. The judgment stays yours. That's the honest version of this shift, and it's a more interesting career than "write more scripts, faster."
Discussion: If an AI agent could run, debug, modify, and retest your automation today, which action would you trust it to take without human approval?
Top comments (2)
The line you draw between technical judgment and product judgment is the part of this most write-ups skip. Renaming a locator because a rebrand PR merged two hours ago is mechanics. Deciding that an assertion no longer needs to hold is someone's product call, and the moment an agent is allowed to make it, your suite stops measuring what anyone chose.
The failure mode I would name explicitly, because I have lived it: self-healing converges on the path of least resistance. Each individual locator fix is defensible and reviewed, and nobody reviews the sum. Six months later the suite is green and tests a DOM nobody designed, with waits that happen to match whatever the app does today. The metric that caught it for me was not pass rate but assertion churn — how many test expectations an agent modified per week, and what fraction of those touched something a human wrote on purpose. Do you have anything like that dashboard in the pipeline, or is the review still per-PR? The per-PR view is where this drift hides, because each diff looks tiny and correct.
The most important line here is that the agent sits on top of the deterministic test stack rather than replacing it. I would add one more boundary: self-healing should create a proposed patch and rerun against quarantined evidence before it can alter the canonical test. Otherwise a green rerun can mean the agent weakened the test instead of fixing it. Measuring "protected assertions preserved" makes that distinction visible.