From CI failure to verified fix: how AI agents can run, diagnose, repair, and retest automated tests.
Your test fails at 2 AM in the CI pipeline. You wake up, open the logs, scroll past 400 lines of stack trace, squint at a screenshot, and mutter "is this a locator issue or did the button actually move?" Twenty minutes later you find it — a data-testid changed during a refactor. You fix it, push, and go back to your actual work.
If that paragraph felt uncomfortably familiar, you already understand the problem agentic testing is trying to solve.
This isn't another "AI will replace QA" post. It's a practical breakdown — from one SDET to another — of what agentic testing actually is, what it can realistically do today, and where it still needs you.
Why Agentic Testing Is Different Now (Not Just Rebranded RPA)
If you've been in QA for more than a few years, you've seen "smart" testing tools come and go — rule-based self-healing, record-and-playback tools that claimed to "understand" your app, RPA scripts marketed as AI. It's fair to be skeptical that this is just another rebrand.
Here's what's genuinely different this time, in concrete terms:
Tool use, not just chat. Earlier "AI in testing" meant a chatbot you pasted logs into. Current LLMs can call functions directly — run a command, read a file, query a DOM, hit an API — and decide which tool to call based on what they find. That's the mechanical difference between "an assistant you operate" and "an agent that operates tools."
Multi-step reasoning that holds up across steps. Rule-based self-healing followed one fixed heuristic ("find nearest matching element"). Today's models can chain several reasoning steps — read a log, form a hypothesis, check a second source to confirm it, revise the hypothesis — which is closer to how you actually debug, not a single pattern match.
Context windows large enough to hold real evidence. Diagnosing a failure well means holding the test code, the DOM snapshot, the network log, and recent git history in view at once. That was previously impractical; it's now routine.
Cheaper, faster inference. Running this reasoning loop on every CI failure would have been cost-prohibitive a few years ago. It's now realistic to run on a meaningful slice of your failures, not just a curated demo.
To be clear about what this is not: it's not evidence that these systems reason the way humans do, and it's not a claim that failure rates have dropped by some measurable percentage — no credible industry-wide numbers exist yet, and be skeptical of anyone quoting one. What's changed is narrower and more mechanical: agents now have the tool access and reasoning chain-length to make the loop from earlier in this article actually work end-to-end, where before it had to be done manually or with brittle heuristics.
What "Agentic Testing" Actually Means
Strip away the buzzword, and agentic testing is this:
An AI agent that can reason about a testing task, use tools to act on it, observe the results, and decide what to do next — instead of just executing a fixed script.
A traditional test does exactly what you wrote, every time, in the same order. It has no awareness that anything went wrong beyond a pass/fail signal.
An agentic test system behaves more like a junior engineer sitting next to you: it runs the test, notices the failure, opens the logs itself, forms a hypothesis ("this looks like a timing issue, not a real bug"), checks the DOM to confirm, tries an adjusted action, and reports back with reasoning — not just a red X.
The key word is loop. Agentic testing isn't one clever trick (like auto-healing a broken locator). It's a repeatable reasoning loop applied to the whole investigation process.
Five Terms People Keep Mixing Up
This confusion is the single biggest source of AI-testing hype. Let's separate them cleanly.
Junior engineers often think self-healing is agentic testing. It isn't. Self-healing swaps a broken selector; it doesn't ask "did the product actually break, or did my test just get stale?" Agentic testing asks that question — and that's the whole point.
LLM/AI vs AI-Assisted Testing vs Agentic Testing — What Came First, and What Each One Actually Does
This is where most articles blur three very different things into one "AI testing" blob. They're not interchangeable — they're layers, and each one entered your testing life in a different order, doing a different job.
The order they showed up in your actual workflow:
LLM/AI came first — it's just a reasoning engine. It has no idea your test suite exists unless you copy-paste context into it manually. This is where most SDETs already live today: pasting a stack trace into an AI chat and asking "what's likely going on here?"
AI-assisted testing came next — someone wired that same reasoning engine into your actual tooling (IDE plugin, test generator, log summarizer), but a human is still the one clicking "run," reading the suggestion, and deciding what to do with it. The AI never acts on its own.
Agentic testing is the third layer — the same reasoning engine, now given tools it can call itself (run the test, read the DOM, check the network tab, open a diff) and permission to move through the investigate → fix → validate loop without you manually feeding it each piece of context at every step. You still set the boundaries; it works inside them.
The progression is really about who holds the tools and who presses "go" at each step — not about the AI getting "smarter" in some vague sense. Same underlying reasoning capability, increasing levels of tool access and autonomy.
The Agentic Testing Loop
Here's the loop in plain terms — the same one you already run manually when debugging a failure, except an agent runs it as a first pass. Each stage below is tagged with who's actually doing it, so it's clear where the "agentic" part really is:
Where the human sits in this loop: goal-setting at the start (you decide what "correct" means), and approval/audit at the two decision points — Change/Retry and Validate — especially for anything business-critical. Everything in between (Plan → Act → Observe → Diagnose) is where the agent is doing genuinely new work compared to a traditional script: it's reasoning across evidence, not just executing steps.
Notice: Report is not optional, and Validate always exists — a responsible agentic system never silently commits a "fix" without confirming it actually holds up, and never hides its reasoning from the human reviewing it.
A Realistic Example: Login Test Failure
Let's make this concrete instead of theoretical. Say you have a straightforward login test.
test('user can log in with valid credentials', async ({ page }) => {
await page.goto('/login');
await page.fill('#email', 'userk@example.com');
await page.fill('#password', 'Test1234!');
await page.click('#login-submit-btn');
await expect(page.locator('.dashboard-header')).toBeVisible();
});
It fails in CI. Here's what a traditional pipeline gives you:
FAIL: user can log in with valid credentials
Error: Timeout 30000ms exceeded waiting for locator '.dashboard-header'
That's it. You now have to go do the investigation yourself.
What an agentic system does instead
Step 1 — Observe. The agent captures the DOM snapshot at failure time, the network log, and a screenshot instead of just the timeout message.
Step 2 — Diagnose. It compares the captured DOM against the expected state and finds this:
Expected element: .dashboard-header
DOM at failure time contains: .dashboard__header (double underscore, renamed in last deploy)
Network log: POST /api/login → 200 OK, redirect to /dashboard confirmed
Screenshot: Dashboard page IS visible, correctly rendered
Step 3 — Form a hypothesis. Login actually succeeded — the assertion is broken, not the product. The class name changed in a recent CSS refactor (a real PR diff would confirm this, if the agent has repo access).
Step 4 — Propose a change (not silently apply it).
Agent suggestion:
Confidence: High (87%)
Cause: Locator drift (class renamed, unrelated to auth logic)
Suggested fix: update locator to '.dashboard__header'
Action taken: none yet - awaiting human approval (or auto-applied
only if this test is flagged as "low-risk, auto-fixable" in config)
Step 5 — Validate. If approved, it reruns the test against the new locator, confirms a clean pass, and logs the change with a diff — visible in the PR, not buried in a black box.
Step 6 — Report.
SUMMARY
Test: user can log in with valid credentials
Result: Locator drift, not a functional defect
Evidence: network 200 OK, dashboard rendered, class renamed in commit abc123
Fix applied: locator updated, rerun passed 3/3
Human review: recommended (auth-related test, high business risk)
This is the entire value proposition in one example: faster, evidence-backed triage — not unsupervised decision-making.
Compare that to what a self-healing tool alone would have done: it would have just swapped .dashboard-header for the nearest matching element and moved on — no diagnosis, no report explaining why, no distinction made between "cosmetic class rename" and "the dashboard didn't actually load." That distinction is exactly why agentic reasoning matters more than pattern-matching self-healing.
The Full CI Workflow: From Failure to Verified Fix, With a Real Git Diff
The login example above shows the reasoning. This section shows how that reasoning actually plugs into a pipeline you'd run in production — because "the agent diagnoses it" means nothing until you see where it sits relative to your CI, your git history, and your merge gate.
The part worth underlining: the agent never pushes directly to your main branch. Its output is a draft PR with an attached evidence bundle — the same artifact a human contributor would produce, reviewable the same way. This is what makes "agent modifies the test" fundamentally different from "agent silently mutates CI state." If your implementation skips the draft-PR step and lets an agent commit straight to a protected branch, you've removed the one checkpoint that makes this whole workflow trustworthy.
What Agents Can Realistically Do Today
Framed as illustrative capability, not a benchmarked claim:
- Inspect DOM/page state, accessibility tree, and console errors at the moment of failure
- Parse logs, stack traces, and network requests to narrow down a likely cause
- Correlate a failure with a recent code or config change, if given repo/CI access
- Distinguish common failure classes: locator drift, timing/race conditions, stale test data, environment issues
- Propose or apply a scoped fix (locator update, wait strategy, data refresh)
- Rerun the test and compare before/after outcomes
- Produce a structured, evidence-based failure summary instead of a raw stack trace
What Agents Silently Cannot Guarantee
This is the section most AI-testing content skips, and it's the most important one for your credibility as an SDET:
- Business correctness. An agent doesn't know that "checkout total should include a loyalty discount for tier-3 customers" — unless that rule is explicitly encoded somewhere it can see.
- Detecting every functional defect. It's diagnosing why a test failed, not independently discovering bugs the test wasn't written to catch.
- Knowing intended product behavior. If the requirement was never documented or the ticket is vague, the agent has no ground truth to compare against.
- Safely changing tests without masking real defects. An agent that "fixes" a failing assertion by loosening it can accidentally hide a genuine regression. This risk is real and needs guardrails (below).
- Replacing human risk judgment. Deciding whether a payment flow test failure is worth blocking a release is a judgment call about business risk — not a pattern-matching task.
An agent that can fix a test is not the same as an agent that can prove the product is correct. Fixing a broken locator tells you the test is executable again. It tells you nothing about whether the feature behind that locator does what the business needs it to do. Treat every auto-applied fix as "test restored to a runnable state," not "feature verified correct."
Comparison Table: Autonomy, Reasoning, and Risk
Architecture of a Practical Agentic Testing System
The guardrails box is not decoration — it's the component that decides which failures an agent may auto-fix versus which must always stop and wait for a human. Without it, you don't have agentic testing; you have an unsupervised script with extra confidence.
What "Tools" Actually Mean for an Agent (No Magic Here)
"The agent inspects the DOM" sounds like it understands your app the way you do. It doesn't. An agent only has a defined, finite list of functions it's allowed to call — nothing more. If you didn't wire up a tool for it, it cannot do that thing, full stop. This matters because it demystifies the whole system: there's no hidden capability, just a list of functions with a name, inputs, and outputs.
A realistic tool list for a testing agent looks like this:
Tool: run_test(test_name)
→ executes one test, returns pass/fail + exit code
Tool: get_dom_snapshot(page)
→ returns the current HTML/accessibility tree at time of call
Tool: read_logs(run_id)
→ returns console, network, and framework logs for a run
Tool: get_screenshot(run_id)
→ returns a screenshot captured at failure time
Tool: git_diff(since_commit)
→ returns code changes since a given commit/tag
Tool: read_file(path)
→ returns file contents (read-only, scoped to repo paths)
Tool: propose_fix(file, change)
→ does NOT apply the change - stages it for human approval
Tool: apply_fix(file, change) [gated - only callable if
autonomy config allows it
for this specific test]
Each tool is just a function with a schema — the agent picks which one to call based on its reasoning, the same way you'd decide "let me check the network tab" versus "let me check git blame" while debugging. The agent isn't smarter than the tools you gave it access to. If git_diff isn't wired up, it can't correlate a failure with a recent code change — it'll guess instead, and guessing is exactly the failure mode you're trying to avoid. The quality of your agentic testing setup is mostly a function of how good and how scoped your tool list is — not how good the underlying model is.
Notice apply_fix is deliberately gated separately from propose_fix. That split — "the agent can always suggest, but can only act where explicitly permitted" — is the single most important design decision in the whole system, and it's what your instruction file (next section) actually configures.
Hands-On: How to Actually Implement This in Your Framework
Everything above explains the concept. This section is for the engineer who closes this article and asks "okay, but how do I actually set this up on Monday?" Here's a concrete, three-part breakdown.
1. Pick an AI Coding Agent: Claude Code / Codex / Local LLM
You need a "coding agent" — something that can read your repo, run commands, and call tools — not just a chat window. The three realistic options today:
How to actually decide:
- Does your test data or codebase contain sensitive/regulated information? → Local LLM, or a vendor with a data-retention agreement you've verified.
- Do you need strong reasoning over messy logs and DOM diffs? → Hosted agent (Claude Code or Codex) generally outperforms smaller local models on this kind of multi-step reasoning today.
- Is budget the constraint, not data sensitivity? → Start with whichever hosted agent your team already pays for elsewhere — don't add a second subscription just for this.
Whichever you pick, verify the current subscription tiers and rate limits directly on the provider's docs before committing a budget line — pricing and included usage change frequently.
2. The Instruction/Agent File — and Its Lifecycle
This is the single most important artifact in the whole setup. It's a file (commonly named AGENTS.md, CLAUDE.md, or a system-prompt config) that tells the agent: what its job is, what tools it may use, what it must never do without approval, and what format to report in.
A minimal version looks like this:
# Test Agent Instructions
## Role
You are a test-failure triage agent for the checkout and login test suites.
## You MAY:
- Read test code, logs, DOM snapshots, and network traces
- Rerun a failing test up to 2 times
- Propose locator/wait/data fixes with a confidence score
## You MUST NOT:
- Modify test files without an approval flag set to true
- Touch tests tagged `payment-critical` or `compliance` without human sign-off
- Loosen an assertion without explicitly flagging it as a "risk: masking possible defect"
## Report format
Always output: cause, evidence, confidence %, suggested fix, risk flag.
You don't write this once and forget it — it has a lifecycle, the same way test code does:
Treat step 6 as non-negotiable. Every time an agent gets something wrong — masks a real bug, misdiagnoses a cause — that's a signal to update the instruction file, not just to override the output once and move on. Version-control this file the same way you version-control test code, with PR review on changes.
3. Multi-Agent Prompt Setup: Splitting the Work Across Specialized Agents
A single "do everything" agent gets unreliable fast. In practice, this works better as a small team of narrowly-scoped agents that hand off to each other — similar to how you'd split responsibilities across QA roles on a real team.
Illustrative role prompts for each agent (adapt scope/tools to your stack):
A few more worth adding as your setup matures:
- Flaky Test Detector — tracks pass/fail history over time and flags tests that fail intermittently regardless of code changes, so they get fixed at the root instead of being repeatedly "healed."
- Test Data Manager — checks whether a failure traces back to stale/expired test fixtures or seed data, a very common false-positive source that's easy to under-diagnose.
- Regression Risk Scorer — cross-references a failing test against recent deploys/PRs to estimate whether this failure correlates with a specific change, speeding up root-cause attribution.
Keep each agent's prompt scoped to one job. The temptation is to write one giant prompt that does analysis, scripting, healing, and escalation together — resist it. Narrow scope is what makes each agent's output reviewable, and reviewability is what makes the whole system trustworthy enough to actually put in CI.
4. A First Experiment You Can Actually Run This Week
Don't start by wiring all five agents into CI. Start with one test and one narrow question. Here's a scoped experiment using a Playwright suite you already have:
- Pick one flaky or recently-broken test — not a payment or auth test. A low-risk UI check is ideal for a first run.
- Manually gather the evidence bundle the way an agent would: the failure's stack trace, a DOM snapshot at failure time (
page.content()in Playwright), the relevant network log, and git diff since the last known-green commit. - Paste all of it into a single prompt to whichever AI coding agent you chose in step 1 (Claude Code, Codex, or a local model), something close to:
Here is a failing Playwright test, its stack trace, a DOM snapshot at failure time,
and the git diff since the last passing run. Classify the failure as: locator drift,
timing issue, data issue, or likely real defect. State your confidence and cite the
specific evidence you used. Do not modify any files - diagnosis only.
- Judge the output yourself — not on whether it "sounds smart," but on whether its cited evidence actually supports its conclusion. This is the reviewer skill from the "What This Means for SDETs" section, and this experiment is where you practice it.
- Only if the diagnosis holds up, take the second step: ask it to propose (not apply) a minimal fix, and manually verify that fix against a rerun yourself.
- Log what happened — was the diagnosis right, wrong, or partially right? That log is the seed of your instruction file's guardrails from the lifecycle diagram earlier in this section.
This costs you one afternoon, no CI changes, and no new tooling investment — and it tells you, concretely, whether your test suite and your chosen agent are actually a good match before you build any automation around it. If the diagnosis is consistently solid on your first five attempts, you have a real candidate for the CI-integrated workflow above. If it's shaky, that's useful information too — it tells you your logs/evidence capture needs work before an agent (or a human) can diagnose reliably from them.
What This Means for SDETs
For a junior engineer, this is genuinely good news day-to-day: instead of spending 40 minutes tracing why a test failed, you review a structured summary and decide in 5 minutes whether the fix is sound. But that 5 minutes requires you to actually understand why the agent reached its conclusion — you're becoming a reviewer of reasoning, not just a writer of scripts. That's a skill you build the same way you built debugging skill: by doing it, deliberately, on real failures.
5-Minute Checklist: Is This Test a Good Candidate for Agentic Testing?
Use this before handing any test suite to an agentic workflow:
- Is the failure mode usually mechanical (locators, timing, stale data) rather than deep business logic?
- Does the test have clear, deterministic pass/fail criteria (not a fuzzy or flaky assertion)?
- Is there low-to-medium business risk if a wrong auto-fix temporarily masks a regression?
- Do you have log/DOM/network access rich enough for the agent to actually diagnose, not guess?
- Is there an audit trail requirement you can satisfy (every change logged, reviewable)?
- Can you define a human approval gate for anything touching payments, auth, or compliance-sensitive flows?
Score 5–6 yes: good candidate for agent-assisted auto-fix with light review.
Score 3–4 yes: good candidate for agent-assisted diagnosis only, human applies the fix.
Score 0–2 yes: keep this test fully manual — the risk of a silent wrong fix outweighs the time saved.
Decision Matrix: When NOT to Use Agentic Testing
A Realistic Adoption Path
Nobody should go from "manual debugging" to "fully autonomous agent commits fixes to main" in one step. A sane rollout looks like this:
- Start AI-assisted. Use AI to summarize failures and suggest causes; a human still fixes everything.
- Automate the repetitive investigation. Let the agent gather DOM/logs/network evidence automatically, even if a human still decides the fix.
- Introduce controlled agents on low-risk suites. Smoke tests, non-critical UI checks — auto-fix with logged rationale, human spot-checks.
- Add evaluation and guardrails. Define autonomy scopes, blocked test lists, and confidence thresholds below which the agent must always escalate.
- Integrate with CI/CD. Agent runs as part of the pipeline, gated by the guardrails from step 4, not bypassing them.
- Measure outcomes. Track false-fix rate, time-to-triage, and — critically — how often a "fix" later correlated with a real regression that got masked. Adjust scope based on that data, not on vibes.
The Takeaway
Agentic testing is a real shift in how the debugging half of QA work gets done — not because AI is infallible, but because a structured reasoning loop over logs, DOM state, and code history can triage mechanical failures faster than a human doing it manually every time.
It does not replace the parts of QA that were never about typing test steps in the first place: understanding what "correct" means for the business, deciding what risk is acceptable to ship, and knowing when a green checkmark is lying to you.
Treat agentic testing as augmentation of your engineering judgment, not a substitute for it. The agents get faster at the loop. The judgment about when to trust the loop — that's still yours.
Would you trust an AI agent to debug, modify, and retest your automation without human approval? Where would you draw the line?
Drop your take below — especially if you've drawn that line somewhere different from where you expected to.
Top comments (0)