DEV Community: Pranta Kundu

Race Conditions Don't Happen Once. They Return With Every New Feature.

Pranta Kundu — Tue, 29 Sep 2026 06:24:00 +0000

Your feature passed every test. Then two users clicked at the same moment. Why concurrency bugs keep coming back, and what developers and QA should each do about it.

The Feature Worked. The System Failed.

Here is a story I have seen in many forms. (This is a hypothetical scenario built from common real-world patterns.)

A team ships a "last item in stock" feature. It passes functional testing, API testing, automation, regression, and staging. QA signs off.

On the first busy evening, two customers press "Buy" within a few milliseconds of each other. Both get a success message. The warehouse has one item and two orders.

The feature worked. The system failed under concurrency.

Next sprint, a coupon limit ships with the same bug: one coupon, used twice. Then a wallet balance. Then a booking slot. Different features, same root problem.

A race condition is not a bug you fix once. It returns whenever a new feature touches shared state.

Let's take it in order: when concurrent work happens, how it becomes a race condition, what database and design decisions create the risk, and what developers and QA should each do.

Part 1: When Does Concurrent Work Happen?

Concurrent work means more than one thing is happening at the same time on the same system. This is normal. It is not a bug. Every real product is concurrent all day.

A simple picture: two people stand at two different ATMs and withdraw money from the same joint account at the same moment. The bank must handle both correctly. That is concurrency, and the shared account is the "shared state."

Real situations in software:

Two users, one resource. Two customers buy the last item. Two agents book the same seat.
One user, two requests. A user double-clicks "Pay," or a mobile app retries after a slow network. The server receives the same action twice.
Two admins, one record. Two people open the same order and both change its status.
User and background job. A customer cancels an order while a scheduled job is shipping it.
Two workers, one event. A queue delivers the same message and two workers process it.
Many servers, one database. Production runs several copies of your app, all writing to the same rows.
A slow job that overlaps itself. A nightly job runs longer than expected and the next run starts before it ends.

None of these needs a huge traffic spike. Two requests are enough. Concurrency is not only a performance topic. It is a correctness topic.

Concurrent work is normal. It becomes a problem only when it touches shared state without protection.

Part 2: How Concurrent Work Becomes a Race Condition

A race condition happens when the result depends on which request finishes first, or on how the steps of two requests interleave.

Three ingredients must be present:

Shared state: something both requests read or change (stock, balance, status).
Overlapping timing: both requests are in progress at the same time.
A gap between checking and changing: the code reads a value, decides, and writes later, with no protection in between.

Remove any one ingredient and the race disappears.

Example 1: Overselling the last item

Stock is 1. The code does this:

Step 1 → Read the stock (returns 1)
Step 2 → Check: is stock greater than 0? Yes, continue
Step 3 → Update stock to 0
Step 4 → Create the order

Request A and Request B both finish Step 1 before either reaches Step 3. Both see stock = 1. Both pass the check. Both create an order.

sequenceDiagram
    participant A as Request A
    participant DB as Database (stock = 1)
    participant B as Request B

    A->>DB: Read stock
    DB-->>A: stock = 1
    B->>DB: Read stock
    DB-->>B: stock = 1
    Note over A,B: Both pass the "stock > 0" check
    A->>DB: Update stock = 0
    A->>DB: Create order A
    B->>DB: Update stock = 0
    B->>DB: Create order B
    Note over DB: Result: stock = 0, but 2 orders exist for 1 item

The time between the read (Step 1) and the write (Step 3) is called the race window. It may last only a few milliseconds, but production traffic will find it.

Example 2: The lost update

A wallet has a balance of 100. Two deposits of 50 arrive together.

Request A reads 100 → Request B reads 100 → A writes 150 → B writes 150

The customer should have 200. They have 150. No error was shown. A lost update is dangerous because it fails silently.

Example 3: Invalid state

An order is PAID. A customer cancels while the warehouse marks it SHIPPED. Both updates succeed. The order ends as CANCELLED, but the package is already on a truck. Each update was valid alone. The combination is not.

Example 4: Duplicate processing

A payment request times out. The client retries. But the first request was still running and finished. The customer is charged twice.

Why is it so hard to catch?

It depends on exact timing, so it is intermittent.
Sequential tests never create the overlap.
It often passes in staging, where traffic is low.
It leaves quiet damage (wrong numbers, duplicates, bad state), not crashes.

Why does it keep returning?

Each new feature has its own code path and its own read-check-write. The fix for the stock feature protects only the stock feature. The coupon feature starts with no protection, because nobody asked the concurrency question for it.

Every feature that changes shared state should trigger one question: what happens if two actors do this at the same time?

Part 3: How It Arises: Database and System Design Concerns

A race condition is created in two places: the database layer and the system design.

Database concerns

1. Read-modify-write outside a safe boundary. The most common cause. The read and the write are separate steps with a gap between them.

2. Transaction boundary in the wrong place. If the read is outside the transaction and the write is inside, the gap is still open. A transaction that is too wide also hurts: it holds locks longer and causes waiting and deadlocks.

3. Isolation level. Isolation controls what one running transaction can see of another. Different levels allow different anomalies. A test that passes on one setting can fail on another. Check that your test database uses the same isolation setting as production.

4. Missing constraints. If a rule matters ("one coupon per user," "one active booking per slot"), the database should enforce it with a unique constraint. Application code can have bugs or forgotten code paths. A constraint cannot be bypassed.

5. Locking choices. Row-level locks block other writers on the same row. They protect data but make requests wait. Two transactions can also lock each other, causing a deadlock, where the database kills one of them.

6. Optimistic vs pessimistic locking.

Pessimistic: lock first, work after. Safe, but slower under contention.
Optimistic: no lock, but a version number. If the version changed, the update is rejected. Fast, but the app must handle the conflict error.

System design concerns

A request travels through several layers, and each one carries its own concurrency risk:

flowchart LR
    U[User] --> F[Frontend]
    F --> G[API Gateway]
    G --> AI["App instances<br/>(multiple copies)"]
    AI --> BL[Business logic]
    BL --> DB[(Database)]
    BL --> Q["Queue & workers"]
    BL --> EXT[External service]

1. Multiple app instances. A lock in one server's memory does nothing for the other servers. Protection must live in shared places, like the database or a shared lock service.

2. Queues that deliver at least once. Many message systems can deliver the same message more than once. Consumers must be safe to run twice.

3. Retries and timeouts. When a call times out, the caller does not know if the work happened. A blind retry can duplicate it.

4. Missing idempotency. Idempotent means doing something twice has the same effect as doing it once. Without an idempotency key or a natural unique rule, the system cannot recognize a repeated request.

5. Cache and database out of sync. If the cache says 5 in stock and the database says 0, a decision from the cache is wrong. That is a race between two copies of the truth.

6. Async flows and external services. A payment provider or warehouse system adds delay, and delay widens race windows.

7. Unclear source of truth. If two services can both change the same data, races between services are very hard to prevent.

Part 4: Developer Concerns

Developers own the implementation. Here is what I would expect, and what I would gladly review.

1. Write down the rule first. "Stock can never go below zero." "One coupon per user." If the rule is not clear, nobody can protect it.

2. Make the check and the change one step. For the stock example, do it in one atomic database statement:

UPDATE products
SET stock = stock - 1
WHERE id = 101 AND stock > 0;

Then check how many rows were updated. One row means success. Zero rows means someone else got it, so return a clear "out of stock" response.

3. Choose the right control for the problem. There is no universal answer. Common options:

Atomic update: best for counters and stock.
Row lock (pessimistic): useful when several steps must happen together.

  BEGIN;
  SELECT * FROM products WHERE id = 101 FOR UPDATE;
  -- check stock, then update
  UPDATE products SET stock = stock - 1 WHERE id = 101;
  COMMIT;

Optimistic version check: useful for records people edit.

  UPDATE orders
  SET status = 'SHIPPED', version = version + 1
  WHERE id = 55 AND version = 7;

If zero rows changed, someone else edited first.

Unique constraint plus idempotency key: best for "do this only once." A repeat request fails on the unique index, and you return the first result.
Serialize through a queue: all changes for one item go through one ordered path.

The right choice depends on your database, transaction boundary, business rule, isolation level, traffic, and how strict consistency must be. One SQL pattern does not solve every case.

4. Do not rely on in-memory locks in a system with multiple instances.

5. Handle conflict errors kindly. A losing request should get a clear response, not a 500 error. Decide when it is safe to retry.

6. Keep transactions short. Long transactions hold locks and cause waiting and deadlocks.

7. Make consumers and retries safe. Any handler that a queue or client may call twice must be idempotent.

8. Add request IDs to logs. When something breaks at 2 a.m., correlation IDs are how you reconstruct what happened.

9. Tell QA where the risk is. A short note in the pull request, like "this updates stock inside a transaction using X," helps QA target tests.

Part 5: QA Concerns

This is where I spend most of my time. The SDET does not have to write the locking code. The SDET has to notice the risk, design the tests, and check the truth in the data.

Step 1: Spot the risk before testing

The most valuable sentence an SDET can say in grooming:

"This feature changes shared state, so concurrency is part of its risk profile."

Use this checklist for every feature.

Shared state

Do multiple users touch the same resource?
Is stock, balance, or status involved?

Parallel requests

Can two requests arrive at nearly the same time?
Can a user double-click, or a client retry?

Async processing

Are queues or workers involved?
Can a job run twice?

Database

Is there read → modify → write logic?
Which constraint protects the rule?

External dependencies

What if a payment call times out? Can the client retry?

Distributed system

Are there multiple instances? Is a cache involved?

Several "yes" or "not sure" answers mean the feature needs concurrency tests.

Step 2: Test in four levels

Most teams stop at level 2.

Level 1, Functional: Request A → Success
Level 2, Sequential: Request A completes → then Request B completes
Level 3, Concurrent: Request A and Request B hit the API at the same moment
Level 4, High concurrency: 10 users → 50 requests → 100 requests → 500 requests

Measure more than the status code: correctness, duplicate records, lost updates, wrong state, transaction failures, deadlocks, timeouts, retry behavior, error rate, response time, and throughput.

A concurrency test is not passed because the API returned HTTP 200. Two 200 responses for one item is a failure. The final database state and business outcome are the real result.

Step 3: A practical test design

Scenario: only one user can buy the final item.

Initial state: Stock = 1
Concurrent requests: User A and User B both send POST /orders
Expected: one request succeeds, one fails gracefully (for example, "out of stock")
Database: stock = 0, exactly 1 successful order

There must be no negative stock, no duplicate reservation, no inconsistent order status, and no double payment.

How to run it: write a small script (JavaScript or Python) that fires 20 requests at the same instant using a shared start time, and prints every status code. You do not need a full load-testing platform for a first reproduction.

Then verify the truth in the database:

Query the stock for the product. It must be 0, never negative.
Count the successful orders for the product. It must be exactly 1.
Group orders by user and look for any user with more than one.

Check five things:

API response: one success, the rest handled cleanly, no 500 errors
Database state: stock is 0, never negative
Queue or event state: one event published, not two
Logs: no deadlocks or unexpected exceptions
Business outcome: one order, one charge, one shipment

Step 4: Reproduce it reliably

Race conditions are intermittent, so make them appear on purpose:

Start all requests at the same instant
Use more parallel workers
Repeat the test many times
Add a small artificial delay between the read and the write in a test build, to widen the race window
Hit the same row from many requests
Simulate retries and network delay
Reset test data to a known state before each run
Tag every request with a unique ID for log tracing

Flaky test or race condition?

Run once → Pass. Run 100 times → Failure. This does not automatically mean the test is flaky.

A flaky test is unreliable because of the test itself: bad waits, shared test data, an unstable environment.
A race condition is real product behavior that depends on timing. The test is reliable and is telling you the truth.

To tell them apart, fix the test setup (isolated data, deterministic waits) and rerun. If it still fails under concurrency, it is the product. Never label a concurrency failure "flaky, ignore." That is how these bugs reach production.

Step 5: Use tools by purpose

To generate concurrent requests

A small custom script is often best for the first reproduction
k6 or JMeter when you need many virtual users and metrics
Postman/Newman or Playwright API tests when your suite already exists

To investigate the database

Lock and transaction views for waiting queries
Query, slow query, and deadlock logs
Direct queries comparing final state to expected state

To investigate the application

Request and correlation IDs
Timestamps in logs
Distributed tracing, if available
Queue dashboards and worker logs

In CI/CD
A small concurrency check can run before release. Even 20 concurrent requests that verify final database state can catch a broken feature on every deployment.

Step 6: Follow a feature-level workflow

flowchart TD
    A[New feature] --> B[Identify shared state]
    B --> C[Identify concurrent actors]
    C --> D[Identify race window]
    D --> E[Review system design]
    E --> F[Review database & transaction strategy]
    F --> G[Create concurrent test]
    G --> H[Run controlled reproduction]
    H --> I[Increase concurrency]
    I --> J[Verify API, database, and business state]
    J --> K[Analyze logs and tracing]
    K --> L[Regression]
    L --> M[Release decision]

The first four steps happen before any test is written. Concurrency thinking is a design-time activity. If you start at "create concurrent test," you are already late.

Step 7: Ask these in feature grooming

What shared state does this feature modify?
Can two actors modify it at the same time?
What prevents duplicate processing?
Is this operation idempotent?
Where does the transaction start and end?
What happens if the request times out after the server has processed it?
Can the queue deliver the same message twice?
Which database constraint protects this business rule?
What happens when two workers pick up the same entity?
How can QA reproduce the expected concurrency behavior?

If nobody can answer question 10, you have found a testability problem. Better to find it now than after release.

Step 8: Release readiness

Before release, a feature that changes shared state should have answers to these:

What happens when two requests arrive at once?
What happens when the same request is retried?
What happens when a worker processes an event twice?
What happens when an update conflicts?
What happens when an external service times out and the client retries?
Is the operation idempotent where it needs to be?
Can invalid state or duplicate records exist?
Is the final database state correct?
Are failures visible in logs or alerts?

Release readiness is not only "does the happy path work?" It is "does the system keep its business rules true when concurrency goes wrong?"

Be honest about where this effort is not worth it. A single-user internal tool, data written by only one process, or a read-only feature does not need the same depth. Match the effort to the risk.

Who Owns What

Nobody owns this alone. It is built across the chain:

Product: states the business rule clearly
Architect: decides the source of truth and how requests are ordered
Developer: implements the control and documents the transaction boundary
Database engineer: reviews constraints, indexes, isolation, and locking
QA / SDET: spots the risk, designs the concurrent tests, verifies final data
DevOps / SRE: makes failures visible with logs, metrics, alerts, and correct retry and timeout settings

Closing

Every new feature is a new opportunity for a concurrency bug.

That is why QA should ask two questions, not one. The first is the one we always ask: "Does this feature work?" The second is the one that catches these bugs: "What happens when multiple actors try to change the same thing at the same time?"

Good SDET work is not only proving that the system works. It is discovering the conditions under which the system stops being correct, and making those conditions visible before production does.

What was the last concurrency bug your team shipped, and which check would have caught it earlier? I'd like to compare notes in the comments.

From Automation Frameworks to Agentic QA: What Changes When AI Can Take Action?

Pranta Kundu — Mon, 14 Sep 2026 18:12:24 +0000

A Staff SDET's field guide to the architectural shift — for juniors who want to actually understand it, not just hear the buzzword.

Your test fails at 2 AM. The pipeline goes red. A human wakes up, opens the trace viewer, stares at a screenshot, greps the logs, guesses it's a selector change, fixes it, reruns, goes back to sleep.

Now imagine the pipeline doesn't wait for that human. It looks at the failure itself, forms a hypothesis, checks the DOM, checks the API response, decides whether the app broke or the test broke, patches the test if it's safe to, reruns it, and only wakes a human up if it's not sure.

That's not "AI will replace testers." That's a new layer sitting on top of the framework you already built. This article is about exactly what that layer is made of, what it can and can't be trusted with, and what you — as a QA engineer — need to learn next.

No hype. No "70% faster" stats pulled from nowhere. Just architecture, trade-offs, and one worked example you can map onto your own codebase.

1. The Shift in One Picture

Traditional automation loop:

flowchart LR
    A[Write] --> B[Run]
    B --> C[Fail]
    C --> D["Human investigates"]
    D --> E[Fix]
    E --> F[Rerun]

Every arrow after "Fail" requires a person. The framework can tell you that something broke. It cannot tell you why, and it definitely can't decide what to do about it.

Agentic QA loop:

flowchart LR
    A[Goal] --> B[Plan]
    B --> C[Act]
    C --> D[Observe]
    D --> E[Diagnose]
    E --> F[Modify]
    F --> G[Retest]
    G --> H[Report]
    G -.loop until confident or escalated.-> C

The difference isn't "more AI." It's that the system now has a goal instead of a script, and a loop instead of a dead end. A traditional test is a fixed sequence of steps someone wrote in advance. An agent is given an objective — "verify checkout completes for a logged-in user" — and decides, step by step, how to pursue it, using the actual state of the application in front of it.

Keep that distinction in your head for the rest of this article: script = predetermined steps. agent = goal + reasoning + tools + feedback loop.

2. What a Traditional Automation Framework Actually Does

Before touching agents, it's worth being precise about what you already have, because the agent doesn't replace any of this — it sits on top of it.

A mature framework is a stack of responsibilities:

Layer	Responsibility	Example
Test code	Encodes expected behavior as assertions	`expect(page.locator('#total')).toHaveText('$49.99')`
Page Objects / Components	Abstract UI structure from test logic	`CheckoutPage.submitOrder()`
Utilities	Shared helpers (auth, data setup, waits)	`loginAs(user)`
Test data	Inputs and fixtures	JSON/DB seed data, factories
Assertions	Pass/fail decision logic	Playwright's web-first `expect`
Configuration	Environments, browsers, retries	`playwright.config.ts`
Reporting	Human-readable results	HTML report, Allure, trace viewer
CI/CD	Trigger, execute, gate	GitHub Actions, Jenkins

This stack is deterministic by design. That determinism is a feature — it's why automation is trustworthy. An agent doesn't remove any of these layers. It adds a reasoning layer that can operate these layers instead of a human doing it manually when things go wrong.

3. What Changes When an Agent Enters the Architecture

Here's the honest list of new capabilities — and new problems — that show up the moment you add an agent.

Reasoning — the system forms a hypothesis ("this looks like a timing issue, not a locator change") instead of just reporting a stack trace.
Tool use — the agent can call real tools: run a test, read a log file, query an API, inspect the DOM, open a git diff — not just generate text about them.
Context — it needs to know the application's structure, the test's intent, recent commits, and prior failures, not just the current error.
Observation — it reads back the result of its own actions (a screenshot, an accessibility snapshot, an HTTP status) and updates its plan.
State — it tracks what it has already tried across a multi-step investigation, instead of starting fresh each time.
Decision-making — it chooses between several next actions (retry / inspect further / modify code / escalate) based on confidence, not a fixed script.
Controlled action — and this is the one that actually matters — it can change things: rerun a suite, edit a locator, open a PR, post to Slack. This is what separates "AI-assisted" from "agentic," and it's exactly where guardrails become non-negotiable.

That last point is the whole ballgame. An LLM that suggests a fix in a chat window is low risk. An agent that applies the fix, commits it, and reruns the pipeline is a different risk category entirely — even if the underlying model is identical.

4. Traditional vs. AI-Assisted vs. Agentic vs. Self-Healing — The Comparison Table

This is the table people usually get wrong, because "self-healing" and "agentic" get used interchangeably. They are not the same thing.

Dimension	Traditional Automation	AI-Assisted Automation	Self-Healing Automation	Agentic QA
Autonomy	None — runs exactly what's written	None — suggests, human applies	Narrow — auto-fixes locators only	Broad — plans and executes multi-step investigation
Decision-making	Fixed pass/fail assertion	Human decides after AI suggestion	Rule-based similarity match (e.g., nearest DOM match)	Model reasons over observed state to choose next action
Tool usage	Test runner only	None (chat/IDE suggestion)	Internal locator-matching heuristic	Multiple tools: browser, runner, logs, API, source code, git
Debugging	Human reads logs/trace	AI explains logs, human interprets	N/A — doesn't debug, just adapts selectors	Agent inspects logs, DOM, network, diffs to form root cause
Retesting	Manual rerun	Manual rerun	Automatic (same test, healed locator)	Agent decides if/how to rerun, possibly with modified test
Risk profile	Low — deterministic, predictable	Low — advisory only	Medium — can mask a real UI regression as a "fix"	Medium–High — can take unreviewed action if unguarded
Human involvement	Full ownership of triage & fix	Full ownership, AI as assistant	Configuration + periodic audit	Approval gates, escalation review, strategy decisions
Best for	Stable, well-understood flows	Speeding up authoring/debugging	Cosmetic UI drift (renamed classes, reordered DOM)	Ambiguous failures needing multi-step investigation

Trade-off to internalize: self-healing automation solves a narrow, well-bounded problem (locator drift) with a narrow, well-bounded mechanism (similarity matching). Agentic QA solves a much wider problem (arbitrary failure investigation) with a much wider mechanism (an LLM planning over tools) — which means wider power and wider blast radius. Don't reach for an agent to solve what self-healing already solves cheaply and safely.

5. One Failure, Three Ways to Handle It — Login Workflow

Scenario: your CI run just failed on should log in with valid credentials. Here's how each generation of automation handles it, with actual JS/Playwright code.

The test:

// login.spec.js
import { test, expect } from '@playwright/test';

test('should log in with valid credentials', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('user@example.com');
  await page.getByLabel('Password').fill('Secret123!');
  await page.getByRole('button', { name: 'Sign in' }).click();
  await expect(page.getByTestId('dashboard-welcome')).toBeVisible();
});

It fails with: TimeoutError: element not found — [data-testid="dashboard-welcome"]

A) Traditional Automation

The pipeline goes red. That's it. The framework's job ends at "assertion failed." A human opens the HTML report, downloads the trace, scrubs through screenshots, and manually decides: is this a real regression, a flaky wait, or a renamed test-id? Time to resolution depends entirely on who's on call and how fast they can context-switch.

B) AI-Assisted Automation

Same failure, but now an engineer pastes the stack trace and screenshot into a chat interface. The model reads the trace and says something like: "The welcome banner's data-testid may have changed — check the latest frontend diff for the dashboard component." Useful. Faster than manual grepping. But the human still opens the repo, checks the diff, decides, and edits the test by hand. The AI never touched the pipeline.

C) Agentic QA Workflow

The agent is wired into the pipeline with tool access. On failure it:

Observes — pulls the Playwright trace, screenshot, and accessibility snapshot at the point of failure.
Diagnoses — queries the app's current DOM via a Playwright MCP-style tool call and compares it against the expected data-testid. Finds dashboard-welcome-v2 now exists where dashboard-welcome used to.
Checks intent — pulls the last few commits touching the dashboard component to see if this was an intentional rename (it was — part of a rebrand PR merged two hours ago).
Decides — this is a cosmetic, intentional rename, not a functional regression. Confidence is high enough to act.
Acts — updates the locator in the test file, opens a draft PR (does not merge it), and reruns the test against the fix.
Reports — posts to the team channel: "login.spec.js failed due to a test-id rename in PR #482 (dashboard rebrand). Draft fix opened in PR #611, test now passing locally. Awaiting review."

Notice what the agent did not do: it did not merge its own fix, it did not silently loosen the assertion, and it did not decide on its own that the rebrand was "fine" from a product standpoint — it only judged that the locator mismatch was cosmetic. That distinction — technical judgment vs. product judgment — is exactly the line covered in Section 8.

6. Practical Architecture: How the Pieces Actually Fit Together

Why each box matters:

Tools, not vision: In 2026, the dominant pattern for browser-based agents is reading a structured accessibility snapshot rather than a screenshot — Microsoft's open-source Playwright MCP server exposes dozens of browser-control tools this way, which makes agent actions deterministic and auditable instead of vision-model guesswork.
Observe is not optional: an agent that acts without reading back the result of its action is just automation with extra steps and extra risk. The observe step is what turns "action" into a controlled feedback loop instead of a blind write.
Evaluate is the guardrail chokepoint: this is where confidence thresholds, allow-lists of permitted actions, and human-approval gates live. Everything downstream of "Evaluate" should be scoped by policy, not by the model's own judgment alone.
Escalate is a feature, not a failure: a good agent architecture treats "I'm not sure, ask a human" as a successful outcome, not a fallback of last resort. Teams that punish escalation (by treating it as "the AI didn't work") end up training the system, implicitly, to guess more confidently — which is the opposite of what you want.

7. What "Taking Action" Actually Means

"Agentic" gets thrown around loosely. Concretely, action means the agent can do some subset of the following, through real tool calls:

Running tests — triggering a specific spec or suite, not just reading results.
Inspecting failures — pulling traces, console logs, network waterfalls, accessibility snapshots.
Navigating the application — clicking through the live app to check whether a flow still works, independent of the failing test's script.
Collecting artifacts — screenshots, HAR files, DB state snapshots, timestamps — building an evidence trail, not just a verdict.
Modifying test code — updating a locator, adjusting a wait strategy, adding a missing await — scoped to test code, ideally as a draft PR, never a direct merge.
Rerunning tests — after a change, to confirm the fix actually resolves the failure rather than just silencing it.
Escalating uncertain failures — routing to a human with a structured summary when confidence is low or the change touches assertions/business logic.

A simplified version of a modify-and-retest tool call, using a Playwright MCP–style setup, looks like this conceptually:

// agent-tools.js — simplified illustrative tool wrapper
// (structure mirrors how an MCP-based agent would call Playwright tools)

async function inspectFailure(page, testId) {
  const snapshot = await page.accessibility.snapshot();
  const trace = await page.context().tracing.stop({ path: `traces/${testId}.zip` });
  return { snapshot, trace };
}

async function proposeLocatorFix(oldLocator, domSnapshot) {
  // agent reasoning step happens here (LLM call, not shown),
  // returns a candidate replacement based on structural similarity
  return findClosestMatch(oldLocator, domSnapshot);
}

async function applyFixAsDraftPR(filePath, oldLocator, newLocator) {
  // scoped to test files only — never application source code
  await git.checkoutBranch('agent/fix-login-locator');
  await editFile(filePath, oldLocator, newLocator);
  await git.commit('fix(test): update stale locator after rebrand (agent-proposed)');
  await git.openDraftPR({ reviewers: ['qa-team'] });
}

async function rerunAndConfirm(specFile) {
  const result = await runPlaywright(specFile);
  return result.status === 'passed';
}

Notice the shape: every action is narrow, logged, and reversible. That's not incidental — it's the actual design requirement for any agent you let near your pipeline.

8. What Agents Still Cannot Reliably Determine

This is the section junior engineers should bookmark. An agent can be extremely good at technical diagnosis and still be the wrong entity to make these calls:

Business intent — whether a behavior change is "expected" often depends on a roadmap conversation, a support ticket, or a product decision the agent has no visibility into.
Acceptable risk — a flaky test on a marketing page and a flaky test on a payment confirmation step are not the same risk, even if the failure signature looks identical.
Whether a test should exist at all — an agent can tell you a test is failing; it can't reliably tell you the test is testing the wrong thing, or that a whole flow is no longer worth covering.
Whether changing an assertion masks a real defect — this is the single most dangerous action an agent can take unsupervised. Loosening toHaveText('$49.99') to toBeVisible() makes the test pass. It does not mean checkout still charges the right amount.
Whether a passing test provides meaningful coverage — a green suite full of tests that got quietly "healed" into shallow assertions is a worse state than a red suite, because it's a false signal that's actively hiding risk.

Staff-level framing: the job isn't shrinking, it's moving. The scarce skill is no longer writing automation steps — it's engineering reliable systems that can reason and act around automation, and knowing exactly where to put the fence around them. That's a systems-design skill, not a scripting skill, and it's a more senior skill than what most of us started our careers doing.

9. What SDETs Need to Learn Next

Area	Why it matters	Where to start
Agent fundamentals	Understand planner/reasoning loops, not just prompts	Study ReAct-style agent loops, ADK/agent-framework docs
Tool calling	Agents are only as good as the tools they can invoke	Practice writing well-scoped, single-purpose tool functions
MCP (Model Context Protocol)	The emerging standard for exposing tools/context to LLMs	Read the MCP spec; try Microsoft's Playwright MCP server locally
Browser automation internals	You need to know what the agent is actually calling	Deepen Playwright fundamentals — accessibility tree, tracing, network interception
Context & state management	Agents fail silently when context is stale or too large	Learn context-window budgeting, memory/session design
Agent evaluation	"It worked once" isn't evidence	Learn eval harnesses — golden datasets of known failures, pass/fail scoring for agent decisions
Observability & tracing	You'll debug the agent, not just the app	OpenTelemetry basics, structured logging for multi-step agent runs
Guardrails & policy	This is the actual engineering discipline here	Allow-lists, approval gates, sandboxed branches, rollback design
CI/CD integration	Agents live in pipelines, not just chat windows	GitHub Actions/Jenkins integration patterns, draft-PR workflows
Test strategy	The highest-leverage skill, and the one AI can't do for you	Risk-based test design, coverage strategy, what not to automate

10. What Not to Automate With an Agent

Concrete no-go list, not a vague warning:

Direct writes to production data or production environments — an agent's "checkout works" investigation should never run against real customer accounts or real payment rails.
Merging its own pull requests — draft only, human-reviewed, always.
Deleting or disabling failing tests to make CI green — this is the agentic equivalent of .skip()-ing everything, and it's worse when it's automated because it happens silently and at scale.
Loosening assertions on regulated or financial flows (pricing, tax calculation, payment confirmation) without a named human sign-off.
Security- and auth-sensitive test paths — session handling, permission boundaries, PII fields — where a "plausible fix" can quietly create a real vulnerability.
Irreversible destructive actions — dropping test databases, rotating shared credentials, force-pushing to shared branches.
Anything where "looks the same" isn't good enough — visual-only comparisons on financial documents, legal text, or compliance-critical copy, where subtle wording changes matter more than layout.

Rule of thumb: if a wrong decision here is expensive, irreversible, or invisible until a customer hits it — keep a human explicitly in the loop, not just "available."

11. Five-Minute Checklist: Is My Framework Ready for Agentic QA?

Score yourself honestly:

Do my tests produce structured, machine-readable artifacts (traces, accessibility snapshots, JSON reports) — not just pass/fail in a console log?
Is my test code organized well enough (Page Objects, clear naming, isolated specs) that an automated diff to one file wouldn't ripple unpredictably elsewhere?
Do I have CI/CD hooks where an external process could safely trigger a rerun or open a draft PR without direct merge rights?
Do I have a clear, written list of what should never be auto-modified (assertions on financial/security flows, production config)?
Is there a human-review gate (required PR review, branch protection) that a bot account cannot bypass?
Do I have observability into why a test failed beyond the assertion message (network logs, DOM state, recent commits)?
Could I explain, in one sentence, the difference between what my self-healing tooling already handles and what would require full agentic reasoning?

Five or more checked: you're structurally ready to pilot agentic workflows on a low-risk test suite. Fewer than five: fix the framework fundamentals first — an agent layered on a fragile framework just automates the fragility faster.

12. The Maturity Path

flowchart TD
    A["Traditional SDET"] --> B["AI-Assisted SDET<br/>adds AI as a debugging/authoring assistant"]
    B --> C["AI-Augmented SDET<br/>integrates AI suggestions into daily workflow, still manual application"]
    C --> D["Agentic Test Engineer<br/>builds/configures tool-using agents for bounded tasks, e.g. locator healing"]
    D --> E["AI QA Agent Engineer<br/>designs multi-step agent workflows, guardrails, escalation policy"]
    E --> F["Owns the architecture: planners, tool contracts, evals, observability, policy"]

You don't skip stages by reading one article — but you can see, clearly, that every stage is still a quality engineering skill. The tools changed. The judgment requirement didn't.

The Actual Point

Agentic QA doesn't remove the need for a human who understands the product, the risk, and the business — it removes the tedium of manually operating the investigation loop a skilled QA engineer already runs in their head. The framework gets more capable. The judgment stays yours. That's the honest version of this shift, and it's a more interesting career than "write more scripts, faster."

Discussion: If an AI agent could run, debug, modify, and retest your automation today, which action would you trust it to take without human approval?

Agentic Testing Is Changing QA: When AI Agents Can Run, Debug, and Retest Your Tests

Pranta Kundu — Sat, 12 Sep 2026 14:25:34 +0000

From CI failure to verified fix: how AI agents can run, diagnose, repair, and retest automated tests.

Your test fails at 2 AM in the CI pipeline. You wake up, open the logs, scroll past 400 lines of stack trace, squint at a screenshot, and mutter "is this a locator issue or did the button actually move?" Twenty minutes later you find it — a data-testid changed during a refactor. You fix it, push, and go back to your actual work.

If that paragraph felt uncomfortably familiar, you already understand the problem agentic testing is trying to solve.

This isn't another "AI will replace QA" post. It's a practical breakdown — from one SDET to another — of what agentic testing actually is, what it can realistically do today, and where it still needs you.

Why Agentic Testing Is Different Now (Not Just Rebranded RPA)

If you've been in QA for more than a few years, you've seen "smart" testing tools come and go — rule-based self-healing, record-and-playback tools that claimed to "understand" your app, RPA scripts marketed as AI. It's fair to be skeptical that this is just another rebrand.

Here's what's genuinely different this time, in concrete terms:

Tool use, not just chat. Earlier "AI in testing" meant a chatbot you pasted logs into. Current LLMs can call functions directly — run a command, read a file, query a DOM, hit an API — and decide which tool to call based on what they find. That's the mechanical difference between "an assistant you operate" and "an agent that operates tools."

Multi-step reasoning that holds up across steps. Rule-based self-healing followed one fixed heuristic ("find nearest matching element"). Today's models can chain several reasoning steps — read a log, form a hypothesis, check a second source to confirm it, revise the hypothesis — which is closer to how you actually debug, not a single pattern match.

Context windows large enough to hold real evidence. Diagnosing a failure well means holding the test code, the DOM snapshot, the network log, and recent git history in view at once. That was previously impractical; it's now routine.

Cheaper, faster inference. Running this reasoning loop on every CI failure would have been cost-prohibitive a few years ago. It's now realistic to run on a meaningful slice of your failures, not just a curated demo.

To be clear about what this is not: it's not evidence that these systems reason the way humans do, and it's not a claim that failure rates have dropped by some measurable percentage — no credible industry-wide numbers exist yet, and be skeptical of anyone quoting one. What's changed is narrower and more mechanical: agents now have the tool access and reasoning chain-length to make the loop from earlier in this article actually work end-to-end, where before it had to be done manually or with brittle heuristics.

What "Agentic Testing" Actually Means

Strip away the buzzword, and agentic testing is this:

An AI agent that can reason about a testing task, use tools to act on it, observe the results, and decide what to do next — instead of just executing a fixed script.

A traditional test does exactly what you wrote, every time, in the same order. It has no awareness that anything went wrong beyond a pass/fail signal.

An agentic test system behaves more like a junior engineer sitting next to you: it runs the test, notices the failure, opens the logs itself, forms a hypothesis ("this looks like a timing issue, not a real bug"), checks the DOM to confirm, tries an adjusted action, and reports back with reasoning — not just a red X.

The key word is loop. Agentic testing isn't one clever trick (like auto-healing a broken locator). It's a repeatable reasoning loop applied to the whole investigation process.

Five Terms People Keep Mixing Up

This confusion is the single biggest source of AI-testing hype. Let's separate them cleanly.

Junior engineers often think self-healing is agentic testing. It isn't. Self-healing swaps a broken selector; it doesn't ask "did the product actually break, or did my test just get stale?" Agentic testing asks that question — and that's the whole point.

LLM/AI vs AI-Assisted Testing vs Agentic Testing — What Came First, and What Each One Actually Does

This is where most articles blur three very different things into one "AI testing" blob. They're not interchangeable — they're layers, and each one entered your testing life in a different order, doing a different job.

The order they showed up in your actual workflow:

LLM/AI came first — it's just a reasoning engine. It has no idea your test suite exists unless you copy-paste context into it manually. This is where most SDETs already live today: pasting a stack trace into an AI chat and asking "what's likely going on here?"

AI-assisted testing came next — someone wired that same reasoning engine into your actual tooling (IDE plugin, test generator, log summarizer), but a human is still the one clicking "run," reading the suggestion, and deciding what to do with it. The AI never acts on its own.

Agentic testing is the third layer — the same reasoning engine, now given tools it can call itself (run the test, read the DOM, check the network tab, open a diff) and permission to move through the investigate → fix → validate loop without you manually feeding it each piece of context at every step. You still set the boundaries; it works inside them.

The progression is really about who holds the tools and who presses "go" at each step — not about the AI getting "smarter" in some vague sense. Same underlying reasoning capability, increasing levels of tool access and autonomy.

The Agentic Testing Loop

Here's the loop in plain terms — the same one you already run manually when debugging a failure, except an agent runs it as a first pass. Each stage below is tagged with who's actually doing it, so it's clear where the "agentic" part really is:

Where the human sits in this loop: goal-setting at the start (you decide what "correct" means), and approval/audit at the two decision points — Change/Retry and Validate — especially for anything business-critical. Everything in between (Plan → Act → Observe → Diagnose) is where the agent is doing genuinely new work compared to a traditional script: it's reasoning across evidence, not just executing steps.

Notice: Report is not optional, and Validate always exists — a responsible agentic system never silently commits a "fix" without confirming it actually holds up, and never hides its reasoning from the human reviewing it.

A Realistic Example: Login Test Failure

Let's make this concrete instead of theoretical. Say you have a straightforward login test.

test('user can log in with valid credentials', async ({ page }) => {
  await page.goto('/login');
  await page.fill('#email', 'userk@example.com');
  await page.fill('#password', 'Test1234!');
  await page.click('#login-submit-btn');
  await expect(page.locator('.dashboard-header')).toBeVisible();
});

It fails in CI. Here's what a traditional pipeline gives you:

FAIL: user can log in with valid credentials
Error: Timeout 30000ms exceeded waiting for locator '.dashboard-header'

That's it. You now have to go do the investigation yourself.

What an agentic system does instead

Step 1 — Observe. The agent captures the DOM snapshot at failure time, the network log, and a screenshot instead of just the timeout message.

Step 2 — Diagnose. It compares the captured DOM against the expected state and finds this:

Expected element: .dashboard-header
DOM at failure time contains: .dashboard__header  (double underscore, renamed in last deploy)
Network log: POST /api/login → 200 OK, redirect to /dashboard confirmed
Screenshot: Dashboard page IS visible, correctly rendered

Step 3 — Form a hypothesis. Login actually succeeded — the assertion is broken, not the product. The class name changed in a recent CSS refactor (a real PR diff would confirm this, if the agent has repo access).

Step 4 — Propose a change (not silently apply it).

Agent suggestion:
  Confidence: High (87%)
  Cause: Locator drift (class renamed, unrelated to auth logic)
  Suggested fix: update locator to '.dashboard__header'
  Action taken: none yet - awaiting human approval (or auto-applied
  only if this test is flagged as "low-risk, auto-fixable" in config)

Step 5 — Validate. If approved, it reruns the test against the new locator, confirms a clean pass, and logs the change with a diff — visible in the PR, not buried in a black box.

Step 6 — Report.

SUMMARY
Test: user can log in with valid credentials
Result: Locator drift, not a functional defect
Evidence: network 200 OK, dashboard rendered, class renamed in commit abc123
Fix applied: locator updated, rerun passed 3/3
Human review: recommended (auth-related test, high business risk)

This is the entire value proposition in one example: faster, evidence-backed triage — not unsupervised decision-making.

Compare that to what a self-healing tool alone would have done: it would have just swapped .dashboard-header for the nearest matching element and moved on — no diagnosis, no report explaining why, no distinction made between "cosmetic class rename" and "the dashboard didn't actually load." That distinction is exactly why agentic reasoning matters more than pattern-matching self-healing.

The Full CI Workflow: From Failure to Verified Fix, With a Real Git Diff

The login example above shows the reasoning. This section shows how that reasoning actually plugs into a pipeline you'd run in production — because "the agent diagnoses it" means nothing until you see where it sits relative to your CI, your git history, and your merge gate.

The part worth underlining: the agent never pushes directly to your main branch. Its output is a draft PR with an attached evidence bundle — the same artifact a human contributor would produce, reviewable the same way. This is what makes "agent modifies the test" fundamentally different from "agent silently mutates CI state." If your implementation skips the draft-PR step and lets an agent commit straight to a protected branch, you've removed the one checkpoint that makes this whole workflow trustworthy.

What Agents Can Realistically Do Today

Framed as illustrative capability, not a benchmarked claim:

Inspect DOM/page state, accessibility tree, and console errors at the moment of failure
Parse logs, stack traces, and network requests to narrow down a likely cause
Correlate a failure with a recent code or config change, if given repo/CI access
Distinguish common failure classes: locator drift, timing/race conditions, stale test data, environment issues
Propose or apply a scoped fix (locator update, wait strategy, data refresh)
Rerun the test and compare before/after outcomes
Produce a structured, evidence-based failure summary instead of a raw stack trace

What Agents Silently Cannot Guarantee

This is the section most AI-testing content skips, and it's the most important one for your credibility as an SDET:

Business correctness. An agent doesn't know that "checkout total should include a loyalty discount for tier-3 customers" — unless that rule is explicitly encoded somewhere it can see.
Detecting every functional defect. It's diagnosing why a test failed, not independently discovering bugs the test wasn't written to catch.
Knowing intended product behavior. If the requirement was never documented or the ticket is vague, the agent has no ground truth to compare against.
Safely changing tests without masking real defects. An agent that "fixes" a failing assertion by loosening it can accidentally hide a genuine regression. This risk is real and needs guardrails (below).
Replacing human risk judgment. Deciding whether a payment flow test failure is worth blocking a release is a judgment call about business risk — not a pattern-matching task.

An agent that can fix a test is not the same as an agent that can prove the product is correct. Fixing a broken locator tells you the test is executable again. It tells you nothing about whether the feature behind that locator does what the business needs it to do. Treat every auto-applied fix as "test restored to a runnable state," not "feature verified correct."

Comparison Table: Autonomy, Reasoning, and Risk

Architecture of a Practical Agentic Testing System

The guardrails box is not decoration — it's the component that decides which failures an agent may auto-fix versus which must always stop and wait for a human. Without it, you don't have agentic testing; you have an unsupervised script with extra confidence.

What "Tools" Actually Mean for an Agent (No Magic Here)

"The agent inspects the DOM" sounds like it understands your app the way you do. It doesn't. An agent only has a defined, finite list of functions it's allowed to call — nothing more. If you didn't wire up a tool for it, it cannot do that thing, full stop. This matters because it demystifies the whole system: there's no hidden capability, just a list of functions with a name, inputs, and outputs.

A realistic tool list for a testing agent looks like this:

Tool: run_test(test_name)
  → executes one test, returns pass/fail + exit code

Tool: get_dom_snapshot(page)
  → returns the current HTML/accessibility tree at time of call

Tool: read_logs(run_id)
  → returns console, network, and framework logs for a run

Tool: get_screenshot(run_id)
  → returns a screenshot captured at failure time

Tool: git_diff(since_commit)
  → returns code changes since a given commit/tag

Tool: read_file(path)
  → returns file contents (read-only, scoped to repo paths)

Tool: propose_fix(file, change)
  → does NOT apply the change - stages it for human approval

Tool: apply_fix(file, change)   [gated - only callable if
                                 autonomy config allows it
                                 for this specific test]

Each tool is just a function with a schema — the agent picks which one to call based on its reasoning, the same way you'd decide "let me check the network tab" versus "let me check git blame" while debugging. The agent isn't smarter than the tools you gave it access to. If git_diff isn't wired up, it can't correlate a failure with a recent code change — it'll guess instead, and guessing is exactly the failure mode you're trying to avoid. The quality of your agentic testing setup is mostly a function of how good and how scoped your tool list is — not how good the underlying model is.

Notice apply_fix is deliberately gated separately from propose_fix. That split — "the agent can always suggest, but can only act where explicitly permitted" — is the single most important design decision in the whole system, and it's what your instruction file (next section) actually configures.

Hands-On: How to Actually Implement This in Your Framework

Everything above explains the concept. This section is for the engineer who closes this article and asks "okay, but how do I actually set this up on Monday?" Here's a concrete, three-part breakdown.

1. Pick an AI Coding Agent: Claude Code / Codex / Local LLM

You need a "coding agent" — something that can read your repo, run commands, and call tools — not just a chat window. The three realistic options today:

How to actually decide:

Does your test data or codebase contain sensitive/regulated information? → Local LLM, or a vendor with a data-retention agreement you've verified.
Do you need strong reasoning over messy logs and DOM diffs? → Hosted agent (Claude Code or Codex) generally outperforms smaller local models on this kind of multi-step reasoning today.
Is budget the constraint, not data sensitivity? → Start with whichever hosted agent your team already pays for elsewhere — don't add a second subscription just for this.

Whichever you pick, verify the current subscription tiers and rate limits directly on the provider's docs before committing a budget line — pricing and included usage change frequently.

2. The Instruction/Agent File — and Its Lifecycle

This is the single most important artifact in the whole setup. It's a file (commonly named AGENTS.md, CLAUDE.md, or a system-prompt config) that tells the agent: what its job is, what tools it may use, what it must never do without approval, and what format to report in.

A minimal version looks like this:

# Test Agent Instructions

## Role
You are a test-failure triage agent for the checkout and login test suites.

## You MAY:
- Read test code, logs, DOM snapshots, and network traces
- Rerun a failing test up to 2 times
- Propose locator/wait/data fixes with a confidence score

## You MUST NOT:
- Modify test files without an approval flag set to true
- Touch tests tagged `payment-critical` or `compliance` without human sign-off
- Loosen an assertion without explicitly flagging it as a "risk: masking possible defect"

## Report format
Always output: cause, evidence, confidence %, suggested fix, risk flag.

You don't write this once and forget it — it has a lifecycle, the same way test code does:

Treat step 6 as non-negotiable. Every time an agent gets something wrong — masks a real bug, misdiagnoses a cause — that's a signal to update the instruction file, not just to override the output once and move on. Version-control this file the same way you version-control test code, with PR review on changes.

3. Multi-Agent Prompt Setup: Splitting the Work Across Specialized Agents

A single "do everything" agent gets unreliable fast. In practice, this works better as a small team of narrowly-scoped agents that hand off to each other — similar to how you'd split responsibilities across QA roles on a real team.

Illustrative role prompts for each agent (adapt scope/tools to your stack):

A few more worth adding as your setup matures:

Flaky Test Detector — tracks pass/fail history over time and flags tests that fail intermittently regardless of code changes, so they get fixed at the root instead of being repeatedly "healed."
Test Data Manager — checks whether a failure traces back to stale/expired test fixtures or seed data, a very common false-positive source that's easy to under-diagnose.
Regression Risk Scorer — cross-references a failing test against recent deploys/PRs to estimate whether this failure correlates with a specific change, speeding up root-cause attribution.

Keep each agent's prompt scoped to one job. The temptation is to write one giant prompt that does analysis, scripting, healing, and escalation together — resist it. Narrow scope is what makes each agent's output reviewable, and reviewability is what makes the whole system trustworthy enough to actually put in CI.

4. A First Experiment You Can Actually Run This Week

Don't start by wiring all five agents into CI. Start with one test and one narrow question. Here's a scoped experiment using a Playwright suite you already have:

Pick one flaky or recently-broken test — not a payment or auth test. A low-risk UI check is ideal for a first run.
Manually gather the evidence bundle the way an agent would: the failure's stack trace, a DOM snapshot at failure time (page.content() in Playwright), the relevant network log, and git diff since the last known-green commit.
Paste all of it into a single prompt to whichever AI coding agent you chose in step 1 (Claude Code, Codex, or a local model), something close to:

Here is a failing Playwright test, its stack trace, a DOM snapshot at failure time,
and the git diff since the last passing run. Classify the failure as: locator drift,
timing issue, data issue, or likely real defect. State your confidence and cite the
specific evidence you used. Do not modify any files - diagnosis only.

Judge the output yourself — not on whether it "sounds smart," but on whether its cited evidence actually supports its conclusion. This is the reviewer skill from the "What This Means for SDETs" section, and this experiment is where you practice it.
Only if the diagnosis holds up, take the second step: ask it to propose (not apply) a minimal fix, and manually verify that fix against a rerun yourself.
Log what happened — was the diagnosis right, wrong, or partially right? That log is the seed of your instruction file's guardrails from the lifecycle diagram earlier in this section.

This costs you one afternoon, no CI changes, and no new tooling investment — and it tells you, concretely, whether your test suite and your chosen agent are actually a good match before you build any automation around it. If the diagnosis is consistently solid on your first five attempts, you have a real candidate for the CI-integrated workflow above. If it's shaky, that's useful information too — it tells you your logs/evidence capture needs work before an agent (or a human) can diagnose reliably from them.

What This Means for SDETs

For a junior engineer, this is genuinely good news day-to-day: instead of spending 40 minutes tracing why a test failed, you review a structured summary and decide in 5 minutes whether the fix is sound. But that 5 minutes requires you to actually understand why the agent reached its conclusion — you're becoming a reviewer of reasoning, not just a writer of scripts. That's a skill you build the same way you built debugging skill: by doing it, deliberately, on real failures.

5-Minute Checklist: Is This Test a Good Candidate for Agentic Testing?

Use this before handing any test suite to an agentic workflow:

Is the failure mode usually mechanical (locators, timing, stale data) rather than deep business logic?
Does the test have clear, deterministic pass/fail criteria (not a fuzzy or flaky assertion)?
Is there low-to-medium business risk if a wrong auto-fix temporarily masks a regression?
Do you have log/DOM/network access rich enough for the agent to actually diagnose, not guess?
Is there an audit trail requirement you can satisfy (every change logged, reviewable)?
Can you define a human approval gate for anything touching payments, auth, or compliance-sensitive flows?

Score 5–6 yes: good candidate for agent-assisted auto-fix with light review.
Score 3–4 yes: good candidate for agent-assisted diagnosis only, human applies the fix.
Score 0–2 yes: keep this test fully manual — the risk of a silent wrong fix outweighs the time saved.

Decision Matrix: When NOT to Use Agentic Testing

A Realistic Adoption Path

Nobody should go from "manual debugging" to "fully autonomous agent commits fixes to main" in one step. A sane rollout looks like this:

Start AI-assisted. Use AI to summarize failures and suggest causes; a human still fixes everything.
Automate the repetitive investigation. Let the agent gather DOM/logs/network evidence automatically, even if a human still decides the fix.
Introduce controlled agents on low-risk suites. Smoke tests, non-critical UI checks — auto-fix with logged rationale, human spot-checks.
Add evaluation and guardrails. Define autonomy scopes, blocked test lists, and confidence thresholds below which the agent must always escalate.
Integrate with CI/CD. Agent runs as part of the pipeline, gated by the guardrails from step 4, not bypassing them.
Measure outcomes. Track false-fix rate, time-to-triage, and — critically — how often a "fix" later correlated with a real regression that got masked. Adjust scope based on that data, not on vibes.

The Takeaway

Agentic testing is a real shift in how the debugging half of QA work gets done — not because AI is infallible, but because a structured reasoning loop over logs, DOM state, and code history can triage mechanical failures faster than a human doing it manually every time.

It does not replace the parts of QA that were never about typing test steps in the first place: understanding what "correct" means for the business, deciding what risk is acceptable to ship, and knowing when a green checkmark is lying to you.

Treat agentic testing as augmentation of your engineering judgment, not a substitute for it. The agents get faster at the loop. The judgment about when to trust the loop — that's still yours.

Would you trust an AI agent to debug, modify, and retest your automation without human approval? Where would you draw the line?

Drop your take below — especially if you've drawn that line somewhere different from where you expected to.

When Should AI Write Your Tests? A Staff SDET's Decision Framework + Prompts

Pranta Kundu — Thu, 10 Sep 2026 17:12:25 +0000

The most expensive mistake in AI-assisted QA isn't a bad test. It's not knowing why you wrote it.

Every week I watch engineers open ChatGPT, paste a user story, and type "write test cases for this." Ten seconds later they have 40 lines of confident-looking test code. Nobody asked what actually needed to be tested. Nobody asked what could actually break. The AI didn't fail — the question did. A generated test that verifies the wrong thing is worse than no test at all, because it gives you a green checkmark and a false sense of safety.

This isn't an anti-AI article. I use AI to write tests every single day. But after 8+ years designing automation frameworks and watching teams adopt AI tooling, I've learned that the skill that matters now isn't "can you prompt an LLM" — it's "can you tell, before you prompt it, whether this is even a job for AI." That's the decision most junior engineers skip. Let's fix that.

The Core Question Nobody Asks

Not "can AI write this test?" — it almost always can.

The real question is: "Should AI own this decision, or just the typing?"

Writing a test involves two separate jobs that get collapsed into one prompt:

Test strategy — deciding what needs to be verified, under what conditions, and why it matters.
Test implementation — turning that decision into code.

AI is frequently excellent at #2. It is inconsistent — and sometimes dangerous — at #1. The framework below exists to help you separate these two jobs before you ever open a prompt window.

The Staff SDET Decision Framework: 8 Dimensions to Evaluate First

Before you decide whether AI writes, assists, or stays out of a test, run it through these eight lenses. This takes less time to do than to read.

Dimension	Question to Ask	Why It Matters
Business risk	If this fails in production, what breaks — a typo or a transaction?	High-risk flows (payments, auth, data integrity) need human-owned assertions, not AI guesses.
Test complexity	Is this a single assertion, or a multi-step state machine?	AI handles linear flows well; it loses the thread in deep conditional logic.
Requirements clarity	Is the expected behavior documented, or does it live in someone's head?	AI cannot infer undocumented business rules — it will invent plausible-sounding ones instead.
Repetition/volume	Am I writing 1 unique test or 30 variations of the same pattern?	High-volume, low-variance work is where AI has the best cost/benefit ratio.
Framework maturity	Do we have established patterns, page objects, fixtures AI can follow?	AI writing into a mature framework is safe. AI designing the framework from scratch is not.
Test-data complexity	Does this need realistic, interdependent, or regulated data?	AI often invents data that "looks right" but violates real constraints (e.g. invalid state combinations).
Debugging/maintenance cost	If this test is flaky in 3 months, who has to understand it?	Code no one on the team understands is a liability, no matter who — or what — wrote it.
Need for human judgment	Does "correct" depend on product intuition, UX nuance, or edge-case tradeoffs?	Judgment calls are exactly where AI produces confident, wrong answers.

Run through these in under a minute. You're not filling out a form — you're building the instinct a Staff SDET already has.

The Decision Tree: "Should AI Write This Test?"

Here's the shape that instinct takes once it's explicit.

Four honest outcomes — not "AI vs. human," but a spectrum:

AI Writes — generates the test independently; you review before merge.
AI Assists — you design the test, AI helps with syntax, boilerplate, or edge-case suggestions.
Human-Led — you write and own it; AI might draft a skeleton, nothing more.
Do Not Automate Yet — the real problem is unclear requirements, not test authorship.

That last branch is the one juniors skip most often — and it's usually where the actual bug in the process lives.

Where AI Is Genuinely Excellent

No hype needed here — these are the cases where AI reliably saves real hours:

Repetitive scenario expansion — same flow, different inputs (valid email formats, currency values, locale variations).
API test variations — status codes, header combinations, pagination edge cases.
Boundary and negative cases — empty strings, max-length fields, null values, off-by-one boundaries.
Boilerplate framework code — Playwright/Selenium page objects, fixture setup, wait handling, config scaffolding.
Data-driven test tables — generating CSV/JSON test matrices once the logic is defined by a human.
Regression suite expansion — converting an existing manual test case into automated coverage using an established pattern.

Common thread: the decision about what to test was already made by a human. AI is filling in the matrix, not drawing it.

Where AI Should NOT Be Trusted to Decide Independently

Unclear or conflicting requirements — AI will resolve ambiguity by guessing, and the guess will look confident.
Critical business workflows — checkout, billing, auth, data migrations. The cost of a wrong assumption is too high.
Complex state transitions — multi-step workflows with conditional branches (e.g., order states: draft → pending → paid → refunded → disputed).
Security-sensitive behavior — auth bypass attempts, permission boundaries, injection handling. Getting the threat model wrong is worse than not testing at all.
Anything where a wrong assertion creates false confidence — a test that passes but checks the wrong thing is more dangerous than a missing test, because it hides the gap.

The risk isn't that AI writes broken code. It's that AI writes code that runs, passes, and proves nothing.

7 Practical Prompts You Can Copy-Paste (Browser ChatGPT)

Each prompt includes when to use it, what to feed it, what to expect back, and what you must verify yourself — because the output is a draft, not a decision.

1. Generating test cases from requirements

Prompt:

Act as a QA analyst. Here are the requirements for [feature]:
[paste requirements / user story / acceptance criteria]

List test cases grouped by: positive, negative, boundary, and edge cases.
For each test case, state the precondition, the action, and the expected result.
Do not write automation code yet — this is test design only.
Flag any requirement that is ambiguous or missing information instead of guessing.

When to use: Early in test design, right after requirements are written or refined.
Input needed: The actual requirement or user story — not a summary of it.
Expected output: A structured list of test case ideas with expected results.
You must verify: That every "ambiguous" flag AI raises is actually resolved by a human before automation starts. Check for missing business rules AI wouldn't know (permissions, regional logic, edge policies).

2. Converting test cases into Playwright tests

Prompt:

Convert the following test cases into Playwright tests using TypeScript.
Follow this existing pattern from our framework: [paste an existing test file as a style reference]
Use data-testid selectors where possible. Do not invent selectors — mark any selector
you're unsure about with a TODO comment instead of guessing.

Test cases:
[paste test cases]

When to use: Once test cases are approved and you're ready to implement.
Input needed: Approved test cases + a real example from your codebase for pattern consistency.
Expected output: Playwright test code matching your team's conventions.
You must verify: Every selector against the real DOM — AI will hallucinate plausible-looking selectors that don't exist. Confirm wait strategies aren't just page.waitForTimeout() hacks.

3. Finding missing edge cases

Prompt:

Here is a feature description and the test cases we already have:
[paste feature description]
[paste existing test list]

Identify edge cases or scenarios that are NOT covered.
Focus on: concurrency, empty/null states, permission boundaries, and unusual but
realistic user sequences. Do not repeat cases we already have.

When to use: As a pre-review step before finalizing test coverage.
Input needed: Your existing test list — the value here is in the gap, not a fresh list.
Expected output: A short, targeted list of genuinely new scenarios.
You must verify: Relevance — AI sometimes proposes edge cases that don't apply to your actual architecture (e.g., suggesting race-condition tests for a single-threaded batch job).

4. Generating negative API test scenarios

Prompt:

Here is an API contract: [paste OpenAPI spec / endpoint docs / example request-response]

Generate negative test scenarios covering: invalid payloads, missing required fields,
wrong data types, unauthorized access, and unexpected status codes.
For each scenario, state the request, the expected status code, and expected error behavior
per the documented contract only — do not assume behavior that isn't documented.

When to use: When you have a documented API contract (Swagger/OpenAPI/internal spec).
Input needed: The actual contract, not a paraphrase of it.
Expected output: A scenario list mapped to documented expected behavior.
You must verify: That expected status codes match the real API, not just what's "typical" — AI defaults to REST conventions even if your API doesn't follow them.

5. Reviewing AI-generated (or any) tests

Prompt:

Review this test code as a senior QA engineer would:
[paste test code]

Check for: weak or missing assertions, hardcoded waits, brittle selectors,
tests that would pass even if the feature were broken, and duplicated coverage
with tests I already have (if I paste them below).
Be specific about what's wrong and why — don't just say "looks good."

When to use: Before merging any AI- or human-written test.
Input needed: The test code, plus related existing tests if checking for duplication.
Expected output: A critique — ideally uncomfortable, not a rubber stamp.
You must verify: Run the critique against the actual application behavior. AI can flag a weak assertion in theory but can't confirm whether the strengthened assertion is actually correct for your app.

6. Improving weak assertions

Prompt:

Here is a test with an assertion I think is too weak:
[paste test code]

Suggest a stronger assertion that would actually fail if [describe the real bug
this test is supposed to catch]. Explain what the current assertion misses.

When to use: When a test "passes" but you suspect it wouldn't catch a real regression.
Input needed: The test and a clear description of the failure mode you're worried about.
Expected output: A tightened assertion plus reasoning.
You must verify: That the new assertion doesn't just become brittle instead of weak — overly strict assertions fail on harmless changes and erode trust in the suite.

7. Identifying redundant or flaky tests

Prompt:

Here is a list of test names and short descriptions from our suite:
[paste list]

Identify: (1) tests that likely overlap in coverage, (2) tests that sound like they
depend on timing/network state and are flaky-prone, (3) tests that test implementation
details rather than behavior. Explain your reasoning for each flag.

When to use: Periodic suite health reviews, or before a big regression run.
Input needed: Test names/descriptions — full code helps but isn't required for a first pass.
Expected output: Flags with reasoning, not automatic deletions.
You must verify: Every flag manually. "Sounds redundant" isn't the same as "is redundant" — two tests can look similar but cover different failure paths.

Before / After: A Real Prompt Upgrade

Junior engineer's prompt:

"Write Playwright tests for the login page."

What comes back: Generic tests for a login form that may not match your actual UI, using invented selectors, testing only the happy path plus one wrong-password case, with no negative testing, no consideration of lockout policy, and no connection to your existing framework patterns.

Staff SDET-quality prompt:

"Here's our login page requirements doc [pasted], our existing Playwright page object for the signup flow as a style reference [pasted], and our account lockout policy (5 failed attempts locks for 15 minutes) [pasted]. Generate Playwright tests in TypeScript covering: successful login, invalid password, non-existent email, account lockout after 5 failed attempts, and session expiry redirect. Use data-testid selectors matching our existing pattern. Mark any selector you're not certain about with a TODO instead of guessing. Do not test password strength rules — that's covered in the signup suite."

Why the difference matters: The second prompt gives AI real constraints (existing patterns, actual business rules, explicit scope boundaries) instead of asking it to invent them. The output quality gap isn't about "better AI" — it's about the human doing the strategy work before the prompt, not hoping the prompt does it for them.

The Principle to Internalize

AI can generate test code. It cannot automatically determine whether the test proves the right thing.

Correctness of syntax and correctness of intent are two different problems. AI is increasingly reliable at the first. The second still requires someone who understands the business, the risk, and the failure modes — because "the test passed" and "the feature works" are not the same sentence.

The risks worth naming plainly:

Hallucinated selectors — code that looks right, targets nothing real.
Weak assertions — tests that pass regardless of whether the feature is broken.
Duplicated coverage — AI doesn't know what already exists unless you tell it.
Incorrect assumptions — AI fills gaps in unclear requirements with plausible guesses, silently.
False confidence — the most expensive risk of all: a green suite that isn't actually protecting you.

The 5-Minute AI Test Generation Checklist (Save This)

Before you accept any AI-generated test:

Do I know why this test exists — what specific failure it should catch?
Are the requirements behind this test documented, not assumed?
Have I checked every selector against the real DOM/UI?
Would this test actually fail if the real bug happened? (Mentally break the feature and check.)
Does this duplicate existing coverage?
Is the test data realistic and valid for our actual system constraints?
If this test is flaky in 3 months, can someone on my team debug it without me?
Did I make the risk/judgment call, or did the AI make it for me by default?

If you can't check every box, the test isn't ready — regardless of who wrote it.

How This Skill Evolves

AI-assisted testing isn't a single skill — it's a maturity curve most teams move through:

Manual test design → AI-assisted test design → AI-generated automation → AI-reviewed automation → Agentic testing

Manual test design — humans decide what and how, no AI involved.
AI-assisted test design — AI helps brainstorm cases and gaps; humans still decide.
AI-generated automation — AI writes the code for well-scoped, well-understood cases.
AI-reviewed automation — AI flags weak assertions, duplication, and flakiness risk across the suite.
Agentic testing — AI systems that can plan, execute, and adapt test runs with human-defined guardrails.

Most teams today are somewhere between stage 2 and 3. Stage 5 gets a lot of hype, but it doesn't remove the need for stage 1's judgment — it just moves that judgment further upstream, into how you define the guardrails. The dimensions in this framework don't disappear as the tooling gets more capable. They become more important, because the cost of an unexamined mistake scales with how much autonomy you hand over.

The Staff-Level Takeaway

AI didn't change what makes a good test. It changed how fast a bad decision can get typed into code. The engineers who get real leverage from AI in QA aren't the ones prompting the most — they're the ones who can look at a testing problem for thirty seconds and know exactly which parts belong to a human and which parts belong to a machine. That judgment call is now the actual skill. The code was never the hard part.

So here's the discussion I want to have: What would you never let AI decide when writing your tests? Drop it in the comments — I'll respond to the interesting ones.

The AI-in-QA Decision Framework, With Real Prompts

Pranta Kundu — Thu, 10 Sep 2026 08:56:19 +0000

A Staff SDET's field guide to knowing when to use AI in testing — and the exact prompts to use when you do.

Most "AI for QA" content shows you a shiny demo: paste a requirement, get 20 test cases, applause.

Nobody shows you the moment after that — when the AI hallucinates a login flow that doesn't exist, or generates a Playwright test that passes locally and fails in CI because it never accounted for a loading spinner.

I wrote an earlier article introducing a 7-question framework for deciding whether AI should touch a QA problem in the first place. This is the follow-up I promised: every question turned into a real, copy-paste-ready prompt, with the engineering thinking behind it — the part most "AI + testing" content skips entirely.

If you're a junior QA engineer wondering how to actually use AI on Monday morning without breaking your pipeline, this is for you.

Why This Framework Exists

AI didn't remove the hardest part of QA. It just moved it.

The hard part used to be: "How do I write this test?"

The hard part now is: "Should AI write this test, and how do I know if it's right?"

A junior engineer asks: "Can AI do this?" A Staff SDET asks: "Should AI do this, what risk does it introduce, how will I validate it, and what happens when the AI is wrong?"

That gap — between can and should — is the entire framework.

The 7-Question Decision Framework (Quick View)

1. What does the AI technology/tool actually do?
2. What QA problem does it solve?
3. Is AI actually necessary, or would deterministic automation be better?
4. What AI capability is being used?
5. What tools/integration are required?
6. How do I validate the AI's output?
7. What are the risks, limitations, and human responsibilities?

Below, each question gets a real scenario, a real prompt, and a real validation step. No generic "generate test cases for login" prompts here.

Question 1 — What does the AI technology/tool actually do?

Why an SDET should ask it: You cannot evaluate output you don't understand the mechanics of. An LLM predicts plausible text; it does not "know" your application. If you don't internalize that, you will trust output you shouldn't.

Scenario: A junior engineer wants to use an LLM to generate test cases for a new "Apply Coupon" feature from a Jira ticket.

AI tool/use case: General-purpose LLM (Claude, GPT-class model) used for natural-language requirement analysis.

Prompt:

You are a Senior SDET reviewing a requirement before test design.

Requirement:
"Users can apply one coupon code per order at checkout. Invalid or
expired codes should show an error. Coupon discount should apply to
subtotal before tax."

Do the following, in order:
1. List all assumptions you are making about this requirement.
2. List any ambiguities or missing information a developer/QA should
   clarify before test design begins.
3. Do NOT generate test cases yet. Only output assumptions and gaps.

Example input/context: One Jira ticket, no linked designs, no acceptance criteria attached.

Expected AI output type: A list like:

Assumption: "one coupon per order" means additional codes are silently rejected, not queued.
Gap: What happens if the coupon is valid but the cart becomes empty before submit?
Gap: Is there a limit on coupon value relative to order value (can discount exceed subtotal)?

How to verify: Cross-check every assumption against the actual ticket, designs, and product owner — not against your memory of "how coupons usually work." Treat the list as a starting point for a 10-minute clarification conversation, not a finished artifact.

When NOT to use AI here: If the requirement is trivial and unambiguous (e.g., "field must be required"), skip this — you're adding process overhead for zero risk reduction.

Key lesson: AI is strongest at surfacing ambiguity, not at resolving it. Use it to ask better questions, not to skip asking them.

Question 2 — What QA problem does it solve?

Why an SDET should ask it: "We should use AI" is not a QA problem. "Our regression suite takes 40 minutes to triage after each nightly run" is. Tie every AI use case to a named, measurable pain point.

Scenario: Your team's CI pipeline produces 200+ automated test results nightly, and someone manually scans logs every morning to find real failures vs. flaky noise.

AI tool/use case: LLM-based log summarization and clustering, either via a script feeding failure logs to an LLM API, or a CI-integrated AI log tool.

Prompt:

You are a CI triage assistant for a test automation pipeline.

Below are failure logs from tonight's regression run. For each failure:
1. Extract the failing test name.
2. Classify the failure type: [Assertion Failure, Timeout,
   Element Not Found, Network/API Error, Environment Issue, Unknown].
3. Group failures that share the same likely root cause.
4. Flag any failure pattern that resembles known flaky behavior
   (e.g., timing-related, intermittent selector issues).
5. Output a ranked list: which failures need human investigation
   FIRST, and why.

Logs:
<paste raw CI failure logs here>

Example input/context: Raw stack traces and console logs from a Jenkins/GitHub Actions run, 15–30 failed tests.

Expected AI output type: A clustered table — e.g., "12 failures share a TimeoutError on #checkout-button, likely a shared root cause (staging environment slow to load) — investigate first."

How to verify: Spot-check 2–3 clusters manually against the raw logs. Confirm the "likely root cause" is a hypothesis, not a diagnosis — the AI is pattern-matching text, not executing your app.

When NOT to use AI here: If your suite only has 5–10 tests, manual triage is faster and more reliable than setting up an AI pipeline for it.

Key lesson: AI earns its place on volume and repetition, not novelty. Triage at scale is a textbook fit; triage of five tests is not.

Question 3 — Is AI actually necessary, or would deterministic automation be better?

Why an SDET should ask it: This is the question most engineers skip, and it's the one that separates a Staff-level thinker from someone chasing a trend. Deterministic code is cheaper, faster, and 100% reproducible for problems that don't require language understanding or judgment.

Scenario: You need to verify that an API response always returns status: "active" for a newly created user.

Why AI is the WRONG tool here:

❌ Don't do this: "Ask an LLM to check if the API response looks correct."

An LLM introduces non-determinism, latency, and cost into a check that a one-line assertion solves perfectly:

// ✅ Deterministic automation - correct choice expect(response.body.status).toBe("active");

When AI legitimately enters this scenario: Generating the initial draft of the test scaffolding, or generating edge-case ideas for the response schema (e.g., "what other states might a user API expose that we haven't tested?") — then a human converts those into deterministic assertions.

Key lesson (say this out loud in code review):

"If the correct answer is a fixed rule, write a rule. If the correct answer requires judgment or language understanding, that's where AI earns its seat."

This is the single most important discipline in the entire framework. AI-assisted testing should never replace an expect() statement that can be written in five seconds.

Question 4 — What AI capability is being used?

Why an SDET should ask it: "AI" is not one thing. Code generation, natural language reasoning, visual/image analysis, and autonomous multi-step agents have different failure modes. Naming the capability tells you what kind of validation you need.

Scenario: You want AI to generate a Playwright test for a multi-step checkout flow from a manual test case.

AI capability being used: Code generation grounded in natural-language instructions (not visual testing, not agentic browsing).

Prompt:

You are a Senior SDET writing a Playwright test in TypeScript.

Context:
- Framework: Playwright + TypeScript
- Page Object Model is used; page objects live in /pages
- Existing selector convention: data-testid attributes only
- Test must run in CI headless, must not use hardcoded waits (no
  page.waitForTimeout)

Manual test case:
"1. Go to checkout page with 1 item in cart.
 2. Apply coupon code 'SAVE10'.
 3. Verify discounted total reflects 10% off subtotal (not total
    with tax).
 4. Click 'Place Order'.
 5. Verify order confirmation page shows correct order ID format."

Write the Playwright test. Requirements:
- Use Page Object Model, assume a CheckoutPage class exists with
  methods: applyCoupon(code), getDiscountedTotal(), placeOrder()
- Use explicit waits tied to network/DOM state, never fixed timeouts
- Add one negative-path test: invalid coupon code
- Call out any missing selectors or page object methods you assumed
  exist, at the end, as a comment block

Example input/context: One manual test case + a short description of existing project conventions (this context matters enormously — a prompt without your project's real conventions produces unusable code).

Expected AI output type: A .spec.ts file using your POM structure, plus a clearly labeled comment block listing assumed methods/selectors that don't exist yet — this is the AI being honest about its own guesses, which you explicitly asked it to do.

How to verify:

Run it locally before it ever touches CI.
Check every assumed selector/method against the real codebase.
Confirm the "no hardcoded waits" instruction was actually followed — AI models frequently insert waitForTimeout even when told not to.
Review assertions for false positives (e.g., checking element exists instead of checking element value).

When NOT to use AI here: For extremely simple tests (a single field validation), writing it yourself is faster than prompting, reviewing, and correcting AI output.

Key lesson: The quality of AI-generated test code is a direct function of how much real project context you give it. A prompt with no conventions produces demo-quality code, not production-quality code.

Question 5 — What tools/integration are required?

Why an SDET should ask it: A good prompt is useless if it doesn't fit your actual pipeline. Before adopting an AI use case, map exactly where it plugs into your stack — IDE, CI runner, test management tool, or a standalone script.

Scenario: You want AI-generated API test scenarios to plug directly into your existing Postman/Newman or REST-assured suite, not live in a separate doc nobody maintains.

AI tool/use case: LLM prompt run through a script (or IDE plugin) that outputs structured JSON, consumed directly by your test generator.

Prompt:

You are generating API test scenarios for an SDET's automated suite.

Endpoint: POST /api/v1/orders
Request schema:
{
  "userId": "string (UUID)",
  "items": [{ "sku": "string", "quantity": "integer > 0" }],
  "couponCode": "string, optional"
}

Generate test scenarios covering:
- Valid requests (happy path)
- Boundary values (quantity = 0, quantity = max int, empty items array)
- Invalid types (quantity as string, missing userId)
- Security-relevant edge cases (SQL-injection-style strings in sku,
  oversized payloads)
- Business-rule edge cases (couponCode present but user has no cart)

Output as a JSON array, each object with fields:
{ "scenario_name": "", "request_body": {}, "expected_status": "",
  "expected_behavior": "", "risk_level": "low|medium|high" }

Only output valid JSON. No explanation text.

Integration point: This structured JSON output is designed to feed directly into a script that converts each object into a REST-assured or Newman test case — this is what makes it "integration-ready" instead of a wall of text you retype by hand.

How to verify: Validate the JSON schema programmatically before trusting it (a malformed field breaks your generator silently). Manually review the risk_level field — AI risk-ranking is a starting opinion, not a security audit.

When NOT to use AI here: If your API surface is small and stable (under ~10 endpoints, rarely changing), manually maintained scenarios are easier to keep accurate than an AI-regeneration pipeline.

Key lesson: The real ROI of AI in QA comes from integration, not novelty. A brilliant prompt that produces output you copy-paste by hand doesn't scale — structure your prompts to feed your existing tools.

Question 6 — How do I validate the AI's output?

Why an SDET should ask it: This is the question that determines whether you're doing "AI-assisted testing" or "AI-assisted incident creation." Validation is not optional, and it is not the same as "it compiled."

Scenario: A test that was passing for three months starts failing intermittently — 1 in 8 runs — and you want AI to help investigate before you spend an afternoon on it.

AI tool/use case: LLM-based root-cause reasoning over logs, diffs, and historical run data.

Prompt:

You are helping an SDET investigate a flaky test.

Test: "should display updated cart total after quantity change"
Framework: Cypress
Failure rate: ~1 in 8 CI runs, always in headless CI, never
reproduces locally.

Here is the failing run's log/error output:
<paste error + stack trace>

Here is the relevant test code:
<paste test code>

Here is the relevant application code being tested (cart update
handler):
<paste app code>

Analyze and respond in this structure:
1. Most likely root cause category (race condition, animation/timing,
   test isolation issue, environment resource contention, incorrect
   assertion, backend non-determinism) - with confidence level
   (low/medium/high).
2. Specific evidence from the logs/code that supports this.
3. Two alternative hypotheses, ranked by likelihood.
4. A minimal experiment I could run to confirm or rule out the top
   hypothesis (not a rewrite - an experiment).
5. What you are NOT confident about, explicitly.

Example input/context: Real logs, real test code, real handler code — vague or partial context produces vague or partial (and confidently wrong) diagnoses.

Expected AI output type: A ranked hypothesis (e.g., "high confidence: race condition — the assertion checks total before the debounced API call resolves") plus a small, falsifiable experiment ("add a network-idle wait tied to the specific PATCH call and rerun 20x in CI").

How to verify — this is the core discipline:

Run the suggested experiment. Do not accept the hypothesis until it's been tested against the real, flaky test at least 10–20 times.
If the "fix" makes the test pass once, that is not validation — flaky tests pass "by luck" too. Require a meaningfully higher pass rate over repeated runs.
Never merge an AI-suggested fix into a shared test without your own understanding of why it works. If you can't explain it in code review, don't ship it.

When NOT to use AI here: If the flake is already well-understood by your team (e.g., "we know this suite has animation timing issues on Fridays"), skip the AI investigation and apply the known fix.

Key lesson: AI-generated root cause analysis is a hypothesis engine, not a verdict. Validation is the SDET's job, every time, with no exceptions — this is the line between engineering and guessing.

Question 7 — What are the risks, limitations, and human responsibilities?

Why an SDET should ask it: Every AI use case has a failure mode. Naming it in advance is how you prevent it from becoming an incident.

Scenario: Using AI to generate synthetic test data for a staging environment load test.

AI tool/use case: LLM-based synthetic data generation.

Prompt:

Generate 50 synthetic user records for load-testing a signup API.

Constraints:
- Fields: fullName, email, dateOfBirth (must include users aged
  17, 18, and 100+ for boundary testing), country (mix of at least
  8 countries, including ones with non-Latin characters), password
  (must include realistic weak AND strong examples)
- Do NOT use real people's names or real, resolvable email domains -
  use example.com or example.org only
- Include 5 intentionally malformed records (invalid email format,
  future date of birth, empty required field) explicitly labeled
  "INVALID - for negative testing"
- Output as CSV

Named risks for this use case:

How to verify: Scan the CSV for real-looking PII patterns, confirm invalid records are actually invalid in the ways intended, and confirm the file never gets referenced outside the test/staging scope.

When NOT to use AI here: Regulated domains (healthcare, finance) where synthetic data must meet specific compliance patterns — use a purpose-built synthetic data tool with compliance guarantees, not a general LLM prompt.

Key lesson: The risk isn't that AI gets something wrong. The risk is a team building a habit of not checking because AI usually gets it right. Usually is where incidents live.

AI Prompt ≠ AI Decision

This is the sentence I'd want every junior SDET to tattoo on their monitor:

A good SDET does not blindly execute whatever AI recommends.

The real workflow — the one that separates "using AI" from "engineering with AI" — looks like this:

flowchart TD
    A[Requirement] --> B["Problem: What are we actually trying to verify or solve?"]
    B --> C["AI Suitability Check — Would deterministic automation be better?"]
    C --> D["Prompt: structured, project-aware, explicit about constraints"]
    D --> E["AI Output (draft — never treated as final)"]
    E --> F["Validation: run it, test it, cross-check assumptions, check risk"]
    F --> G["Human Decision: accept / modify / reject — with a reason"]
    G --> H["Automation / CI Integration — only validated, understood work ships"]

Every step in this chain is a checkpoint. Skipping AI Suitability Check gets you AI-generated tests for problems a one-line assertion would've solved. Skipping Validation gets you flaky CI. Skipping Human Decision gets you tests nobody on the team can explain in six months.

One-liner for your team's Slack channel:

"AI can write the test. It can't own the test."

Staff-level read of this table: notice reliability goes down and required human involvement goes up as autonomy increases. That's not a flaw in AI tooling — that's the correct engineering relationship between autonomy and oversight. Anyone selling you "fully autonomous QA, zero human review" is selling you an incident.

The 5-Minute AI Adoption Checklist

Before you introduce any AI tool into a QA workflow, run through this:

Named problem — Can I state the QA problem in one sentence without mentioning AI?
Deterministic check — Have I confirmed a fixed rule/script can't solve this more reliably?
Capability match — Do I know exactly which AI capability I'm using (text generation, code generation, reasoning, agentic action)?
Context supplied — Does my prompt include real project conventions, not generic instructions?
Validation plan — Do I know before I run the prompt how I will verify the output?
Blast radius — If this output is wrong and I miss it, what's the worst that happens (flaky CI vs. a shipped bug vs. leaked data)?
Owner — Is there a named human who reviews and approves before this reaches CI/production?
Reversibility — Can I easily disable/roll back this AI-generated artifact if it causes problems?

If you can't check every box in five minutes, you're not ready to automate this with AI yet — and that's a useful, not embarrassing, conclusion.

How I Would Explain This to a Junior SDET

Imagine you hire a very fast, very well-read intern. They've read almost every testing blog, every framework doc, every Stack Overflow answer — but they've never actually run your app, never seen your production incidents, and they will confidently guess when they don't know something instead of saying "I don't know."

Would you let that intern:

Draft your test cases? Yes — review before you use them.
Write your first-pass Playwright script? Yes — review every selector.
Decide alone that a flaky test is now fixed? No — make them prove it, ten times, in CI.
Push straight to your production test suite unsupervised? Never.

That intern is exactly what AI is in your QA workflow. Brilliant draft-writer. Terrible unsupervised decision-maker. Treat it exactly that way and you'll get real value without real risk.

How a Staff SDET Thinks Differently

That last row is the whole game. AI doesn't get blamed for a bad test in a postmortem. The engineer who merged it does. Staff-level thinking means you never let a prompt make a decision you haven't personally signed off on.

Where This Is Heading (Without the Hype)

None of this means testers get replaced. It means the skill set keeps layering:

flowchart TD
    A["AI-assisted SDET — uses AI as a tool inside existing manual/automated workflows"] --> B["AI-augmented SDET — AI embedded across the workflow: test design, debugging, data, triage - with strong validation habits"]
    B --> C["Agentic Test Engineer — designs and supervises AI agents that execute multi-step testing tasks with defined guardrails"]
    C --> D["AI QA Agent Engineer — builds, evaluates, and governs the AI systems/agents QA teams rely on"]
    D --> E["AI Quality Engineering — quality engineering discipline expands to include AI system quality itself: evaluating models, prompts, and agents as first-class testable systems"]

To be clear: these are evolving skill directions, not standardized job titles you'll see on every job board tomorrow. Nobody should treat this as a fixed career ladder — treat it as a map of where the skills are heading, and start building the ones nearest to where you already are.

The testers who do well here aren't the ones who trust AI the most. They're the ones who validate it the best.

Try This Today

Pick one task from this article — requirement ambiguity analysis, log triage, or flaky test investigation — and run the real prompt on a real problem you have this week. Don't use it on a toy example. Use it on something in your actual backlog, then apply the validation step before you trust a single line of it.

That's the whole framework in practice: not "AI or no AI," but problem → suitability → prompt → validation → decision, every single time.

Which QA task would you trust AI with first — and which one would you never hand over to AI?

Drop it in the comments. I'll respond with how I'd validate (or reject) it.