When Should AI Write Your Tests? A Staff SDET's Decision Framework + Prompts

The most expensive mistake in AI-assisted QA isn't a bad test. It's not knowing why you wrote it.

Every week I watch engineers open ChatGPT, paste a user story, and type "write test cases for this." Ten seconds later they have 40 lines of confident-looking test code. Nobody asked what actually needed to be tested. Nobody asked what could actually break. The AI didn't fail — the question did. A generated test that verifies the wrong thing is worse than no test at all, because it gives you a green checkmark and a false sense of safety.

This isn't an anti-AI article. I use AI to write tests every single day. But after 8+ years designing automation frameworks and watching teams adopt AI tooling, I've learned that the skill that matters now isn't "can you prompt an LLM" — it's "can you tell, before you prompt it, whether this is even a job for AI." That's the decision most junior engineers skip. Let's fix that.

---

The Core Question Nobody Asks

Not "can AI write this test?" — it almost always can.

The real question is: "Should AI own this decision, or just the typing?"

Writing a test involves two separate jobs that get collapsed into one prompt:

1. Test strategy — deciding what needs to be verified, under what conditions, and why it matters.
2. Test implementation — turning that decision into code.

AI is frequently excellent at #2. It is inconsistent — and sometimes dangerous — at #1. The framework below exists to help you separate these two jobs before you ever open a prompt window.

---

The Staff SDET Decision Framework: 8 Dimensions to Evaluate First

Before you decide whether AI writes, assists, or stays out of a test, run it through these eight lenses. This takes less time to do than to read.

Dimension	Question to Ask	Why It Matters
Business risk	If this fails in production, what breaks — a typo or a transaction?	High-risk flows (payments, auth, data integrity) need human-owned assertions, not AI guesses.
Test complexity	Is this a single assertion, or a multi-step state machine?	AI handles linear flows well; it loses the thread in deep conditional logic.
Requirements clarity	Is the expected behavior documented, or does it live in someone's head?	AI cannot infer undocumented business rules — it will invent plausible-sounding ones instead.
Repetition/volume	Am I writing 1 unique test or 30 variations of the same pattern?	High-volume, low-variance work is where AI has the best cost/benefit ratio.
Framework maturity	Do we have established patterns, page objects, fixtures AI can follow?	AI writing into a mature framework is safe. AI designing the framework from scratch is not.
Test-data complexity	Does this need realistic, interdependent, or regulated data?	AI often invents data that "looks right" but violates real constraints (e.g. invalid state combinations).
Debugging/maintenance cost	If this test is flaky in 3 months, who has to understand it?	Code no one on the team understands is a liability, no matter who — or what — wrote it.
Need for human judgment	Does "correct" depend on product intuition, UX nuance, or edge-case tradeoffs?	Judgment calls are exactly where AI produces confident, wrong answers.

Run through these in under a minute. You're not filling out a form — you're building the instinct a Staff SDET already has.

---

The Decision Tree: "Should AI Write This Test?"

Here's the shape that instinct takes once it's explicit.

Four honest outcomes — not "AI vs. human," but a spectrum:

• AI Writes — generates the test independently; you review before merge.
• AI Assists — you design the test, AI helps with syntax, boilerplate, or edge-case suggestions.
• Human-Led — you write and own it; AI might draft a skeleton, nothing more.
• Do Not Automate Yet — the real problem is unclear requirements, not test authorship.

That last branch is the one juniors skip most often — and it's usually where the actual bug in the process lives.

---

Where AI Is Genuinely Excellent

No hype needed here — these are the cases where AI reliably saves real hours:

• Repetitive scenario expansion — same flow, different inputs (valid email formats, currency values, locale variations).
• API test variations — status codes, header combinations, pagination edge cases.
• Boundary and negative cases — empty strings, max-length fields, null values, off-by-one boundaries.
• Boilerplate framework code — Playwright/Selenium page objects, fixture setup, wait handling, config scaffolding.
• Data-driven test tables — generating CSV/JSON test matrices once the logic is defined by a human.
• Regression suite expansion — converting an existing manual test case into automated coverage using an established pattern.

Common thread: the decision about what to test was already made by a human. AI is filling in the matrix, not drawing it.

---

Where AI Should NOT Be Trusted to Decide Independently

• Unclear or conflicting requirements — AI will resolve ambiguity by guessing, and the guess will look confident.
• Critical business workflows — checkout, billing, auth, data migrations. The cost of a wrong assumption is too high.
• Complex state transitions — multi-step workflows with conditional branches (e.g., order states: draft → pending → paid → refunded → disputed).
• Security-sensitive behavior — auth bypass attempts, permission boundaries, injection handling. Getting the threat model wrong is worse than not testing at all.
• Anything where a wrong assertion creates false confidence — a test that passes but checks the wrong thing is more dangerous than a missing test, because it hides the gap.

The risk isn't that AI writes broken code. It's that AI writes code that runs, passes, and proves nothing.

---

---

7 Practical Prompts You Can Copy-Paste (Browser ChatGPT)

Each prompt includes when to use it, what to feed it, what to expect back, and what you must verify yourself — because the output is a draft, not a decision.

1. Generating test cases from requirements

Prompt:

Act as a QA analyst. Here are the requirements for [feature]:
[paste requirements / user story / acceptance criteria]

List test cases grouped by: positive, negative, boundary, and edge cases.
For each test case, state the precondition, the action, and the expected result.
Do not write automation code yet — this is test design only.
Flag any requirement that is ambiguous or missing information instead of guessing.

• When to use: Early in test design, right after requirements are written or refined.
• Input needed: The actual requirement or user story — not a summary of it.
• Expected output: A structured list of test case ideas with expected results.
• You must verify: That every "ambiguous" flag AI raises is actually resolved by a human before automation starts. Check for missing business rules AI wouldn't know (permissions, regional logic, edge policies).

2. Converting test cases into Playwright tests

Prompt:

Convert the following test cases into Playwright tests using TypeScript.
Follow this existing pattern from our framework: [paste an existing test file as a style reference]
Use data-testid selectors where possible. Do not invent selectors — mark any selector
you're unsure about with a TODO comment instead of guessing.

Test cases:
[paste test cases]

• When to use: Once test cases are approved and you're ready to implement.
• Input needed: Approved test cases + a real example from your codebase for pattern consistency.
• Expected output: Playwright test code matching your team's conventions.
• You must verify: Every selector against the real DOM — AI will hallucinate plausible-looking selectors that don't exist. Confirm wait strategies aren't just page.waitForTimeout() hacks.

3. Finding missing edge cases

Prompt:

Here is a feature description and the test cases we already have:
[paste feature description]
[paste existing test list]

Identify edge cases or scenarios that are NOT covered.
Focus on: concurrency, empty/null states, permission boundaries, and unusual but
realistic user sequences. Do not repeat cases we already have.

• When to use: As a pre-review step before finalizing test coverage.
• Input needed: Your existing test list — the value here is in the gap, not a fresh list.
• Expected output: A short, targeted list of genuinely new scenarios.
• You must verify: Relevance — AI sometimes proposes edge cases that don't apply to your actual architecture (e.g., suggesting race-condition tests for a single-threaded batch job).

4. Generating negative API test scenarios

Prompt:

Here is an API contract: [paste OpenAPI spec / endpoint docs / example request-response]

Generate negative test scenarios covering: invalid payloads, missing required fields,
wrong data types, unauthorized access, and unexpected status codes.
For each scenario, state the request, the expected status code, and expected error behavior
per the documented contract only — do not assume behavior that isn't documented.

• When to use: When you have a documented API contract (Swagger/OpenAPI/internal spec).
• Input needed: The actual contract, not a paraphrase of it.
• Expected output: A scenario list mapped to documented expected behavior.
• You must verify: That expected status codes match the real API, not just what's "typical" — AI defaults to REST conventions even if your API doesn't follow them.

5. Reviewing AI-generated (or any) tests

Prompt:

Review this test code as a senior QA engineer would:
[paste test code]

Check for: weak or missing assertions, hardcoded waits, brittle selectors,
tests that would pass even if the feature were broken, and duplicated coverage
with tests I already have (if I paste them below).
Be specific about what's wrong and why — don't just say "looks good."

• When to use: Before merging any AI- or human-written test.
• Input needed: The test code, plus related existing tests if checking for duplication.
• Expected output: A critique — ideally uncomfortable, not a rubber stamp.
• You must verify: Run the critique against the actual application behavior. AI can flag a weak assertion in theory but can't confirm whether the strengthened assertion is actually correct for your app.

6. Improving weak assertions

Prompt:

Here is a test with an assertion I think is too weak:
[paste test code]

Suggest a stronger assertion that would actually fail if [describe the real bug
this test is supposed to catch]. Explain what the current assertion misses.

• When to use: When a test "passes" but you suspect it wouldn't catch a real regression.
• Input needed: The test and a clear description of the failure mode you're worried about.
• Expected output: A tightened assertion plus reasoning.
• You must verify: That the new assertion doesn't just become brittle instead of weak — overly strict assertions fail on harmless changes and erode trust in the suite.

7. Identifying redundant or flaky tests

Prompt:

Here is a list of test names and short descriptions from our suite:
[paste list]

Identify: (1) tests that likely overlap in coverage, (2) tests that sound like they
depend on timing/network state and are flaky-prone, (3) tests that test implementation
details rather than behavior. Explain your reasoning for each flag.

• When to use: Periodic suite health reviews, or before a big regression run.
• Input needed: Test names/descriptions — full code helps but isn't required for a first pass.
• Expected output: Flags with reasoning, not automatic deletions.
• You must verify: Every flag manually. "Sounds redundant" isn't the same as "is redundant" — two tests can look similar but cover different failure paths.

---

Before / After: A Real Prompt Upgrade

Junior engineer's prompt:

"Write Playwright tests for the login page."

What comes back: Generic tests for a login form that may not match your actual UI, using invented selectors, testing only the happy path plus one wrong-password case, with no negative testing, no consideration of lockout policy, and no connection to your existing framework patterns.

Staff SDET-quality prompt:

"Here's our login page requirements doc [pasted], our existing Playwright page object for the signup flow as a style reference [pasted], and our account lockout policy (5 failed attempts locks for 15 minutes) [pasted]. Generate Playwright tests in TypeScript covering: successful login, invalid password, non-existent email, account lockout after 5 failed attempts, and session expiry redirect. Use data-testid selectors matching our existing pattern. Mark any selector you're not certain about with a TODO instead of guessing. Do not test password strength rules — that's covered in the signup suite."

Why the difference matters: The second prompt gives AI real constraints (existing patterns, actual business rules, explicit scope boundaries) instead of asking it to invent them. The output quality gap isn't about "better AI" — it's about the human doing the strategy work before the prompt, not hoping the prompt does it for them.

---

The Principle to Internalize

AI can generate test code. It cannot automatically determine whether the test proves the right thing.

Correctness of syntax and correctness of intent are two different problems. AI is increasingly reliable at the first. The second still requires someone who understands the business, the risk, and the failure modes — because "the test passed" and "the feature works" are not the same sentence.

The risks worth naming plainly:

• Hallucinated selectors — code that looks right, targets nothing real.
• Weak assertions — tests that pass regardless of whether the feature is broken.
• Duplicated coverage — AI doesn't know what already exists unless you tell it.
• Incorrect assumptions — AI fills gaps in unclear requirements with plausible guesses, silently.
• False confidence — the most expensive risk of all: a green suite that isn't actually protecting you.

---

The 5-Minute AI Test Generation Checklist (Save This)

Before you accept any AI-generated test:

• Do I know why this test exists — what specific failure it should catch?
• Are the requirements behind this test documented, not assumed?
• Have I checked every selector against the real DOM/UI?
• Would this test actually fail if the real bug happened? (Mentally break the feature and check.)
• Does this duplicate existing coverage?
• Is the test data realistic and valid for our actual system constraints?
• If this test is flaky in 3 months, can someone on my team debug it without me?
• Did I make the risk/judgment call, or did the AI make it for me by default?

If you can't check every box, the test isn't ready — regardless of who wrote it.

---

How This Skill Evolves

AI-assisted testing isn't a single skill — it's a maturity curve most teams move through:

Manual test design → AI-assisted test design → AI-generated automation → AI-reviewed automation → Agentic testing

1. Manual test design — humans decide what and how, no AI involved.
2. AI-assisted test design — AI helps brainstorm cases and gaps; humans still decide.
3. AI-generated automation — AI writes the code for well-scoped, well-understood cases.
4. AI-reviewed automation — AI flags weak assertions, duplication, and flakiness risk across the suite.
5. Agentic testing — AI systems that can plan, execute, and adapt test runs with human-defined guardrails.

Most teams today are somewhere between stage 2 and 3. Stage 5 gets a lot of hype, but it doesn't remove the need for stage 1's judgment — it just moves that judgment further upstream, into how you define the guardrails. The dimensions in this framework don't disappear as the tooling gets more capable. They become more important, because the cost of an unexamined mistake scales with how much autonomy you hand over.

---

The Staff-Level Takeaway

AI didn't change what makes a good test. It changed how fast a bad decision can get typed into code. The engineers who get real leverage from AI in QA aren't the ones prompting the most — they're the ones who can look at a testing problem for thirty seconds and know exactly which parts belong to a human and which parts belong to a machine. That judgment call is now the actual skill. The code was never the hard part.

So here's the discussion I want to have: What would you never let AI decide when writing your tests? Drop it in the comments — I'll respond to the interesting ones.