The AI-in-QA Decision Framework, With Real Prompts

A Staff SDET's field guide to knowing when to use AI in testing — and the exact prompts to use when you do.

Most "AI for QA" content shows you a shiny demo: paste a requirement, get 20 test cases, applause.

Nobody shows you the moment after that — when the AI hallucinates a login flow that doesn't exist, or generates a Playwright test that passes locally and fails in CI because it never accounted for a loading spinner.

I wrote an earlier article introducing a 7-question framework for deciding whether AI should touch a QA problem in the first place. This is the follow-up I promised: every question turned into a real, copy-paste-ready prompt, with the engineering thinking behind it — the part most "AI + testing" content skips entirely.

If you're a junior QA engineer wondering how to actually use AI on Monday morning without breaking your pipeline, this is for you.

Why This Framework Exists

AI didn't remove the hardest part of QA. It just moved it.

The hard part used to be: "How do I write this test?"

The hard part now is: "Should AI write this test, and how do I know if it's right?"

A junior engineer asks: "Can AI do this?" A Staff SDET asks: "Should AI do this, what risk does it introduce, how will I validate it, and what happens when the AI is wrong?"

That gap — between can and should — is the entire framework.

The 7-Question Decision Framework (Quick View)

1. What does the AI technology/tool actually do?
2. What QA problem does it solve?
3. Is AI actually necessary, or would deterministic automation be better?
4. What AI capability is being used?
5. What tools/integration are required?
6. How do I validate the AI's output?
7. What are the risks, limitations, and human responsibilities?

Below, each question gets a real scenario, a real prompt, and a real validation step. No generic "generate test cases for login" prompts here.

Question 1 — What does the AI technology/tool actually do?

Why an SDET should ask it: You cannot evaluate output you don't understand the mechanics of. An LLM predicts plausible text; it does not "know" your application. If you don't internalize that, you will trust output you shouldn't.

Scenario: A junior engineer wants to use an LLM to generate test cases for a new "Apply Coupon" feature from a Jira ticket.

AI tool/use case: General-purpose LLM (Claude, GPT-class model) used for natural-language requirement analysis.

Prompt:

You are a Senior SDET reviewing a requirement before test design.

Requirement:
"Users can apply one coupon code per order at checkout. Invalid or
expired codes should show an error. Coupon discount should apply to
subtotal before tax."

Do the following, in order:
1. List all assumptions you are making about this requirement.
2. List any ambiguities or missing information a developer/QA should
   clarify before test design begins.
3. Do NOT generate test cases yet. Only output assumptions and gaps.

Example input/context: One Jira ticket, no linked designs, no acceptance criteria attached.

Expected AI output type: A list like:

• Assumption: "one coupon per order" means additional codes are silently rejected, not queued.
• Gap: What happens if the coupon is valid but the cart becomes empty before submit?
• Gap: Is there a limit on coupon value relative to order value (can discount exceed subtotal)?

How to verify: Cross-check every assumption against the actual ticket, designs, and product owner — not against your memory of "how coupons usually work." Treat the list as a starting point for a 10-minute clarification conversation, not a finished artifact.

When NOT to use AI here: If the requirement is trivial and unambiguous (e.g., "field must be required"), skip this — you're adding process overhead for zero risk reduction.

Key lesson: AI is strongest at surfacing ambiguity, not at resolving it. Use it to ask better questions, not to skip asking them.

Question 2 — What QA problem does it solve?

Why an SDET should ask it: "We should use AI" is not a QA problem. "Our regression suite takes 40 minutes to triage after each nightly run" is. Tie every AI use case to a named, measurable pain point.

Scenario: Your team's CI pipeline produces 200+ automated test results nightly, and someone manually scans logs every morning to find real failures vs. flaky noise.

AI tool/use case: LLM-based log summarization and clustering, either via a script feeding failure logs to an LLM API, or a CI-integrated AI log tool.

Prompt:

You are a CI triage assistant for a test automation pipeline.

Below are failure logs from tonight's regression run. For each failure:
1. Extract the failing test name.
2. Classify the failure type: [Assertion Failure, Timeout,
   Element Not Found, Network/API Error, Environment Issue, Unknown].
3. Group failures that share the same likely root cause.
4. Flag any failure pattern that resembles known flaky behavior
   (e.g., timing-related, intermittent selector issues).
5. Output a ranked list: which failures need human investigation
   FIRST, and why.

Logs:
<paste raw CI failure logs here>

Example input/context: Raw stack traces and console logs from a Jenkins/GitHub Actions run, 15–30 failed tests.

Expected AI output type: A clustered table — e.g., "12 failures share a TimeoutError on #checkout-button, likely a shared root cause (staging environment slow to load) — investigate first."

How to verify: Spot-check 2–3 clusters manually against the raw logs. Confirm the "likely root cause" is a hypothesis, not a diagnosis — the AI is pattern-matching text, not executing your app.

When NOT to use AI here: If your suite only has 5–10 tests, manual triage is faster and more reliable than setting up an AI pipeline for it.

Key lesson: AI earns its place on volume and repetition, not novelty. Triage at scale is a textbook fit; triage of five tests is not.

Question 3 — Is AI actually necessary, or would deterministic automation be better?

Why an SDET should ask it: This is the question most engineers skip, and it's the one that separates a Staff-level thinker from someone chasing a trend. Deterministic code is cheaper, faster, and 100% reproducible for problems that don't require language understanding or judgment.

Scenario: You need to verify that an API response always returns status: "active" for a newly created user.

Why AI is the WRONG tool here:


❌ Don't do this:
"Ask an LLM to check if the API response looks correct."

An LLM introduces non-determinism, latency, and cost into a check that a one-line assertion solves perfectly:

// ✅ Deterministic automation - correct choice
expect(response.body.status).toBe("active");

When AI legitimately enters this scenario: Generating the initial draft of the test scaffolding, or generating edge-case ideas for the response schema (e.g., "what other states might a user API expose that we haven't tested?") — then a human converts those into deterministic assertions.

Key lesson (say this out loud in code review):

"If the correct answer is a fixed rule, write a rule. If the correct answer requires judgment or language understanding, that's where AI earns its seat."

This is the single most important discipline in the entire framework. AI-assisted testing should never replace an expect() statement that can be written in five seconds.

Question 4 — What AI capability is being used?

Why an SDET should ask it: "AI" is not one thing. Code generation, natural language reasoning, visual/image analysis, and autonomous multi-step agents have different failure modes. Naming the capability tells you what kind of validation you need.

Scenario: You want AI to generate a Playwright test for a multi-step checkout flow from a manual test case.

AI capability being used: Code generation grounded in natural-language instructions (not visual testing, not agentic browsing).

Prompt:

You are a Senior SDET writing a Playwright test in TypeScript.

Context:
- Framework: Playwright + TypeScript
- Page Object Model is used; page objects live in /pages
- Existing selector convention: data-testid attributes only
- Test must run in CI headless, must not use hardcoded waits (no
  page.waitForTimeout)

Manual test case:
"1. Go to checkout page with 1 item in cart.
 2. Apply coupon code 'SAVE10'.
 3. Verify discounted total reflects 10% off subtotal (not total
    with tax).
 4. Click 'Place Order'.
 5. Verify order confirmation page shows correct order ID format."

Write the Playwright test. Requirements:
- Use Page Object Model, assume a CheckoutPage class exists with
  methods: applyCoupon(code), getDiscountedTotal(), placeOrder()
- Use explicit waits tied to network/DOM state, never fixed timeouts
- Add one negative-path test: invalid coupon code
- Call out any missing selectors or page object methods you assumed
  exist, at the end, as a comment block

Example input/context: One manual test case + a short description of existing project conventions (this context matters enormously — a prompt without your project's real conventions produces unusable code).

Expected AI output type: A .spec.ts file using your POM structure, plus a clearly labeled comment block listing assumed methods/selectors that don't exist yet — this is the AI being honest about its own guesses, which you explicitly asked it to do.

How to verify:

• Run it locally before it ever touches CI.
• Check every assumed selector/method against the real codebase.
• Confirm the "no hardcoded waits" instruction was actually followed — AI models frequently insert waitForTimeout even when told not to.
• Review assertions for false positives (e.g., checking element exists instead of checking element value).

When NOT to use AI here: For extremely simple tests (a single field validation), writing it yourself is faster than prompting, reviewing, and correcting AI output.

Key lesson: The quality of AI-generated test code is a direct function of how much real project context you give it. A prompt with no conventions produces demo-quality code, not production-quality code.

Question 5 — What tools/integration are required?

Why an SDET should ask it: A good prompt is useless if it doesn't fit your actual pipeline. Before adopting an AI use case, map exactly where it plugs into your stack — IDE, CI runner, test management tool, or a standalone script.

Scenario: You want AI-generated API test scenarios to plug directly into your existing Postman/Newman or REST-assured suite, not live in a separate doc nobody maintains.

AI tool/use case: LLM prompt run through a script (or IDE plugin) that outputs structured JSON, consumed directly by your test generator.

Prompt:

You are generating API test scenarios for an SDET's automated suite.

Endpoint: POST /api/v1/orders
Request schema:
{
  "userId": "string (UUID)",
  "items": [{ "sku": "string", "quantity": "integer > 0" }],
  "couponCode": "string, optional"
}

Generate test scenarios covering:
- Valid requests (happy path)
- Boundary values (quantity = 0, quantity = max int, empty items array)
- Invalid types (quantity as string, missing userId)
- Security-relevant edge cases (SQL-injection-style strings in sku,
  oversized payloads)
- Business-rule edge cases (couponCode present but user has no cart)

Output as a JSON array, each object with fields:
{ "scenario_name": "", "request_body": {}, "expected_status": "",
  "expected_behavior": "", "risk_level": "low|medium|high" }

Only output valid JSON. No explanation text.

Integration point: This structured JSON output is designed to feed directly into a script that converts each object into a REST-assured or Newman test case — this is what makes it "integration-ready" instead of a wall of text you retype by hand.

How to verify: Validate the JSON schema programmatically before trusting it (a malformed field breaks your generator silently). Manually review the risk_level field — AI risk-ranking is a starting opinion, not a security audit.

When NOT to use AI here: If your API surface is small and stable (under ~10 endpoints, rarely changing), manually maintained scenarios are easier to keep accurate than an AI-regeneration pipeline.

Key lesson: The real ROI of AI in QA comes from integration, not novelty. A brilliant prompt that produces output you copy-paste by hand doesn't scale — structure your prompts to feed your existing tools.

Question 6 — How do I validate the AI's output?

Why an SDET should ask it: This is the question that determines whether you're doing "AI-assisted testing" or "AI-assisted incident creation." Validation is not optional, and it is not the same as "it compiled."

Scenario: A test that was passing for three months starts failing intermittently — 1 in 8 runs — and you want AI to help investigate before you spend an afternoon on it.

AI tool/use case: LLM-based root-cause reasoning over logs, diffs, and historical run data.

Prompt:

You are helping an SDET investigate a flaky test.

Test: "should display updated cart total after quantity change"
Framework: Cypress
Failure rate: ~1 in 8 CI runs, always in headless CI, never
reproduces locally.

Here is the failing run's log/error output:
<paste error + stack trace>

Here is the relevant test code:
<paste test code>

Here is the relevant application code being tested (cart update
handler):
<paste app code>

Analyze and respond in this structure:
1. Most likely root cause category (race condition, animation/timing,
   test isolation issue, environment resource contention, incorrect
   assertion, backend non-determinism) - with confidence level
   (low/medium/high).
2. Specific evidence from the logs/code that supports this.
3. Two alternative hypotheses, ranked by likelihood.
4. A minimal experiment I could run to confirm or rule out the top
   hypothesis (not a rewrite - an experiment).
5. What you are NOT confident about, explicitly.

Example input/context: Real logs, real test code, real handler code — vague or partial context produces vague or partial (and confidently wrong) diagnoses.

Expected AI output type: A ranked hypothesis (e.g., "high confidence: race condition — the assertion checks total before the debounced API call resolves") plus a small, falsifiable experiment ("add a network-idle wait tied to the specific PATCH call and rerun 20x in CI").

How to verify — this is the core discipline:

• Run the suggested experiment. Do not accept the hypothesis until it's been tested against the real, flaky test at least 10–20 times.
• If the "fix" makes the test pass once, that is not validation — flaky tests pass "by luck" too. Require a meaningfully higher pass rate over repeated runs.
• Never merge an AI-suggested fix into a shared test without your own understanding of why it works. If you can't explain it in code review, don't ship it.

When NOT to use AI here: If the flake is already well-understood by your team (e.g., "we know this suite has animation timing issues on Fridays"), skip the AI investigation and apply the known fix.

Key lesson: AI-generated root cause analysis is a hypothesis engine, not a verdict. Validation is the SDET's job, every time, with no exceptions — this is the line between engineering and guessing.

Question 7 — What are the risks, limitations, and human responsibilities?

Why an SDET should ask it: Every AI use case has a failure mode. Naming it in advance is how you prevent it from becoming an incident.

Scenario: Using AI to generate synthetic test data for a staging environment load test.

AI tool/use case: LLM-based synthetic data generation.

Prompt:

Generate 50 synthetic user records for load-testing a signup API.

Constraints:
- Fields: fullName, email, dateOfBirth (must include users aged
  17, 18, and 100+ for boundary testing), country (mix of at least
  8 countries, including ones with non-Latin characters), password
  (must include realistic weak AND strong examples)
- Do NOT use real people's names or real, resolvable email domains -
  use example.com or example.org only
- Include 5 intentionally malformed records (invalid email format,
  future date of birth, empty required field) explicitly labeled
  "INVALID - for negative testing"
- Output as CSV

Named risks for this use case:

How to verify: Scan the CSV for real-looking PII patterns, confirm invalid records are actually invalid in the ways intended, and confirm the file never gets referenced outside the test/staging scope.

When NOT to use AI here: Regulated domains (healthcare, finance) where synthetic data must meet specific compliance patterns — use a purpose-built synthetic data tool with compliance guarantees, not a general LLM prompt.

Key lesson: The risk isn't that AI gets something wrong. The risk is a team building a habit of not checking because AI usually gets it right. Usually is where incidents live.

AI Prompt ≠ AI Decision

This is the sentence I'd want every junior SDET to tattoo on their monitor:

A good SDET does not blindly execute whatever AI recommends.

The real workflow — the one that separates "using AI" from "engineering with AI" — looks like this:

flowchart TD
    A[Requirement] --> B["Problem: What are we actually trying to verify or solve?"]
    B --> C["AI Suitability Check — Would deterministic automation be better?"]
    C --> D["Prompt: structured, project-aware, explicit about constraints"]
    D --> E["AI Output (draft — never treated as final)"]
    E --> F["Validation: run it, test it, cross-check assumptions, check risk"]
    F --> G["Human Decision: accept / modify / reject — with a reason"]
    G --> H["Automation / CI Integration — only validated, understood work ships"]

Every step in this chain is a checkpoint. Skipping AI Suitability Check gets you AI-generated tests for problems a one-line assertion would've solved. Skipping Validation gets you flaky CI. Skipping Human Decision gets you tests nobody on the team can explain in six months.

One-liner for your team's Slack channel:

"AI can write the test. It can't own the test."

Staff-level read of this table: notice reliability goes down and required human involvement goes up as autonomy increases. That's not a flaw in AI tooling — that's the correct engineering relationship between autonomy and oversight. Anyone selling you "fully autonomous QA, zero human review" is selling you an incident.

The 5-Minute AI Adoption Checklist

Before you introduce any AI tool into a QA workflow, run through this:

• Named problem — Can I state the QA problem in one sentence without mentioning AI?
• Deterministic check — Have I confirmed a fixed rule/script can't solve this more reliably?
• Capability match — Do I know exactly which AI capability I'm using (text generation, code generation, reasoning, agentic action)?
• Context supplied — Does my prompt include real project conventions, not generic instructions?
• Validation plan — Do I know before I run the prompt how I will verify the output?
• Blast radius — If this output is wrong and I miss it, what's the worst that happens (flaky CI vs. a shipped bug vs. leaked data)?
• Owner — Is there a named human who reviews and approves before this reaches CI/production?
• Reversibility — Can I easily disable/roll back this AI-generated artifact if it causes problems?

If you can't check every box in five minutes, you're not ready to automate this with AI yet — and that's a useful, not embarrassing, conclusion.

How I Would Explain This to a Junior SDET

Imagine you hire a very fast, very well-read intern. They've read almost every testing blog, every framework doc, every Stack Overflow answer — but they've never actually run your app, never seen your production incidents, and they will confidently guess when they don't know something instead of saying "I don't know."

Would you let that intern:

• Draft your test cases? Yes — review before you use them.
• Write your first-pass Playwright script? Yes — review every selector.
• Decide alone that a flaky test is now fixed? No — make them prove it, ten times, in CI.
• Push straight to your production test suite unsupervised? Never.

That intern is exactly what AI is in your QA workflow. Brilliant draft-writer. Terrible unsupervised decision-maker. Treat it exactly that way and you'll get real value without real risk.

How a Staff SDET Thinks Differently

That last row is the whole game. AI doesn't get blamed for a bad test in a postmortem. The engineer who merged it does. Staff-level thinking means you never let a prompt make a decision you haven't personally signed off on.

Where This Is Heading (Without the Hype)

None of this means testers get replaced. It means the skill set keeps layering:

flowchart TD
    A["AI-assisted SDET — uses AI as a tool inside existing manual/automated workflows"] --> B["AI-augmented SDET — AI embedded across the workflow: test design, debugging, data, triage - with strong validation habits"]
    B --> C["Agentic Test Engineer — designs and supervises AI agents that execute multi-step testing tasks with defined guardrails"]
    C --> D["AI QA Agent Engineer — builds, evaluates, and governs the AI systems/agents QA teams rely on"]
    D --> E["AI Quality Engineering — quality engineering discipline expands to include AI system quality itself: evaluating models, prompts, and agents as first-class testable systems"]

To be clear: these are evolving skill directions, not standardized job titles you'll see on every job board tomorrow. Nobody should treat this as a fixed career ladder — treat it as a map of where the skills are heading, and start building the ones nearest to where you already are.

The testers who do well here aren't the ones who trust AI the most. They're the ones who validate it the best.

Try This Today

Pick one task from this article — requirement ambiguity analysis, log triage, or flaky test investigation — and run the real prompt on a real problem you have this week. Don't use it on a toy example. Use it on something in your actual backlog, then apply the validation step before you trust a single line of it.

That's the whole framework in practice: not "AI or no AI," but problem → suitability → prompt → validation → decision, every single time.

---

Which QA task would you trust AI with first — and which one would you never hand over to AI?

Drop it in the comments. I'll respond with how I'd validate (or reject) it.

Comments

[Baptiste Le Bouquin]

The "can vs should" framing is the part juniors actually need, because the failure mode I see most is not bad generated tests — it's plausible tests that encode an assumption nobody verified. A test that passes while checking the wrong thing is worse than no test, and the loading-spinner-in-CI example is a perfect illustration of why validation has to happen outside the AI's own context.

Question 3 (deterministic automation first) deserves to be bolded. Most of my AI-assisted test generation ended up being scaffolding for fixtures and edge-case enumeration, with the assertions still hand-written. That's not a failure of the framework — it's where the boundary actually sits.

[Pranta Kundu]

Really well put — "a test that passes while checking the wrong thing is worse than no test" nails the actual danger better than "hallucination" does. It fails quietly instead of loudly, so you don't find out until the bug ships anyway.

And agreed on Q3. That's exactly where the line should sit — AI drafts fixtures and edge cases, you write the assertion, because "what counts as correct" is a judgment call, not something to autocomplete. If you can't point to the assertion as your decision, that's the sign validation got skipped.