One of the most useful test reports I've seen an AI produce came from agents that never saw a single line of the application's code.
At a company I worked at, we tried a simple experiment. We gave the models the business rules of a system that was already built, and nothing else. No repository, no architecture docs, no database schema. The models wrote the tests from the rules, then ran those tests against the running application. At the end we got a list of every place where the application did not do what the business said it should.
That list was more useful than any green test suite, because every line in it was a rule the system broke. And the reason it worked is not that the models were especially clever. It's that they didn't know how the system was implemented, so they couldn't bend their tests around it.
This is black-box testing with AI agents: the agent tests the behavior against the specification, from the outside, the way a customer or an auditor would. This post covers:
- Why agents that can read the code write weaker tests than you think.
- How to set up a black-box pipeline: rules in, test cases, execution, findings report.
- The prompts, file layout and report format I'd use, plus when this approach is the wrong tool.
The problem: tests that confirm the code instead of the requirement
Ask a coding agent to "write tests for RefundService" and it will read RefundService, see what it does, and write tests that check it does exactly that. If the code has an off-by-one in the refund window, the agent sees daysSinceDelivery < 30, and the generated test asserts that day 30 is rejected. Green suite. Wrong behavior. Now locked in by a test.
Testers have a name for the source of "expected": the test oracle, whatever you use to decide whether an observed result is correct. This isn't a model failure. It's what any tester does when the code is the only spec they have: the implementation becomes the oracle, so the tests can only ever confirm it.
It gets worse when the same agent wrote the code and the tests. Both come from the same understanding of the problem, so any misunderstanding shows up twice and cancels out. I covered a version of this in the context of dark factories: a check is only worth something when the thing being checked can't shape it.
| Oracle derived from the code | Oracle derived from the rules | |
|---|---|---|
| Source of "expected" | What the code currently does | What the business rule says |
| Off-by-one in a rule | Encoded as correct | Flagged as a failure |
| Missing rule in the code | Easy to miss (no code to suggest a test) | Found: the rule exists, the behavior doesn't |
| Ambiguous rule | Silently resolved by the implementation | Surfaced as an open question |
| Refactoring the code | Tests often break | Tests survive, behavior is the contract |
That third row is the one people miss. If nobody implemented "only the order owner can request a refund", there's no code that suggests the test. A black-box agent reads the rule and tries it.
To be fair to white-box testing: an agent with the code and an independent list of requirements could catch this too. Code access isn't the defect. When an agent derives its expectations from the implementation, it reproduces the misunderstandings already in the code. In my view, keeping the code away from the testing agent is the simplest way to stop that from happening.
What black-box testing means (and what agents add to it)
Black-box testing is old. The ISTQB calls it specification-based testing: you derive test cases from the requirements and the observable behavior, without looking at the internal structure. The classic techniques are still the right ones (chapter 4 of the ISTQB Foundation Level syllabus, free to download):
- Equivalence partitioning: group inputs that should behave the same and test one from each group.
- Boundary value analysis: test at and right next to each limit (29, 30 and 31 days for a 30-day window).
- Decision tables: list every combination of conditions in a rule that has several of them.
- State transition testing: for anything with a lifecycle, test valid and invalid transitions.
The techniques were never the hard part. The hard part was the labor: turning fifty business rules into a few hundred precise cases is slow, boring work, and people skip the boring edges. That's exactly where LLMs are useful. They're good at reading a rule in plain language and systematically expanding it into partitions, boundaries and combinations, and they don't get bored at case 140.
What the agent must not do is invent the expected result. The value of the whole approach depends on expectations coming from the rules, so the pipeline has to make that a hard boundary, not a polite request.
The pipeline: rules in, findings out
This is the setup I'd build today. After a one-time setup (step 0, isolation), it has four steps: write the rules, design the cases, generate the tests, then run and triage. Three agents do the work, and humans review at two checkpoints.
business rules ──▶ [1. test designer] ──▶ test cases + open questions
│
human review
│
[2. test generator] ◀──────┘
│
executable tests ──▶ run against staging (public interfaces only)
│
[3. triage] ◀──────────────┘
│
findings report ──▶ human decides: bug, rule gap or bad test
Step 0 (setup): isolate the agents from the code
The simplest way to stop an agent from reading the code is to make sure the code isn't there. I wouldn't rely on instructions ("don't open src/") or on a deny list that may not cover every tool. Give the testing agents their own workspace:
qa-workspace/ # everything the testing agents can see
├── rules/refunds.md # business rules, each with an ID
├── contracts/openapi.yaml # the public API contract clients already get
├── cases/ # generated test cases (Markdown)
├── tests/ # generated Playwright tests
└── .env.test # BASE_URL + test-only credentials (gitignored)
# The application repository is not cloned here.
The agents talk to the system only through what a real client would use: the public API and the UI of a staging environment. If the agent can't reach it from outside, it isn't part of the test.
Isolation is necessary, but it doesn't make the oracle independent on its own. That depends on where the expectations come from. If someone writes the rules by watching what the current system does, or "fixes" a failing expectation to match what the system returned, the oracle depends on the implementation again, just one step removed. Expectations come from rules that the business has reviewed, and a failing test changes only when the rule changes.
Step 1: write the rules so they can be tested
Garbage in, garbage out applies hard here. The rules don't need a formal language, but each one needs an ID and a single, checkable statement. Here's a hypothetical refund policy for an online store, which I'll use for the rest of the post:
# Refund rules
- BR-01: A customer can request a refund up to 30 days after delivery.
- BR-02: Refunds above 500.00 EUR require manager approval before payout.
- BR-03: Digital items cannot be refunded after they have been downloaded.
- BR-04: The total refunded for an order can never exceed the amount paid.
- BR-05: Only the customer who placed the order can request its refund.
Notice that these are written the way a product owner would write them, not the way a developer would. That's the point. If your rules only exist in people's heads or in Jira comments, collecting them into a file like this is the first deliverable, and often a valuable one on its own.
Step 2: the test designer agent
The first agent turns rules into cases. The prompt does two jobs: it tells the model which techniques to apply, and it forbids guessing.
You are a QA analyst. You have not seen the application's code and you
must not ask for it.
Input: the business rules below, each with an ID.
For every rule:
1. Derive test cases using equivalence partitioning, boundary values,
decision tables (for combined conditions) and state transitions
(for lifecycles). Include invalid and unauthorized cases.
2. Write each case as: case ID, rule ID(s), technique, preconditions,
Given/When/Then, expected result.
3. Expected results must come only from the rules. If a rule does not
say what should happen, do not guess: add the case to an
"Open questions" list instead.
Output: a Markdown table of cases, then the open questions.
For BR-01 and BR-02, a good output looks something like this (trimmed):
| Case | Rule | Technique | Given / When | Expected |
|---|---|---|---|---|
| TC-01 | BR-01 | Boundary | Delivered 29 days ago, refund requested | Accepted |
| TC-02 | BR-01 | Boundary | Delivered 30 days ago, refund requested | Accepted |
| TC-03 | BR-01 | Boundary | Delivered 31 days ago, refund requested | Rejected |
| TC-04 | BR-02 | Boundary | Refund of 500.01 EUR | No payout until a manager approves |
| TC-05 | BR-02 | Boundary | Refund of 500.00 EUR | No manager approval required |
| TC-06 | BR-05 | Partition | User B requests refund on user A's order | Rejected |
And the open questions list:
Q1 (BR-01): Is "30 days" counted in calendar days, and in which time zone?
Q2 (BR-03): Does a partial download count as "downloaded"?
Q3 (BR-02 + BR-04): Do two refunds of 300 EUR on the same order need approval?
Q4 (BR-02): At 500.00 EUR or less, is the payout immediate or still queued?
The open questions are a deliverable, not noise. Every one of them is a place where two developers could have implemented different behavior and both claimed they followed the spec. A product owner answers them in minutes, and the answers go back into the rules file.
This is the first human checkpoint: someone who knows the business skims the cases and answers the questions before anything runs. It's a much cheaper review than reading test code.
Step 3: generate executable tests, not live improvisation
You could let an agent click through the app and decide on the fly whether things look right. I wouldn't. A run you can't repeat is a run you can't trust, and you want to re-run the same suite after every fix.
So the second agent turns approved cases into ordinary test code, here with Playwright, which covers both API and UI tests:
import { test, expect } from "@playwright/test";
import { createDeliveredOrder, loginAs } from "./fixtures";
// TC-02 / BR-01: day 30 after delivery is still inside the refund window
test("TC-02 BR-01: refund accepted on day 30 after delivery", async ({ request }) => {
const customer = await loginAs(request, "customer-a");
const order = await createDeliveredOrder(request, customer, { deliveredDaysAgo: 30 });
const response = await request.post(`/api/orders/${order.id}/refunds`, {
headers: customer.headers,
data: { amount: order.total },
});
// BR-01 says "accepted"; 201 is how the API contract says "accepted"
expect(response.status()).toBe(201);
});
This snippet is illustrative: loginAs and createDeliveredOrder are fixtures you have to provide, and baseURL comes from the Playwright config. The test name carries the case and rule IDs, which is what makes the final report traceable back to the business.
Notice where the 201 comes from. The business rule only says the refund is accepted; it says nothing about HTTP. The status code comes from the public API contract in the workspace, which is still a black-box source: it's what any client integrating with the API reads. So the generator gets two inputs, the approved cases (the behavior) and the contract (how that behavior shows up on the wire), and neither one is the implementation. If the contract doesn't say how "accepted" looks, that's another open question, not a guess.
If you test through the UI, Playwright's own test agents (planner, generator and healer, set up with npx playwright init-agents --loop=claude, or the loop for your tool) can do much of this step. The planner can take a product requirements document as context, which fits this approach well. Review what the healer changes, though: in a black-box setup, a test that fails because the app is wrong must stay failing.
Step 4: run, triage and report
Run the suite against staging. Then a third agent, or the same one in a fresh session, turns the raw results into the report people actually read. For each failure it records the rule, the case, what was expected, what happened, and the evidence (response body, screenshot or trace).
The agent also proposes a verdict, because not every red test is a bug:
- Bug: the system contradicts a clear rule.
- Rule gap: the rule doesn't say, and the system picked a behavior. Goes to the product owner.
- Bad test: the agent misread the rule or the test setup is wrong. Goes back to step 2.
The proposed verdict is a starting point; the second human checkpoint is someone confirming it before anything becomes a ticket.
A worked example: what the findings report looks like
Here's what a report for the refund rules could look like. This is a hypothetical run, built to show the format and the kinds of issues the approach catches:
## Findings: refund rules, staging
| # | Rule | Case | Expected | Actual | Verdict |
|---|-------|----------------------------------------------|-----------------|------------------------------|----------------|
| 1 | BR-01 | Refund on day 30 after delivery | Accepted | 422 "refund window expired" | Bug |
| 2 | BR-04 | Two partial refunds totalling 110% of order | Second rejected | Both accepted | Bug |
| 3 | BR-05 | User B refunds user A's order | Rejected | 201 Created | Bug (security) |
| 4 | BR-02 | Two refunds of 300 EUR on one order | Unclear (Q3) | Paid without approval | Rule gap |
Look at what each line represents. Finding 1 is the off-by-one a white-box agent would have locked in. Finding 2 only shows up when you combine two rules, which is exactly what decision tables are for. Finding 3 is a broken access control bug: an ownership rule written in business language turned into a security test without anyone asking for a "security test". Finding 4 isn't a bug at all, it's a business decision nobody made.
This is the kind of output I saw at that company, and it's why the approach stuck. Developers argue with test code. It's much harder to argue with "the rule says X, the system did Y, here's the response". The report speaks the language of the people who own the rules, so it gets acted on.
Trade-offs: when black-box agents are the wrong tool
This approach complements your existing tests. It doesn't replace them.
| Black-box agents | White-box agent tests | Human exploratory QA | |
|---|---|---|---|
| Finds missing behavior | Yes | No | Sometimes |
| Finds edge cases in internal logic | Only if observable | Yes | Rarely |
| Speed of feedback | Minutes to hours (needs a deployed env) | Seconds | Days |
| Setup cost | Rules file, staging env, test data fixtures | Low | People's time |
| Best at | "Does the system do what the business asked?" | "Does this function do what it should?" | "Does this feel right to a user?" |
Skip it, or don't expect much from it, when:
- The rules aren't written down. If the spec is the code, the agent has nothing independent to test against. Writing the rules comes first.
- You can't set up state from the outside. Black-box tests need a way to create "an order delivered 30 days ago" through an API, a seed endpoint or fixtures. Without that, most boundary cases are untestable.
- The behavior isn't observable. Caching, retries, internal queues: if the effect doesn't show up in an API response, a UI or an email, a black-box test can't see it. Those belong in unit and integration tests.
- The question is about UX, performance or load. Those need other tools and, for UX, people.
Two costs are easy to underestimate. First, false failures: the agent will sometimes misread a rule, and every bad test costs someone time to triage. The open-questions step and the human review of cases are what keep that rate down. Second, a stable environment: a flaky staging environment turns every run into noise, and noise is how teams stop reading the report.
FAQ
Isn't this just BDD with extra steps?
It shares the idea that behavior is the contract, and Given/When/Then is a fine format for the cases. The difference is who writes the scenarios and who can see what. In BDD the team writing the code usually writes the scenarios too. Here, the agent writing the tests never sees the code.
Can the same model write the code and the black-box tests?
The same model, yes. The same session or context, no. What matters is that the test designer's input is the rules and nothing derived from the implementation. A fresh session in a separate workspace with no repository is enough.
Doesn't the agent need the code to find UI selectors?
No. Playwright locators work best with roles and visible labels (getByRole, getByLabel), which come from what the user sees, not from the source. If a button can't be found by its role and name, that's often an accessibility finding in itself.
How is this related to holdout scenarios?
It's the same principle. StrongDM's team, as described by Simon Willison, keeps end-to-end scenarios outside the codebase so coding agents can't see them. Black-box testing from rules is a way to build that holdout set without hand-writing every scenario.
What about secrets and test credentials?
Use test-only accounts on a staging environment, keep them in a gitignored env file, and never point the agents at production. The agents only need what a regular customer would have.
Takeaway
- An oracle derived from the code confirms bugs instead of finding them. Expectations must come from somewhere the implementation can't shape.
- Give the testing agent the rules and the public contract, nothing from the implementation. Physical isolation (no repository in its workspace) beats instructions, but reviewed rules are what make the oracle independent.
- Make expected results come only from the rules. Anything the rules don't cover becomes an open question, not a guess.
- Generate repeatable test code with rule IDs in the test names, so every failure traces back to a business rule.
- Deliver a findings report, not a test log. "Rule X, expected Y, got Z" is what gets bugs fixed.
- Keep humans at two points: approving the cases and confirming the verdicts.
Generating more tests was never the hard part. What AI agents make cheap is an independent check of whether the software keeps the promises the business made, written in the business's own words.
Have you tried letting agents test a system without showing them the code? I'd like to hear what the first findings report surprised you with, or why you decided against it.
I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where every article becomes a visual carousel you can save, like the 4-step version of this pipeline. This article also has a video walkthrough on my YouTube channel, Diogo Soares Dev, so subscribe there for the next ones. Let's also connect on LinkedIn, or check out diogosoares.dev.
Top comments (4)
The refund-window example is the cleanest statement of the oracle problem — an agent that reads daysSinceDelivery < 30 will lock day 30's rejection in as correct and call the suite green. Keeping the code away from the testing agent is a blunt but honest fix, since a shared misunderstanding between code-writer and test-writer cancels out instead of catching anything. Where I'd push: ambiguous rules surface as open questions in your pipeline, but who resolves them — does the human checkpoint rewrite the rule, or does the triage agent decide the rule was the bug?
Thanks, that's a fair push, and I should have made it more explicit in the post. In my view, it should never be the agent. The triage agent can propose "rule gap" as a verdict, but I wouldn't let it edit a rule or decide the rule was wrong. That goes to whoever owns the rule (usually the product owner), who answers the open question, and the answer goes back into the rules file under the same ID. Only then would I regenerate the cases.
My reasoning is the same oracle argument: if an agent could rewrite a rule to match what the system returned, the expectations would depend on the implementation again, just one step removed. So the split I'd aim for is: humans own the rules, agents own the expansion and the evidence.
That split is the clean version — humans own the rules, agents own the expansion and the evidence, and the answer going back under the same ID is the detail that makes it stick, since the cases regenerate against a versioned rule rather than a quietly rewritten expectation. Your oracle argument closes it: an agent rewriting a rule is just the implementation grading itself one step removed. The one thing I'd watch is the open-question queue itself — does an unanswered question block case generation, or can a rule sit open while new cases keep piling up around it?
Good question. In my view, an open question should block only the cases that depend on it, not the whole rule. In the refund example, the time zone question only affects the cases right at the 30-day boundary; days 29 and 31 don't depend on it and can run. The dependent cases stay marked as pending, with the question ID, so nobody turns them into tests by guessing.
To keep the queue from quietly growing, I'd put the open questions in the findings report itself, with how long each has been waiting. A rule that has had an unanswered question for weeks is a finding too: it means nobody actually owns that behavior.