One of the most useful test reports I've seen an AI produce came from agents that never saw a single line of the application's code.
At a company ...
For further actions, you may consider blocking this person and/or reporting abuse
The refund-window example is the cleanest statement of the oracle problem — an agent that reads daysSinceDelivery < 30 will lock day 30's rejection in as correct and call the suite green. Keeping the code away from the testing agent is a blunt but honest fix, since a shared misunderstanding between code-writer and test-writer cancels out instead of catching anything. Where I'd push: ambiguous rules surface as open questions in your pipeline, but who resolves them — does the human checkpoint rewrite the rule, or does the triage agent decide the rule was the bug?
Thanks, that's a fair push, and I should have made it more explicit in the post. In my view, it should never be the agent. The triage agent can propose "rule gap" as a verdict, but I wouldn't let it edit a rule or decide the rule was wrong. That goes to whoever owns the rule (usually the product owner), who answers the open question, and the answer goes back into the rules file under the same ID. Only then would I regenerate the cases.
My reasoning is the same oracle argument: if an agent could rewrite a rule to match what the system returned, the expectations would depend on the implementation again, just one step removed. So the split I'd aim for is: humans own the rules, agents own the expansion and the evidence.
That split is the clean version — humans own the rules, agents own the expansion and the evidence, and the answer going back under the same ID is the detail that makes it stick, since the cases regenerate against a versioned rule rather than a quietly rewritten expectation. Your oracle argument closes it: an agent rewriting a rule is just the implementation grading itself one step removed. The one thing I'd watch is the open-question queue itself — does an unanswered question block case generation, or can a rule sit open while new cases keep piling up around it?
Good question. In my view, an open question should block only the cases that depend on it, not the whole rule. In the refund example, the time zone question only affects the cases right at the 30-day boundary; days 29 and 31 don't depend on it and can run. The dependent cases stay marked as pending, with the question ID, so nobody turns them into tests by guessing.
To keep the queue from quietly growing, I'd put the open questions in the findings report itself, with how long each has been waiting. A rule that has had an unanswered question for weeks is a finding too: it means nobody actually owns that behavior.
Dependency-scoped blocking is the right call — the whole rule stalling on one boundary question would be worse than the ambiguity itself, and marking dependents pending with the question ID stops anyone from quietly guessing their way to green. Putting the queue's age in the findings report is the part that makes it work: an unanswered question stops being invisible and becomes an ownership finding. One question on the mechanics: when the owner finally answers, do the pending cases regenerate automatically, or does someone have to notice and re-trigger them?
I'd make it automatic, but never silent. The rules file lives in git, so answering a question is a commit that updates the rule under the same ID and closes the question. That commit triggers regeneration of exactly the cases that referenced it, and the new cases go through the same human checkpoint before they run. Nobody has to remember to re-trigger anything, and nothing reaches the suite without review.
Thanks for the thoughtful questions, this thread filled a gap the post left open.
Bringing Black Box Testing back with AI is a really neat concept, and being able to test the rules against what is implemented is a great automatic way of testing a requirements matrix against what was built. I agree keeping the tests away from the code also helps ensure independent oversight of what was built, away from having other tests ensure that existing tests are catching the right things.
My only gap is how many fixtures are you making? The idea of letting the rules direct the agent to do the test code seems a good way of having the tests built without someone matching behavior, and saving the tedious work of writing out all the test code, which makes me consider that the fixtures will match behavior not expectation. How are you also keeping them separate from implementation?
Thanks, and that’s a fair gap to point at. You can have an independent oracle and still leak the implementation back in through the test data.
The way I think about it: fixtures only set up preconditions, never expectations. Each test case already states the state it needs (“an order delivered 30 days ago, paid €600, owned by user A”), and the fixtures are derived from that list, inside the testing workspace, not from a database dump or the dev team’s seed scripts. If a fixture is wrong, the test tends to fail or come out inconclusive. It can’t make a wrong behaviour look correct, because the expected outcome still comes from the rule.
Setup also goes through the outside: the public API, or a test-setup endpoint documented in the contract, never direct inserts based on the schema.
As for how many, I’d aim for one fixture per precondition class, not one per test. Equivalence partitions keep it small: for the refund rules, a few order ages around the 30-day boundary, amounts around €500, a downloaded and a not-downloaded digital item, and two users.
Where the state can’t be built from outside (controlling the clock for “delivered 30 days ago”, for example), that’s exactly where I’d say this approach stops fitting, or where it becomes an open question for whoever owns the environment.
The isolation-from-repo argument is right, but the oracle can still leak implementation through the back door: if the OpenAPI contract is generated from the running code (common), then an expectation like 'day 30 → 201' is code-derived one step removed, so 'physical isolation beats instructions' only holds if the contract is hand-maintained. Same risk in your fixtures — a createDeliveredOrder that seeds state through an internal or admin endpoint can hide the exact access-control bug your BR-05 case is hunting, because the test never exercises the path a real customer takes. I'd add a check that the contract and the seed endpoints are themselves black-box artifacts, otherwise the pipeline quietly re-imports the assumptions it was built to exclude.
That's a sharp catch, and I agree with both, with one distinction each.
On the contract: in the pipeline it only decides how "accepted" shows up on the wire, never whether day 30 is accepted, so a generated contract can't turn the off-by-one into the truth. But you're right that it's still implementation-derived, so in my view it belongs on the same side of the line: reviewed and versioned, ideally written contract-first, before the code. Spec-before-code is what this pipeline depends on.
On fixtures: seeding through a test-only path is hard to avoid (you can't wait 30 days for a delivered order), but I'd draw the line at the action under test. Setup may take shortcuts; the request being tested and its assertions go through the real customer path with real customer credentials. For BR-05, the order would be seeded for customer A, but the refund request would go through the public endpoint as customer B, which is what lets it catch the access-control bug. A fixture that leaks an admin token into that step would hide it, so that's worth checking explicitly.
A black-box suite benefits from mutation-style checks on the specification itself: change one boundary expectation and make sure the test fails. That guards against examples that merely mirror the current implementation or an overly permissive oracle.
Great addition, thanks. In my view this is the black-box version of mutation testing: since the testing agent never touches the code, you mutate the oracle instead. Flip one expectation, rerun, and if the suite stays green, that test was never measuring the rule.
The boundary triples get part of this for free. Days 29, 30 and 31 have different expected outcomes, so an “anything goes” assertion fails at 31. Spec mutation extends that to rules without a natural boundary, like ownership checks or manager approval, where a weak assertion is easier to miss.
One caveat I’d keep in mind: a mutation check proves a test can fail, not that its expectation is right. So I’d see it as an automated check after the generation step, next to the human review of the cases, not as a replacement for it.
That separation suggests a useful audit trail: retain the original oracle, each single-rule mutation, and the observed failing assertion. Review can then distinguish a test that is sensitive to a changed rule from one that merely fails for an unrelated setup reason.
The refund window example is a good choice because it is so ordinary. daysSinceDelivery < 30 looks right in review, and only a test written from the rule, refunds allowed within 30 days, shows whether day 30 is in or out. That is the oracle problem in one line.
Agentic Affairs makes the point I would underline: humans own the rules, and open questions block only the work that depends on them. That keeps the agent moving without letting it settle ambiguity by guessing.
Qingfeng's warning about a generated OpenAPI contract leaking implementation is worth repeating. Once the contract describes how the code works instead of what the business needs, the tests start agreeing with the code by design.
Alex Shev's idea of mutating the oracle and Michael's rule that fixtures set preconditions, not expectations, would both fit in a definition of done for client work. Have you found a format for the rules that business owners will actually read and sign?