There is a ceiling on your test coverage and it is not technical.
Running tests has been a solved problem for a long time. Runners are fast, parallelism is cheap, CI is a commodity. None of that touches the actual limit, which is that somebody has to sit down, read the requirement, work out what could go wrong, and write the cases.
So coverage ends up being a function of how much of that a person had time to do. Not what could break. Not what changed in this release. What fitted in the sprint.
A worked example
Take one line of acceptance criteria:
As a customer I can apply a promo code at checkout.
Ask a competent engineer to cover it and you will reliably get three cases:
- a valid code is accepted
- an invalid code is rejected
- an expired code is rejected
Those three are not a failure of skill. They are the three cases stated in the requirement. The requirement has one happy path and two named error conditions, so you get one happy path and two named error conditions.
Now here is a different list for the same line:
- the same code applied twice in one basket
- a code at exactly the minimum basket value
- a code combined with a gift card
- a code on a partially refunded order
- a code applied in a second currency
- a code that expires between basket and payment
Every one of those is a defect I would bet on. And not one of them is in the requirement.
The pattern
Those six are not a grab bag. They fall into categories, and the categories generalise to anything you are testing:
| Category | The question it asks | In the example |
|---|---|---|
| Idempotency | What if this happens twice? | the same code applied twice |
| Boundary | What happens exactly at the threshold? | exactly the minimum basket value |
| Composition | What if another feature is active at the same time? | code plus gift card |
| Prior state | What if the entity has history? | a partially refunded order |
| Locale | What if the assumptions about format are wrong? | a second currency |
| Time-of-check to time-of-use | What if state changes between validation and commit? | expires between basket and payment |
That last row is the interesting one, because it is a genuine race and it is the one least likely to be written by hand. Validation happens when the code is entered. Commitment happens at payment. If your expiry check runs only at entry, the bug is invisible in every test that treats the basket as instantaneous - which is every test somebody writes at their desk.
The requirement will never mention it. It is not a requirement. It is a consequence of the requirement meeting a real system.
This is the actual gap. Not execution. Not maintenance. Enumeration.
So generate them - but that is where it gets interesting
Producing that second list from the requirement plus the code is tractable. Language models are good at enumerating category x feature combinations, and the categories above are a decent prompt on their own. You can try it this afternoon.
The hard part is that a generated case is only worth having if four things are true.
1. It is traceable to something. A case you cannot tie back to a clause in the requirement or
a branch in the code is a case nobody can review, and a case nobody reviews will not survive its first false failure. Whatever generates the case has to carry the provenance with it.
2. It is prioritised. Six extra cases per acceptance criterion, across a real backlog, is not
coverage - it is a suite nobody runs. Generation without ranking converts an enumeration problem into a scheduling problem. Something has to say these two matter for this change.
3. It survives maintenance. This is where self-healing selectors earn their place, and it is
worth being precise about why. Self-healing on a hand-written suite is a convenience. Self-healing on a generated suite is a precondition - because the whole proposition is more tests, and more tests hand-maintained is just more maintenance. Healing is the floor that makes the volume viable. It is not the feature.
4. It fails loudly when it is unsure. Any mechanism that relocates a moved element is making a
judgement. A tool that quietly picks the nearest match when nothing scores well has not saved you a failure; it has converted a red test into a green one that proves nothing. A heal nobody can review is a silent pass, and a silent pass is worse than a broken test because you stop looking.
The honest trade
Generation does not remove human judgement from testing. It moves it - from authoring cases to
reviewing them. That is a genuinely better place for it to sit, because reviewing a proposed
case against a requirement is faster than inventing one from scratch, and because the categories above are easier to check than to remember.
But it is a trade, not a free win. If your team has no capacity to review, generation will produce a large suite of plausible tests with unexamined assumptions baked in, and you will have moved the problem rather than solved it.
Start with one acceptance criterion. Generate against the six categories. See how many of the results you would actually have written. That number is your coverage ceiling, and it is usually uncomfortable.
Disclosure: I work at 2SD Technologies, where we build TAI - a testing platform that does the generation, prioritisation and healing described above.
Top comments (0)