Your closing question also has a countable answer, and the count decides whether the next assertion is worth reading.
I ran your fixture against eight wrong implementations, one per contract axis instead of one per token: return orders instead of list(orders); not statuses; filter ignored entirely; order["status"] == statuses; reversed output; one pass per requested status (so ["paid", "paid"] returns duplicates); a re-sort of the caller's list; and append-a-sentinel (every returned value is correct, the caller's list silently grows). Rejected:
suite
wrong implementations rejected
your two checks
3/8
+ the empty-list check
4/8
+ f(ORDERS) is not ORDERS
5/8
+ order preservation
7/8
+ "the caller's list is unchanged"
8/8
Two things fall out. The value of an added assertion is exactly the number of enumerated wrong implementations that only it rejects - my duplicate-status check (f(ORDERS, ["paid", "paid"]) == [ORDERS[0]]) rejected nothing the order check did not already reject, so on this fixture it is decoration. And the surviving axes are the ones operators do not generate: list(orders) to return orders is a one-click diff that passes your two checks, your third, and the fails-before/passes-after test; only an identity check, or a caller that mutates what it got back, separates them. Your fix writes list(orders) deliberately and nothing in the fixture observes that decision.
That is also the reading I would take from vinhnguyenthanhdn's mutmut run above: 5 mutants, 5 killed, identical for your corrected implementation and for the if not statuses version you already showed is wrong. Token-level operators on a five-line function mostly produce mutants that raise (list(None)) or move a string literal; they do not produce "right symptom, wrong contract", which is the failure mode this post is about. A 100% mutation score here is not evidence about the suite, it is evidence that the mutants were drawn from the code rather than from the requirements.
For the agent loop I would upgrade step 1 accordingly: write the expected behavior down as three to five separable wrong implementations per requirement, with the axis named - empty vs. missing, identity, order, side effects on inputs - and treat the suite as acceptable only if it rejects each of them. The list is cheap to keep next to the issue, and it tells you which assertion to add: the one that rejects an entry no current check rejects.
This is a much stronger framing than my original step 1. I like the idea of treating each requirement as a small set of named wrong implementations, then adding assertions only when they kill a survivor. That also explains why 100% mutation coverage can still say almost nothing about semantic coverage here.
Would you keep those wrong implementations as explicit fixtures in the repo, or generate them from the contract during the agent loop?
Dug into that, because the generate-vs-pin split turns out to be measurable. I applied a deliberately generous single-operator set to the corrected implementation - comparison and identity flips, membership flips, wrapper removal (list(orders) to orders), comprehension-direction reversal, in to ==, return-value substitution, literal replacement - and asked which of my eight axis implementations some mutant actually reproduces. Four of eight. Under the five mutants vinhnguyenthanhdn listed above, zero.
The four nothing reaches are unreachable by construction: operators remove or replace, they never insert. The duplicate-status axis needs a loop that isn't there, the caller-list-grows axis is an added statement, the in-place sort is a new call. And no operator rewrites a condition into a different condition, which is why statuses is None never becomes not statuses - that is a different reading of the requirement, not an edit to the same expression. Where a single operator does land on one of my axes, it's because the axis coincides with a wrong output on probes that already exist, which is a weaker thing than naming the clause.
So, neither pole: generate to discover, pin to regress. Let the agent propose candidates from the requirement text rather than from the code inside the loop, and pin the proposals that survive review as fixtures next to the contract with their axis named - because the artefact you maintain is the requirement, and the fixture is the executable form of a clause nobody had written down. The acceptance criterion then has to be mechanical rather than a judgement: a proposal counts only if it is a real implementation (it must not raise), it satisfies every other requirement, and the current suite still rejects it. Any proposal that no check kills is either a missing requirement or a missing check, and you have to say which - that rule is what keeps the loop from congratulating itself on the kill count.
One pruning cost to budget for: in my run one of my own added checks killed nothing the order check hadn't already killed, so it was decoration. Per fixture, ask which check exists solely to kill it; no answer, drop the fixture. And the enumeration only stays tractable while it's bounded - one axis per clause plus the three that cross every clause (identity, ordering, side effects) is roughly what I used, and I don't have a good story for a contract with fifty clauses.
Generate to discover, pin to regress is a useful split. I’d keep the violated clause, candidate implementation, and rejecting assertion together. One caution on pruning: a check that adds no unique kills in today’s catalogue may still cover behavior the catalogue misses. I’d call it redundant against those eight candidates, rather than redundant against the contract. And identity or side-effect requirements need to be explicit before we score implementations against them.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Your closing question also has a countable answer, and the count decides whether the next assertion is worth reading.
I ran your fixture against eight wrong implementations, one per contract axis instead of one per token:
return ordersinstead oflist(orders);not statuses; filter ignored entirely;order["status"] == statuses; reversed output; one pass per requested status (so["paid", "paid"]returns duplicates); a re-sort of the caller's list; and append-a-sentinel (every returned value is correct, the caller's list silently grows). Rejected:f(ORDERS) is not ORDERSTwo things fall out. The value of an added assertion is exactly the number of enumerated wrong implementations that only it rejects - my duplicate-status check (
f(ORDERS, ["paid", "paid"]) == [ORDERS[0]]) rejected nothing the order check did not already reject, so on this fixture it is decoration. And the surviving axes are the ones operators do not generate:list(orders)toreturn ordersis a one-click diff that passes your two checks, your third, and the fails-before/passes-after test; only an identity check, or a caller that mutates what it got back, separates them. Your fix writeslist(orders)deliberately and nothing in the fixture observes that decision.That is also the reading I would take from vinhnguyenthanhdn's mutmut run above: 5 mutants, 5 killed, identical for your corrected implementation and for the
if not statusesversion you already showed is wrong. Token-level operators on a five-line function mostly produce mutants that raise (list(None)) or move a string literal; they do not produce "right symptom, wrong contract", which is the failure mode this post is about. A 100% mutation score here is not evidence about the suite, it is evidence that the mutants were drawn from the code rather than from the requirements.For the agent loop I would upgrade step 1 accordingly: write the expected behavior down as three to five separable wrong implementations per requirement, with the axis named - empty vs. missing, identity, order, side effects on inputs - and treat the suite as acceptable only if it rejects each of them. The list is cheap to keep next to the issue, and it tells you which assertion to add: the one that rejects an entry no current check rejects.
This is a much stronger framing than my original step 1. I like the idea of treating each requirement as a small set of named wrong implementations, then adding assertions only when they kill a survivor. That also explains why 100% mutation coverage can still say almost nothing about semantic coverage here.
Would you keep those wrong implementations as explicit fixtures in the repo, or generate them from the contract during the agent loop?
Dug into that, because the generate-vs-pin split turns out to be measurable. I applied a deliberately generous single-operator set to the corrected implementation - comparison and identity flips, membership flips, wrapper removal (
list(orders)toorders), comprehension-direction reversal,into==, return-value substitution, literal replacement - and asked which of my eight axis implementations some mutant actually reproduces. Four of eight. Under the five mutants vinhnguyenthanhdn listed above, zero.The four nothing reaches are unreachable by construction: operators remove or replace, they never insert. The duplicate-status axis needs a loop that isn't there, the caller-list-grows axis is an added statement, the in-place sort is a new call. And no operator rewrites a condition into a different condition, which is why
statuses is Nonenever becomesnot statuses- that is a different reading of the requirement, not an edit to the same expression. Where a single operator does land on one of my axes, it's because the axis coincides with a wrong output on probes that already exist, which is a weaker thing than naming the clause.So, neither pole: generate to discover, pin to regress. Let the agent propose candidates from the requirement text rather than from the code inside the loop, and pin the proposals that survive review as fixtures next to the contract with their axis named - because the artefact you maintain is the requirement, and the fixture is the executable form of a clause nobody had written down. The acceptance criterion then has to be mechanical rather than a judgement: a proposal counts only if it is a real implementation (it must not raise), it satisfies every other requirement, and the current suite still rejects it. Any proposal that no check kills is either a missing requirement or a missing check, and you have to say which - that rule is what keeps the loop from congratulating itself on the kill count.
One pruning cost to budget for: in my run one of my own added checks killed nothing the order check hadn't already killed, so it was decoration. Per fixture, ask which check exists solely to kill it; no answer, drop the fixture. And the enumeration only stays tractable while it's bounded - one axis per clause plus the three that cross every clause (identity, ordering, side effects) is roughly what I used, and I don't have a good story for a contract with fifty clauses.
Generate to discover, pin to regress is a useful split. I’d keep the violated clause, candidate implementation, and rejecting assertion together. One caution on pruning: a check that adds no unique kills in today’s catalogue may still cover behavior the catalogue misses. I’d call it redundant against those eight candidates, rather than redundant against the contract. And identity or side-effect requirements need to be explicit before we score implementations against them.