Solution architect, 16 years in software, last 3 on AI automation. I write about making automated work verifiable: evaluation harnesses, audit trails, and results published even when the answer is no.
The mutation-testing note is the one part I would put a boundary on, because I ran it against your fixture. mutmut 3.7.0 on CPython 3.14.6, your corrected implementation as the only source file and just the first two checks as the suite: 5 mutants, 5 killed, 0 survivors. Same run against the plausible patch, the one with if not statuses that you already showed is wrong: also 5 mutants, 5 killed, 0 survivors. The operators it generated were is None to is not None, list(orders) to list(None), two string mutations on the status key, and in to not in — the mutation you name as the useful one, statuses is None to not statuses, is not in that set, since the operators rewrite a node in place rather than swap an identity test for a truthiness test. So the mutation score gives a perfect grade to both implementations under the suite that cannot separate them, which leaves your review question doing all the work rather than the tool.
This is a really useful catch. You’re right, mutmut can give a perfect score here while missing the semantic mutation that actually matters. That makes the “plausible wrong implementation” check more important than the mutation score itself. Would you treat mutation tools as a supplement, with semantic mutants derived from the contract?
Solution architect, 16 years in software, last 3 on AI automation. I write about making automated work verifiable: evaluation harnesses, audit trails, and results published even when the answer is no.
Supplement, and specifically as a carrier rather than a generator. I ran mutmut 3.7.0 on the corrected implementation against both suites: 5 mutants, 5/5 killed with the two checks, and 5/5 killed again once test_empty is added, so the score is flat across exactly the check this article is about. It cannot rank the two suites any better than it could rank the two implementations. The contract mutant is what separates them: statuses is None becomes not statuses, which passes the two-check suite and fails test_empty in the three-check suite, so the catalogue reads 5/6 against 6/6 and every bit of the ranking power comes from the one mutant you have to derive from the requirement by hand. Boundary: CPython 3.14.6, mutmut default operators, this fixture only.
That comparison makes the limitation very concrete: adding the missing requirement check improves the suite, while the default mutation score stays unchanged. The useful unit to preserve is the named contract violation and the assertion that rejects it. Thanks for running both suites and stating the environment; that makes the result much easier to interpret without generalizing beyond this fixture.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The mutation-testing note is the one part I would put a boundary on, because I ran it against your fixture. mutmut 3.7.0 on CPython 3.14.6, your corrected implementation as the only source file and just the first two checks as the suite: 5 mutants, 5 killed, 0 survivors. Same run against the plausible patch, the one with
if not statusesthat you already showed is wrong: also 5 mutants, 5 killed, 0 survivors. The operators it generated wereis Nonetois not None,list(orders)tolist(None), two string mutations on thestatuskey, andintonot in— the mutation you name as the useful one,statuses is Nonetonot statuses, is not in that set, since the operators rewrite a node in place rather than swap an identity test for a truthiness test. So the mutation score gives a perfect grade to both implementations under the suite that cannot separate them, which leaves your review question doing all the work rather than the tool.This is a really useful catch. You’re right, mutmut can give a perfect score here while missing the semantic mutation that actually matters. That makes the “plausible wrong implementation” check more important than the mutation score itself. Would you treat mutation tools as a supplement, with semantic mutants derived from the contract?
Supplement, and specifically as a carrier rather than a generator. I ran mutmut 3.7.0 on the corrected implementation against both suites: 5 mutants, 5/5 killed with the two checks, and 5/5 killed again once
test_emptyis added, so the score is flat across exactly the check this article is about. It cannot rank the two suites any better than it could rank the two implementations. The contract mutant is what separates them:statuses is Nonebecomesnot statuses, which passes the two-check suite and failstest_emptyin the three-check suite, so the catalogue reads 5/6 against 6/6 and every bit of the ranking power comes from the one mutant you have to derive from the requirement by hand. Boundary: CPython 3.14.6, mutmut default operators, this fixture only.That comparison makes the limitation very concrete: adding the missing requirement check improves the suite, while the default mutation score stays unchanged. The useful unit to preserve is the named contract violation and the assertion that rejects it. Thanks for running both suites and stating the environment; that makes the result much easier to interpret without generalizing beyond this fixture.