Breaking CI Guards on Purpose to Prove They Can Fail

Code: Megapixel99/canfail

A CI guard that has never failed may be incapable of failing. A lint rule disabled by a config merge, a type check whose glob stopped matching, a schema validation step pointed at the wrong directory, a security scanner with an empty ruleset. Every one of them is green forever, and green is what you were looking for. For code guarded by tests, mutation tools like Stryker generate the breaks automatically and you should use them. canfail is for the guards they do not cover, where the thing being guarded is YAML, Terraform, a Dockerfile, or anything else without a function to mutate. You declare a break (a file, an anchor string, its replacement, and what failure you expect). The tool applies it, runs your check, and reports whether the check noticed.

pip install canfail
canfail canfail.json

{
  "checks": [{
    "name": "unit tests", "run": ["python3", "example/test_prices.py", "-q"],
    "breaks": [{
      "name": "total: multiply -> add",
      "file": "example/src/prices.py",
      "replace": "item[\"price\"] * item[\"quantity\"]",
      "with": "item[\"price\"] + item[\"quantity\"]",
      "expect": "assert|Error"
    }]
  }]
}

It is a mutation harness, a very small one aimed at CI configuration, so it inherits the rules that make one trustworthy. The check must pass on the clean tree first; a check that was already red tells you nothing when you break something. A failure is not a catch: it has to fail for the reason you named. A break that just makes the file unparseable is scored wrong-failure, and a check that hit its timeout exits 124, which looks exactly like going red and settles nothing. The anchor must match exactly once. Zero matches means the break never happened, and two means the check was asked about code nobody was thinking of. And the file must come back, verified by digest, which is restore-verified's whole job. canfail used to carry 78 lines of that guard inline, about a quarter of the module, so it could claim no dependencies. Deleting the copy once the real package existed was the right trade: a second copy of a guarantee is a second thing to get wrong, and the copy is the one that never gets the upstream's tests.

The example configuration in the repository declares four breaks against one check and yields four different outcomes in a single run, closing with the tally 4 declared break(s): 1 caught, 1 not caught, 2 not settled. CI asserts that tally line rather than just the exit code, because exit 1 alone would be satisfied by finding the wrong thing.

The paragraph I most want to keep is about a fix of mine that measurement killed. The first working version reported a genuinely blind guard as catching, but only when its break ran second. After the first break, Python had written __pycache__ bytecode from the broken source, and the second break's run executed that stale bytecode, failing for the previous break's reason. My fix was to stop restoring mtime on the guarded file. Then canfail got the treatment it gives other people's guards, deliberate breaks to its own source with its suite as the check, and that mutation pass showed the fix does nothing. Re-enabling bytecode caching breaks the test that pins the ordering bug whether or not mtime is restored. mtime invalidation has one-second granularity, this tool edits, runs and restores in milliseconds, and a .pyc written from the broken source therefore looks fresh either way. PYTHONDONTWRITEBYTECODE is the load-bearing guard; not restoring mtime is a cheap belt beside it. The general form is worth carrying to any tool with a fast edit cycle: anything keyed on mtime (bytecode caches, make, ninja, file watchers) is blind on sub-second edits.

blind also means two different things unless you make it say which: the check ran and did not notice, or the check never ran at all. The first says your guard is weak and the second says it is missing. They send you to opposite ends of the CI file. Declaring evidence on a check hands that question to didrun, so a break that stops the check from running becomes a refusal rather than a finding. Omit it and the report says outright that it cannot tell the two apart. An evidence object naming no known predicate is a config error rather than a no-op. A misspelled key silently downgrading the check is this package's own failure mode pointed at itself.

That self-directed suspicion earned its keep. The current README carries a section titled "Things this got wrong about itself", and every entry is a verdict the tool reported without having earned it. A killed check scored as a catch, because exit 124 is non-zero. A wrote evidence path resolved in the tool's directory while the check ran in --cwd, so a check that wrote its report perfectly was reported as never having run. An uncompilable expect regex reached the interpreter as a traceback and exited 1, the code that means a guard is blind. A break applied to a CRLF file came back as a whole-file diff, because universal-newline reading stripped the returns. Each is now pinned by a test that fails without the fix.

The score so far is six mutations against the original properties: five caught, and the sixth (restoring mtime) survived, which is how the paragraph above got corrected. Six more were applied against the later fixes, and all six were caught. The adoption cost is honest too: you have to write the breaks yourself. For Python code guarded by a Python suite, the README points you at PyPI's mutation-testing instead, which swaps __code__ objects and never touches the disk at all. Everything canfail carries about restores and signals is apparatus it needs only because it edits real files. That is what buys it the config formats nothing else covers.