Mutation Testing That Actually Fits Into Your Workflow
Mutation testing tools (mutmut, Cosmic Ray, MutPy) exist, but most teams don't use them in practice. The bottleneck is runtime: waiting 20 minutes per mutation run makes the feedback loop too long to fit inside a normal TDD cycle.
pytest-gremlins v1.3.0 ships today with UX fixes and correctness patches that were blocking production workflows.
What Is pytest-gremlins?
pytest-gremlins is a mutation testing plugin for pytest. It injects small code modifications into your source (flipping > to >=, changing and to or, negating return values), then runs your tests to see which mutations survive. Survivors indicate code that is covered but not validated: the tests execute it, but no assertion distinguishes the mutated behavior from the original.
Four mechanisms drive its speed:
- Mutation switching: instrument the code once with all mutations embedded, then toggle active mutations via environment variable. No file rewrites, no module reloads between runs.
- Coverage-guided test selection: only run the tests that cover the mutated line, reducing test executions by 10-100x per mutation.
- Incremental caching: skip re-testing mutations on unchanged code. Cached results are keyed by content hash.
- Parallel execution: distribute mutation subprocesses across CPU cores. Mutation switching makes this safe because there is no shared mutable state between workers.
In parallel mode, pytest-gremlins is 3.73x faster than mutmut. With the cache warm on a second run, that rises to 13.82x faster. It also finds more mutations: 117 vs. mutmut's 86, with a 98% kill rate vs. 86%.
Getting Started
pip install pytest-gremlins
# Run mutation testing on your project
pytest --gremlins
# Parallel execution (recommended for speed)
pytest --gremlins --gremlin-parallel
# With coverage report alongside
pytest --gremlins --gremlin-parallel --cov
pytest-gremlins auto-discovers your source paths from pyproject.toml setuptools metadata, falling back to src/ if metadata is absent. No config file required for the common case.
What's New in v1.3.0
--gremlin-workers Now Implies Parallel Mode
In v1.2.x, enabling parallel execution required two flags:
pytest --gremlins --gremlin-parallel --gremlin-workers=4
In v1.3.0, specifying a worker count enables parallel mode implicitly:
pytest --gremlins --gremlin-workers=4
pytest --gremlins --gremlin-parallel # use all CPU cores
The --gremlin-parallel flag remains valid as an explicit opt-in when you want all cores without pinning a count.
--gremlins --cov Actually Works Now
Combining pytest-gremlins with pytest-cov previously corrupted the .coverage data file. The mutation pre-scan subprocess wrote to .coverage without isolation, overwriting or merging data from the main test run. The coverage report was either wrong or missing.
# v1.3.0: mutation results and accurate coverage report together
pytest --gremlins --gremlin-parallel --cov --cov-report=html
The pre-scan subprocess now suppresses coverage instrumentation, so the .coverage file written by the main run is not clobbered.
Clear Error When Combining with pytest-xdist
Before v1.3.0, running pytest --gremlins -n auto produced zero mutations tested and zero output, with no indication of the conflict.
v1.3.0 fails immediately with a diagnostic:
ERROR: --gremlins and -n (pytest-xdist) cannot be combined.
Use --gremlin-workers for parallel mutation execution.
The two flags solve different problems and cannot operate simultaneously. -n parallelizes test collection and execution by distributing them across worker processes that share a single pytest session. --gremlin-workers launches isolated mutation subprocesses, each running a complete pytest session with a single mutation active. Running both at once breaks the subprocess isolation that mutation switching depends on.
Windows Path Fix
A path separator bug in WorkerPool's sources.json caused worker failures on Windows. The fix is in place and cross-platform support is now covered by CI.
Subprocess Isolation
Mutation subprocesses now suppress addopts and coverage instrumentation inherited from the host environment. Previously, host pytest configuration could leak into mutation subprocess runs, producing incorrect results on projects with complex addopts entries.
Typical Output
================== pytest-gremlins mutation report ==================
Zapped: 142 gremlins (85%) <- tests caught the mutation
Survived: 18 gremlins (11%) <- test gaps to investigate
Timeout: 5 gremlins (3%)
Error: 2 gremlins (1%)
Mutation score: 85%
Coverage: 94% of lines covered
========================== 167 gremlins ==========================
pyproject.toml Configuration
[tool.pytest-gremlins]
paths = ["src"] # directories to mutate
exclude = ["**/migrations/*"] # exclude patterns
operators = ["comparison", "boolean", "arithmetic"] # which mutations to run
Performance Numbers
Benchmarked against mutmut (Python 3.12):
| Mode | Time | vs mutmut |
|---|---|---|
--gremlins (sequential) |
17.79s | 0.84x |
--gremlins --gremlin-parallel |
3.99s | 3.73x faster |
--gremlins --gremlin-parallel --gremlin-cache |
1.08s | 13.82x faster |
Sequential mode runs slightly slower than mutmut because of subprocess isolation overhead per mutation. The tradeoff is correctness and mutation coverage: 117 mutations found vs. mutmut's 86, with a 98% kill rate vs. 86%.
Top comments (0)