DEV Community

Blake Yang
Blake Yang

Posted on

The Mutant Run: Catching Weakened Tests in an OSS Patch With Three Git Worktrees

#opensource #testing #git #ai

A patch lands on a small utility library. It adds the missing test for a paging helper, fixes a ceiling-division bug, and, in the same commit, rewrites two older assertions into looser >= comparisons. CI is green, the diff reads cleanly, and the review takes ninety seconds. Six weeks later, a pagination regression slips through the exact code path those softened assertions used to guard.

That scenario is a composite, not a specific incident, and it can be rebuilt locally in about ten minutes. Everything below runs against a four-file fixture repository that any reader can construct with the commands shown. The goal is a mechanical check that a patch's tests actually observe the patch's behavior, instead of merely coexisting with it.

Why a green suite proves less than it appears

A passing suite answers one narrow question: the assertions that exist did not fail. It says nothing about whether those assertions are strict enough, or whether the new test would also pass on the unfixed code. Reviewers usually compensate with attention, which is exactly the resource that runs out on a Friday afternoon.

Three cheap signals restore most of the missing information. The first is whether the new test fails on the base commit. The second is whether the suite fails when the fix is removed but the tests are kept. The third is how many assertion lines the patch deletes or relaxes.

The three states every patch should visit

Each state is a detached Git worktree, so no branch juggling and no stash risk is involved:

  1. Base state — upstream tip plus the contributor's test changes only. The new test must fail here, and it must fail for the intended reason.
  2. Patched state — the contributor's full diff applied. The suite must pass here.
  3. Mutant state — the patched tree with src/ reverted to base and the tests left alone. The suite must fail here, or the tests do not observe the fix at all.

State three is the one most review workflows skip, and it is the only state that detects a test written to describe the fix rather than to constrain it.

Step 1: build the fixture repository

The fixture is deliberately small: one module, one test file, one bug. The bug is a missing ceiling division that returns zero pages for any total smaller than the page size.

mkdir tamper-demo && cd tamper-demo
git init -q
mkdir -p src tests bin
Enter fullscreen mode Exit fullscreen mode

src/paging.py on the base commit:

def pages(total, size):
    return total // size
Enter fullscreen mode Exit fullscreen mode

tests/test_paging.py on the base commit:

from src.paging import pages


def test_exact_multiple():
    assert pages(10, 5) == 2


def test_large_total():
    assert pages(11, 5) == 2


def test_zero_total():
    assert pages(0, 5) == 0
Enter fullscreen mode Exit fullscreen mode

Commit that as main, then create a patch branch that fixes the division, adds a fourth test, and weakens two existing assertions. The added test uses pages(10, 5) == 2, which passes on the buggy code, so it captures nothing.

Step 2: the three-state script

The script below assumes pytest is installed and that src/ is the only directory carrying the fix. Both assumptions are adjustable, and the limitations section returns to them.

#!/usr/bin/env bash
# bin/three-state-check.sh <base-ref> <patch-ref>
set -euo pipefail

BASE_REF="${1:-origin/main}"
PATCH_REF="${2:-HEAD}"
PY="${PY:-python3}"
WORK="$(mktemp -d)"
LOG="$WORK/report.txt"
status=0

cleanup() {
  for d in base patched mutant; do
    git worktree remove --force "$WORK/$d" 2>/dev/null || true
  done
}
trap cleanup EXIT

note() { printf '%s\n' "$*" | tee -a "$LOG"; }

suite() { ( cd "$1" && "$PY" -m pytest -q 2>&1 | tail -n 25 ); }

git worktree add -q --detach "$WORK/base"    "$BASE_REF"
git worktree add -q --detach "$WORK/patched" "$PATCH_REF"
git worktree add -q --detach "$WORK/mutant"  "$PATCH_REF"
git -C "$WORK/mutant" checkout -q "$BASE_REF" -- src/

note "== state 1: base ($BASE_REF) =="
base_out="$(suite "$WORK/base")"
printf '%s\n' "$base_out" | tee -a "$LOG"
if printf '%s' "$base_out" | grep -qE '[0-9]+ passed' && \
   ! printf '%s' "$base_out" | grep -qE '[0-9]+ failed'; then
  note "WARN base is green; the failing reproduction is missing"
fi

note "== state 2: patched ($PATCH_REF) =="
patched_out="$(suite "$WORK/patched")"
printf '%s\n' "$patched_out" | tee -a "$LOG"
if printf '%s' "$patched_out" | grep -qE '[0-9]+ failed'; then
  note "FAIL the patch does not make the suite pass"
  status=1
fi

note "== state 3: mutant (fix reverted, tests kept) =="
mutant_out="$(suite "$WORK/mutant")"
printf '%s\n' "$mutant_out" | tee -a "$LOG"
if ! printf '%s' "$mutant_out" | grep -qE '[0-9]+ failed'; then
  note "FAIL the suite passes without the fix; the tests observe nothing"
  status=1
fi

note "== assertion churn =="
added="$(git diff -U0 "$BASE_REF" "$PATCH_REF" -- 'tests/' \
  | grep -E '^\+[^+]' | grep -c 'assert' || true)"
removed="$(git diff -U0 "$BASE_REF" "$PATCH_REF" -- 'tests/' \
  | grep -E '^-[^-]' | grep -c 'assert' || true)"
note "assertions added=$added removed=$removed"
if [ "${removed:-0}" -gt 0 ]; then
  note "REVIEW the test diff first: $removed assertion line(s) removed"
fi

exit "$status"
Enter fullscreen mode Exit fullscreen mode

Step 3: read the output like a reviewer

Running bin/three-state-check.sh main patch on the fixture produces a report close to the following. The exact pytest counts vary with the fixture, and the verdict lines do not.

== state 1: base (main) ==
1 failed, 3 passed in 0.11s
== state 2: patched (patch) ==
4 passed in 0.09s
== state 3: mutant (fix reverted, tests kept) ==
4 passed in 0.09s
FAIL the suite passes without the fix; the tests observe nothing
== assertion churn ==
assertions added=1 removed=2
REVIEW the test diff first: 2 assertion line(s) removed
Enter fullscreen mode Exit fullscreen mode

Two independent signals point at the same weakness. The mutant run proves the new test cannot distinguish fixed from broken code, and the churn count shows that two earlier guarantees were softened. Neither signal is conclusive on its own, but together they justify a request for changes instead of an approval.

The failure message matters as much as the failure itself. A mutant run that fails on an import error has proved nothing about the assertion, so the log should be checked for AssertionError or the named test before the result counts.

Where free models genuinely help, and where they do not

A free model is useful for the parts of this loop that are text-shaped and low-risk, and it is a poor substitute for the maintainer's judgment. The division of labor that has held up in practice looks like this:

  • Model-assisted: rewriting a failing mutant log into a two-sentence PR comment, drafting the missing test from the diff, naming the observable behavior a patch changes, and proposing which existing assertions a refactor legitimately touches.
  • Maintainer-only: deciding whether a weakened assertion is justified, judging whether a suite is flaky rather than wrong, and approving the merge.

A prompt skeleton that keeps the model inside those bounds:

You are given: a unified diff, the base test log, the patched test log,
and the mutant test log. Do not propose a merge decision.
1. List the observable behaviors the diff changes.
2. For each new test, state whether it would fail without the diff.
3. Propose one additional test that would fail in the mutant state.
4. Mark anything you cannot tell from the logs as unknown.
Enter fullscreen mode Exit fullscreen mode

Free model access through MonkeyCode covers the drafting steps above without a paid key, and the free server option gives the three worktrees a disposable host instead of the reviewer's laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Capacity and terms for both the free model access and the free server option are operator-supplied and can change, so check the currently published terms before depending on them in a team workflow.

The disposable host is a real convenience rather than a marketing point. Three worktrees plus a virtualenv plus a test cache accumulate quickly, and a host that can be destroyed after the report is written makes "start from a clean state" the default rather than a chore. Running the script there and copying only the report log back keeps the review artifact small and auditable.

git clone --quiet https://example.invalid/your-fork.git verify
cd verify && git fetch --quiet origin pull/1234/head:pr-1234
PY=python3 bin/three-state-check.sh origin/main pr-1234 > report.txt 2>&1 || true
Enter fullscreen mode Exit fullscreen mode

A decision table for patch triage

The table converts the script output into a repeatable first response, which matters more than any single verdict.

Signal Merge Ask the author Block
New test fails at base, passes patched, fails mutant yes — —
Mutant state passes — — yes
Assertion lines removed — yes, with a written reason —
Test files exceed 30% of the diff — yes —
Base state already green — yes, reproduction missing —

Limitations and failure modes

The mutant run is a heuristic, and it degrades in predictable ways that reviewers should anticipate:

  • Reverting src/ can break imports, so the mutant failure may originate outside the assertion under test.
  • Suites that assert on mocks or log strings often stay green in the mutant state even when behavior changed.
  • Flaky tests make state three noisy, and a retry policy is needed before the signal is trusted.
  • Large monorepos need affected-test selection, which reduces the mutant run's coverage to the selected subset.
  • Patches that legitimately rename or restructure tests will trip the churn counter and require manual reading.

Who should skip this workflow

Repositories without a working test command should not start here, because the script only amplifies a suite that already exists. Teams that treat tests as documentation rather than constraints will find state three permanently red and will disable it within a week. Maintainers reviewing one small patch a month will get more value from reading the test diff carefully than from maintaining three worktrees and a shell script.

The workflow earns its keep on repositories where patch volume outpaces reviewer attention, and where a weakened assertion is expensive to discover late. Three worktrees, one script, and a log file are enough to make the check routine, and readers who want a throwaway host for the run can trial MonkeyCode's free server option on a scratch fork and compare the report against their local output.

chmod +x bin/three-state-check.sh
./bin/three-state-check.sh main patch; echo "exit=$?"
Enter fullscreen mode Exit fullscreen mode

Top comments (1)

Collapse
 
vishisht_choudhary_a12d08 profile image
Vishisht Choudhary •

The mutant state is such a clean idea, basically one-shot mutation testing aimed at exactly the lines the patch says it fixes. A small gotcha if you script it: if the patch adds new files under src/, git checkout base -- src/ leaves them in place, so the mutant can quietly keep part of the fix. Deleting src/ before checking it out from base avoids that. Does your relaxed-assertion count catch swapped matchers too (like toEqual becoming toContain), or just comparison operators?