Turns out that a very old test was broken for a while and wasn't running in the automatic build, whoops.
It would be very helpful to find the exact commit that broke the test. Well, git-bisect to the rescue: git bisect will binary search the commit history to find the bad commit. And we can use pytest to tell git if the test is passing or not for any possible commit.
Here is a toy repository with a simple function and a unit test:
$ ls my_script.py README
And this is
from functools import reduce from operator import mul def product(factors): return reduce(lambda x , y: x - y, factors) def test_product(): assert product([1, 2, 3,]) == 6
Pretty simple so far.
test_product broke for some reason.
Let's look at the commit history:
$ git log --oneline 7e26872 Unrelated commit # 21 f67f19e Unrelated commit # 20 ... 2b37410 Unrelated commit # 15 80956a1 Unrelated commit # 14 d2a3327 Some innocent change that certainly did not break anything 8cc3f8b Unrelated commit # 13 1b09ced Unrelated commit # 12 ... 3e73dfa Unrelated commit # 2 790ba87 Unrelated commit # 1 0c75c27 unrelated commit d7af196 Added product function with a unit-test
There are several ways to use
bisect. We are going to tell it when the test was passing, and how to run the test, and it will take care of the rest.
We begin with
Tell it that our current commit is broken:
$ git bisect bad HEAD
Let's find a commit where we know the test was passing. Say, the commit that introduced the test:
$ git blame my_script.py | grep test_product d7af196a (oren 2019-06-24 12:48:35 -0700 8) def test_product():
bisect this commit is good:
$ git bisect good d7af196a Bisecting: 11 revisions left to test after this (roughly 4 steps) [97c3a24f28f72bec1885445eb3947e446d0804fe] Unrelated commit # 10
Now we start bisect off with a command to run the test, bisect uses the exit code of the command to tell whether each commit is good or bad, that works because
pytest has a non-zero exit code if tests fail (And,
pytest can run a specific function):
git bisect run pytest -qqq my_script.py::test_product .... .... .... d2a3327df55cb9bf9b43780f9adb223e93ba093d is the first bad commit commit d2a3327df55cb9bf9b43780f9adb223e93ba093d Author: oren <----@----.---> Date: Mon Jun 24 12:50:58 2019 -0700 Some innocent change that certainly did not brake anything :100644 100644 0be2cd4c2d0e450026507ee86e058785de56cadc 98c1b14d74c4f46b538df5cc1d32b9bb42afc791 M my_script.py bisect run success
And indeed, this is the bad commit:
$ git show d2a3327df55cb9bf9b43780f9adb223e93ba093d commit d2a3327df55cb9bf9b43780f9adb223e93ba093d Author: oren <----@----.---> Date: Mon Jun 24 12:50:58 2019 -0700 Some innocent change that certainly did not brake anything ....... ....... def product(factors): - return reduce(mul, factors) + return reduce(lambda x , y: x - y, factors)
Note: if you are using a script as a command to
git bisect run. Make sure this script is not tracked by git, otherwise it might change when git changes revisions.