Improving Python code incrementally

Photo: Matěj Baťha / CC-BY-SA 2.5

---

Many of us have adopted tools to improve code quality, like

• code formatters,
• linters, and
• test coverage measurement.

For Python, such tools include for example:

• Black (code formatting)
• isort (import sorting & formatting)
• Pylint (code quality)
• Flake8 (code style)
• Mypy (static typing)
• coverage.py (test coverage)

For new and small projects, it's simple to enforce such automatic code modifications and quality checks for the whole code base. It can be done locally with Git hooks, as well as in a CI pipeline.

But if you have a large existing code base, or are contributing to an Open Source project, it may not be feasible to turn all checks green in one go.

One good reason to not reformat a code base in one go is that authorship information is lost for all reformatted lines, which may make collaboration on bugs and features trickier from then on. While git --ignore-rev offers one solution for this problem, this blog post presents another alternative.

Introducing Darker – apply Black formatting gradually

The idea behind Darker is to reformat code using Black (and optionally isort), but only apply new formatting to regions which have been modified by the developer.

By applying only changes suggested by Darker, the codebase will gradually converge towards a point where all code is formatted according to Black rules.

Darker can also run quality checks. The violations are filtered so you only see ones touching newly modified regions.

Finally, the pytest-darker plugin integrates all this with Pytest so reformatting and checks are run along with unit tests.

I'm sold, how to use Darker?

In the following example, I'll show first what Darker would have done in a real-world situation for a popular Open Source project when used manually on the command line.

I'll also show how to integrate it with the test suite using the pytest-darker plugin.

Case study: dateutil

Let's first clone the dateutil package:

$ git clone https://github.com/dateutil/dateutil.git
$ cd dateutil

In January 2020, there was a tiny feature branch which we can use as a guinea pig for Darker (see dateutil/dateutil#993):

$ export GIT_PAGER=cat

$ git log --graph --oneline a332879..6edfecd
* 6edfecd Update documentation links
*   110a09b Merge pull request #993 from pganssle/parsererror_repr
|\  
| * ea7f441 Fix custom repr for ParserError
|/  
* 21fe6e9 Merge pull request #987 from eastface/issue_981

We can now recreate the situation before the pganssle/parsererror_repr branch was merged by resetting master back to before the branch...

$ git reset --hard 21fe6e9
HEAD is now at 21fe6e9 Merge pull request #987 from eastface/issue_981

...and recreating the pganssle/parsererror_repr branch at the point before the merge was done:

$ git checkout -b pganssle/parsererror_repr ea7f441
Switched to a new branch 'pganssle/parsererror_repr'

Our repository is now in a similar state as before the branch was merged:

$ git log --graph --oneline master^..
* ea7f441 (HEAD -> pganssle/parsererror_repr) Fix custom repr for ParserError
* 21fe6e9 (master) Merge pull request #987 from eastface/issue_981
* a332879 Added Mark Bailey to AUTHORS file
* 79d2e48 Fix TypeError in parser wrapper logic

The changes introduced by the pganssle/parsererror_repr branch are:

$ git diff -U0 master
diff --git a/changelog.d/991.bugfix.rst b/changelog.d/991.bugfix.rst
new file mode 100644
index 0000000..473082e
--- /dev/null
+++ b/changelog.d/991.bugfix.rst
@@ -0,0 +1 @@
+Fixed the custom ``repr`` for ``dateutil.parser.ParserError``, which was not defined due to an indentation error. (gh issue #991, gh pr #993)
diff --git a/dateutil/parser/_parser.py b/dateutil/parser/_parser.py
index 7fcfa54..8d67584 100644
--- a/dateutil/parser/_parser.py
+++ b/dateutil/parser/_parser.py
@@ -1603,2 +1603,3 @@ class ParserError(ValueError):
-        def __repr__(self):
-            return "%s(%s)" % (self.__class__.__name__, str(self))
+    def __repr__(self):
+        args = ", ".join("'%s'" % arg for arg in self.args)
+        return "%s(%s)" % (self.__class__.__name__, args)
diff --git a/dateutil/test/test_parser.py b/dateutil/test/test_parser.py
index 605705e..cfa4bbb 100644
--- a/dateutil/test/test_parser.py
+++ b/dateutil/test/test_parser.py
@@ -945,0 +946,9 @@ def test_decimal_error(value):
+
+def test_parsererror_repr():
+    # GH 991 — the __repr__ was not properly indented and so was never defined.
+    # This tests the current behavior of the ParserError __repr__, but the
+    # precise format is not guaranteed to be stable and may change even in
+    # minor versions. This test exists to avoid regressions.
+    s = repr(ParserError("Problem with string: %s", "2019-01-01"))
+
+    assert s == "ParserError('Problem with string: %s', '2019-01-01')"

On the surface, all the changes look Black compliant. If the whole code base at master was already Black formatted, we could simply run Black to see if we missed anything in the pganssle/parsererror_repr branch. But alas:

$ pip install -q black

$ black --diff .
### ... LOTS of output redacted ...
All done! ✨ 🍰 ✨
35 files would be reformatted, 3 files would be left unchanged.

Ok thanks Black, but we're only interested in whether the changes in this branch are well formatted.

Unfortunately Black doesn't offer this functionality out of the box. Instead we'll use Darker:

$ pip install -q darker

$ darker --diff --revision master... .
--- dateutil/test/test_parser.py
+++ dateutil/test/test_parser.py
@@ -944,6 +944,7 @@
     with pytest.raises(ParserError):
         parse(value)

+
 def test_parsererror_repr():
     # GH 991 — the __repr__ was not properly indented and so was never defined.
     # This tests the current behavior of the ParserError __repr__, but the

Ha! So we did miss one newline before a function definition!

We can now just run Darker without the --diff flag so it actually writes that change to the source code:

$ darker --revision master... . 

$ git diff -U1
diff --git a/dateutil/test/test_parser.py b/dateutil/test/test_parser.py
index cfa4bbb..53a02de 100644
--- a/dateutil/test/test_parser.py
+++ b/dateutil/test/test_parser.py
@@ -946,2 +946,3 @@ def test_decimal_error(value):

+
 def test_parsererror_repr():

Running Darker as part of the test suite

We can now install pytest-darker:

$ pip install -q pytest-darker

We'll configure Darker so it always checks but doesn't modify source files. This is done in the pyproject.toml file:

[tool.darker]
src = [
    "."
]
check = true
diff = true
revision = "master..."

Now we can run the test suite and have it check the formatting of the feature branch as well:

$ pip install --quiet --requirement requirement-dev.txt
$ pytest --darker
====================== test session starts =======================
# ... lots of output ...
dateutil/test/test_parser.py F............................ [ 38%]
.......................................................... [ 40%]
.......................................................... [ 43%]
.......................................................... [ 46%]
...................xxxxxxxxxxxxx..........                 [ 48%]
# ... more output ...
============================ FAILURES ============================
______________________ Darker format check _______________________
Edited lines are not Black formatted
---------------------- Captured stdout call ----------------------
--- dateutil/test/test_parser.py
+++ dateutil/test/test_parser.py
@@ -944,6 +944,7 @@
     with pytest.raises(ParserError):
         parse(value)

+
 def test_parsererror_repr():
     # GH 991 — the __repr__ was not properly indented and so was never defined.
     # This tests the current behavior of the ParserError __repr__, but the
==================== short test summary info =====================
FAILED dateutil/test/test_parser.py::DARKER
===== 1 failed, 2012 passed, 84 skipped, 19 xfailed in 7.12s =====

More goodies

I've shown how to run Darker on the command line as well as integrated in the test suite. In future blog posts, I plan to

• guide you to integrate Darker with an IDE,
• show how to have Travis CI on GitHub run Darker, and
• explain the mysterious --revision master... command line option.
• show how to sort imports and do code linting through Darker

Contribute!

We're a small happy bunch of developers contributing to Darker on its GitHub repository – drop by, say hello and lend your hand reviewing and submitting pull requests and making bug reports! We also sometimes have virtual meetups spanning continents and time zones, and we'd happy to see you there as well!