Writing hooks for "pre-commit" framework

Grygorii Iermolenko — Tue, 18 Jun 2019 04:55:26 +0000

There are already hundreds of great hooks around. But what if we want one more? Well, reading official docs is the great place to start (and reference in the future).

What I want is to:

show by example how particular idea can be developed
describe pitfalls which surprised me the most (and how to overcome them)
bring attention to such a wonderful tool that pre-commit is

The theme I chose for illustration is assignment expressions (PEP572). And down the line I will create three different hooks for that.

Preparations

For experiments with hooks already existing in the wild you would need only one git repository. And then you would configure it to install/use hooks from somewhere else.

But to write your own, there should be two separate repositories.

Let's begin.

If you didn't install pre-commit into system/environment - see documentation on how to do that.
Start with basic structure like this (two folders with git init)

[~/code/temp/try-pre-commit]
$ tree -a -L 2
.
├── code
│   └── .git
└── hooks
    └── .git

Create dummy hooks/.pre-commit-hooks.yaml

$ touch hooks/.pre-commit-hooks.yaml

Create dummy code/experiments.txt

$ touch code/experiments.txt

cd into code folder and initialize pre-commit (don't let this second install confuse you, it is different from installation into system/environment. It's scope is repository-wide, one git hook file, actually)

$ cd code
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

Here is how it should look now

[~/code/temp/try-pre-commit]
$ tree -a -L 2
.
├── code
│   ├── experiments.txt
│   └── .git
└── hooks
    ├── .git
    └── .pre-commit-hooks.yaml

Example 1: grep

The first hook will be very basic, leveraging existing possibilities of the framework. I will call it "walrus-grep".

Update hooks/.pre-commit-hooks.yaml

- id: walrus-grep
  name: walrus-grep
  description: Warn about assignment expressions
  entry: ":="
  language: pygrep

pygrep is a cross-platform version of grep.

entry takes regexp of something that you don't want to commit. In my case I used ":="-pattern without actual regexp magic or escaping symbols.

Update code/experiments.txt

text file
a := 1

The most convenient way to experiment is with try-repo command. It allows to test hooks without commiting .pre-commit-hooks.yaml every time, and without pinning to revision later on in .pre-commit-config.yaml.

We can either stage files each time before running checks or use try-repo command with --all-files flag.

[master][~/code/temp/try-pre-commit/code]
$ pre-commit try-repo ../hooks walrus-grep --verbose --all-files

This happened because hooks repo doesn't have a single commit - there is nothing HEAD could point to. Commit .pre-commit-hooks.yaml and repeat.

See "(no files to check)"? Often this would be true, checks could have pre-defined limits e.g. file extension, path, ignore options. But we have our experiments.txt file and do not expect any of those!

Gotcha: --all-files flag is more like "all tracked, not only staged files". So either way you should add files to git before running checks.

At last! "Failed" is what we needed to prove that grep hook catches ":=" inside some random staged file. So here failure is a good thing.

But to think about it situation is far from ideal:

first, we don't actually want to catch ":=" inside text files, only in python code
second, what if ":=" happens inside comments, docstrings, strings?

For "python code only" part there is types: [python] option which I will add to .pre-commit-hooks.yaml

- id: walrus-grep
  ...
  types: [python]

We see "(no files to check)Skipped" again, but now it is expected.

To double-check

$ mv experiments.txt experiments.py
$ git add experiments.py
$ pre-commit try-repo ../hooks walrus-grep --verbose --all-files

and we have our beloved failure back.

As for the second concern, ":=" inside comments or docstrings, we would require something different.

Example 2: AST

The second hook will be analyzing syntax tree to find exact node type. I will call it "walrus-ast".

Disclaimer: This one will work only in Python 3.8 environment (beta1 is currently available for testing).

Update hooks/.pre-commit-hooks.yaml

...
- id: walrus-ast
  name: walrus-ast
  description: Warn about assignment expressions
  entry: walrus-ast
  language: python
  types: [python]

and experiments.py

# .py file with `:=` (Python3.8 syntax)
(x := 123)

The hook itself will be at hooks/walrus_ast.py

import argparse
import ast


def check(filename):
    with open(filename) as f:
        contents = f.read()

    try:
        tree = ast.parse(contents)
    except SyntaxError:
        print('SyntaxError, continue')
        return

    for node in ast.walk(tree):
        if isinstance(node, ast.NamedExpr):
            return node.lineno


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('filenames', nargs='*')
    args = parser.parse_args()

    for filename in args.filenames:
        lineno = check(filename)
        if lineno:
            print(f"{filename}:{lineno}")
            return 1

    return 0


if __name__ == '__main__':
    exit(main())

What we can or can't do here? The rules per documentation are simple

The hook must exit nonzero on failure or modify files in the working directory

Things to note:

exit() with some code ("0" is when everything OK) - will decide whether hook fails or not
checks like if __name__ == '__main__': and names like main() or check() are not required (i.e. you can have useless file with exit(1) as its only contents)
parsing filenames is needed if you want to be able to limit by filename. Not required but nice to have

Now run try-repo with new hook name

$ pre-commit try-repo ../hooks walrus-ast --verbose --all-files

"Directory '.' is not installable". What happened?

It turns out that (in case of python scripts) we should describe how to install it as usual, with setup.py or pyproject.toml.

Create hooks/setup.py

from setuptools import setup
setup(
    name="hooks",
    py_modules=["walrus_ast",],
    entry_points={
        'console_scripts': [
            "walrus-ast=walrus_ast:main",
        ],
    },
)

Note: instead of py_modules you can use something like packages=find_packages(".") or anything else you are used to do in setup.py

Track both new files in git

$ git add setup.py
$ git add walrus_ast.py

Now working directory should look like this

[master][~/code/temp/try-pre-commit/hooks]
$ tree -a -L 1
.
├── .git
├── .pre-commit-hooks.yaml
├── setup.py
└── walrus_ast.py

Try running it again

$ pre-commit try-repo ../hooks walrus-ast --verbose --all-files

Reminder on how experiments.py looks:

# .py file with `:=` (Python3.8 syntax)
(x := 123)

The ":=" is indeed on the second line, and the one from comments is not reported. Thanks, AST!

Note: And if you forgot that this is Python3.8 only thing - you'll get (hopefully) some indication. In that case hook doesn't work, but shouldn't stop others from doing their job either.

Example 3: Less negativity, more opportunity!

It was not my intention to stop you from using assignment expressions, rather to play with new things and describe the process.

Let's approach the idea from the opposite direction and find a place where walrus can fit. Thanks to this tweet and very interesting library by Chase Stevens we can find such places.

Create hooks/walrus_opportunity.py

import astpath

def main():
    search_path = "//Assign[targets/Name/@id = following-sibling::*[1][name(.) = 'If']/test/Name/@id]"
    found = astpath.search('.', search_path, print_matches=True, after_context=1)
    return len(found)

if __name__ == "__main__":
    exit(main())

Here astpath uses xpath to find AST nodes.

Update setup.py

from setuptools import setup

setup(
    name="hooks",
    py_modules=["walrus_ast", "walrus_opportunity"],
    entry_points={
        'console_scripts': [
            "walrus-ast=walrus_ast:main",
            "walrus-opportunity=walrus_opportunity:main"
        ],
    },
    install_requires=[
        # Dep. for `walrus-opportunity`
        "astpath[xpath]",
    ],
)

Update hooks/.pre-commit-hooks.yaml

- id: walrus-opportunity
  name: walrus-opportunity
  description: Warn if you could have used ":=" somewhere
  entry: walrus-opportunity
  language: python
  types: [python]

Update code/experiments.py

# .py file with `:=` (Python3.8 syntax)
(x := 123)

# missed chance to use `:=`
x = calculate()
if x:
    print(f"found {x}")

and run it

$ pre-commit try-repo ../hooks walrus-opportunity --verbose --all-files

Wrapping up

If you decide that local development is done:
Push hooks folder to your-repo (e.g. on github), create code/.pre-commit-config.yaml

repos:
- repo: https://github.com/<your-repo>/pre-commit-hooks
  rev: <commit-sha> or <tag>
  hooks:
  - id: walrus-grep
  - id: walrus-ast
  - id: walrus-opportunity

And now you should be able to use git from code repo as usual: change files, stage and commit them. Before each commit git will run a script from code/.git/hooks/pre-commit. And pre-commit framework should show report and allow commit if everything is OK, or show report and abort commit if some checks have returned non-zero or files were changed.

Addendum

More about AST visitors and working on the tree here.
Thanks to Anthony Sottile for such a wonderful framework.

DEV Community: Grygorii Iermolenko

Writing hooks for "pre-commit" framework

Preparations

Example 1: grep

Example 2: AST

Example 3: Less negativity, more opportunity!

Wrapping up

Addendum