DEV Community

Batuhan Osman Taşkaya
Batuhan Osman Taşkaya

Posted on

Inspecting python with Inspector Tiger

Working with multiple developers on a big codebase can cause consistency problems. These consistency problems generally handled during code-reviews by a person or linters. But in some specific cases, built-in handlers that come with linter isn't sufficient enough. It is where the inspector tiger idea born.

Inspector Tiger is a code review framework and a linter. It comes with a series of built-in handlers (like finding yield statements that can be replaced with yield from). But the real aim is using its framework side, developing codebase-specific plugins. It is what we are going to build through this tutorial.

class Foo(SomeObjects):
    def bar(self, x=[], y: Union[int, None] = None):
        x.append(1)
        for _ in range(3):
            try:
                super(bla, bla).foo_baz()
            except Exception:
                print("An exception")
            except AttributeError:
                print("An attribute error")
[Inspector Tiger] INFO - InspectorTiger inspected 🔎 and found these problems;
[Inspector Tiger] INFO -
[misc]
  - ../t.py:2:4     => DEFAULT_MUTABLE_ARG
  - ../t.py:5:12    => UNREACHABLE_EXCEPT
[upgradeable]
  - ../t.py:2:27    => OPTIONAL
  - ../t.py:6:16    => SUPER_ARGS

How it works

Like most of the linting tools, it is a static analyzer. It parses your code without executing it and generates a syntax tree from it. For this process, we use CPython's ast module. After the AST generation, the plugins are loaded into the Inspector. Heart of Inspector Tiger. Every plugin offers Inspector a series of handlers. A handler is a simple function that registers itself to a specific AST node. When that node comes in, Inspector will invoke that handler and get a True or False value according to the handler's concept.

Let's Start

We are going to implement PEP 601 (which is applied to PEP 8 after rejection). It is about forbidding return, break and continue statements within a finally suite where they would break out of the finally.

Plugins in Inspector Tiger are python packages (not modules). For starting we need a python package layout (with setup.py/setup.cfg).

$ tree
├── pep601
│   ├── __init__.py
│   └── pep601.py
└── setup.py
from setuptools import setup, find_packages

setup(
    name="inspectortiger-pep601",
    packages=find_packages()
)

We generally name our plugins with inspectortiger- prefix, that is just an optional standard.

Before Writing the handler

Developing handlers requires an underlying understanding of AST nodes. If you aren't familiar with AST, you should check out Green Tree Snakes. It is "the missing Python AST docs".

The handler itself

We'll start writing our extension by registering our handler to an AST node. Whenever a try/except/finally/else statement comes in, Inspector will invoke our handler with an ast.Try node and a reference to an internal database where handlers can share information between them. By the way, choose your function name carefully, it is going to be used as an error code.

@Inspector.register(ast.Try)
def control_flow_inside_finally(node, db):

Next thing we are going to do is examining what is a ast.Try node look like and proceed.

>>> import ast
>>> code = """
def foo():
    try:
        foo()
    finally:
        return
"""
>>> tree = ast.parse(code)
>>> try_stmt = tree.body[0].body[0]

After getting the try statement from the function body, we are going to dump it. If you are not using python3.9 or higher I suggest you check out astpretty.

>>> print(ast.dump(try_stmt, indent=4))
Try(
    body=[
        Expr(
            value=Call(
                func=Name(id='foo', ctx=Load()),
                args=[],
                keywords=[]))],
    handlers=[],
    orelse=[],
    finalbody=[
        Return(value=None)])

As it is seen, try nodes has a finalbody field that contains the body of finally statements. Lets iterate over that finalbody and find occurrences of return/break/continue

    for subnode in node.finalbody:
        for child in ast.walk(subnode):

The next step is deciding if a child is a control flow statement and if that statement will break out of the finally. The second part is important because you can write a function inside of finally and that could contain a return which wouldn't break out of the finally.

def invalid():
    try:
        pass
    finally:
        return

def valid():
    try:
        pass
    finally:
        def foo():
            return

For finding such things, we need to know which context a node belongs to. Luckily Inspector Tiger comes with a built-in context management plugin. Lets try that out by creating a control for if the child node is a return statement and the context of child node is same with context of our try/finally statement.

            if isinstance(child, ast.Return) and get_context(
                child, db
            ) is get_context(node, db):
                return True

And if that is the case, we'll return something that is not falsy. The second check is if the child is a continue or break statement and it doesn't inside of another for loop. This example can give you a better understanding of which case is valid which case isn't

def invalid():
    for _ in baz:
        try:
            pass
        finally:
            continue

def valid():
    for _ in baz:
        try:
            pass
        finally:
            for _ in bar:
                continue

For doing such a check we need to traverse our node's parents to the finally and check if there is any another for loop. And luckily again we have a plugin for traversing parents. parentize offers a utility named parent_to. It will yield every parent from child node to the given parent node. If there are no occurrences of a for loop between the continue/break and our try statement we can say that statement will break out of the finally.

            elif isinstance(child, (ast.Break, ast.Continue)) and not any(
                isinstance(parent, ast.For)
                for parent in parent_to(child, node)
            ):
                return True

And that was it, we're done.

Configuration

The last thing is setting up an .inspector.rc on your home directory and specifying your plugin.

{
    "plugins": {
        "pep601": ["pep601"],
        "package": ["module_that_contains_handlers"]
    }
}

Discussion (0)