Static Analysis Tooling: how three fancy words are my new best friends

This week in my OSD600 adventures, we're learning about Static Anaylsis Tooling: Three fancy words which basically equates to the smart kid in the class (who you secretly hate) that proof reads your poorly written paper and points out more errors than you'd like.

We don't really speak about this kid anymore. Does it really matter that Marvin now works as a software engineer for a FAANG? No, because Marvin is getting paid six figures to write quality code, drive a Tesla, put a down payment on a house, but you get to work with linting software, and quality sunday lectures.

---

Static Analysis Tools are pieces of software that are used in Static Code Analysis, a process of determing if the code you're about push into a dumpster is pizza crust, or rotten pizza crust. 🍕🗑️🔥

For this week's lab I had to implement two tools my ever evolving link checker project: a code formatter called Black, and a code linter called Flake8.

Black formats your code in such a way that makes it more readable and easier to understand. It turns your spaghetti into penne. Your mess into order. It's great.

Flake8 looks through your code and scans it for suspicious lines of code. These lines might be stylistic errors that may end up as a bug down the road.

Basically, you can think of Black as a spell checker, and Flake8 as a grammar checker.

Once I chose my tools it was time to RTFM and set them up in my project.

Note: I am using Windows and thus these programs and set up files will vary based on your OS.

---

The first step was to install and set up Black: pip install git+git://github.com/psf/black

Since my project uses a setup.py file, I also added Black to the install_requires portion so that anyone who wants to set up the project simply types pip install .

The next step is testing: black sample/hdj_util.py which amazing just worked, a common phase I mutter creepily to myself when working with Python. Until things don't. Then I mutter profanities. It automatically prett-ified my hdj_util.py file.

Afterwards I set up PyCharm to run black on my project upon every save to whatever file I'm currently working on:

1. File -> Settings -> Tools -> External Tools Click the + icon to add a new external tool and add the following values:

Name: Black
Description: Black is the uncompromising Python code formatter.
Program: C:\Users\Chris\AppData\Local\Programs\Python\Python39\Scripts
Arguments: $FilePath$

Note the Program: path will obviously be different based on your machine. Simply locate the black.exe file on your system and point it to that.

From here you can go to Tools -> External Tools -> black to run black on the currently open file, or you can configure PyCharm to run Black on every file save. This requires the File Watchers plugin to work.

1. Go to Preferences or Settings -> Tools -> File Watchers and click + to add a new custom watcher:

Name: Black
File type: Python
Scope: Project Files
Program: C:\Users\Chris\AppData\Local\Programs\Python\Python39\Scripts\black.exe
Arguments: $FilePath$
Output paths to refresh: $FilePath$
Working directory: $ProjectFileDir$

Uncheck “Auto-save edited files to trigger the watcher” in Advanced Options

1. Create a pyproject.toml file to configure my project's black:

[tool.black]
line-length = 88
include = '\.pyi?$'
exclude = '''
/(
    \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''

line-length: Formats lines over a length of 88 characters to a prettier version. You can see this occur in the gif above.
include runs black on only .py files (thus excluding files like .gitignore.)
exclude is pretty self explanatory: "Don't run black on these folders or files".

---

With Black set up, let's move onto our next step and set up Flake8: python -m pip install flake8.

Again, my project uses a setup.py file, So I also added Flake8 to the install_requires portion.

The next step is testing: flake8 sample (sample is the directory which contains my code) which yet again just worked... albeit maybe too well... grumble grumble

The process to get this to run on save took a bit of google-fu but I managed to set it up.

We must simply add a file watcher just like above:

1. Go to Preferences or Settings -> Tools -> File Watchers and click + to add a new custom watcher:

Name: Flake8
File type: Python
Scope: Current File
Program: C:\Users\Chris\AppData\Local\Programs\Python\Python39\Scripts\flake8.exe
Arguments: $FileDir$/$FileName$

Check “Auto-save edited files to trigger the watcher” in Advanced Options
Output filters: $FILE_PATH$:$LINE$:$COLUMN$: $MESSAGE$

1. Go to Preferences or Settings -> Editor -> Inspections -> File Watchers -> File watcher problems, and change the severity to Warning.

Now Flake8 will auto run and highlight linting errors in yellow!

This is either wonderful or terrible depending entirely on your viewpoint, or however many glasses of wine you've had that evening, which for this author is, respectively, terrible and many.

1. Create a .flake8 file to configure my project's flake8:

[flake8]
ignore=E501, W503
exclude =
    .idea,
    .git,
    __pycache__,
    assets,
    build,
    dist,
    docs

ignore=E501, W503: E501 simply ignores the line too long warning, and W503 ignores the Line break occurred before a binary operator rule. Because I'm anal but not that anal.
exclude again, is pretty self explanatory: "Don't run flake8 on these folders or files".

Great! We're all done.

---

Between the two tools neither of them find too many issues. Black altered all four Python files in my project, which was expected (pretend this says changed):

...and Flake8 screamed about... well, a lot of things. But it's fine. These were just either minor things, or things that I wanted to fix up anyway.

All in all, I didn't have too much to fix up. It's also pretty cool that with the code formatter and linter installed globally I can run either of them on the CLI and just specify a directory or file to run them on. Truly batteries included.

slaps monitor Now it's time to get these bad boys to run upon every commit, because suffering truly builds character. Or software.

---

Git hooks are awesome. They really are. Imagine forcing people to run a script before they can even commit something to your project. Amazing. I bet there's a lot of funny gimmick hooks out there that just insult the user upon a commit... writes down.

It was suprisingly easy to set up a git hook to run these programs upon commit, it just required a bit of reading. We're going to leverage something called a pre-commit framework to do this. It's also configured with YAML, which is pretty cool. It's yet another mark up language that I've never used:

1. Install pre-commit: pip install pre-commit .`
2. Added pre-commit to my setup.py.
3. Create .pre-commit-config.yaml with the hooks needed in the pipeline: ` repos:
4. repo: https://github.com/ambv/black rev: stable hooks:
   • id: black language_version: python3.9
5. repo: https://gitlab.com/pycqa/flake8 rev: 3.8.4 hooks:
   • id: flake8 `
6. Run pre-commit install which installs git hooks in .git/ directory of my project.

I mostly had to dig around for the proper repos and versions to use. I thought I'd have to config this some more because I'm using the new version of Python but surprisingly I didn't have to.

Neat.

---

I learned a LOT in this lab, and just as I predicted the tooling labs so far are really interesting and fun.

I also unfortunately learned about another tool called Pylama (What is it with the Python community and animals?) which is both a code formatter and a linter! I'll have to keep it in mind for my next big project.

I think we're doing CI tools next week! It's gonna be great, I can't wait.

The struggled wasn't too bad, but it's all worth it for that sweet, sweet fast forward rebase and merge.