DEV Community

Cover image for Why Wait for CI? Shift Left with Pre-commit Hooks
Ashok Nagaraj
Ashok Nagaraj

Posted on

Why Wait for CI? Shift Left with Pre-commit Hooks

#ci #python #git

In the world of DevOps and modern software engineering, automation and consistency are key. Continuous Integration (CI) pipelines help enforce these principles by automating tests, builds, and deployments. However, many issues—like inconsistent formatting, trailing whitespace, or forgotten debug statements—can and should be caught before code even reaches the CI server.

This is where the pre-commit framework comes in. It allows you to define a set of checks (called hooks) that run automatically before each commit. These hooks can catch common issues early, saving time and reducing friction in code reviews and CI failures.

Problem Statement

  • How can we ensure that all code committed to a Git repository adheres to defined quality standards (e.g., linting, formatting, security checks) before it even gets pushed or merged?
  • How can we better utilize the dev infrastructure to do CI?
  • How do we avoid getting started/blocked by non-availability of CI infrastructure?

Solution: pre-commit Hooks

The pre-commit framework provides a unified way to manage and maintain multi-language pre-commit hooks. It integrates seamlessly with Git and can be used both locally and in CI environments.

How to setup

pre-commit installation

Install pre-commit globally or within your Python environment:

# pip
$ pip install pre-commit

# poetry: add it to pyproject.toml
[tool.poetry.dependencies]
pre-commit = "^3.0.0"
Enter fullscreen mode Exit fullscreen mode

Hook definition

Create a .pre-commit-config.yaml file in the root of your repository. Here's a typical configuration for a Python project:

$ cat .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v5.0.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-added-large-files

  - repo: https://github.com/psf/black
    rev: 23.3.0
    hooks:
      - id: black
        language_version: python3

  - repo: https://github.com/pycqa/flake8
    rev: 7.2.0
    hooks:
      - id: flake8
        additional_dependencies: ['flake8-bugbear']

  - repo: https://github.com/PyCQA/isort
    rev: 5.12.0
    hooks:
      - id: isort
        name: Sort Imports

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.15.0
    hooks:
      - id: mypy
Enter fullscreen mode Exit fullscreen mode

What Each Hook Does

  • Black: Formats Python code to a consistent style.
  • Flake8: Lints Python code for style and logical errors.
  • Isort: Automatically sorts imports.
  • Mypy: Performs static type checking.
  • Trailing Whitespace / EOF Fixer: Cleans up whitespace issues.
  • Check YAML: Validates YAML syntax.
  • Check Large Files: Prevents accidental commits of large files.

Hook installation

Initialize CI into your repo by running:

$ pre-commit install
Enter fullscreen mode Exit fullscreen mode

Usage

Invoking pre-commit

  • To check all files in the repository: 
$ pre-commit run --all-files
Enter fullscreen mode Exit fullscreen mode
  • To check specific files: 
$ pre-commit run --files <file1> <file2>
Enter fullscreen mode Exit fullscreen mode
  • To check only staged files (default on commit): 
$ git add <files>
$ git commit -m "your message"
# pre-commit will run automatically
Enter fullscreen mode Exit fullscreen mode
Fixing issues

Some hooks will auto-fix issues (e.g., formatting). After running pre-commit, re-add any fixed files:

git add <fixed-files>
git commit
Enter fullscreen mode Exit fullscreen mode
Bypassing pre-commit
  • To bypass specific hooks: 
# find the hook ID in `.pre-commit-config.yaml` and run:
SKIP=<hook_id> git commit -m "your message"
Enter fullscreen mode Exit fullscreen mode
  • To bypass all hooks (not recommended): 
git commit --no-verify -m "your message"
Enter fullscreen mode Exit fullscreen mode
Example runs

test.py (before)

import sys
import os

def add(a, b):
    return a+b

add(7, 3)
Enter fullscreen mode Exit fullscreen mode

run

pre-commit run --files test.py
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
debug statements (python)................................................Passed
fix double quoted strings................................................Passed
python tests naming..................................(no files to check)Skipped
fix requirements.txt.................................(no files to check)Skipped
check for merge conflicts................................................Passed
check json...........................................(no files to check)Skipped
shellcheck...........................................(no files to check)Skipped
Makefile linter/analyzer.............................(no files to check)Skipped
Reorder python imports...................................................Failed
- hook id: reorder-python-imports
- exit code: 1

Reordering imports in test.py

Add trailing commas......................................................Passed
autopep8.................................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

test.py:3:1: F401 'os' imported but unused
test.py:4:1: F401 'sys' imported but unused
Enter fullscreen mode Exit fullscreen mode

test.py (after)

from __future__ import annotations

import os
import sys


def add(a, b):
    return a+b


add(7, 3)
Enter fullscreen mode Exit fullscreen mode

CI integration

Even though pre-commit runs locally, it’s essential to enforce it in CI to catch skipped hooks. Here's an example GitHub Actions step:

- name: Run pre-commit checks
  run: |
    pip install pre-commit
    pre-commit run --all-files
Enter fullscreen mode Exit fullscreen mode

Customization

You can define your own hooks for project-specific checks. For example, to block TODOs in code:

# .pre-commit-config.yaml
- repo: local
  hooks:
    - id: check-todo
      name: Check for TODOs
      entry: python scripts/check_todo.py
      language: system
      files: \.py$
Enter fullscreen mode Exit fullscreen mode 
# scripts/check_todo.py
import sys

for in sys.argv[1:]:
    with open(filename) as f:
        for i, line in enumerate(f, 1):
            if "TODO" in line:
                print(f"{filename}:{i}: Found TODO")
                sys.exit(1)
Enter fullscreen mode Exit fullscreen mode

References

Top comments (0)