RobustTrueTry

Posted on May 18

Automating Python Code Reviews with Free LLMs

#python #ai #automation

Introduction

Code reviews are a bottleneck for many Python teams. While human reviewers catch logical bugs and style issues, they are limited by time and availability. Fortunately, recent open‑source large language models (LLMs) can provide instant feedback on code quality, suggest improvements, and enforce style guides—all without a paid API key.

In this article you’ll build a GitHub Actions workflow that:

Checks out the PR code.
Runs a free LLM (e.g., Mistral‑7B via the ollama runtime) to generate a review comment.
Posts the comment back to the pull request.

By the end, every PR will receive an automated review that highlights:

PEP‑8 violations
Potential bugs (e.g., mutable default arguments)
Opportunities for refactoring

Prerequisite: Basic familiarity with GitHub Actions, Docker, and Python.

1. Choose a Free LLM Runtime

Several projects let you run LLMs locally for free:

Ollama – simple CLI, supports Mistral, Llama 3, etc.
vLLM – high‑throughput server for many GPUs.
LM Studio – desktop UI with a built‑in server.

For CI we need a headless, container‑friendly solution. Ollama fits perfectly because it ships a lightweight Docker image that can pull the model on first run.

# Pull the official Ollama image
docker pull ollama/ollama:latest

We'll use the Mistral‑7B‑Instruct model, which is under an Apache‑2.0 license and works well for code tasks.

2. Create a Small Review Script

The script will:

Receive a list of changed Python files.
Send each file's content to the LLM with a prompt.
Collect the responses and format them as a GitHub comment.

Save this as reviewer.py in the repository root.

import os, json, subprocess, textwrap
from pathlib import Path

# Prompt template – keep it short for speed
PROMPT = textwrap.dedent('''
You are a senior Python developer reviewing a pull request. For each file, provide:
- PEP‑8 style issues (line numbers)
- Possible bugs or anti‑patterns
- One concrete suggestion to improve readability or performance
Only output a markdown list. If no issues, say "No problems found."

File: {filename}
---
{content}
---
''')

def run_ollama(prompt: str) -> str:
    """Call the local Ollama server and return the model's response."""
    # Use the `ollama` CLI; it reads JSON from stdin and writes JSON to stdout
    request = {"model": "mistral", "prompt": prompt, "stream": False}
    result = subprocess.run(
        ["ollama", "run", "--json"],
        input=json.dumps(request).encode(),
        capture_output=True,
        check=True,
    )
    response = json.loads(result.stdout)
    return response.get("response", "")

def main():
    # GitHub provides the list of changed files via the `GITHUB_EVENT_PATH` JSON
    event_path = os.getenv("GITHUB_EVENT_PATH")
    if not event_path:
        raise SystemExit("Missing GITHUB_EVENT_PATH")
    with open(event_path) as f:
        event = json.load(f)
    files = [f["filename"] for f in event["pull_request"]["files"] if f["filename"].endswith('.py')]
    if not files:
        print("::notice::No Python files changed.")
        return
    comments = []
    for file in files:
        content = Path(file).read_text()
        prompt = PROMPT.format(filename=file, content=content)
        review = run_ollama(prompt)
        comments.append(f"### Review for `{file}`\n{review}\n")
    # Write the combined comment to a file for the Action step to read
    Path("/tmp/review_comment.md").write_text("\n---\n".join(comments))

if __name__ == "__main__":
    main()

Explanation

The script reads the GitHub event payload to discover changed Python files.
For each file it builds a concise prompt and calls ollama run.
Results are concatenated into a markdown file that the workflow later posts.

3. Dockerize the Review Environment

GitHub Actions runs in a clean VM, so we need a container that includes:

Python 3.11
ollama runtime with the model pre‑downloaded
Our reviewer.py

Create a Dockerfile:

FROM python:3.11-slim

# Install curl (needed by Ollama) and git
RUN apt-get update && apt-get install -y curl git && rm -rf /var/lib/apt/lists/*

# Install Ollama
RUN curl -fsSL https://ollama.com/install.sh | sh

# Pull the model (takes a few minutes on first run)
RUN ollama pull mistral

# Copy the reviewer script
WORKDIR /app
COPY reviewer.py .

# Install any Python deps (none needed now, but keep the step)
RUN pip install --no-cache-dir pyyaml

ENTRYPOINT ["python", "reviewer.py"]

Build and push the image to GitHub Container Registry (GHCR) from your local machine or a CI job:

# Authenticate with GHCR
echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin

docker build -t ghcr.io/<owner>/<repo>/code-reviewer:latest .

docker push ghcr.io/<owner>/<repo>/code-reviewer:latest

Replace <owner> and <repo> with your GitHub namespace.

4. Define the GitHub Action Workflow

Create .github/workflows/auto-code-review.yml:

name: Automated Code Review
on:
  pull_request:
    paths:
      - '**/*.py'

jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write  # needed to post comments
    steps:
      - name: Checkout PR
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Pull reviewer image
        run: |
          docker pull ghcr.io/${{ github.repository_owner }}/${{ github.event.repository.name }}/code-reviewer:latest

      - name: Run reviewer container
        env:
          GITHUB_EVENT_PATH: ${{ github.event_path }}
        run: |
          docker run --rm \
            -e GITHUB_EVENT_PATH=/github/event.json \
            -v ${{ github.event_path }}:/github/event.json:ro \
            -v ${{ github.workspace }}:/app \
            ghcr.io/${{ github.repository_owner }}/${{ github.event.repository.name }}/code-reviewer:latest

      - name: Post comment
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const comment = fs.readFileSync('/tmp/review_comment.md', 'utf8');
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.payload.pull_request.number,
              body: comment
            });

Key points

The workflow triggers on any PR that modifies *.py files.
It checks out the code, pulls the pre‑built container, and runs it with the event payload mounted.
The container writes the markdown review to /tmp/review_comment.md (shared via the host filesystem).
Finally, actions/github-script posts the comment back to the PR.

5. Fine‑Tuning the Prompt (Optional)

If you notice the model missing certain patterns, adjust the prompt:

Add a section for type‑hint suggestions.
Include a short example of the desired output format.
Limit the token budget (max_tokens) via the Ollama request if the run is slow.

Iterate until the feedback aligns with your team's standards.

6. Benefits and Limitations

Benefit	Limitation
Instant feedback – reviewers get a comment as soon as the PR is opened.	LLM may hallucinate; always treat output as a suggestion, not a rule.
Zero API cost – runs entirely on free, open‑source models.	Model size (7 B) consumes ~4 GB RAM; ensure the CI runner has enough resources.
Customizable – change the prompt or swap the model without code changes.	No deep static analysis (e.g., data‑flow) – combine with tools like `ruff` for completeness.

7. Next Steps

Combine with linters – run ruff or flake8 in the same container and merge their output.
Cache the model – store the Ollama model layer in a separate Docker layer to speed up CI.
Add a review badge – show a status check that the automated review passed.
Experiment with larger models – if your CI budget allows, try Llama 3‑8B for richer suggestions.

Takeaway

By leveraging a free, locally hosted LLM like Mistral‑7B through Ollama, you can automate the first pass of Python code reviews directly in GitHub Actions. The setup is lightweight, cost‑free, and extensible—giving developers faster feedback while keeping human reviewers focused on higher‑level design decisions.

Try it today: add the Dockerfile and workflow to a test repository, open a PR with a simple Python change, and watch the AI‑driven review appear instantly.

DEV Community