Give Every Dependabot PR a Risk Label Before You Click Review

Dependabot opens five PRs a week. You merge the patch version, panic over the major release, and manually open changelogs to guess whether a type signature changed. That's tedious and error-prone.

A small LLM can do that triage for you: read the changelog, inspect the diff stats, and output a risk label plus a one-sentence reason. Run it on a schedule with free model credits and a free server, and you get a living dependency radar without adding to your cloud bill.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The context gap in dependency updates

Dependabot tells you what changed — version X to Y — but not whether it matters. The release notes might be huge, yet the actual API difference might be trivial. Or the changelog says "minor internal refactor" while the diff changes a public function's return type.

That gray zone is where bugs are born. Humans can't read every changelog, and deterministic checks only catch breaking version bumps. A model can summarize the risk signal in a structured form, and a small CI gate can present it the moment a PR appears.

What we're building

A Python script that:

• Fetches a Dependabot PR's metadata and diff stats from the GitHub API.
• Extracts the changelog from the PR description or the linked release page (when available).
• Calls a free LLM with a strict JSON prompt, asking for a risk assessment: low, medium, high, or breaking.
• Writes a GitHub check or PR comment with the label and reason.

The script is designed to run anywhere — a GitHub Action, a cron job, or a free server slot. No proprietary SDK required; it uses plain HTTP.

Step 1: The risk assessment script

Save this as dependabot_triage.py. Replace the API endpoint and key with whatever provider you're using; MonkeyCode's free models work with this pattern.

import json
import os
import re
import sys
import time
from urllib import request, error


def call_llm(prompt: str) -> dict:
    api_key = os.environ["MK_API_KEY"]
    payload = {
        "prompt": prompt,
        "max_tokens": 200,
        "temperature": 0.2,
    }
    req = request.Request(
        os.environ.get("MK_API_URL", "https://api.monkeycode.ai/v1/completions"),
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        },
    )
    try:
        with request.urlopen(req, timeout=45) as resp:
            data = json.loads(resp.read().decode())
            text = data["choices"][0]["text"]
            # extract first JSON object
            start = text.find("{")
            end = text.rfind("}")
            return json.loads(text[start:end + 1])
    except Exception:
        # deterministic fallback: call it unknown, not broken
        return {"risk": "unknown", "reason": "call failed, human review required"}


def fetch_pr(repo: str, pr_number: int, token: str) -> dict:
    req = request.Request(
        f"https://api.github.com/repos/{repo}/pulls/{pr_number}",
        headers={"Authorization": f"Bearer {token}", "Accept": "application/vnd.github+json"},
    )
    with request.urlopen(req, timeout=30) as resp:
        pr = json.loads(resp.read().decode())
    req2 = request.Request(
        f"https://api.github.com/repos/{repo}/pulls/{pr_number}/files",
        headers={"Authorization": f"Bearer {token}", "Accept": "application/vnd.github+json"},
    )
    with request.urlopen(req2, timeout=30) as resp:
        files = json.loads(resp.read().decode())

    additions = sum(f"additions" for f in files)
    deletions = sum(f["deletions"] for f in files)
    changed = [f["filename"] for f in files[:20]]
    return {
        "title": pr["title"],
        "body": (pr.get("body") or "")[:2000],
        "additions": additions,
        "deletions": deletions,
        "changed_files": changed,
    }


def build_prompt(pr: dict) -> str:
    return f"""You are a dependency risk analyst. Given a PR update, label the risk of merging it.
Return JSON only: {{"risk": "low|medium|high|breaking", "reason": "one short sentence"}}

PR title: {pr['title']}

PR body snippet:
{pr['body'][:1000]}

Diff stats: +{pr['additions']} -{pr['deletions']}
Changed files: {', '.join(pr['changed_files'][:10])}

Rules:
- low: patch or minor with no touching core types
- medium: behavior might change, needs manual smoke test
- high: major version, many files, risky packages
- breaking: clearly says BREAKING or renames public API"""


def triage(repo: str, pr_number: int) -> dict:
    token = os.environ["GITHUB_TOKEN"]
    pr = fetch_pr(repo, pr_number, token)
    prompt = build_prompt(pr)

    # retry pattern: the model may respond with malformed JSON
    for _ in range(3):
        result = call_llm(prompt)
        if "risk" in result:
            return result
        time.sleep(2)
    return {"risk": "unknown", "reason": "could not parse model output"}


if __name__ == "__main__":
    repo, pr_number = sys.argv[1], int(sys.argv[2])
    result = triage(repo, pr_number)
    print(json.dumps(result, ensure_ascii=False))

The key line is the fallback: if the model call fails, you get unknown instead of a crash. That's the difference between a helpful automation and a support ticket generator.

Step 2: Turn the label into a GitHub check

You want this visible where developers already look. A simple Action can run the script and leave a PR comment.

name: dependabot-triage
on:
  pull_request_target:
    types: [opened, edited, synchronize]

jobs:
  triage:
    if: github.event.pull_request.user.login == 'dependabot[bot]'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run triage
        id: triage
        env:
          MK_API_KEY: ${{ secrets.MK_API_KEY }}
          MK_API_URL: ${{ secrets.MK_API_URL }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          python dependabot_triage.py ${{ github.repository }} ${{ github.event.pull_request.number }} > result.json
          echo "::set-output name=risk::$(jq -r .risk result.json)"
          echo "::set-output name=reason::$(jq -r .reason result.json)"
      - name: Comment on PR
        uses: actions/github-script@v6
        with:
          script: |
            const risk = "${{ steps.triage.outputs.risk }}";
            const reason = "${{ steps.triage.outputs.reason }}";
            github.rest.issues.createComment({
              ...context.repo,
              issue_number: context.issue.number,
              body: `**Risk: ${risk}** — ${reason}`
            });

Note the if condition: this only runs on Dependabot PRs, so you're not paying for model calls on your own feature branches. Free-tier quotas last much longer that way.

Step 3: Turn the label into a policy

The label alone is not a merge gate. You still need humans for medium and high. A decision table makes the action obvious:

Risk label	Meaning	Required action
low	Patch or trivial minor, no core type changes	Merge after CI passes
medium	Behavior might shift; needs smoke test	Run test suite, manual sanity check
high	Major version or many files changed	Full review, feature manual test
breaking	Public API renamed/removed	Schedule migration, update call sites first
unknown	Model failed or gave bad JSON	Manual triage, ignore automation suggestion

You can encode this in Branch Protection rules or just as a team convention. The model doesn't make the decision; it hands you the evidence in a form you can act on.

Step 4: Run it on a free server

GitHub Actions works, but you might want a daily digest of all open dependency PRs across a whole org. That's where a free server with cron comes in.

On the free server:

pip install requests  # or nothing else needed for the script above
crontab -e
# add line:
0 8 * * * cd /opt/dep-triage && for pr in $(list_prs); do python3 dependabot_triage.py org/repo $pr >> digest.md; done

Then send digest.md to your team's chat or leave it as a file in the repo. The script is lightweight enough that a single free server slot handles hundreds of PRs a day without breaking a sweat.

Limitations

• Hallucinated labels: a model can misread a changelog. Always keep the final merge decision with a human.
• Changelog quality: if the PR body is empty and the repo has no release notes, the model works from diff stats only, which is a weak signal.
• Prompt drift: your prompt might need tuning as you change LLM providers. Pin the prompt and version it.
• Rate limits: free tiers have quota. The pull_request_target filter and cooldown timers keep usage reasonable.

Who should not use this

• Teams with zero test coverage: a risk label won't save you if every upgrade breaks anyway. Fix the safety net first.
• Regulated environments: no generated risk label is audit-grade. You still need a documented human review.
• Repos with huge release notes every week: feeding 50KB of changelog into a free model gets expensive and slow. Truncate smarter or only process major bumps.

The human stays in the loop

Automation should compress the boring part of dependency upgrades — reading three changelogs and squinting at a diff — while leaving the judgment call for a person. A cheap model on a cron schedule is the right tool for that compression.

Next time Dependabot opens a PR, spend 30 seconds wiring up a triage label instead of clicking merge with your eyes half closed. The label won't make the decision for you, but it will make sure you never skip the update that deserves your attention.