PullGuard: a GitHub Action that scores PR review risk for OSS maintainers

OSS maintainers spend reviewer hours on the same patterns over and over. Near-empty fixes, broad unrelated changes, refactors that drop tests, PR descriptions that don't match the diff. Some PRs are AI-generated, some are just careless. Either way they end up in the same triage queue.

PullGuard is a GitHub Action that scores pull request review risk so maintainers can route their attention. BYOK (OpenAI or Anthropic). It can comment, label, or close a PR based on policy.

LINK

What it flags

PullGuard does not try to detect whether a PR was written by an AI. That signal is too noisy and the question is the wrong one. Maintainers actually care about review cost. The rule set looks for:

• missing or weakly extended tests
• broad unrelated changes in one PR
• risky refactors without obvious motivation
• duplicated logic across files
• weak or templated PR descriptions
• suspiciously shallow fixes (one-line patches to deep issues)

Output is a 0 to 100 score plus a short comment listing top findings, capped at four to keep noise down.

Install

npx pullguard init

That writes .github/workflows/pullguard.yml and .github/pullguard.yml, then prints exact next steps. Add OPENAI_API_KEY or ANTHROPIC_API_KEY as a repository secret.

Configuration

model:
  provider: openai
  name: gpt-5.4-mini-2026-03-17

trigger:
  mode: comment
  comment: /pullguard
  allowedCommentAuthorAssociations:
    - OWNER
    - MEMBER
    - COLLABORATOR

actions:
  comment:
    enabled: true
  labels:
    enabled: true
    rules:
      - threshold: 50
        label: needs-human-review
      - threshold: 80
        label: high-risk-pr
  close:
    enabled: false

Trigger modes

Three options:

• always: run on every configured PR event.
• label: run when a specific label is applied.
• comment: run when an allowed maintainer comments /pullguard.

Default is comment, restricted to maintainers via allowedCommentAuthorAssociations. Random commenters cannot spend the repository's API credits.

Maintainers can override a single run from the comment:

/pullguard --depth codebase --close 95
/pullguard --provider anthropic --model claude-sonnet-4-20250514

Three independent actions

comment, labels, and close are separate switches. Common combinations:

• Comment-only: post a short review summary, no labels, no closing.
• Label-only: route to humans by score threshold.
• Observe-only: compute the score for downstream workflow steps without touching the PR.
• All three: the full pipeline for projects buried in low-quality PRs.

Closing is off by default. In public OSS the cost of an incorrect close is much higher than a missed PR.

Why BYOK

The action does not call any third-party service of mine. Provider key, model choice, and rate are under the maintainer's control. Token spend is bounded by maxPatchCharsPerFile, maxFiles, and maxFindings, so a single run on a normal PR is cheap.

Disclaimer

Early work. Score calibration is still being adjusted against real OSS PRs and the comment text needs polishing. Two useful things if you try it:

1. Tell me when the score is obviously wrong in either direction.
2. Tell me when the comment is confusing or unhelpful.

Repo: https://github.com/vladzima/pullguard

Comments

[Gilder Miller]

The trigger modes are a smart design. Restricting execution to maintainer comments by default saves API costs and prevents random users from burning through your budget. The depth parameter for token management shows practical production thinking too.
Curious about the scoring calibration. Have you found certain patterns that consistently produce false positives? Duplicated logic across files seems like it could flag legitimate shared utilities. How do you handle those cases without watering down the signal?