DEV Community

Vladimir Ternovoy
Vladimir Ternovoy

Posted on

I built a read-only GitHub Action to flag risky PRs before merge

#github #testing #devops #qa

Most CI checks answer questions like:

  • Does the code compile?
  • Do the tests pass?
  • Does linting pass?
  • Is formatting correct?

Those checks are useful, but they often miss a different question:

Did this pull request touch a risky product area?

That was the reason I built QA Boutique PR Risk Scanner — a free read-only GitHub Action that flags risky PR areas before merge.

For example, a pull request that changes checkout, pricing, authentication, permissions, API routes, or billing logic probably deserves a different level of review than a small UI copy change.

The idea

The action is intentionally simple.

It does not try to replace code review.

It does not generate tests.

It does not use AI.

It does not upload code anywhere.

Instead, it looks at the changed files and diff signals in a pull request and writes a lightweight PR risk summary into the GitHub Actions Summary.

The goal is to give reviewers a useful checklist before merge.

What it checks

The action looks for changes in areas such as:

  • billing
  • pricing
  • checkout
  • subscriptions
  • authentication
  • authorization
  • permissions
  • API routes
  • backend logic
  • critical user flows
  • test / CI changes

It also warns when production files changed but no related test files were detected.

This is not a perfect signal, of course.

But it is often enough to make reviewers stop and ask:

Should this PR have regression tests?
Should QA review this flow?
Could this affect billing, auth, or checkout?

Why read-only matters

For this kind of tool, I wanted the free GitHub Action to be safe by default.

So the action is:

  • read-only
  • rule-based
  • non-AI
  • no code upload
  • no external API calls
  • no write access required

It only reads PR metadata and changed files through GitHub Actions.

That makes it easier to try in real repositories without giving an external service write access or sending source code outside GitHub.

Demo: risky checkout PR

I created a small public demo repository to show the idea.

The demo PR adds a checkout endpoint:

app/api/checkout/route.ts
Enter fullscreen mode Exit fullscreen mode

The endpoint intentionally contains common business-logic risks:

  • hardcoded plan
  • hardcoded price
  • hardcoded discount
  • no request body parsing
  • no parameter validation
  • no related regression tests

The free GitHub Action flags the PR as high risk because it touches checkout, pricing, and API logic.

It also warns that no related tests were changed.

Where the free action stops

The free action is intentionally lightweight.

It can say:

This PR touches a risky area.

But it does not deeply understand business logic.

That is where a deeper AI review can help.

When I ran the same demo PR through QA Boutique’s AI review, it found concrete issues:

  1. The checkout endpoint returns a fixed startup plan, price 99, and discount 0.2 regardless of user input.
  2. The discount is embedded directly in the checkout URL without validation or server-side verification.
  3. The endpoint ignores the POST request body entirely.
  4. There is no plan lookup, price calculation, or parameter validation.
  5. Regression tests should verify different plan selections, discount eligibility, invalid plans, and manipulated checkout URLs.

That difference is important:

  • Free Action = lightweight PR risk awareness
  • QA Boutique AI Review = deeper PR analysis with concrete findings and suggested regression tests

Why this matters

A pull request can pass normal syntax checks and still break a critical business flow.

For example:

  • users may be charged the wrong price
  • all users may receive the same hardcoded plan
  • discount logic may be bypassed
  • checkout URLs may be manipulated
  • important regression tests may be missing

These are not always syntax problems.

They are product and business-logic problems.

That is why I think PR review tools should not only ask whether code is valid, but also whether the change touches a risky area.

Try it

The GitHub Action is available on GitHub Marketplace:

https://github.com/marketplace/actions/qa-boutique-pr-risk-scanner

Source code:

https://github.com/TerFree70/qa-boutique-pr-risk-scanner

Public demo repository:

https://github.com/TerFree70/qa-boutique-demo-checkout-risk

Free AI PR Audit:

https://qaboutique.com/#free-pr-audit

Feedback welcome

I am especially interested in feedback from developers, QA engineers, and engineering managers:

  • Would a lightweight PR risk checklist be useful in your workflow?
  • What risky areas should the action detect better?
  • Would you prefer this as a GitHub Action, a PR comment, or both?

Top comments (1)

Collapse
 
terfree profile image
Vladimir Ternovoy •

I’m especially interested in feedback from people using GitHub Actions in real teams: would you prefer this kind of PR risk signal only in GitHub Actions Summary, or also as an optional PR comment?