DEV Community

Cover image for Your AI agent shipped a billing bug. ProdVerdict blocks it in CI.
Matthew
Matthew

Posted on

Your AI agent shipped a billing bug. ProdVerdict blocks it in CI.

#saas #github #opensource #ai

AI coding tools are fast. Tests pass. The PR looks fine. Then production quietly leaks revenue.

The pattern I kept seeing in vibecoded SaaS:

  1. Billing logic lands in a PR
  2. Webhooks get skipped or half-wired
  3. Stripe says active while Postgres says has_paid_access = false

That's not a lint issue. That's money walking out the door.

ProdVerdict is deterministic CI for production contracts — not another AI code reviewer. Zero LLM in the evaluation path. Read-only observations (Stripe/Paddle + your DB) and fixed rules. Same input, same output every time.

Three contracts in v0.5

1. Access — billing vs database

Compares live subscription state (Stripe or Paddle) against your Postgres users table. Catches:

  • Revenue leak — active subscription, has_paid_access = false
  • Wrongful access — cancelled user still has paid features
  • Plan drift — price ID maps to pro, row says starter
  • Duplicate customer — same billing ID on multiple app users

2. Config — env var drift

Scans process.env references in your code vs .env.example and CI secrets. The classic "it worked locally" killer.

3. Migration — unsafe Postgres DDL

Static SQL rules for agent-generated migrations: blocking CREATE INDEX without CONCURRENTLY, dangerous NOT NULL on hot tables, and more.

Try it in 10 seconds — no API keys

Clone the SDK once for fixture paths, then run:

git clone --depth=1 https://github.com/prodv-dev/prodverdict-sdk.git
cd prodverdict-sdk

npx prodverdict check access \
  --config examples/paddle-stripe/prodverdict.yml \
  --fixtures \
  --fixtures-dir examples/paddle-stripe/scenarios/fail-revenue-leak
Enter fullscreen mode Exit fullscreen mode

You should see a FAIL with a concrete fix hint:


[HIGH] user:usr_alice — Active subscription but has_paid_access is false
fix: Set has_paid_access=true in your webhook handler
VERDICT: FAIL
Enter fullscreen mode Exit fullscreen mode

Passing scenario:

npx prodverdict check access \
  --config examples/nextjs-stripe/prodverdict.yml \
  --fixtures \
  --fixtures-dir examples/nextjs-stripe/scenarios/pass
Enter fullscreen mode Exit fullscreen mode

Add to your repo 

npx prodverdict init --stack nextjs-stripe
# or: npx prodverdict init --stack paddle-stripe
Enter fullscreen mode Exit fullscreen mode

Commit prodverdict.yml, add secrets, and wire the GitHub Action:

- uses: prodv-dev/prodverdict-action@v0.5.0
  with:
    config: ./prodverdict.yml
    contract: access
  env:
    STRIPE_SECRET_KEY: ${{ secrets.STRIPE_SECRET_KEY }}
    DATABASE_URL: ${{ secrets.DATABASE_URL }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Enter fullscreen mode Exit fullscreen mode

Runs on every PR. Secrets never leave your runner. Optional dashboard stores verdict metadata only — not subscription rows or PII.

Install from GitHub Marketplace · npm · MCP for Cursor/Claude

How this differs from AI code review

AI code review ProdVerdict
Evaluation LLM suggestions Deterministic rules
Input Source code diff Live billing + DB state (or fixtures)
Output "Consider checking…" PASS / WARN / FAIL with fix hints
CI behavior Optional, noisy Merge-blocking on HIGH

Agents can call ProdVerdict via MCP before opening a PR. Enforcement still happens in CI.

Pricing

  • Free — public repos, CLI, GitHub Action, MCP, Access + Config contracts
  • Pro ($39/project/mo) — private repos, scheduled checks, dashboard history

We're on Product Hunt — use code PRODUCTHUNT at checkout for 3 months free Pro.

What's next?

Webhook contract, API boundary checks (mass-assignment / forbidden fields), and more stack templates. Open an issue on prodverdict-sdk if you want a rule prioritized.

What's the scariest prod bug an agent ever shipped for you? Drop it in the comments — I'm collecting patterns for the next contracts.

ProdVerdict is MIT-licensed (SDK + Action). prodverdict.com · Discord

Top comments (0)