DEV Community

Darius Hermes
Darius Hermes

Posted on

How I review AI-generated PRs for missing environment variables

#devops #ai #githubactions #opensource

AI-assisted PRs often change application code, deployment config, and CI at the same time.

That is useful, but it creates a boring failure mode that is easy to miss in review:

the PR starts using a new environment variable, but .env.example or .env.dist is not updated.

No raw secret values need to be leaked for this to break a deploy. The problem is metadata drift: the repo now depends on a variable that the declared environment contract does not mention.

Here is the review flow I use for that kind of PR.

Scenario

An AI agent opens a PR that looks reasonable:

  • adds a Stripe-backed checkout deploy step to GitHub Actions;
  • adds a Redis-backed worker in Docker Compose;
  • updates runtime code paths;
  • forgets to update .env.example.

The diff might pass normal code review because each individual change looks small.

1. Start with the env contract

The template file is the contract future contributors and reviewers see:

NEXT_PUBLIC_APP_URL=https://example.com
DATABASE_URL=
Enter fullscreen mode Exit fullscreen mode

This says the project expects NEXT_PUBLIC_APP_URL and DATABASE_URL.

It does not say anything about Stripe or Redis.

2. Check CI and deployment config, not only source code

The workflow now references a new secret:

env:
  DATABASE_URL: ${{ secrets.DATABASE_URL }}
  STRIPE_SECRET_KEY: ${{ secrets.STRIPE_SECRET_KEY }}
Enter fullscreen mode Exit fullscreen mode

Docker Compose also expects Redis:

services:
  web:
    environment:
      DATABASE_URL: ${DATABASE_URL}
      REDIS_URL: ${REDIS_URL}

  worker:
    environment:
      REDIS_URL: ${REDIS_URL}
Enter fullscreen mode Exit fullscreen mode

Both changes are normal. The bug is that STRIPE_SECRET_KEY and REDIS_URL became required deployment inputs without being added to the env template.

3. Run a metadata-only drift check

In the Secret Coverage repo, these fixtures model the two review findings:

pnpm scan -- --path examples/demos/github-actions-missing-secret --ci
pnpm scan -- --path examples/demos/docker-compose-missing-redis-url --ci
Enter fullscreen mode Exit fullscreen mode

For a consumer repo using the published package:

pnpm dlx @leviro-ai/secret-coverage scan --path . --ci
Enter fullscreen mode Exit fullscreen mode

The check compares variable names referenced by CI/CD, Docker, and config files with variable names declared in env templates. It does not need production secret values.

4. Read the findings like review comments

GitHub Actions drift:

## Critical

- **STRIPE_SECRET_KEY** — STRIPE_SECRET_KEY is used in .github/workflows/deploy.yml but missing from an env template.
  - Context: `.github/workflows/deploy.yml` · `missing-from-template`
  - Fix: Add STRIPE_SECRET_KEY= to an env template and configure the value in your deployment environment.
Enter fullscreen mode Exit fullscreen mode

Docker Compose drift:

## Critical

- **REDIS_URL** — REDIS_URL is used in docker-compose.yml but missing from an env template.
  - Context: `docker-compose.yml` · `missing-from-template`
  - Fix: Add REDIS_URL= to an env template and configure the value in your deployment environment.
Enter fullscreen mode Exit fullscreen mode

That is a useful PR-review signal because it points to the missing contract update, not to a vague deployment risk.

5. Ask for the smallest safe fix

The fix is usually a tiny template update:

NEXT_PUBLIC_APP_URL=https://example.com
DATABASE_URL=
STRIPE_SECRET_KEY=
REDIS_URL=
Enter fullscreen mode Exit fullscreen mode

The real values still belong in the deployment platform:

  • GitHub Actions secrets for CI/CD secrets;
  • Docker or Compose runtime environment for service variables;
  • Vercel, CircleCI, or another platform's environment settings where relevant.

Do not put raw secret values into .env.example, docs, screenshots, or review comments.

6. Add the check before merge

A small CI check can fail the PR before the deploy job discovers the mismatch:

name: secret-coverage

on:
  pull_request:
  push:
    branches: [main]

jobs:
  env-contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v6
        with:
          node-version: 20
      - run: pnpm dlx @leviro-ai/secret-coverage scan --path . --ci
Enter fullscreen mode Exit fullscreen mode

PR review checklist

Use this when an AI-assisted PR touches deployment config, CI, Docker, workers, or framework runtime config:

  • Did the PR add a new process.env.NAME, ${NAME}, ${{ secrets.NAME }}, or CI env reference?
  • Is every new required variable declared in .env.example, .env.dist, or the chosen template file?
  • Are demo/docs outputs metadata-only, without raw secret values?
  • Does the CI check run before deploy?
  • Does the finding tell the contributor which template entry to add?

Secret Coverage is the local-first tool I am using for this check. It is deterministic, metadata-only, and does not need a cloud account.

Links:

Top comments (0)