Stop Letting AI Write Tests You Won’t Run: Enforce SDLC Gates That Catch LLM Hallucinations, Drift, and Silent Regressions

Originally published on tamiz.pro.

The Problem: AI Tests That Never Run

LLMs generate test code faster than developers can review it. But unchecked, these tests introduce hallucinated assertions, behavioral drift, and silent regressions that only surface in production. The fix isn’t banning AI—it’s enforcing SDLC gates that verify generated tests actually pass, cover real code paths, and match expected behavior before they merge.

The Gate Strategy

Instead of trusting generated tests, treat them as untrusted input. Every AI-generated test must pass through four gates:

1. Execution Gate — Tests must compile and run successfully.
2. Coverage Gate — New tests must increase meaningful coverage.
3. Behavior Gate — Tests must fail when target behavior changes (mutation testing).
4. Drift Gate — Tests must not assert on hallucinated or irrelevant logic.

Prerequisites

• A CI pipeline (GitHub Actions, GitLab CI, etc.)
• A test runner (Jest, Pytest, Go test, etc.)
• A coverage tool (nyc, coverage.py, go test -cover)
• Optionally: a mutation testing tool (Stryker, Cosmic-Ray, go-mutesting)

Step 1: Enforce the Execution Gate

AI-generated tests often contain syntax errors or reference non-existent APIs. Block merges where tests fail to run.

GitHub Actions Example

name: Test Execution Gate
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies
        run: npm ci
      - name: Run tests
        run: npm test -- --ci --maxWorkers=2

Why This Matters

If a test doesn’t run, it provides zero protection. This gate ensures every generated test is at least syntactically valid and executable.

Step 2: Enforce the Coverage Gate

AI can generate tests that execute code without asserting meaningful behavior. Require new tests to increase line and branch coverage.

Jest Coverage Threshold

// package.json
{
  "jest": {
    "collectCoverageFrom": ["src/**/*.{js,ts}"],
    "coverageThreshold": {
      "global": {
        "branches": 80,
        "functions": 80,
        "lines": 80,
        "statements": 80
      }
    }
  }
}

GitLab CI Coverage Check

test:
  script:
    - npm test -- --coverage
  coverage: '/Statements\s*:\s*(\d+\.?\d*)%'/```
{% endraw %}


### Why This Matters

Coverage thresholds prevent low-value tests from sneaking in. If an AI test doesn’t meaningfully increase coverage, it’s likely asserting on trivial or hallucinated paths.

## Step 3: Enforce the Behavior Gate (Mutation Testing)

The strongest guard against hallucinated assertions is mutation testing. If a test doesn’t detect a mutated version of the code, it’s not actually validating behavior.

### StrykerJS Example
{% raw %}


```bash
# package.json
turbo run mutate --filter=src/**/*.test.js

# .strykerrc.json
{
  "mutate": ["src/**/*.ts"],
  "testRunner": "jest",
  "thresholdHigh": 80,
  "thresholdLow": 60,
  "thresholdBreak": 0
}

GitHub Actions

mutation:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: 18
    - run: npm ci
    - run: npx stryker run --configuration .strykerrc.json

Why This Matters

Mutation testing proves tests fail when behavior changes. An AI test that survives mutations is asserting on irrelevant details, not real invariants.

Step 4: Enforce the Drift Gate

AI tests can drift from intended behavior by asserting on implementation details or hallucinated logic. Use snapshot testing or golden-master techniques to lock in expected outputs.

Jest Snapshot Testing

// user.service.test.js
test('returns active users only', () => {
  const result = userService.getActiveUsers();
  expect(result).toMatchSnapshot();
});

If the snapshot changes, the CI gate fails unless the developer explicitly approves the diff. This prevents silent behavioral drift.

Approval Tests Pattern

For non-JS ecosystems, use approval tests:

# Python example
from approvaltests import verify

def test_process_order():
    result = order_service.process_order(order_data)
    verify(result)

Why This Matters

Drift gates catch unintended behavioral changes. AI-generated tests can accidentally encode wrong assumptions—this gate forces explicit approval when assumptions change.

Step 5: Combine Gates in a Single Pipeline

Run all gates in parallel to catch failures early without slowing feedback.

name: AI Test Validation Pipeline
on: [pull_request]
jobs:
  execution:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm test -- --ci
  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm test -- --coverage
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx stryker run

A PR merges only when all gates pass. AI-generated tests that don’t survive execution, coverage, mutation, or drift checks are rejected automatically.

Best Practices

• Review generated code, but don’t trust it. Gates replace manual review for correctness.
• Fail fast. Run execution and coverage gates before mutation testing to save time.
• Use baselines. Track coverage and mutation scores over time to detect degradation.
• Automate approvals. For snapshot/diff-based gates, require explicit developer approval for changes.
• Log failures. Capture which gate failed and why, so teams can improve AI prompts or training data.

Frequently Asked Questions

**Q: Won’t mutation testing slow down CI?
A: Yes, significantly. Run it on a scheduled basis or for high-risk changes only. Use execution, coverage, and drift gates as the primary PR gates.

**Q: How do I handle AI-generated tests that are intentionally exploratory?
A: Separate exploratory tests from committed tests. Only enforce gates on committed test files.

**Q: What if my coverage threshold is too strict?
A: Start with a low threshold (e.g., 50%) and increase it gradually as tests improve. The goal is to catch zero-value tests, not enforce arbitrary numbers.

The Bottom Line

AI-generated tests are a productivity lever, not a quality guarantee. By enforcing execution, coverage, behavior, and drift gates in your SDLC pipeline, you turn untrusted AI output into verified, reliable test coverage. The result: faster development without silent regressions slipping into production.

For deeper insights on test reliability and pipeline design, see Tamiz's Insights.