Automating Test Generation in CI/CD Pipelines with AI

If you've shipped code in the last year, you've probably noticed the shift. AI is no longer just an autocomplete tool sitting in your editor — it's becoming a real participant in the pipeline itself. One of the most practical (and underrated) use cases is AI-generated test coverage running directly inside CI/CD.

In this article, we'll break down why this matters, how to actually wire it into a pipeline, and walk through working code you can drop into your own project today.

Overview

Traditionally, test writing has been the bottleneck between "code works on my machine" and "code is safe to ship." Developers write features fast, then either skip tests under deadline pressure or spend hours writing boilerplate unit tests that mostly check the obvious paths.

AI-assisted test generation flips this. Instead of a human writing every test case by hand, an LLM reads the function or module, understands its inputs/outputs and edge cases, and generates a first draft of tests — which a developer then reviews, trims, and commits. This isn't about replacing test-writing judgment; it's about removing the blank-page problem so engineers spend their time reviewing logic instead of typing boilerplate assertions.

Teams running this in production pipelines report catching edge cases (null inputs, boundary values, malformed payloads) that manual test suites often miss simply because nobody thought to write them.

If you're setting this up for your team, it's worth browsing a Software Hub first to compare which AI testing tools plug into your existing stack instead of building a custom integration from scratch.

How It Fits Into a Pipeline

The general flow looks like this:

1. A pull request is opened.
2. CI detects changed files.
3. An AI test-generation step analyzes the diff and produces test stubs for uncovered functions.
4. Generated tests run alongside the existing suite.
5. Coverage report is posted as a PR comment for human review.

Here's a simplified GitHub Actions workflow that implements this pattern using a Node.js project and an LLM API call to generate missing unit tests:

name: AI Test Generation

on:
  pull_request:
    branches: [main]

jobs:
  generate-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Get changed source files
        id: diff
        run: |
          git diff --name-only origin/main...HEAD -- '*.js' '*.ts' > changed_files.txt
          cat changed_files.txt

      - name: Generate tests for uncovered functions
        run: node scripts/generate-tests.js changed_files.txt

      - name: Run full test suite
        run: npm test -- --coverage

      - name: Upload coverage report
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage/

The Test Generation Script

The core logic lives in a small Node.js script that reads each changed file, sends the function signatures and body to an LLM, and writes out a test file if one doesn't already exist.

// scripts/generate-tests.js
const fs = require('fs');
const path = require('path');

const CHANGED_FILES_LIST = process.argv[2];

async function generateTestForFile(filePath) {
  const sourceCode = fs.readFileSync(filePath, 'utf-8');
  const testPath = filePath.replace(/\.js$/, '.test.js');

  if (fs.existsSync(testPath)) {
    console.log(`Skipping ${filePath}, tests already exist.`);
    return;
  }

  const prompt = `
    Analyze the following JavaScript module and generate a Jest test file
    covering normal cases, edge cases, and error handling.
    Only return valid JavaScript code, no explanations.

    Source:
    ${sourceCode}
  `;

  const response = await fetch('https://api.anthropic.com/v1/messages', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'claude-sonnet-4-6',
      max_tokens: 1500,
      messages: [{ role: 'user', content: prompt }],
    }),
  });

  const data = await response.json();
  const generatedTest = data.content
    .filter((block) => block.type === 'text')
    .map((block) => block.text)
    .join('\n');

  fs.writeFileSync(testPath, generatedTest);
  console.log(`Generated tests at ${testPath}`);
}

async function main() {
  const changedFiles = fs
    .readFileSync(CHANGED_FILES_LIST, 'utf-8')
    .split('\n')
    .filter((line) => line.trim().endsWith('.js'));

  for (const file of changedFiles) {
    if (fs.existsSync(file)) {
      await generateTestForFile(file);
    }
  }
}

main().catch((err) => {
  console.error('Test generation failed:', err);
  process.exit(1);
});

A few things worth noting about this setup:

• Review is non-negotiable. Generated tests go into the PR as regular files, so reviewers see them and can push back on weak assertions before merge.
• Cost control matters. Only run generation on changed files, not the whole repo, or your API bill will scale with every commit instead of every diff.
• Deterministic fallback. If the AI step fails or times out, the pipeline should still run the existing test suite rather than blocking the whole PR.

Handling Flaky or Low-Quality Generated Tests

Not every generated test is good out of the box. A simple guard is to run generated tests in isolation first and fail the step (without blocking the PR) if they don't compile or immediately throw:

// scripts/validate-generated-tests.js
const { execSync } = require('child_process');

function validateTestFile(testPath) {
  try {
    execSync(`npx jest ${testPath} --silent`, { stdio: 'pipe' });
    return true;
  } catch (err) {
    console.warn(`Generated test ${testPath} failed validation, flagging for manual review.`);
    return false;
  }
}

module.exports = { validateTestFile };

This keeps the pipeline honest — generated tests either prove themselves or get flagged, they don't silently rot in the repo.

Tooling Costs and Alternatives

Most teams reach for a paid SaaS product to handle the LLM orchestration, prompt tuning, and coverage dashboards out of the box. That's a reasonable shortcut if budget allows it. But if you're a solo developer or a small team testing this workflow before committing budget, it's worth checking a free alternative of paid software before paying for a full platform — many of the core features (diff parsing, prompt templating, coverage merging) can be self-hosted with the open-source tools already in your stack.

Wrapping Up

AI-generated tests aren't a replacement for thoughtful test design, but they're a genuinely useful way to close coverage gaps that get skipped under deadline pressure. The pattern above — detect changed files, generate targeted tests, validate before merge, surface coverage in the PR — is lightweight enough to bolt onto almost any existing CI/CD setup without a full platform migration.

If you're experimenting with this, start small: wire it up for one service, watch what the AI gets wrong, and tighten your prompts from there. The blank-page problem is the one you're actually solving — everything else is refinement.