How to Add Visual Regression Testing to Your CI Pipeline

How to Add Visual Regression Testing to Your CI Pipeline

Your CSS change looked fine in the browser. You merged it. The deployment went out. Then someone noticed the pricing page hero section collapsed on mobile. The button text wrapped. The testimonial card lost its shadow.

Visual regression testing catches these before they reach production. Here's how to set it up without overcomplicating your pipeline.

The Concept

Take screenshots of key pages before and after a code change. Compare them pixel by pixel. If the difference exceeds a threshold, fail the build.

It's the visual equivalent of unit tests — you define what "correct" looks like, and the system tells you when something deviates.

Baseline Screenshots

First, capture reference screenshots of your pages in a known-good state. These are your baselines:

# capture-baselines.sh
PAGES=(
  "/"
  "/pricing"
  "/features"
  "/docs/getting-started"
  "/blog"
)

for page in "${PAGES[@]}"; do
  slug=$(echo "$page" | sed 's/\//_/g' | sed 's/^_//')
  [ -z "$slug" ] && slug="index"
  
  curl -s "https://screenshotrun.com/api/v1/screenshot?\
url=https://staging.yourapp.com${page}&\
width=1280&\
format=png&\
full_page=false" \
    -H "Authorization: Bearer $SCREENSHOT_API_KEY" \
    -o "baselines/${slug}.png"
    
  echo "Captured baseline: $slug"
done

Store baselines in your repository or artifact storage. They update only when you intentionally approve visual changes.

CI Comparison Script

On each pull request, capture the same pages from your preview/staging environment and compare against baselines:

#!/usr/bin/env python3
"""
visual_regression.py — compare current screenshots against baselines.
Exit code 1 if any page exceeds the diff threshold.
"""

import os
import sys
import requests
from pathlib import Path

try:
    from PIL import Image
    import numpy as np
except ImportError:
    print("pip install Pillow numpy")
    sys.exit(1)

API_URL = "https://screenshotrun.com/api/v1/screenshot"
API_KEY = os.environ["SCREENSHOT_API_KEY"]
STAGING_URL = os.environ.get("STAGING_URL", "https://staging.yourapp.com")
THRESHOLD = float(os.environ.get("VISUAL_DIFF_THRESHOLD", "2.0"))

PAGES = {
    "index": "/",
    "pricing": "/pricing",
    "features": "/features",
    "docs": "/docs/getting-started",
    "blog": "/blog",
}


def capture(url: str) -> bytes:
    params = {
        "url": url,
        "width": 1280,
        "format": "png",
        "full_page": False,
        "delay": 2,
    }
    resp = requests.get(
        API_URL,
        params=params,
        headers={"Authorization": f"Bearer {API_KEY}"},
        timeout=60,
    )
    resp.raise_for_status()
    return resp.content


def compare_images(baseline_path: str, current_bytes: bytes) -> float:
    baseline = np.array(Image.open(baseline_path).convert("RGB"))
    current = np.array(Image.open(
        __import__("io").BytesIO(current_bytes)
    ).convert("RGB"))

    # Resize if dimensions don't match
    if baseline.shape != current.shape:
        h = min(baseline.shape[0], current.shape[0])
        w = min(baseline.shape[1], current.shape[1])
        baseline = baseline[:h, :w]
        current = current[:h, :w]

    diff = np.abs(baseline.astype(int) - current.astype(int))
    changed_pixels = np.any(diff > 25, axis=2).sum()
    total_pixels = diff.shape[0] * diff.shape[1]
    return (changed_pixels / total_pixels) * 100


def main():
    baselines_dir = Path("baselines")
    failures = []

    for slug, path in PAGES.items():
        baseline_file = baselines_dir / f"{slug}.png"
        if not baseline_file.exists():
            print(f"  SKIP {slug}: no baseline")
            continue

        url = f"{STAGING_URL}{path}"
        print(f"  Checking {slug} ({url})...")

        img_bytes = capture(url)
        diff_pct = compare_images(str(baseline_file), img_bytes)

        if diff_pct > THRESHOLD:
            failures.append((slug, diff_pct))
            print(f"  FAIL {slug}: {diff_pct:.2f}% changed (threshold: {THRESHOLD}%)")

            # Save the current screenshot for review
            Path("diffs").mkdir(exist_ok=True)
            (Path("diffs") / f"{slug}_current.png").write_bytes(img_bytes)
        else:
            print(f"  OK   {slug}: {diff_pct:.2f}% changed")

    if failures:
        print(f"\n{len(failures)} page(s) exceeded visual diff threshold.")
        sys.exit(1)

    print("\nAll pages passed visual regression check.")


if __name__ == "__main__":
    main()

GitHub Actions Integration

name: Visual Regression
on: [pull_request]

jobs:
  visual-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      
      - name: Install dependencies
        run: pip install requests Pillow numpy
      
      - name: Run visual regression tests
        env:
          SCREENSHOT_API_KEY: ${{ secrets.SCREENSHOT_API_KEY }}
          STAGING_URL: ${{ env.PREVIEW_URL }}
          VISUAL_DIFF_THRESHOLD: "2.0"
        run: python visual_regression.py
      
      - name: Upload diff artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: visual-diffs
          path: diffs/

When the check fails, reviewers can download the diff artifacts and see exactly which pages changed and how.

Choosing What to Test

Don't screenshot every page. Focus on:

• Revenue-critical pages — pricing, checkout, signup
• High-traffic pages — homepage, main landing pages
• Component-heavy pages — pages that use many shared components, so a single CSS change cascades

Skip pages with dynamic content (dashboards, feeds) unless you can seed them with consistent test data.

Managing Baseline Updates

When you intentionally change a page's appearance, the visual test will fail. That's expected. The workflow:

1. PR changes CSS/layout
2. Visual test fails (expected)
3. Review the diff artifacts to confirm changes look correct
4. Update baselines: ./capture-baselines.sh
5. Commit new baselines
6. Visual test passes

Some teams automate baseline updates with a bot comment — reply "update baselines" on the PR, and a workflow captures fresh screenshots and commits them.

Threshold Tuning

A 0% threshold catches everything, including subpixel rendering differences between environments. Too noisy.

A 5% threshold misses subtle changes — a shifted button, a truncated label. Too loose.

Start at 1-2% and adjust based on your false positive rate. Different pages might need different thresholds — a text-heavy docs page is more stable than a marketing page with animations.

Cost and Performance

Each screenshot API call takes 2-5 seconds. Testing 10 pages adds under a minute to your CI pipeline. The cost is minimal compared to the engineering time spent debugging visual bugs that shipped to production.

The tricky part isn't the technology — it's the discipline of maintaining baselines and reviewing diffs. Treat visual tests like any other test: if they fail, investigate before merging.

Using ScreenshotRun for the capture step keeps the infrastructure simple — no need to maintain your own headless browser pool in CI. Send a URL, get a screenshot, compare locally.