Yuri Gerasymov

Posted on Jun 4

Catch Visual Bugs Before They Hit Production: Automated Visual Regression Testing for Astro with Diffy and Cloudflare

#astro #github #diffy

You push a CSS tweak. It looks fine locally. The PR gets approved. You deploy — and two hours later someone messages you: "Hey, the navigation is broken on mobile."

Sound familiar? This is exactly the gap that visual regression testing fills. And if you're building with Astro and deploying to Cloudflare Pages, you can close that gap automatically on every pull request — without any manual QA steps.

This post walks through setting up Diffy with your Astro + GitHub + Cloudflare workflow so that every PR gets its own visual comparison against production before it's merged.

What Is Diffy?

Diffy is a screenshot-based visual regression testing tool. It takes screenshots of your pages across multiple breakpoints, compares them against a baseline (typically production), and highlights any visual differences — layout shifts, missing elements, broken styles, unexpected changes.

What sets Diffy apart from rolling your own Playwright/Puppetteer setup is its focus on zero false positives. Dynamic content like ads, carousels, and video embeds are the big problem of visual diffing — they change every time and flood you with noise. Diffy handles this by masking and freezing dynamic elements so your diffs stay meaningful.

The Stack

Here's what this integration connects:

Astro — your framework, with pages deployed as a static/SSR site
Cloudflare Pages — which spins up a unique preview URL for every branch/PR automatically
GitHub Actions — which orchestrates the workflow
Diffy — which does the visual comparison and posts results back to the PR

The flow looks like this:

PR opened → Cloudflare deploys preview → GitHub Action fires →
Diffy CLI compares preview vs. production → Result posted to PR check

Step 1: Set Up Your Diffy Project

First, create a project in Diffy. During setup you'll:

Add the URLs you want to test (your key pages — home, blog, landing pages, etc.)
Configure breakpoints (e.g. 375px mobile, 1280px desktop)
Take an initial set of screenshots from your production environment — this becomes the baseline

Once your project is created, grab two things you'll need later:

Your Project ID (from the project settings URL)
An API key (from https://app.diffy.website/#/keys)

Step 2: Connect Diffy to Your GitHub Repo

In Diffy, go to Project Settings → Notifications → GitHub and add your repository URL in the format:

https://github.com/{username}/{repo}

Then install the Diffy GitHub App on your repository. This is what gives Diffy permission to post check results back to your pull requests.

Step 3: Add Secrets to GitHub

In your GitHub repository, go to Settings → Secrets and variables → Actions and add:

A secret called DIFFY_API_KEY with your API key
A variable called DIFFY_PROJECT_ID with your project ID

(Secrets are encrypted and hidden in logs; variables are visible but not sensitive — hence the split.)

Step 4: Add the GitHub Actions Workflow

Create .github/workflows/visual-regression.yml in your repo:

name: Visual Regression

on:
  check_run:
    types: [completed]

jobs:
  compare:
    name: Run Diffy Comparison
    # Only fire on successful Cloudflare Workers and Pages check runs.
    if: >-
      github.event.check_run.check_suite.conclusion == 'success' &&
      github.event.check_run.app.name == 'Cloudflare Workers and Pages'
    runs-on: ubuntu-latest
    timeout-minutes: 15
    permissions:
      contents: read

    steps:
      # Extract the preview URL from the check run summary and resolve the PR number.
      - name: Get deployment URL and PR number
        id: info
        uses: actions/github-script@v9
        with:
          script: |
            const summary = context.payload.check_run.output.summary || '';
            const urlMatch = summary.match(/https:\/\/[a-z0-9-\.]+\.workers\.dev*/);
            if (!urlMatch) {
              core.setFailed('Could not find a *.workers.dev URL in the check run summary');
              return;
            }
            core.setOutput('deployment_url', urlMatch[0]);

            const branchName = context.payload.check_run.check_suite.head_branch || ''
            if (!branchName) {
              core.setFailed('Could not find a head branch name');
              return;
            }
            core.setOutput('branch_name', branchName);

      - name: Download Diffy CLI and trigger comparison job
        run: |
          wget -q -O /usr/local/bin/diffy \
            https://github.com/diffywebsite/diffy-cli/releases/latest/download/diffy.phar
          chmod +x /usr/local/bin/diffy

          diffy auth:login "${DIFFY_API_KEY}"

          diffy project:compare "${DIFFY_PROJECT_ID}" prod custom \
            --env2Url="${{ steps.info.outputs.deployment_url }}" \
            --commit-sha="${{ github.event.check_run.head_sha }}" \
            --name="PR ${{ steps.info.outputs.branch_name }}"
        env:
          DIFFY_API_KEY: ${{ secrets.DIFFY_API_KEY }}
          DIFFY_PROJECT_ID: ${{ vars.DIFFY_PROJECT_ID }}

Let's break down what's happening:

The trigger (check_run: completed) means this workflow fires whenever any check run finishes on the repo. The if: condition filters it down to only Cloudflare deployments that completed successfully — so you're not burning CI minutes on failed builds.

The first step uses actions/github-script to parse the Cloudflare check run's output summary and extract the preview URL (which looks like your-branch.your-project.pages.dev). It also grabs the branch name for labeling the comparison.

The second step downloads the Diffy CLI, authenticates, then runs project:compare with your production environment as the baseline and the Cloudflare preview URL as the comparison target. The --commit-sha flag is what links the Diffy result back to the specific commit so GitHub knows which PR check to update.

What You Get

After a PR is opened and Cloudflare finishes its deployment, the workflow automatically kicks off. Within a few minutes you'll see a new check appear on your PR:

✅ Passed — no visual differences detected, safe to merge
❌ Failed — visual differences found, with a link to the Diffy comparison

Clicking through takes you to a side-by-side diff view in Diffy where you can inspect exactly which pages changed, across each breakpoint you configured. If a change is intentional (you did update the hero section after all), you can approve it in Diffy to bring the check back to passed.

Why This Matters for Astro Projects

Astro's component model makes it easy to share layout components across dozens of pages. That's a superpower — but it also means a CSS change in a shared component can ripple across your entire site in ways that are hard to catch in code review.

If you make your changes to the site with Claude (or any other AI) you would want to have a safety net that will catch any unintentional changes.

Visual regression testing is that net. With this setup, every PR on your Astro site gets automatically checked against production before it ever has a chance to merge — with zero manual steps required from you or your reviewers.

DEV Community