DEV Community: Vitalii Holben

I replaced Selenium screenshots with a 3-line API call

Vitalii Holben — Fri, 16 Oct 2026 14:01:41 +0000

Had a Selenium setup for taking screenshots in our CI pipeline. It worked, but maintaining it was a whole thing.

The Docker image was 800MB because it bundled Chromium. Build times were 4 minutes just for the screenshot step. And every few weeks something broke -- ChromeDriver version mismatch, a new Chrome flag that changed default behavior, font rendering differences between CI runners.

Spent more time babysitting the screenshot infrastructure than actually using the screenshots.

The switch

Ripped out the entire Selenium setup and replaced it with an HTTP call:

const resp = await fetch(
  `https://api.screenshotrun.com/capture?url=${encodeURIComponent(targetUrl)}&width=1440&format=png`,
  { headers: { "Authorization": `Bearer ${API_KEY}` } }
);
const buffer = await resp.arrayBuffer();
fs.writeFileSync("screenshot.png", Buffer.from(buffer));

Thats it. No browser binary, no driver compatibility matrix, no 800MB Docker layer.

What changed

Build times dropped from 4 minutes to about 12 seconds for the screenshot step. The Docker image went from 800MB to 120MB. Zero maintenance on the rendering side -- thats someone elses problem now.

The screenshots are also more consistent. Same rendering environment every time, regardless of which CI runner picks up the job. No more "works on runner A, looks different on runner B" debugging sessions.

Tradeoffs

Costs money, obviously. For our volume (maybe 200 captures per day across staging and PR previews) its a few bucks a month. Way less than the engineer-hours I was spending on Selenium maintenance.

You also depend on an external service. If their API is down during your CI run, your build fails. Added a simple retry with 3 attempts and a 5-second backoff. Hasnt been an issue in practice but its worth knowing about.

If youre doing fewer than 50 screenshots a day and your Selenium setup is stable, probably not worth switching. But if youre fighting Chrome version mismatches every sprint like I was -- yeah, just use an API.

I Caught a Production CSS Bug Before Users Did

Vitalii Holben — Fri, 02 Oct 2026 13:20:34 +0000

title: I Caught a Production CSS Bug Before Users Did — Here's the Visual Regression Setup published: true tags: testing, webdev, css, devops

We shipped a "minor" font-size change last month. Updated the base size from 16px to 15px in our design tokens. Seemed harmless — the designer wanted tighter spacing.

It broke the checkout flow.

The payment form had a hardcoded min-height based on content size. With the smaller font, the content shifted up, and the submit button overlapped the terms checkbox on mobile viewports. Users on iPhone SE literally couldn't complete a purchase.

We didn't catch it in code review because the CSS diff looked fine. We didn't catch it in unit tests because we don't render real layouts in Jest. We caught it 4 hours after deploy when support tickets started coming in.

What visual regression testing actually looks like

The concept: screenshot key pages before deploy, screenshot them after, diff the images. If pixels changed where they shouldn't have, block the deploy.

In practice it's messier than that.

# pseudo pipeline
1. Take "baseline" screenshots of production
2. Deploy to staging
3. Take "candidate" screenshots of staging
4. Compare each pair, flag differences above threshold
5. Human reviews flagged diffs
6. Approve or reject deploy

Step 4 is where most tools fall apart. Pixel-perfect comparison produces way too many false positives — browser antialiasing, font rendering differences, dynamic content like timestamps, ads, user avatars.

What I actually use

After trying a few setups (BackstopJS, Percy, custom Puppeteer scripts), I landed on a simpler approach:

Monitor key pages periodically with SnapshotArchive — it takes scheduled screenshots and stores the history
When we deploy, I compare the latest pre-deploy screenshot against the staging version
Significant visual changes get flagged in Slack

The advantage over CI-integrated tools: I get visual monitoring even when there's NO deploy. So if a third-party script changes, or a CDN serves different assets, or a CMS editor modifies content — I still see it.

The false positive problem

Every visual testing tool struggles with this. Here's what reduced noise for me:

Mask dynamic regions — hide elements that change on every load (timestamps, personalized content, ads). Most tools support CSS selectors for this.

Perceptual diffing over pixel diffing — a 1px antialiasing difference isn't a real change. Tools that use perceptual comparison (SSIM or similar) produce way fewer false alarms.

Separate critical from cosmetic — the checkout page gets a 0.1% threshold. The blog page gets 5%. Not everything needs the same sensitivity.

My current setup in numbers

12 pages monitored (homepage, pricing, checkout flow, key landing pages)
Screenshots every 6 hours outside of deploys
~2 real alerts per week after tuning thresholds
0 CSS incidents reaching production since setting this up

The checkout font-size bug would've shown up as a visual diff on staging before we merged. Instead it cost us 4 hours of lost conversions and a hotfix.

The irony: the monitoring setup took an afternoon. The incident cost way more.

What's your visual testing setup look like? Or are you still yolo-deploying CSS changes like we used to?

I Built a Screenshot Pipeline That Handles 10K URLs Daily

Vitalii Holben — Fri, 02 Oct 2026 13:20:24 +0000

title: I Built a Screenshot Pipeline That Handles 10K URLs Daily (Here's What Broke First) published: true tags: webdev, api, automation, javascript

Last quarter I needed to generate preview thumbnails for a link aggregation tool. The requirement seemed simple: given a URL, produce a 1200x630 screenshot. At first it was maybe 50 URLs a day. Then the product grew.

At around 2,000 URLs per day things started falling apart.

The first bottleneck: memory

Each Puppeteer page instance uses somewhere between 50-150MB of RAM depending on the page complexity. Running 10 concurrent pages meant 1.5GB just for Chrome, plus Node overhead. Our 4GB server started swapping.

// the naive approach that doesn't scale
const pages = urls.map(async (url) => {
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: 'networkidle0' });
  await page.screenshot({ path: `${hash(url)}.png` });
  await page.close();
});
await Promise.all(pages); // boom, 10K pages at once

The fix was a simple semaphore — limit to 4 concurrent pages and queue the rest. But even with that, Chrome would occasionally leak memory and need a restart every ~500 screenshots.

The second bottleneck: timeouts

Some URLs just don't load. Or they load but never reach networkidle0 because of persistent WebSocket connections or infinite polling. I was losing 15-20% of my daily runs to timeout-related failures.

What worked:

await page.goto(url, {
  waitUntil: 'domcontentloaded', // not networkidle0
  timeout: 15000
});
// then wait for specific readiness signal
await page.waitForFunction(
  () => document.readyState === 'complete',
  { timeout: 5000 }
).catch(() => {}); // screenshot whatever we have

Switching from networkidle0 to domcontentloaded + a short secondary wait cut my failure rate from 18% to under 3%.

The third bottleneck: me

At some point I was spending more time maintaining the screenshot infrastructure than building the actual product. Chrome updates breaking things, memory leak debugging, managing the screenshot queue, handling retries, storage rotation...

I switched the whole thing to ScreenshotRun — send a URL via API, get back an image. No browser management, no server scaling, no 3am Chrome crashes.

// entire screenshot pipeline, after
const response = await fetch('https://api.screenshotrun.com/capture', {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${API_KEY}` },
  body: JSON.stringify({ url, width: 1200, height: 630 })
});
const image = await response.buffer();

My screenshot server went from a dedicated 8GB instance to zero infrastructure. The API handles the browser pool, the timeouts, the retries, the scaling.

What I'd do differently

If I was starting over and knew I'd hit scale:

Don't self-host screenshots past ~500/day — the operational overhead isn't worth it unless you need very specific browser customization
Always use domcontentloaded over networkidle0 — the few pages that need network idle aren't worth the timeout penalty on everything else
Store WebP, not PNG — 60-70% smaller files, virtually identical quality for thumbnails
Build the retry logic from day one — transient failures are guaranteed at scale, not edge cases

Has anyone else hit similar scaling issues with browser-based tools? Curious what concurrency limits others settled on.

How to Capture Full-Page Screenshots of Infinite Scroll Pages

Vitalii Holben — Wed, 23 Sep 2026 09:04:07 +0000

title: How to Capture Full-Page Screenshots of Infinite Scroll Pages tags: webdev, javascript, puppeteer, testing

Infinite scroll pages don't have a "bottom." That makes full-page screenshots tricky — the browser doesn't know how tall the page actually is until you've scrolled through everything.

Here's how I handle it, and where the common approaches break.

The Naive Approach (and Why It Fails)

Puppeteer's fullPage: true option works by measuring document.body.scrollHeight and setting the viewport to that height before capturing. On a normal page, this is fine.

On an infinite scroll page, scrollHeight only reflects content that's already loaded. If the page lazy-loads 50 more items when you reach the bottom... those items aren't in the DOM yet when the screenshot fires.

You get a screenshot of the first batch of content and nothing else.

Scroll, Wait, Repeat

The workaround is manual scrolling:

async function scrollToBottom(page) {
  let previousHeight = 0;
  let currentHeight = await page.evaluate(() => document.body.scrollHeight);
  
  while (previousHeight !== currentHeight) {
    previousHeight = currentHeight;
    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
    await page.waitForTimeout(1500);
    currentHeight = await page.evaluate(() => document.body.scrollHeight);
  }
}

Scroll to bottom, wait for new content, check if height changed. When it stops changing, you've hit the end. Then take your fullPage screenshot.

The waitForTimeout(1500) is a guess. Some pages load fast, some make API calls that take 3+ seconds. I've seen people use waitForNetworkIdle here, but that can hang forever on pages with analytics pings and websocket connections.

What I do: wait for timeout, then also check if new DOM elements appeared:

const itemCount = await page.evaluate(
  () => document.querySelectorAll('.feed-item').length
);
// scroll...
const newCount = await page.evaluate(
  () => document.querySelectorAll('.feed-item').length
);
if (newCount === itemCount) break;

More reliable than height checks for some layouts.

The Memory Problem

A page with 500 loaded items is big. Setting the viewport to 1920x50000 and capturing as PNG creates a massive image. Chrome might crash, or your server runs out of memory.

Two options:

Cap the scroll depth. Don't try to capture everything. Scroll N times and stop:

const MAX_SCROLLS = 20;
for (let i = 0; i < MAX_SCROLLS; i++) {
  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await page.waitForTimeout(1000);
}

Use JPEG instead of PNG. For long screenshots, JPEG at quality 80 is usually 5-10x smaller:

await page.screenshot({ 
  fullPage: true, 
  type: 'jpeg', 
  quality: 80 
});

Intersection Observer Gotchas

Modern sites use IntersectionObserver for lazy loading. The observer fires when elements enter the viewport. In headless Chrome, scrolling programmatically does trigger these observers — but only if you actually scroll through the viewport, not just jump to the bottom.

If you window.scrollTo(0, 99999) in one jump, observers for elements in the middle of the page might not fire. Scroll incrementally:

const scrollStep = 500;
let scrollPos = 0;
const maxHeight = await page.evaluate(() => document.body.scrollHeight);

while (scrollPos < maxHeight) {
  scrollPos += scrollStep;
  await page.evaluate((y) => window.scrollTo(0, y), scrollPos);
  await page.waitForTimeout(200);
}

Slower but catches all the lazy-loaded content.

When to Give Up

Some pages are genuinely infinite — social media feeds, search results that go on forever. Set a reasonable limit — either by scroll count, total height, or time spent — and document it. "We capture the first 50 items" is a perfectly valid approach.

How to Capture Dark Mode Screenshots Programmatically

Vitalii Holben — Wed, 16 Sep 2026 12:23:17 +0000

Needed to generate both light and dark mode screenshots of a marketing site for a client's brand guidelines doc. Turns out forcing dark mode in a headless browser isn't as straightforward as setting a CSS class.

Here's how I got it working with Playwright (same approach works with Puppeteer).

The prefers-color-scheme approach

Most modern sites use the prefers-color-scheme media query for dark mode. You can emulate this in Playwright:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch();

  // Light mode capture
  const lightCtx = await browser.newContext({
    colorScheme: 'light',
    viewport: { width: 1280, height: 800 }
  });
  const lightPage = await lightCtx.newPage();
  await lightPage.goto('https://example.com');
  await lightPage.screenshot({ path: 'light.png', fullPage: true });

  // Dark mode capture
  const darkCtx = await browser.newContext({
    colorScheme: 'dark',
    viewport: { width: 1280, height: 800 }
  });
  const darkPage = await darkCtx.newPage();
  await darkPage.goto('https://example.com');
  await darkPage.screenshot({ path: 'dark.png', fullPage: true });

  await browser.close();
})();

The key is colorScheme: 'dark' in the browser context. This sets prefers-color-scheme: dark at the OS level, so the page responds exactly as it would on a user's machine with dark mode enabled.

When the site uses a toggle instead

Some sites don't rely on the media query. They use a JavaScript toggle that sets a class on <body> or <html> — something like class="dark" or data-theme="dark".

For these, you need to interact with the page:

const page = await browser.newPage();
await page.goto('https://example.com');

// Take light mode screenshot first
await page.screenshot({ path: 'light.png' });

// Click the theme toggle
await page.click('[data-testid="theme-toggle"]');

// Wait for transition animations to finish
await page.waitForTimeout(500);

await page.screenshot({ path: 'dark.png' });

If you don't know the toggle's selector, you can force it by injecting the class directly:

await page.evaluate(() => {
  document.documentElement.classList.add('dark');
  // or
  document.documentElement.setAttribute('data-theme', 'dark');
});

// Give CSS transitions time to complete
await page.waitForTimeout(300);

await page.screenshot({ path: 'dark-forced.png' });

This is hacky but reliable when you can't find the toggle button.

Handling both approaches at once

Real-world sites sometimes use a combination — media query as default, with a manual override stored in localStorage. Here's what I ended up with for a batch capture script:

async function captureWithTheme(url, theme, outputPath) {
  const context = await browser.newContext({
    colorScheme: theme,
    viewport: { width: 1280, height: 800 }
  });

  const page = await context.newPage();
  await page.goto(url, { waitUntil: 'networkidle' });

  // Also set common localStorage/cookie theme preferences
  await page.evaluate((t) => {
    localStorage.setItem('theme', t);
    localStorage.setItem('color-mode', t);
    document.cookie = `theme=${t}; path=/`;
  }, theme);

  // Reload to pick up localStorage-based themes
  await page.reload({ waitUntil: 'networkidle' });

  // Force the class as a fallback
  await page.evaluate((t) => {
    document.documentElement.classList.remove('light', 'dark');
    document.documentElement.classList.add(t);
    document.documentElement.setAttribute('data-theme', t);
  }, theme);

  await page.waitForTimeout(500);
  await page.screenshot({ path: outputPath, fullPage: true });
  await context.close();
}

// Usage
await captureWithTheme('https://example.com', 'dark', 'dark.png');
await captureWithTheme('https://example.com', 'light', 'light.png');

Bit aggressive with the overrides, but it catches most theme implementations I've encountered.

The Puppeteer equivalent

If you're using Puppeteer instead of Playwright, the media query emulation looks different:

const page = await browser.newPage();

// Emulate dark mode
await page.emulateMediaFeatures([
  { name: 'prefers-color-scheme', value: 'dark' }
]);

await page.goto('https://example.com');
await page.screenshot({ path: 'dark.png' });

emulateMediaFeatures works well but you need to call it before navigating to the page. If you call it after, some components might not re-render until you force a repaint.

Gotchas I ran into

CSS transitions mess up screenshots. If the site has a smooth dark-to-light transition (0.3s fade), your screenshot might catch it mid-transition. Either wait for the animation to complete or disable transitions entirely:

await page.addStyleTag({
  content: '*, *::before, *::after { transition: none !important; animation: none !important; }'
});

Images don't always switch. Some sites serve different image assets for light/dark mode using the <picture> element with prefers-color-scheme in the <source> media attribute. The colorScheme context option handles this correctly. The JavaScript class-forcing approach does not — you'll get light mode images with a dark background.

Third-party widgets ignore your theme. Embedded chat widgets, analytics consent banners, and social embeds use their own theme logic. Your dark mode screenshot might have a blinding white cookie popup in the corner. Remove these elements before capturing:

await page.evaluate(() => {
  const selectors = ['[class*="cookie"]', '[class*="consent"]',
                     '[class*="chat-widget"]', 'iframe[src*="chatbot"]'];
  selectors.forEach(sel => {
    document.querySelectorAll(sel).forEach(el => el.remove());
  });
});

System dark mode affects more than colors. On macOS, dark mode also changes scrollbar appearance, form control styling, and the default background color of transparent elements. If pixel-perfect consistency between light and dark captures matters, set a fixed background on <html> before capturing.

When to use this

Visual regression testing is the obvious case — you want to verify that both themes render correctly after every deploy. But I've also used it for generating marketing assets (light mode for print, dark mode for social media previews) and documentation screenshots where you want to show both variants side by side.

The Playwright approach with colorScheme in the browser context is the cleanest solution for media-query-based themes. For everything else, the JavaScript override approach gets the job done even if it feels a bit brute-force.

How to Handle Anti-Bot Measures When Taking Screenshots Programmatically

Vitalii Holben — Thu, 03 Sep 2026 12:33:16 +0000

How to Handle Anti-Bot Measures When Taking Screenshots Programmatically

You send a request. The page loads. The screenshot comes back blank, or shows a CAPTCHA, or captures a "Please verify you're human" wall. This is one of the most common problems when building any screenshot pipeline.

Here's what's actually happening and how to deal with it.

Why headless browsers get flagged

Bot detection works by looking for patterns that differ from real users. Headless Chrome has several tells:

navigator.webdriver returns true by default
Missing Chrome-specific properties like window.chrome
Inconsistent screen dimensions (no monitor attached means no GPU info)
Mouse events fire at pixel-perfect coordinates with no jitter
Font fingerprints differ from headed browsers

Modern detection services (Cloudflare, Akamai, Datadome) look for combinations of these signals, not individual flags. Spoofing one without the others often makes the fingerprint more suspicious, not less.

The practical spectrum of detection

Most sites fall into one of three categories:

No active detection — a basic bot check via User-Agent string at most. Simple fix: set a realistic UA.

Passive fingerprinting — loads a detection script, collects signals, blocks on second or third visit. You'll see this on news sites, e-commerce, media platforms.

Active challenges — Cloudflare Turnstile, hCaptcha, reCAPTCHA v3 score-based. These require real interaction or a solving service.

Know which category your target falls into before spending time on it.

Fixes that work for most cases

1. Use a stealth plugin

For Playwright, playwright-extra with puppeteer-extra-plugin-stealth patches the most common fingerprinting vectors:

npm install playwright-extra puppeteer-extra-plugin-stealth

import { chromium } from 'playwright-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';

chromium.use(StealthPlugin());
const browser = await chromium.launch();

This handles navigator.webdriver, window.chrome, and several other flags automatically.

2. Set realistic headers

await page.setExtraHTTPHeaders({
  'Accept-Language': 'en-US,en;q=0.9',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'sec-ch-ua': '"Chromium";v="120", "Google Chrome";v="120"',
  'sec-ch-ua-mobile': '?0',
  'sec-ch-ua-platform': '"macOS"',
});

3. Use a real Chrome binary, not Chromium

Chromium's font set differs from Chrome. Detection services track which fonts are available. Running headful Chrome via executablePath gives you a more convincing fingerprint than the bundled Chromium.

4. Add realistic delays and mouse movement

// Move mouse before clicking
await page.mouse.move(100 + Math.random() * 50, 200 + Math.random() * 30);
await page.waitForTimeout(300 + Math.random() * 500);
await page.click('#target');

Perfectly timed, pixel-precise interactions are a dead giveaway. Add jitter.

When the above isn't enough

Some sites run heavy fingerprinting that's hard to spoof at the browser level. Options at this point:

Residential proxies — rotate real residential IPs. The IP reputation matters as much as the browser fingerprint. Datacenter IPs are often pre-blocked.

Browser profiles — maintain persistent browser profiles with cookies, browsing history, and localStorage. A "fresh" headless browser looks different from a browser with 3 weeks of history.

Playwright with persistent context:

const context = await chromium.launchPersistentContext('./user-data-dir', {
  headless: false, // or true with stealth
});

The ethical and legal line

There's a difference between taking screenshots of public pages for archival, testing, or display purposes — which is generally fine — and bypassing authentication or rate limits to scrape data at scale. The former is what most screenshot tools do. The latter runs into Terms of Service issues and, in some jurisdictions, legal risk.

Worth being clear on which side of that line your use case sits before investing in bypass techniques.

What to monitor

Once you have a working pipeline, track failure reasons explicitly:

const response = await page.goto(url, { waitUntil: 'networkidle2' });
if (response.status() === 403 || response.status() === 429) {
  // Bot detection hit — log, rotate IP, back off
}

const pageTitle = await page.title();
if (pageTitle.toLowerCase().includes('captcha') || pageTitle.includes('verify')) {
  // Challenge page — flag for manual review
}

Silent failures — a 200 status but a CAPTCHA screenshot — are harder to catch. Use a content hash comparison or check for expected elements before saving the screenshot.

The problem doesn't go away entirely, but being systematic about detection and response turns a reliability nightmare into something manageable.

How to Add Visual Regression Testing to Your CI Pipeline

Vitalii Holben — Mon, 31 Aug 2026 11:15:13 +0000

How to Add Visual Regression Testing to Your CI Pipeline

Your CSS change looked fine in the browser. You merged it. The deployment went out. Then someone noticed the pricing page hero section collapsed on mobile. The button text wrapped. The testimonial card lost its shadow.

Visual regression testing catches these before they reach production. Here's how to set it up without overcomplicating your pipeline.

The Concept

Take screenshots of key pages before and after a code change. Compare them pixel by pixel. If the difference exceeds a threshold, fail the build.

It's the visual equivalent of unit tests — you define what "correct" looks like, and the system tells you when something deviates.

Baseline Screenshots

First, capture reference screenshots of your pages in a known-good state. These are your baselines:

# capture-baselines.sh
PAGES=(
  "/"
  "/pricing"
  "/features"
  "/docs/getting-started"
  "/blog"
)

for page in "${PAGES[@]}"; do
  slug=$(echo "$page" | sed 's/\//_/g' | sed 's/^_//')
  [ -z "$slug" ] && slug="index"
  
  curl -s "https://screenshotrun.com/api/v1/screenshot?\
url=https://staging.yourapp.com${page}&\
width=1280&\
format=png&\
full_page=false" \
    -H "Authorization: Bearer $SCREENSHOT_API_KEY" \
    -o "baselines/${slug}.png"
    
  echo "Captured baseline: $slug"
done

Store baselines in your repository or artifact storage. They update only when you intentionally approve visual changes.

CI Comparison Script

On each pull request, capture the same pages from your preview/staging environment and compare against baselines:

#!/usr/bin/env python3
"""
visual_regression.py — compare current screenshots against baselines.
Exit code 1 if any page exceeds the diff threshold.
"""

import os
import sys
import requests
from pathlib import Path

try:
    from PIL import Image
    import numpy as np
except ImportError:
    print("pip install Pillow numpy")
    sys.exit(1)

API_URL = "https://screenshotrun.com/api/v1/screenshot"
API_KEY = os.environ["SCREENSHOT_API_KEY"]
STAGING_URL = os.environ.get("STAGING_URL", "https://staging.yourapp.com")
THRESHOLD = float(os.environ.get("VISUAL_DIFF_THRESHOLD", "2.0"))

PAGES = {
    "index": "/",
    "pricing": "/pricing",
    "features": "/features",
    "docs": "/docs/getting-started",
    "blog": "/blog",
}


def capture(url: str) -> bytes:
    params = {
        "url": url,
        "width": 1280,
        "format": "png",
        "full_page": False,
        "delay": 2,
    }
    resp = requests.get(
        API_URL,
        params=params,
        headers={"Authorization": f"Bearer {API_KEY}"},
        timeout=60,
    )
    resp.raise_for_status()
    return resp.content


def compare_images(baseline_path: str, current_bytes: bytes) -> float:
    baseline = np.array(Image.open(baseline_path).convert("RGB"))
    current = np.array(Image.open(
        __import__("io").BytesIO(current_bytes)
    ).convert("RGB"))

    # Resize if dimensions don't match
    if baseline.shape != current.shape:
        h = min(baseline.shape[0], current.shape[0])
        w = min(baseline.shape[1], current.shape[1])
        baseline = baseline[:h, :w]
        current = current[:h, :w]

    diff = np.abs(baseline.astype(int) - current.astype(int))
    changed_pixels = np.any(diff > 25, axis=2).sum()
    total_pixels = diff.shape[0] * diff.shape[1]
    return (changed_pixels / total_pixels) * 100


def main():
    baselines_dir = Path("baselines")
    failures = []

    for slug, path in PAGES.items():
        baseline_file = baselines_dir / f"{slug}.png"
        if not baseline_file.exists():
            print(f"  SKIP {slug}: no baseline")
            continue

        url = f"{STAGING_URL}{path}"
        print(f"  Checking {slug} ({url})...")

        img_bytes = capture(url)
        diff_pct = compare_images(str(baseline_file), img_bytes)

        if diff_pct > THRESHOLD:
            failures.append((slug, diff_pct))
            print(f"  FAIL {slug}: {diff_pct:.2f}% changed (threshold: {THRESHOLD}%)")

            # Save the current screenshot for review
            Path("diffs").mkdir(exist_ok=True)
            (Path("diffs") / f"{slug}_current.png").write_bytes(img_bytes)
        else:
            print(f"  OK   {slug}: {diff_pct:.2f}% changed")

    if failures:
        print(f"\n{len(failures)} page(s) exceeded visual diff threshold.")
        sys.exit(1)

    print("\nAll pages passed visual regression check.")


if __name__ == "__main__":
    main()

GitHub Actions Integration

name: Visual Regression
on: [pull_request]

jobs:
  visual-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      
      - name: Install dependencies
        run: pip install requests Pillow numpy
      
      - name: Run visual regression tests
        env:
          SCREENSHOT_API_KEY: ${{ secrets.SCREENSHOT_API_KEY }}
          STAGING_URL: ${{ env.PREVIEW_URL }}
          VISUAL_DIFF_THRESHOLD: "2.0"
        run: python visual_regression.py
      
      - name: Upload diff artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: visual-diffs
          path: diffs/

When the check fails, reviewers can download the diff artifacts and see exactly which pages changed and how.

Choosing What to Test

Don't screenshot every page. Focus on:

Revenue-critical pages — pricing, checkout, signup
High-traffic pages — homepage, main landing pages
Component-heavy pages — pages that use many shared components, so a single CSS change cascades

Skip pages with dynamic content (dashboards, feeds) unless you can seed them with consistent test data.

Managing Baseline Updates

When you intentionally change a page's appearance, the visual test will fail. That's expected. The workflow:

PR changes CSS/layout
Visual test fails (expected)
Review the diff artifacts to confirm changes look correct
Update baselines: ./capture-baselines.sh
Commit new baselines
Visual test passes

Some teams automate baseline updates with a bot comment — reply "update baselines" on the PR, and a workflow captures fresh screenshots and commits them.

Threshold Tuning

A 0% threshold catches everything, including subpixel rendering differences between environments. Too noisy.

A 5% threshold misses subtle changes — a shifted button, a truncated label. Too loose.

Start at 1-2% and adjust based on your false positive rate. Different pages might need different thresholds — a text-heavy docs page is more stable than a marketing page with animations.

Cost and Performance

Each screenshot API call takes 2-5 seconds. Testing 10 pages adds under a minute to your CI pipeline. The cost is minimal compared to the engineering time spent debugging visual bugs that shipped to production.

The tricky part isn't the technology — it's the discipline of maintaining baselines and reviewing diffs. Treat visual tests like any other test: if they fail, investigate before merging.

Using ScreenshotRun for the capture step keeps the infrastructure simple — no need to maintain your own headless browser pool in CI. Send a URL, get a screenshot, compare locally.

How to Build a Link Preview Thumbnail Service in Node.js

Vitalii Holben — Tue, 25 Aug 2026 12:42:03 +0000

Ever built a link aggregator, bookmarking tool, or dashboard that shows URLs? At some point you need thumbnails for those links. Users paste a URL, your app should show a preview — like Slack, Twitter cards, or Notion bookmarks.

You could self-host a browser instance and manage Puppeteer yourself. I did that for about a year. Scaling it was painful, and the memory leaks were real.

Here's how I rebuilt it with a screenshot API instead.

What we're building

A simple Express service that:

Accepts a URL
Returns a thumbnail image (cached)
Handles errors gracefully

The whole thing is under 80 lines.

Setup

mkdir link-thumbs && cd link-thumbs
npm init -y
npm install express node-cache axios sharp

The service

const express = require("express");
const NodeCache = require("node-cache");
const axios = require("axios");
const sharp = require("sharp");
const crypto = require("crypto");

const app = express();
const cache = new NodeCache({ stdTTL: 86400 }); // 24h cache

const API_KEY = process.env.SCREENSHOT_API_KEY;
const API_URL = "https://screenshotrun.com/api/screenshot";

async function getThumbnail(url, width = 1280, thumbWidth = 400) {
  const cacheKey = crypto
    .createHash("md5")
    .update(url + "-" + width + "-" + thumbWidth)
    .digest("hex");

  const cached = cache.get(cacheKey);
  if (cached) return cached;

  const response = await axios.get(API_URL, {
    params: {
      url,
      width,
      format: "png",
      full_page: false,
    },
    headers: { Authorization: "Bearer " + API_KEY },
    responseType: "arraybuffer",
    timeout: 30000,
  });

  // resize to thumbnail
  const thumb = await sharp(response.data)
    .resize(thumbWidth, null, { fit: "inside" })
    .webp({ quality: 80 })
    .toBuffer();

  cache.set(cacheKey, thumb);
  return thumb;
}

app.get("/thumb", async (req, res) => {
  const { url, w } = req.query;

  if (!url) {
    return res.status(400).json({ error: "url parameter required" });
  }

  // basic URL validation
  try {
    new URL(url);
  } catch {
    return res.status(400).json({ error: "invalid url" });
  }

  try {
    const thumb = await getThumbnail(url, 1280, parseInt(w) || 400);
    res.set("Content-Type", "image/webp");
    res.set("Cache-Control", "public, max-age=86400");
    res.send(thumb);
  } catch (err) {
    console.error("Failed: " + url, err.message);
    res.status(502).json({ error: "screenshot failed" });
  }
});

app.listen(3000, () => console.log("Thumbnail service on :3000"));

Using it

# get a thumbnail
curl "http://localhost:3000/thumb?url=https://github.com" -o github.webp

# custom width
curl "http://localhost:3000/thumb?url=https://dev.to&w=600" -o devto.webp

In your frontend, just use it as an image source:

<img 
  src="/thumb?url=https://example.com" 
  alt="example.com preview"
  loading="lazy"
/>

Why not self-host Puppeteer

I ran a self-hosted setup for a year. Here's what I dealt with:

Memory. Each Chromium instance eats 200-400MB. With 10 concurrent requests, you're looking at 4GB just for the browser processes. My 8GB droplet would OOM about once a week.

Zombie processes. Browsers crash. Tabs hang. Pages with infinite scroll or heavy JavaScript would occasionally lock up a tab. I wrote a watchdog script to kill stuck processes, which itself had bugs.

Font rendering. Missing fonts on Linux servers produce garbage screenshots. I installed a font pack, but CJK sites still looked wrong. Kept finding edge cases for months.

Timeouts. Some pages just take forever. A 30-second timeout seems generous until you hit a page that loads 47 third-party scripts and triggers a cookie consent modal that blocks rendering.

A screenshot API handles all of this. Browser pool management, font libraries, timeout handling, retries — someone else's problem. My service just makes HTTP calls and resizes images.

Improvements worth adding

Rate limiting. Without it, someone will feed your service 10,000 URLs and you'll burn through API credits. I use express-rate-limit — 10 requests per minute per IP is reasonable for most use cases.

const rateLimit = require("express-rate-limit");
app.use("/thumb", rateLimit({ windowMs: 60000, max: 10 }));

Persistent cache. NodeCache is in-memory — restarts kill it. For production, swap to Redis or just write thumbnails to disk with the URL hash as filename. Disk cache is underrated for images.

Fallback image. When a screenshot fails (site is down, blocked by firewall, etc.), return a generic placeholder instead of an error. Your UI shouldn't break because one link preview failed.

const FALLBACK = await sharp({
  create: { width: 400, height: 300, channels: 4, background: "#f0f0f0" }
}).webp().toBuffer();

Queue for bulk requests. If you're generating thumbnails for an import of 500 bookmarks, don't fire 500 concurrent API calls. Use a simple queue — bull or even just a promise pool with concurrency of 5.

What about Open Graph images?

OG images are great when they exist. But in practice:

~30% of URLs have no OG image at all
Many OG images are generic company logos, not page-specific
Some are broken CDN links
Quality varies wildly — some are 100x100 pixels

I use OG images as a first try, falling back to a screenshot when OG is missing or low quality. Best of both worlds — fast when OG exists, accurate when it doesn't.

Production numbers

Running this for a bookmarking app with ~2K daily active users:

Average thumbnail generation: 2-3 seconds (cold),

Generating PDFs From Web Pages Without a PDF Library

Vitalii Holben — Wed, 12 Aug 2026 13:11:02 +0000

Most PDF generation tutorials start with "install this PDF library" and then you spend an hour fighting with fonts, CSS support, and layout quirks. There's a simpler path if your content already lives on a web page.

The idea

Instead of rendering HTML to PDF inside your app, render the page in a real browser and print it to PDF. Chrome's DevTools Protocol has Page.printToPDF — same engine that powers "Save as PDF" in your browser, but automated.

Quick setup with Puppeteer

const puppeteer = require('puppeteer');

async function pageToPdf(url, outputPath) {
  const browser = await puppeteer.launch({ headless: 'new' });
  const page = await browser.newPage();

  await page.goto(url, { waitUntil: 'networkidle0' });

  await page.pdf({
    path: outputPath,
    format: 'A4',
    margin: { top: '20mm', bottom: '20mm', left: '15mm', right: '15mm' },
    printBackground: true
  });

  await browser.close();
}

pageToPdf('https://example.com/invoice/123', 'invoice.pdf');

That's it. Full CSS support, web fonts, flexbox, grid — everything renders exactly as it does in Chrome.

Things that tripped me up

Print stylesheets matter. If the page has @media print rules, they'll apply. This is actually useful — hide navigation, sidebars, cookie banners. But if you're not expecting it, your PDF might look nothing like the page.

@media print {
  nav, footer, .cookie-banner { display: none; }
  .content { width: 100%; margin: 0; }
}

waitUntil: 'networkidle0' isn't always enough. Pages with lazy-loaded images or JS-rendered charts might need explicit waits:

await page.waitForSelector('.chart-container canvas', { timeout: 5000 })
  .catch(() => console.log('chart not found, proceeding anyway'));

I learned this the hard way — generated 200 invoices with blank chart sections before noticing.

Headers and footers. The headerTemplate and footerTemplate options exist but they're fiddly. They use a tiny isolated context with limited CSS. For anything complex, I just bake the header into the page HTML itself.

When this approach makes sense

Invoices, reports, receipts — anything you already display on a web page
Generating PDFs from user-created content (blog posts, documentation)
Any situation where you want the PDF to match the web version pixel-for-pixel

Where it doesn't: high-volume generation (thousands per minute) where the browser overhead adds up, or cases where you need PDF-specific features like form fields or digital signatures.

Running it without managing Chrome

If you don't want to deal with keeping a Chrome instance running on your server — especially in containerized environments where Chromium dependencies are a pain — screenshot APIs like ScreenshotRun handle the browser part. You send a URL, get back a PDF or PNG. Offloads the rendering infrastructure entirely.

One more thing

The page.pdf() method has a scale option. Default is 1. Setting it to 0.8 or 0.9 can help fit content that's slightly too wide for the page. Beats reworking your CSS for print.

await page.pdf({
  path: 'output.pdf',
  format: 'A4',
  scale: 0.85,
  printBackground: true
});

Not something I see mentioned often but it's saved me from the "content gets cut off on the right edge" problem more than once.

How to Screenshot Dynamic Content That Changes on Every Page Load

Vitalii Holben — Mon, 03 Aug 2026 19:47:31 +0000

Some pages look different every time you load them. Randomized hero banners, A/B test variants, personalized product recommendations, rotating testimonials. If you're automating screenshots for testing or documentation, this is a problem.

You capture a baseline. You capture again tomorrow. The diff lights up red — not because anything broke, but because the carousel rotated and a different testimonial is showing.

The two types of dynamic content

There's server-side dynamic (different HTML on each request) and client-side dynamic (same HTML, JavaScript shuffles things around after load). The fix is different for each.

Server-side: you need to control the request. Pass a seed parameter, set a cookie, or hit a specific URL variant. If the page personalizes based on geo-IP, you need to pin your request to a consistent location.

Client-side: you need to run JavaScript before or after the page renders to freeze the randomization.

Freezing Math.random

A lot of front-end randomization uses Math.random(). Override it before the page loads:

await page.evaluateOnNewDocument(() => {
  let seed = 42;
  Math.random = () => {
    seed = (seed * 16807) % 2147483647;
    return (seed - 1) / 2147483646;
  };
});

Now every "random" value is deterministic. The carousel always shows the same slide. The A/B test always picks the same variant. Your screenshots become comparable.

Handling date/time displays

Pages that show "Posted 3 hours ago" or "Last updated: August 3, 2026" will always diff because the timestamp changes. Two options:

Option 1: Freeze the clock

await page.evaluateOnNewDocument(() => {
  const fixed = new Date('2026-01-15T12:00:00Z');
  const OrigDate = Date;
  Date = class extends OrigDate {
    constructor(...args) {
      if (args.length === 0) return super(fixed);
      return super(...args);
    }
    static now() { return fixed.getTime(); }
  };
});

Option 2: Mask the regions

If freezing the clock breaks page logic (timers, session management), mask the date regions in your diff comparison instead. Mark those areas as "ignore" in your visual diff tool.

I prefer option 1 for screenshot capture and option 2 for visual regression testing. Depends on whether you need the screenshot to look realistic or just be consistent.

CSS animations and transitions

An animated element captured mid-transition looks different every time. Kill all animations before capture:

await page.addStyleTag({
  content: `
    *, *::before, *::after {
      animation-duration: 0s !important;
      animation-delay: 0s !important;
      transition-duration: 0s !important;
      transition-delay: 0s !important;
    }
  `
});

This one saved me a ton of debugging. Had a landing page where a fade-in animation on the hero image made every screenshot slightly different in opacity. Took me a while to figure out it wasn't a rendering bug.

Lazy-loaded images and intersection observers

Content that loads on scroll won't appear in a screenshot if the viewport never scrolls to it. For full-page screenshots, scroll the page programmatically first:

await page.evaluate(async () => {
  await new Promise((resolve) => {
    let totalHeight = 0;
    const distance = 300;
    const timer = setInterval(() => {
      window.scrollBy(0, distance);
      totalHeight += distance;
      if (totalHeight >= document.body.scrollHeight) {
        clearInterval(timer);
        window.scrollTo(0, 0);
        resolve();
      }
    }, 100);
  });
});

Scroll to the bottom, trigger all the lazy loads, scroll back to top, then capture. Not elegant but it works reliably.

When you can't control the content

Sometimes you're screenshotting third-party pages you don't own. You can't inject seeds or override Math.random. In that case, accept that the screenshots will vary and adjust your comparison threshold.

Instead of pixel-perfect diffs, use structural comparison. Did the layout change? Did elements move? Is text content different? A 2-3% pixel difference from a rotated banner is noise. A 15% difference from a missing navigation bar is a real issue.

The goal isn't identical screenshots — it's catching meaningful changes while ignoring expected variation.

Building a Visual Changelog for Your Web App

Vitalii Holben — Sun, 26 Jul 2026 22:28:15 +0000

You ship a frontend change, merge the PR, deploy to production. Two days later your PM asks "what exactly changed on the pricing page last Tuesday?" and you're digging through git logs trying to reconstruct what the page looked like before and after.

Git tracks code. It doesn't track what the rendered page actually looked like to users.

A visual changelog fixes that. Capture a screenshot of key pages after every deploy, store them with timestamps, and you've got a browsable history of how your app looked at any point. Here's how to set one up.

The basic idea

After each deploy, a script hits a list of URLs and saves screenshots with metadata:

URL
Timestamp
Git commit hash
Deploy environment

You end up with a folder (or database) of timestamped screenshots you can scroll through. "What did the homepage look like three deploys ago?" becomes a five-second lookup instead of a twenty-minute archaeology project.

A minimal implementation

Here's a Node.js script that captures screenshots of a list of pages using Playwright:

const { chromium } = require('playwright');
const fs = require('fs');
const path = require('path');

const PAGES = [
  { name: 'homepage', url: 'https://yourapp.com' },
  { name: 'pricing', url: 'https://yourapp.com/pricing' },
  { name: 'dashboard', url: 'https://yourapp.com/dashboard' },
];

const COMMIT = process.env.GIT_COMMIT || 'unknown';
const TIMESTAMP = new Date().toISOString().replace(/[:.]/g, '-');
const OUTPUT_DIR = path.join(__dirname, 'changelog', TIMESTAMP);

async function capture() {
  fs.mkdirSync(OUTPUT_DIR, { recursive: true });

  const browser = await chromium.launch();
  const context = await browser.newContext({
    viewport: { width: 1440, height: 900 },
  });

  for (const page of PAGES) {
    const tab = await context.newPage();
    await tab.goto(page.url, { waitUntil: 'networkidle' });
    await tab.screenshot({
      path: path.join(OUTPUT_DIR, `${page.name}.png`),
      fullPage: true,
    });
    console.log(`captured ${page.name}`);
  }

  // save metadata
  fs.writeFileSync(
    path.join(OUTPUT_DIR, 'meta.json'),
    JSON.stringify({ commit: COMMIT, timestamp: TIMESTAMP, pages: PAGES.map(p => p.name) })
  );

  await browser.close();
}

capture().catch(console.error);

Run this as a post-deploy hook. In a GitHub Actions workflow:

- name: Capture visual changelog
  run: node capture-changelog.js
  env:
    GIT_COMMIT: ${{ github.sha }}

The problem with running your own browser

This works. I used a setup like this for about four months. Then I hit the issues:

Memory. Chromium eats 300-500MB per instance. On a CI runner with 4GB RAM, capturing 15 pages sequentially takes a while. Doing it in parallel kills the runner.

Font rendering. Screenshots taken on Ubuntu CI runners look different from macOS or Windows. If you're comparing screenshots across deploys and your CI runner image updated, every single screenshot shows a "diff" that's really just font antialiasing.

Flaky waits. doesn't mean "the page looks right." SPAs that fetch data on mount will show loading spinners in half your captures. You end up writing per-page wait logic and maintaining it forever.

After dealing with all three I switched to hitting a screenshot API (I use ScreenshotRun for this) instead of running Playwright locally. Same script structure, but instead of launching a browser you make an HTTP call:

const fetch = require('node-fetch');
const fs = require('fs');

async function captureViaApi(url, outputPath) {
  const response = await fetch(
    `https://api.screenshotrun.com/capture?` +
    `url=${encodeURIComponent(url)}&width=1440&height=900&full_page=true`,
    { headers: { 'Authorization': 'Bearer YOUR_API_KEY' } }
  );
  const buffer = await response.buffer();
  fs.writeFileSync(outputPath, buffer);
}

No browser to manage, consistent rendering environment, and it handles the wait logic for you. Tradeoff is you're paying per screenshot, but for a changelog that captures 10-20 pages per deploy it's negligible.

Making the changelog browsable

A folder of PNGs isn't great for anyone except the person who set it up. Build a simple viewer.

The laziest version that actually works: generate an HTML file that lists screenshots grouped by deploy.

function generateIndex(changelogDir) {
  const deploys = fs.readdirSync(changelogDir)
    .filter(d => fs.existsSync(path.join(changelogDir, d, 'meta.json')))
    .sort()
    .reverse();

  let html = '';

  for (const deploy of deploys) {
    const meta = JSON.parse(
      fs.readFileSync(path.join(changelogDir, deploy, 'meta.json'))
    );
    html += `<h2>${deploy} — ${meta.commit.slice(0, 8)}</h2>`;

    for (const page of meta.pages) {
      html += `
        <h3>${page}</h3>
        <img src="${deploy}/${page}.png" style="max-width: 100%; border: 1px solid #ccc;" />
      `;
    }
  }

  html += '';
  fs.writeFileSync(path.join(changelogDir, 'index.html'), html);
}

Deploy this to a static hosting bucket or serve it from an internal tool. PMs and designers can now browse the visual history without asking engineering.

Side-by-side diffing

The real power comes when you compare consecutive deploys. For each page, show the previous and current screenshot next to each other.

You can get fancier with pixel-level diffing using a library like :

const pixelmatch = require('pixelmatch');
const { PNG } = require('pngjs');

function diffScreenshots(beforePath, afterPath, diffPath) {
  const before = PNG.sync.read(fs.readFileSync(beforePath));
  const after = PNG.sync.read(fs.readFileSync(afterPath));
  const diff = new PNG({ width: before.width, height: before.height });

  const mismatchedPixels = pixelmatch(
    before.data, after.data, diff.data,
    before.width, before.height,
    { threshold: 0.1 }
  );

  fs.writeFileSync(diffPath, PNG.sync.write(diff));
  return mismatchedPixels;
}

If is above zero, something changed visually. You can even fail the deploy or send a Slack notification when unexpected visual changes are detected.

What pages to track

Don't capture everything. Start with pages that matter most:

Landing page and pricing — these change often and have business impact
Auth flows — login, signup, password reset. Regressions here directly cost you users
Core product screens — the 3-5 pages users spend the most time on
Email templates — if you render them as HTML, screenshot those too

I started with 6 pages and now track about 20. Add pages when something breaks that you wish you'd been tracking. Don't over-engineer it upfront.

What I'd skip

Pages with lots of dynamic content (dashboards with live data, feeds with user-generated content). The screenshots will always look "different" and you'll stop paying attention.
Admin panels. Unless you've got non-technical admins who report UI bugs, probably not worth it.
Every single marketing page. Track the ones that convert, not the ones that exist.

The whole setup takes maybe an afternoon. The first time someone asks "when did the pricing page change" and you pull it up in ten seconds, it pays for itself.

How to Know When Your Cron Job Silently Stops Running

Vitalii Holben — Wed, 22 Jul 2026 14:50:00 +0000

Had a backup script running every night at 2 AM for about six months. Worked great. Then one day I needed to restore something and discovered the backups stopped three weeks ago. The cron entry was still there. No errors in syslog. The script just... wasn't running.

Turned out a server update changed the PATH and the script couldn't find pg_dump anymore. It failed silently because I was redirecting stderr to /dev/null like a genius.

The core problem

Cron doesn't care if your job succeeds. It cares if it runs. And even "runs" is generous — if cron fires the command and it exits immediately with an error, cron considers that a success.

Most setups have zero monitoring on whether scheduled jobs actually complete. You find out they broke when the thing they were supposed to do didn't happen, which could be days or weeks later.

Dead man's switch — the simplest fix

The idea is backwards from normal monitoring. Instead of checking if something fails, you check if it doesn't report success.

At the end of your cron script, ping a URL:

#!/bin/bash
pg_dump mydb > /backups/mydb_$(date +%F).sql

if [ $? -eq 0 ]; then
    curl -fsS --retry 3 https://your-monitoring/ping/abc123 > /dev/null
fi

If the monitoring service doesn't receive that ping within the expected window (say, 30 minutes after 2 AM), it alerts you.

That's it. No agent to install, no daemon running, no config files. One curl at the end of your script.

What trips people up

Forgetting to check the exit code. If you just put the curl at the end without the if check, it pings even when the job fails. Defeats the entire purpose.

Timeout too tight. Your job usually takes 5 minutes but sometimes takes 20. If your monitoring window is 10 minutes, you'll get false alerts. Set it to 2x your worst case.

Multiple schedules. A job that runs every hour needs a different timeout than one that runs daily. I've seen people set up monitoring for a daily job and use a 1-hour timeout — so it alerts 23 times before the next run.

Swallowed output. Stop doing 2>/dev/null on cron jobs. Send stderr somewhere useful:

*/5 * * * * /path/to/script.sh >> /var/log/myjob.log 2>&1

DIY vs hosted

You can build this yourself. A small endpoint that logs pings, a checker that runs every minute and compares last-ping time against expected schedule, and a notification sender. Maybe 200 lines of code.

I did that once. It worked until the monitoring server itself went down and nobody noticed because nothing was monitoring the monitor. Classic.

Hosted services handle the reliability part — redundant infrastructure, multiple notification channels, escalation if you don't acknowledge. Worth it if you have more than a handful of jobs to track.

The minimum I'd set up for any project

Every cron job pings a URL on successful completion
Every ping has a timeout matched to the job's schedule
Failed pings send notifications to Slack or email, not just a dashboard
Cron output goes to a log file, not /dev/null

Takes maybe 15 minutes per job. Saves you from the "wait, when did this stop working?" conversation that nobody enjoys.