How to Capture Full-Page Screenshots of Infinite Scroll Pages

#screenshotapi #webcapture #developertools #api

title: How to Capture Full-Page Screenshots of Infinite Scroll Pages tags: webdev, javascript, puppeteer, testing

Infinite scroll pages don't have a "bottom." That makes full-page screenshots tricky — the browser doesn't know how tall the page actually is until you've scrolled through everything.

Here's how I handle it, and where the common approaches break.

The Naive Approach (and Why It Fails)

Puppeteer's fullPage: true option works by measuring document.body.scrollHeight and setting the viewport to that height before capturing. On a normal page, this is fine.

On an infinite scroll page, scrollHeight only reflects content that's already loaded. If the page lazy-loads 50 more items when you reach the bottom... those items aren't in the DOM yet when the screenshot fires.

You get a screenshot of the first batch of content and nothing else.

Scroll, Wait, Repeat

The workaround is manual scrolling:

async function scrollToBottom(page) {
  let previousHeight = 0;
  let currentHeight = await page.evaluate(() => document.body.scrollHeight);
  
  while (previousHeight !== currentHeight) {
    previousHeight = currentHeight;
    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
    await page.waitForTimeout(1500);
    currentHeight = await page.evaluate(() => document.body.scrollHeight);
  }
}

Scroll to bottom, wait for new content, check if height changed. When it stops changing, you've hit the end. Then take your fullPage screenshot.

The waitForTimeout(1500) is a guess. Some pages load fast, some make API calls that take 3+ seconds. I've seen people use waitForNetworkIdle here, but that can hang forever on pages with analytics pings and websocket connections.

What I do: wait for timeout, then also check if new DOM elements appeared:

const itemCount = await page.evaluate(
  () => document.querySelectorAll('.feed-item').length
);
// scroll...
const newCount = await page.evaluate(
  () => document.querySelectorAll('.feed-item').length
);
if (newCount === itemCount) break;

More reliable than height checks for some layouts.

The Memory Problem

A page with 500 loaded items is big. Setting the viewport to 1920x50000 and capturing as PNG creates a massive image. Chrome might crash, or your server runs out of memory.

Two options:

Cap the scroll depth. Don't try to capture everything. Scroll N times and stop:

const MAX_SCROLLS = 20;
for (let i = 0; i < MAX_SCROLLS; i++) {
  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await page.waitForTimeout(1000);
}

Use JPEG instead of PNG. For long screenshots, JPEG at quality 80 is usually 5-10x smaller:

await page.screenshot({ 
  fullPage: true, 
  type: 'jpeg', 
  quality: 80 
});

Intersection Observer Gotchas

Modern sites use IntersectionObserver for lazy loading. The observer fires when elements enter the viewport. In headless Chrome, scrolling programmatically does trigger these observers — but only if you actually scroll through the viewport, not just jump to the bottom.

If you window.scrollTo(0, 99999) in one jump, observers for elements in the middle of the page might not fire. Scroll incrementally:

const scrollStep = 500;
let scrollPos = 0;
const maxHeight = await page.evaluate(() => document.body.scrollHeight);

while (scrollPos < maxHeight) {
  scrollPos += scrollStep;
  await page.evaluate((y) => window.scrollTo(0, y), scrollPos);
  await page.waitForTimeout(200);
}

Slower but catches all the lazy-loaded content.

When to Give Up

Some pages are genuinely infinite — social media feeds, search results that go on forever. Set a reasonable limit — either by scroll count, total height, or time spent — and document it. "We capture the first 50 items" is a perfectly valid approach.

DEV Community