Screenshot Every Page of a Website from Its Sitemap with One Node.js Script

You're about to relaunch a site and someone asks for a screenshot of every page as it stands today. Or a client wants to sign off 140 pages of copy, or legal wants a dated record of what the pricing page said. Doing it by hand means opening each URL, waiting for the lazy images, closing the cookie banner and saving the file under a name you'll still understand next month. Two minutes a page adds up to most of a working day for a 200-page site.

So I wrote one Node.js script that reads the sitemap, captures every page at desktop and mobile widths in real Chrome, and leaves a folder of named PNGs, a CSV manifest and a contact sheet. Last run: 38 full-page captures in 35 seconds, no failures.

Disclosure: I run HTML to Image, and the script uses its Screenshot API for the captures. This is a cross-post of the full guide on the HTML to Image blog, which is the canonical version and goes further into webhooks, costs and compliance archives.

Prerequisites

• Node.js 18 or later, for the built-in fetch.
• An HTML to Image API key. The free credits are enough to try this on a handful of pages.
• A site with a sitemap, or a text file of URLs from a crawl.

npm install @html2img/client
export HTML2IMG_API_KEY=your_api_key

Step 1: Find the sitemap

Most sites declare their sitemap in robots.txt, so check there first:

curl -s https://example.com/robots.txt | grep -i '^sitemap:'

If nothing comes back, try /sitemap.xml, then /sitemap_index.xml (Yoast and Rank Math on WordPress), then /wp-sitemap.xml (WordPress core since 5.5). Big sites use a sitemap index that points at child sitemaps, some of them gzipped. The script follows all of that, so you only ever pass the top-level URL.

One caveat: a sitemap is the list the CMS chooses to publish. Orphaned landing pages and anything marked noindex won't be in it. For a migration archive, cross-check against a crawl and feed the script a text file instead.

Step 2: The script

Save this as capture-site.mjs. It's long-ish because it handles the boring parts properly: sitemap indexes, .gz files, duplicates, a credit check before spending anything, four captures at a time, retries for errors that can recover, and the reports at the end.

// capture-site.mjs
// node capture-site.mjs https://example.com/sitemap.xml [--dry-run]
// node capture-site.mjs urls.txt
import { mkdir, readFile, writeFile } from 'node:fs/promises';
import { gunzipSync } from 'node:zlib';
import {
  Html2img,
  InsufficientCreditsError,
  TimeoutError,
  ServerError,
  ConnectionError,
} from '@html2img/client';

const SOURCE = process.argv[2];
const DRY_RUN = process.argv.includes('--dry-run');
const OUT = 'shots';
const CONCURRENCY = 4;
const MAX_ATTEMPTS = 3;

// One credit per page per variant.
const VARIANTS = [
  { name: 'desktop', width: 1440 },
  { name: 'mobile', width: 390 },
  // { name: 'pdf', format: 'pdf' }, // A4 document with a real text layer
];

// Sent with every capture.
const CAPTURE = {
  fullpage: true,
  css: `#onetrust-banner-sdk, .onetrust-pc-dark-filter, #CybotCookiebotDialog,
        .qc-cmp2-container, .osano-cm-window, #cookie-law-info-bar,
        .intercom-frame, #crisp-chatbox { display: none !important; }`,
};

const SKIP = /\.(pdf|jpe?g|png|gif|webp|svg|zip|mp4|xml)$/i;
const apiKey = process.env.HTML2IMG_API_KEY;
const client = DRY_RUN ? null : new Html2img(apiKey);

// 1. Every <loc> in a sitemap, following sitemap indexes and .gz files.
async function readSitemap(url, seen = new Set()) {
  if (seen.has(url)) return [];
  seen.add(url);
  const res = await fetch(url);
  if (!res.ok) throw new Error(`${url} returned ${res.status}`);
  let body = Buffer.from(await res.arrayBuffer());
  if (body[0] === 0x1f && body[1] === 0x8b) body = gunzipSync(body);
  const xml = body.toString('utf8');
  const locs = [...xml.matchAll(/<loc>\s*(?:<!\[CDATA\[)?(.*?)(?:\]\]>)?\s*<\/loc>/gs)]
    .map((m) => m[1].replace(/&(amp|apos|quot|lt|gt);/g, (_, e) =>
      ({ amp: '&', apos: "'", quot: '"', lt: '<', gt: '>' })[e]));
  if (!/<sitemapindex[\s>]/.test(xml)) return locs;
  const children = await Promise.all(locs.map((loc) => readSitemap(loc, seen)));
  return children.flat();
}

// 2. Same site only, no files, no duplicates.
const bare = (host) => host.replace(/^www\./, '');
function toPages(urls, onlyHost) {
  const pages = new Set();
  for (const raw of urls) {
    let u;
    try { u = new URL(raw.trim()); } catch { continue; } // header rows, blanks
    if (onlyHost && bare(u.hostname) !== bare(onlyHost)) continue;
    if (SKIP.test(u.pathname)) continue;
    u.hash = '';
    pages.add(u.href);
  }
  return [...pages];
}

// "/blog/launch-notes/" -> "blog__launch-notes"
let multiSite = false;
function fileName(page) {
  const { hostname, pathname, search } = new URL(page);
  const slug = (pathname + search)
    .replace(/^\/+|\/+$/g, '')
    .split('/')
    .map((part) => part.replace(/[^a-z0-9]+/gi, '-').replace(/^-|-$/g, ''))
    .join('__')
    .toLowerCase()
    .slice(0, 150) || 'home';
  return multiSite ? `${bare(hostname)}__${slug}` : slug;
}

async function creditsLeft() {
  const res = await fetch('https://app.html2img.com/api/me', {
    headers: { 'X-API-Key': apiKey },
  });
  if (!res.ok) throw new Error(`Key check failed with ${res.status}`);
  return (await res.json()).credits_remaining;
}

const retryable = (err) =>
  err instanceof TimeoutError ||
  err instanceof ServerError ||
  err instanceof ConnectionError ||
  err.errorCode === 'page_load_timeout';

// 3. One capture, retried on errors that are worth a second attempt.
async function capture({ page, variant }) {
  const { name, ...options } = variant;
  const ext = options.format === 'pdf' ? 'pdf' : 'png';
  const row = { page, variant: name, status: 'failed', file: '', image_url: '', error: '' };
  for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
    try {
      const shot = await client.screenshot({ url: page, ...CAPTURE, ...options });
      row.image_url = shot.url;
      const img = await fetch(shot.url);
      if (!img.ok) throw new Error(`download returned ${img.status}`);
      row.file = `${name}/${fileName(page)}.${ext}`;
      await writeFile(`${OUT}/${row.file}`, Buffer.from(await img.arrayBuffer()));
      return { ...row, status: 'ok', error: '' };
    } catch (err) {
      if (err instanceof InsufficientCreditsError) throw err;
      row.error = err.errorCode ?? err.message;
      if (!retryable(err) || attempt === MAX_ATTEMPTS) return row;
      await new Promise((resolve) => setTimeout(resolve, 3000 * attempt));
    }
  }
}

// 4. A fixed number of captures in flight at once.
async function runAll(jobs) {
  const rows = [];
  let next = 0;
  let stopped = false;
  async function lane() {
    while (!stopped && next < jobs.length) {
      const job = jobs[next++];
      try {
        const row = await capture(job);
        rows.push(row);
        console.log(`${rows.length}/${jobs.length} ${row.status} ${row.variant} ${job.page}`);
      } catch (err) {
        if (!stopped) console.error(`Stopping early: ${err.message}`);
        stopped = true;
      }
    }
  }
  await Promise.all(Array.from({ length: CONCURRENCY }, lane));
  return rows;
}

// 5. A manifest for spreadsheets and an index.html for people.
async function writeReports(rows) {
  const cols = ['page', 'variant', 'status', 'file', 'image_url', 'error'];
  const cell = (v) => `"${String(v).replaceAll('"', '""')}"`;
  const csv = [cols, ...rows.map((r) => cols.map((c) => r[c]))]
    .map((line) => line.map(cell).join(','))
    .join('\n');
  await writeFile(`${OUT}/manifest.csv`, csv);

  const esc = (s) => s.replace(/[&<>"]/g, (c) => `&#${c.charCodeAt(0)};`);
  const cards = rows
    .filter((r) => r.status === 'ok' && r.file.endsWith('.png'))
    .map((r) => `<a href="${esc(r.file)}"><img src="${esc(r.file)}" loading="lazy"><span>${r.variant} · ${esc(r.page)}</span></a>`)
    .join('\n');
  await writeFile(`${OUT}/index.html`, `<!doctype html><meta charset="utf-8"><title>Site capture</title>
<style>body{font:14px system-ui;margin:24px;display:grid;grid-template-columns:repeat(auto-fill,minmax(240px,1fr));gap:20px}
a{color:#334155;text-decoration:none}img{width:100%;height:320px;object-fit:cover;object-position:top;border:1px solid #e2e6ec;border-radius:8px}
span{display:block;margin-top:6px;word-break:break-all}</style>\n${cards}`);
}

const fromSitemap = /^https?:\/\//.test(SOURCE);
const urls = fromSitemap
  ? await readSitemap(SOURCE)
  : (await readFile(SOURCE, 'utf8')).split(/\r?\n/);
const pages = toPages(urls, fromSitemap ? new URL(SOURCE).hostname : null);
multiSite = new Set(pages.map((page) => bare(new URL(page).hostname))).size > 1;

if (DRY_RUN) {
  console.log(pages.join('\n'));
  process.exit(0);
}

const jobs = pages.flatMap((page) => VARIANTS.map((variant) => ({ page, variant })));
const credits = await creditsLeft();
console.log(`${pages.length} pages x ${VARIANTS.length} variants = ${jobs.length} credits. Balance: ${credits}.`);
if (credits < jobs.length) {
  console.error('Not enough credits for the whole run. Trim the list or top up first.');
  process.exit(1);
}

for (const { name } of VARIANTS) await mkdir(`${OUT}/${name}`, { recursive: true });
const rows = await runAll(jobs);
await writeReports(rows);

const failed = rows.filter((r) => r.status !== 'ok').length;
const skipped = jobs.length - rows.length;
console.log(`Saved ${rows.length - failed}, failed ${failed}, skipped ${skipped}. Open ${OUT}/index.html`);

The css string is doing quiet but important work. The renderer arrives as a first-time visitor with no cookies, so every page shows its consent banner. Injecting display: none !important for the common consent providers and chat widgets takes them out before the shot. If a site uses something else, find the banner's ID in dev tools and add it.

Step 3: Dry run first, then capture

--dry-run prints the URLs the script would capture. It doesn't need a key and spends nothing, so use it to filter out tag archives, paginated listings and anything else you don't need a picture of:

node capture-site.mjs https://html2img.com/sitemap.xml --dry-run | grep '/tools/' > urls.txt
node capture-site.mjs urls.txt

That's the real run I mentioned at the top, against the free tools section of my own site:

19 pages x 2 variants = 38 credits. Balance: 100.
1/38 ok desktop https://html2img.com/docs/mcp/tools/
2/38 ok mobile https://html2img.com/docs/mcp/tools/
3/38 ok desktop https://html2img.com/tools/
4/38 ok mobile https://html2img.com/tools/
...
38/38 ok desktop https://html2img.com/tools/youtube-thumbnail/
Saved 38, failed 0, skipped 0. Open shots/index.html

Spot the first line. My lazy grep also matched /docs/mcp/tools/, which I never meant to include. That's exactly why the dry run exists: grep '^https://html2img.com/tools/' would have kept it out.

The desktop captures came back 1,440 pixels wide and between 3,737 and 10,016 pixels tall. Here's the index.html the run produced, each card cropped to the top of its page:

What the script does when things go wrong

Not every page cooperates, so the retry logic only retries what a second attempt can fix:

• 504 timeouts, 500s, slow-loading sites and dropped connections get two more attempts, 3 then 6 seconds apart.
• Dead links, SSL failures and sites that block automated visitors (url_not_found, url_unreachable, connection_refused) are logged in manifest.csv and skipped.
• Running out of credits stops every lane, still writes the reports, and tells you how many captures were skipped.

To re-run only the failures, pull them out of the manifest and run them in a fresh folder so the first run's reports stay intact:

awk -F'","' '$3 == "failed" { print substr($1, 2) }' shots/manifest.csv | sort -u > retry.txt
mkdir retry && cd retry && node ../capture-site.mjs ../retry.txt

Keep CONCURRENCY low for sites you don't own. Every capture is a real page load against somebody's production server.

A few limits worth planning around

• Public pages only. The renderer has no cookies or headers, so anything behind a login captures as the sign-in page.
• Infinite scroll has no bottom. Cap the feed with injected CSS such as main.feed { max-height: 12000px; overflow: hidden; }.
• Closed accordions stay closed. Nobody clicks during a capture, so open hidden panels with CSS if they matter.
• One credit per page per width. Desktop plus mobile doubles the cost, and the PDF variant adds a third.

Where I'd use this

Before a migration, so you have the old site on disk the day the DNS switches. For client sign-off, where one contact sheet beats 200 links. And for compliance, where the commented-out pdf variant gives you searchable A4 documents in a dated folder.

The full guide on the HTML to Image blog also covers handing long captures to a webhook instead of waiting on them, a bash version for short URL lists, and what a full-site capture costs at different sizes.

How are you archiving sites before a relaunch at the moment? Browser extension, Screaming Frog, something homegrown? Let me know in the comments.