Your crawler sees an empty div. Crawling a JavaScript SPA with Playwright

#webdev #node #playwright #javascript

TL;DR: On a client-rendered SPA, the HTML the server sends is a script tag and an empty <div id="root">. An HTTP crawler parses that, finds no links and stops. Load each page in a headless browser (Playwright or Puppeteer), wait for it to render, read a[href] from the live DOM and queue what you find. If you want the data rather than the pages, the JSON API the app calls is usually the better thing to crawl.

The empty shell

I pointed a plain crawler at a React dashboard and got exactly one page back. Here's why:

$ curl -s https://app.example.com/
<!doctype html>
<html>
  <head><script type="module" src="/assets/index-4f2a.js"></script></head>
  <body><div id="root"></div></body>
</html>

Every link on that site exists only after the bundle runs, hits the API and renders. wget --mirror, Scrapy or fetch plus cheerio all see the shell above. You need something that executes JavaScript, which means a browser engine you can drive from code.

The crawler

Setup:

npm i playwright robots-parser
npx playwright install chromium

crawl.mjs:

import { chromium } from "playwright"
import robotsParser from "robots-parser"

const start = new URL(process.argv[2])
const MAX_PAGES = 50
const HASH_ROUTES = false // true for apps with URLs like /#/settings
const SKIP = /logout|sign-?out|delete|remove/i

const robotsUrl = new URL("/robots.txt", start).href
const robotsTxt = await fetch(robotsUrl).then((r) => (r.ok ? r.text() : ""), () => "")
const robots = robotsParser(robotsUrl, robotsTxt)
const DELAY_MS = Math.max(1000, (robots.getCrawlDelay("*") ?? 0) * 1000)

const key = (href) => {
  const url = new URL(href)
  if (!(HASH_ROUTES && url.hash.startsWith("#/"))) url.hash = ""
  return url.href
}

const browser = await chromium.launch()
const page = await browser.newPage()
const apiCalls = new Set()
page.on("request", (req) => {
  if (["fetch", "xhr"].includes(req.resourceType())) apiCalls.add(req.method() + " " + req.url())
})

const queue = [key(start.href)]
const seen = new Set(queue)
const pages = []

while (queue.length > 0 && pages.length < MAX_PAGES) {
  const url = queue.shift()
  if (robots.isAllowed(url, "*") === false) continue
  try {
    await page.goto(url, { waitUntil: "domcontentloaded", timeout: 30_000 })
    // Wait for the app's own API calls to settle, but don't hang on polling.
    await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {})
  } catch (error) {
    console.error("failed " + url + ": " + error.message)
    continue
  }

  const hrefs = await page.$$eval("a[href]", (links) => links.map((a) => a.href))
  pages.push({ url, finalUrl: page.url(), title: await page.title(), links: hrefs.length })

  for (const href of hrefs) {
    if (!href.startsWith("http")) continue
    const next = key(href)
    if (new URL(next).origin !== start.origin || SKIP.test(next) || seen.has(next)) continue
    seen.add(next)
    queue.push(next)
  }
  await page.waitForTimeout(DELAY_MS)
}

await browser.close()
console.log(JSON.stringify({ pages, apiCalls: [...apiCalls] }, null, 2))

Run it with node crawl.mjs https://your-site/. I tested it against a small pushState SPA whose links only appear 300 ms after load. It found every route, skipped the Disallowed one and stayed away from /logout.

What each part is doing:

BFS with a seen set, capped at MAX_PAGES, so it can't run away on a big site.
networkidle with a timeout. It resolves after 500 ms with no network connections. The Playwright docs discourage it for tests, and they're right, but for a crawler it's a decent "the app has fetched and rendered" signal. The timeout covers apps that poll forever.
a.href instead of getAttribute("href"), because the browser hands you the resolved absolute URL.
robots.txt and Crawl-delay via robots-parser, plus a one-second floor between pages.
The apiCalls set. This one turns out to be the most useful output (more on that below).

Using Puppeteer? Same loop. Use waitUntil: "networkidle2" on goto and the same $$eval.

Things that will bite you

Hash routing. With /#/settings-style URLs the fragment never reaches the server and most crawlers strip it, so every route looks like one page. Flip HASH_ROUTES to keep #/ fragments.
Buttons pretending to be links. onClick={() => router.push("/reports")} on a <div> has no href, so a link crawler never finds it. Click candidates and check page.url() afterwards, or read /sitemap.xml, or find the route config in the source.
Infinite scroll. Loop await page.mouse.wheel(0, 5000) plus a short wait until the page height stops changing.
Login. Sign in once, await context.storageState({ path: "auth.json" }), then browser.newContext({ storageState: "auth.json" }) for every crawl. And keep the logout skip, because one stray logout click and you're crawling the login page 50 times.
Query string explosions. Sort and filter params can multiply one list into thousands of URLs. Strip what you don't need in key().

Does Google have the same problem?

Less than it used to. Per Google's JavaScript SEO basics, Googlebot crawls the HTML, queues the page for rendering in an evergreen headless Chromium, and then extracts links from the rendered HTML. Rendering can lag behind crawling, it only follows <a> elements with an href, and Google tells you not to use # fragments to load different content. Other bots may not run JavaScript at all, which is why SSR or pre-rendering is still the advice.

Often you want the API, not the HTML

An SPA builds its pages from JSON. If you're after the data, GET /api/projects?page=2 is more stable than whatever CSS classes this quarter's redesign uses. Watch the Network panel with the Fetch/XHR filter on while you use the app, or just read the apiCalls the crawler logged.

That's also why I ended up using Netraptor (I work on it) for my own apps. Its crawler runs in my real Chrome tab, so the SPA renders with my logged-in session and there's no headless browser to log in. It reads the rendered a[href] links from the start page and visits them one level deep on the same origin. It skips logout and delete links and records the API calls per page into an endpoint map. It doesn't click buttons, scroll or follow hash routes, so the Playwright script above still has a place.

Only crawl sites you own or are allowed to test.

The full guide, with an FAQ, is at netraptor.dev/guides/crawl-javascript-website.

DEV Community