DEV Community

Cover image for When you actually need a headless browser (and when you're just wasting RAM)
Andrii Votiakov
Andrii Votiakov

Posted on

When you actually need a headless browser (and when you're just wasting RAM)

#javascript #webdev #puppeteer #scraping

Most of the store-locator scrapers I build never touch a browser. I open the Network tab, find the JSON endpoint the map calls, hit it with fetch, done. A browser is slow, hungry, and flaky, so I treat it as the last resort.

National Book Tokens is where the last resort actually kicked in. It's a UK book gift-card brand, and their "find a bookshop" page shows a Google-Maps-style board of pins for every shop that takes the card. I wanted the full geocoded list. So I did the usual thing: opened the page, opened DevTools, watched the Network tab reload.

Nothing. No /api/shops, no locations.json, no XHR carrying an array of bookshops. The map just... had pins. That absence is the whole story, and it's the thing worth learning, because it's the signal that tells you a browser is unavoidable instead of just habit.

How to tell a browser is actually required

Here's the test I run before I let myself spin up Chrome. It takes about thirty seconds.

  1. Load the page normally in the browser. See the data on screen? Good.
  2. View the raw HTML. Not the Elements panel, the raw document. Ctrl+U (view-source), or in DevTools open the Network tab, click the document request, look at the Response sub-tab. This is the exact bytes the server sent, before any JavaScript ran.
  3. Ctrl+F in that raw response for something you can see on screen. A shop name, a postcode, a latitude.

If the data is in the raw HTML, you (almost) never need a browser. Server-rendered pages put everything in that first response, and a plain fetch plus an HTML parser gets you the lot. If the data is not in the raw response but is in the live Elements panel, that gap is your answer: the page arrived nearly empty and JavaScript filled it in after the fact. You cannot fetch your way to something that only exists after JS runs.

On the National Book Tokens page I searched the Elements panel for markers and found this:

<div id="markers" style="display:none">
  [[&quot;Foyles&quot;,&quot;51.5155&quot;,&quot;-0.1300&quot;,100,&quot;&lt;div&gt;...&lt;/div&gt;&quot;,...
</div>
Enter fullscreen mode Exit fullscreen mode

A hidden div stuffed with the entire dataset, as an HTML-entity-escaped JSON array. The site's own map code reads this div back, unescapes it, and drops the pins. Then I looked at that same #markers div in view-source, and it was empty. Just <div id="markers"></div>. Client-side JS builds the contents on load.

That's the convict. Data present in the DOM, absent from the raw HTML, sitting in a div that some script populates. No JSON endpoint to call directly. A browser it is.

The browser has exactly one job

This is the part people get wrong. They reach for Puppeteer and then start driving it like a robot user, clicking around, scrolling, taking screenshots, scraping rendered text off the page. That's slow and brittle.

The browser's only real job here is to run the page's JavaScript and let the DOM settle. That's it. The moment #markers is populated, everything after that is plain string work: read the innerHTML, unescape it, JSON.parse, map the array. String work runs fine anywhere and it's fast. So I keep the browser part as small as I possibly can and get out.

I also don't launch Chrome, I connect to one. Running the actual scraper inside a Firebase Function, launching full Chromium is a pain (cold starts, binary size, memory). So I run a headless Chrome somewhere else, a browserless-style service, and attach over a WebSocket with puppeteer-core. puppeteer-core is the same library without the bundled Chromium download, which is exactly what you want when the browser lives elsewhere.

Here's the whole thing.

import puppeteer from "puppeteer-core";

const PAGE_URL = "https://www.nationalbooktokens.com/redeem/find-a-bookshop";

// A remote headless Chrome (browserless-style). Locally you can just use
// full `puppeteer` and puppeteer.launch({ headless: "new" }) instead.
const BROWSER_WS =
  process.env.BROWSER_WS_ENDPOINT; // ws://your-chrome-host/browser/chrome?token=...

// Undo the HTML-entity escaping the site applied before it stored the JSON.
function unEscapeHtml(s) {
  return s
    .replace(/&lt;/g, "<")
    .replace(/&gt;/g, ">")
    .replace(/&quot;/g, '"')
    .replace(/&#39;/g, "'")
    .replace(/&amp;/g, "&"); // do &amp; LAST or you double-decode
}

async function scrapeBookshops() {
  const browser = await puppeteer.connect({
    browserWSEndpoint: BROWSER_WS,
    ignoreHTTPSErrors: true,
  });

  try {
    const page = await browser.newPage();
    await page.goto(PAGE_URL, {
      waitUntil: "domcontentloaded",
      timeout: 30000,
    });

    // 1. the div must exist...
    await page.waitForSelector("#markers", { timeout: 30000 });
    // 2. ...AND be substantially filled. An empty #markers appears early,
    //    so waiting on presence alone parses a half-built div.
    await page.waitForFunction(
      () => document.getElementById("markers").innerHTML.length > 100,
      { timeout: 30000 }
    );

    // Pull the raw escaped blob out of the page. That's all the browser is for.
    const rawInner = await page.evaluate(
      () => document.getElementById("markers").innerHTML
    );
    return rawInner;
  } finally {
    await browser.disconnect(); // always let go of the remote browser
  }
}
Enter fullscreen mode Exit fullscreen mode

Notice waitForFunction. waitForSelector("#markers") on its own is a trap: the empty div often exists in the DOM before the script fills it, so the selector resolves instantly and you scrape nothing. Waiting for innerHTML.length > 100 means "wait until the JSON is actually in there." Pick a threshold above whatever the empty/half state looks like.

Now the plain string work

Everything past this point runs in Node, no browser needed.

function parseMarkers(rawInner) {
  // SSR frameworks sprinkle empty comments into the DOM; they break JSON.parse.
  const cleaned = rawInner.replace(/<!--\s*-->/g, "");
  const json = unEscapeHtml(cleaned);
  const listings = JSON.parse(json); // array of positional arrays

  return listings
    .map((row, i) => {
      // Positional, not keyed: [ name, lat, lng, zIndex, popupHtml, ..., uniqueId ]
      const name = (row[0] || "Unknown").replace(/&amp;/g, "&");
      const latitude = parseFloat(row[1]) || 0;
      const longitude = parseFloat(row[2]) || 0;
      const uniqueId = row[row.length - 1] || i;

      // address lives inside the popup HTML at [4]: first <p>, <br> -> ", "
      let address = "";
      const p = /<p>(.*?)<\/p>/s.exec(row[4] || "");
      if (p) address = p[1].replace(/<br\s*\/?>/g, ", ").replace(/,\s*$/, "");

      return { uniqueId, name, address, latitude, longitude };
    })
    .filter((loc) => loc.latitude && loc.longitude); // drop 0/0 junk
}
Enter fullscreen mode Exit fullscreen mode

One page load returns the entire dataset. There's no pagination, no radius sweep, no tiling. Every pin is already in that single array, which is the upside of the DOM-blob pattern: once you've paid for the browser, you get everything in one shot.

A few things that bit me and will bit you:

  • The double-escape. &amp;amp; shows up when the source escaped an already-escaped string. Decode &amp; last in your unescape chain, or Foyles &amp; Co turns into a broken parse.
  • String coordinates. Lat and lng come as "51.5155", quoted. parseFloat them or your map math silently produces NaN.
  • Positional arrays. There are no keys. row[0] is the name today because that's the order the site emits. If they reorder, your mapping is wrong and nothing throws. Log a sample row and eyeball it before you trust the indices.
  • Half-filled div. Covered above, but it's the number-one reason "it worked in DevTools, empty in the script."

Wrap the connect-navigate-extract cycle in a small retry loop (three attempts is plenty) and always disconnect in a finally. Remote browsers drop connections, and a leaked session is a session you're paying for.

Why this generalizes

The National Book Tokens pattern (Google Maps reading pin data from a hidden element) is everywhere on old-school store-locator pages. But the real transferable move isn't Puppeteer. It's the raw-HTML-versus-live-DOM diff. That one check tells you which of three worlds you're in:

  • Data in the raw HTML: fetch plus a parser, no browser.
  • Data from a JSON XHR you can see in the Network tab: call that endpoint directly, still no browser.
  • Data absent from both, appearing only after JS runs: now, and only now, a headless browser, used purely to run the script and read the settled DOM.

Reach for Chrome last, and when you do, make it do the smallest possible thing. It runs the JavaScript, it hands you a string, and then you fire it and go back to code that doesn't eat 400MB of RAM.

Good-citizen note: these are public store-finder endpoints. Cache the result, don't re-run it every request, and don't hammer someone's map page for data that changes maybe monthly.

I build scrapers like this for a living. My published actors are at apify.com/native_emblem if you'd rather rent the output than maintain the Chrome.

Top comments (0)