DEV Community

Abdullah Sheikh
Abdullah Sheikh

Posted on

How to Build a Web Scraper with Node.js and Puppeteer in 8 Simple Steps

#webscraping #puppeteer #node #javascript

Create a reliable, headless-browser scraper from scratch and extract data instantly

Before We Start: What You'll Walk Away With

When you finish this guide you’ll have a ready‑to‑run Node.js project that spins up Puppeteer without opening a browser window.

You’ll be comfortable picking out page elements, clicking “next” buttons, and dropping the scraped rows into a CSV or JSON file.

From there you can point the same code at any site that follows a similar layout and start pulling data in minutes.

  • Launch a headless browser – think of it as ordering a coffee and having the barista prepare it behind the counter while you watch the receipt appear on your phone.

  • Select and iterate over elements – like using Google Maps to pinpoint every café on a street, then moving from one pin to the next.

  • Export results – similar to packing a suitcase: you gather all the items (data) and neatly place them into a CSV or JSON bag for later use.

  • Install node and npm once; the rest of the setup stays the same.

  • Use await page.$$ to collect groups of elements, just as you’d scan a menu for all dessert options.

  • Handle pagination with a simple loop, mimicking the “next page” button you click when scrolling through product listings.

Cheat sheet

  • npm init -y – creates package.json

  • npm install puppeteer – pulls in the headless browser

  • await page.goto(url, {waitUntil: 'networkidle2'}) – lands you on the page

  • fs.writeFileSync('data.json', JSON.stringify(data, null, 2)) – saves the output

Grab the code, run it, and you’ll be pulling data before your coffee even cools.

What Web Scraping with Puppeteer Actually Is (No Jargon)

Puppeteer is a Node.js library that gives you programmatic control over Chrome or Chromium. It lets your code open a page, wait for content, click buttons, scroll, and then pull out the exact bits of HTML or text you need. In short, a web scraper node.js built with Puppeteer can act like a browser you would use manually, but it runs automatically and at scale.

Imagine a robot with a pair of hands sitting at a café table. You tell it, “Open the menu, click the dessert section, and copy the price of the tiramisu.” The robot follows the steps exactly, even if the site throws a pop‑up or requires a scroll. That’s what Puppeteer does for web pages— it mimics a real user’s actions, so sites that rely on JavaScript or dynamic loading still hand over their data.

Because the robot works inside a real browser, you don’t have to guess how the page renders; you see exactly what a human would see. This means fewer broken scrapers and less time fighting invisible APIs.

Got a list of product pages you need to scrape? Just script the robot to visit each URL, wait for the price element, and write it to a CSV. The same approach works for login flows, infinite scrolls, or extracting tables from dashboards.

Think of Puppeteer as your digital assistant that never gets tired, never clicks the wrong link, and always brings back the data you asked for.

The 3 Mistakes Everyone Makes With Puppeteer Scrapers

Most people hit a wall fast because they miss the three classic traps.

  • Ignoring headless detection defenses – Think of it like ordering food at a drive‑through with a clearly fake ID. The server spots the default Puppeteer fingerprint and refuses service. Spoof the user‑agent, hide the webdriver flag, and randomize screen size to blend in.

  • Over‑complicating selectors – It’s like trying to navigate a city with a handwritten map that marks every alley. Using brittle XPaths makes your scraper break on the next layout tweak. Stick to stable CSS selectors such as div.article > h2.title and test them with page.$$(selector) before committing.

  • Forgetting rate‑limiting – Imagine pounding the door of a house with a hammer; you’ll get shut out fast. Bombarding a site with rapid requests triggers bans. Add await page.waitForTimeout(Math.random()*2000+500) between actions and respect robots.txt where feasible.

Fix these and your web scraper node.js will stay alive longer.

How to Build a Web Scraper with Node.js and Puppeteer: Step‑by‑Step

Let’s get your scraper up and running in eight quick actions.

  • Open a terminal, run npm init -y to bootstrap a fresh Node project, then install Puppeteer with npm i puppeteer. Think of this as ordering the base ingredients before cooking.

  • Create a file scraper.js and add an async function, e.g. async function run(). Inside, launch a headless browser via puppeteer.launch() and open a new page with browser.newPage(). This is like turning the ignition and stepping into the driver’s seat.

  • Before you hit the road, set a realistic userAgent string and apply the puppeteer-extra-plugin-stealth plugin. It masks your scraper the way a disguise hides your identity.

  • Direct the page to your target URL with await page.goto(url, {waitUntil: 'networkidle2'}) and pause until a required selector appears using await page.waitForSelector('.product'). It’s like waiting for the traffic light to turn green.

Pull the data you need:

const items = await page.$$eval('.product', cards => 
  cards.map(card => ({
    title: card.querySelector('.title').innerText,
    price: card.querySelector('.price').innerText
  }))
);
Enter fullscreen mode Exit fullscreen mode

Meet Alex, a market researcher who needs product names and prices. Alex runs the snippet above and gets an array of objects ready for analysis.
Handle pagination by looping while a “Next” button exists:

  • Check await page.$('.next').

  • If found, click it and repeat the extraction.

  • Break the loop when the button disappears.

This works like flipping pages in a book until you reach the end.
Save the gathered array to disk. For CSV:

const {Parser} = require('json2csv');
const parser = new Parser();
const csv = parser.parse(items);
require('fs').writeFileSync('data.csv', csv);
Enter fullscreen mode Exit fullscreen mode

Or use fs.writeFileSync('data.json', JSON.stringify(items, null, 2)) for JSON.

  • Finally, close the browser with await browser.close(). Wrap the whole flow in a try/catch block to log errors and guarantee the browser shuts down even if something goes wrong.

Now you have a functional web scraper node.js ready to adapt to any site.

A Real Example: Scraping Product Prices from ExampleStore.com

Maya wants a script that wakes up each morning, grabs the latest prices from ExampleStore.com, and drops them into prices.csv—nothing more.

Install dependencies

npm i puppeteer csv-writer
Enter fullscreen mode Exit fullscreen mode

Launch the browser

const browser = await puppeteer.launch({headless: true});
Enter fullscreen mode Exit fullscreen mode

Open the target page

const page = await browser.newPage();
await page.goto('https://examplestore.com/category/widgets');
Enter fullscreen mode Exit fullscreen mode

Extract product rows

const rows = await page.$$eval('.product-card', cards => 
  cards.map(c => ({
    name: c.querySelector('.title').innerText.trim(),
    price: c.querySelector('.price').innerText.replace('$','')
  }))
);
Enter fullscreen mode Exit fullscreen mode

Handle pagination – click “Next” until it disappears.

while (await page.$('button.next')) {
  await Promise.all([
    page.click('button.next'),
    page.waitForNavigation({waitUntil: 'networkidle0'})
  ]);
  const more = await page.$$eval('.product-card', ...); // same extraction
  rows.push(...more);
}
Enter fullscreen mode Exit fullscreen mode

Write to CSV

const createCsvWriter = require('csv-writer').createObjectCsvWriter;
const csvWriter = createCsvWriter({
  path: 'prices.csv',
  header: [{id:'name',title:'Product'},{id:'price',title:'Price'}]
});
await csvWriter.writeRecords(rows);
Enter fullscreen mode Exit fullscreen mode

Close everything

await browser.close();
Enter fullscreen mode Exit fullscreen mode

Run it daily – add an npm script and a cron entry.

"scripts": { "scrape": "node scraper.js" }
Enter fullscreen mode Exit fullscreen mode 
0 6 * * * cd /path/to/project && npm run scrape
Enter fullscreen mode Exit fullscreen mode

  • Tip: Test the selector '.product-card .price' in Chrome DevTools before coding.

  • Tip: Use waitForSelector after each page change to avoid race conditions.

  • Tip: Keep prices.csv in a version‑controlled folder for easy diffing.

With these eight steps, Maya can treat her scraper like a coffee‑order bot—click, collect, and serve fresh data every day.

The Tools That Make This Easier

Grab these five freebies and you’ll spend less time hunting for tools and more time actually scraping.

  • Puppeteer (npm) – the headless‑browser engine that does the heavy lifting. Think of it as the kitchen appliance that cooks your data soup while you set the timer.

  • puppeteer‑extra‑stealth – a plugin that hides the browser’s “robot” badge. It’s like ordering a meal with a secret sauce that gets past the picky server.

  • csv‑writer – a tiny library that turns JavaScript objects into clean CSV files. Imagine packing a suitcase: each object is an item, csv-writer neatly folds them into rows.

  • VS Code with Prettier – your IDE plus an auto‑formatter. It’s the Google Maps of code layout: you type, Prettier reroutes you to the tidy‑est path.

  • GitHub Actions (free tier) – schedule your scraper to run nightly without a server. Like setting an alarm clock, it wakes up your script at the right hour.

Quick start commands:

npm install puppeteer puppeteer-extra-stealth csv-writer
Enter fullscreen mode Exit fullscreen mode 
const { createObjectCsvWriter } = require('csv-writer');
const csvWriter = createObjectCsvWriter({
  path: 'out.csv',
  header: [{id:'title',title:'Title'},{id:'price',title:'Price'}]
});
Enter fullscreen mode Exit fullscreen mode

With these tools in place, the next step is wiring up the scraper logic.

Quick Reference: Web Scraper with Puppeteer Cheat Sheet

Grab this list and copy‑paste it when you spin up a new scraper.

  • Setup: npm init -y then npm i puppeteer puppeteer-extra-stealth csv-writer. Think of it like ordering the ingredients before you start cooking.

Launch browser:

async function launchBrowser() {
  return await puppeteer.launch({headless:true});
}
Enter fullscreen mode Exit fullscreen mode

It’s the “turn the stove on” step.
Navigate:

await page.goto(URL, {waitUntil:'networkidle2'});
Enter fullscreen mode Exit fullscreen mode

Like telling Google Maps to drive you to the exact address and wait until traffic clears.
Extract items:

const data = await page.$$eval('.item', els =>
  els.map(e => ({
    title: e.querySelector('.title').innerText,
    price: e.querySelector('.price').innerText
  }))
);
Enter fullscreen mode Exit fullscreen mode

You’re picking the right dishes from a buffet and writing down their names and prices.
Paginate loop (example with Maya): Maya wants every product on a multi‑page catalog.

while (await page.$('.next')) {
  await Promise.all([
    page.click('.next'),
    page.waitForNavigation({waitUntil:'networkidle2'})
  ]);
  // repeat extraction here
}
Enter fullscreen mode Exit fullscreen mode

She clicks “next” just like flipping pages in a book until there’s no more.
Save to CSV:

await csvWriter.writeRecords(data);
Enter fullscreen mode Exit fullscreen mode

Think of it as packing the collected items into a suitcase for easy transport.
Cleanup:

try {
  // main logic
} catch (e) {
  console.error(e);
} finally {
  await browser.close();
}
Enter fullscreen mode Exit fullscreen mode

Wrap everything in a try/catch so the scraper doesn’t leave the kitchen a mess.

Keep this cheat sheet handy and your web scraper node.js project will stay on autopilot.

What to Do Next

Grab the script you just wrote and give it a quick spin on your own machine.

  • Run it, tweak a selector, export a CSV. Think of it like ordering a coffee: you ask for exactly what you want, take a sip, then adjust the sugar if needed. Open a terminal and fire: 
node scraper.js > output.csv
Enter fullscreen mode Exit fullscreen mode

Open output.csv in Excel, confirm the columns line up, then change page.$('selector') until the data matches what you expect.

  • Make the scraper a little sneaky. For sites that start blocking bots, add the puppeteer-extra-plugin-stealth and sprinkle random delays between actions. It’s like slipping through a crowd by pausing to look at your phone—less likely to be noticed.

  • Install the plugin: npm i puppeteer-extra puppeteer-extra-plugin-stealth

  • Wrap actions with await page.waitForTimeout(Math.random()*3000+2000)

  • Deploy and schedule. Push the repo to GitHub, create a GitHub Action (or Railway job) that runs nightly. This is the suitcase‑packing stage: you’re ready to ship your scraper so it works without you hovering over the keyboard.

  • GitHub Action snippet:

name: Daily Scrape
on:
  schedule:
    - cron: '0 2 * * *'
jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm ci
      - run: node scraper.js > data.csv
      - uses: actions/upload-artifact@v3
        with:
          name: daily-data
          path: data.csv
Enter fullscreen mode Exit fullscreen mode

Now you have a web scraper node.js pipeline that runs on its own.

💬 Got a site that’s giving you trouble? Drop a comment with the URL and I’ll help you debug!

About the Author

Abdullah Sheikh is the Founder & CEO at Exteed, where he leads a team of skilled developers specializing in Web2 and Web3 applications, Custom Smart Contracts, and Blockchain solutions.

With 6+ years of experience, Abdullah has built CRMs, Crypto Wallets, DeFi Exchanges, E-Commerce Stores, HIPAA Compliant EMR Systems, and AI-powered systems that drive business efficiency and innovation.

His expertise spans Blockchain, Crypto & Tokenomics, Artificial Intelligence, and Web Applications; building reliable and smooth web apps that fit the client’s goals and requirements.

📧 info@abdullah-sheikh.com · 🔗 LinkedIn · 🌐 abdullah-sheikh.com

Top comments (0)