DEV Community

Cover image for Building a Web Scraper with Node.js: A Practical Guide Using Cheerio and Puppeteer
Lucy
Lucy

Posted on

Building a Web Scraper with Node.js: A Practical Guide Using Cheerio and Puppeteer

#javascript #cheerio #webscraping #node

Web scraping is a significant skill for developers it allows you to scrape any data from a website on autopilot. Whether you're scraping prices of products, doing research, listing availability of jobs, etc. a Node.js web scraper can help make this a fast and efficient task.

In this tutorial you will learn step by step, how to develop your own web scraper using Cheerio and Puppeteer.

What is Web Scraping?

Web scraping is when you fetch a web page, and extract information from that web page. Developers use this technique to automate data collection from sites that do not have an API available.

For example, you can scrape a product listing web page so that you can extract the item names, prices and ratings automatically.

Why Use Node.js for Web Scraping?

Node.js is well-suited for web scraping, as it is lightweight, fast, and has great libraries for interacting with webpages. There are two main libraries:

Cheerio: A minimal library for parsing and manipulating HTML similar to jQuery.

Puppeteer: A headless browser that will render JavaScript-heavy websites before scraping.

Together they're a powerful pairing for scraping both static and dynamic websites.

Step 1: Set Up Your Node.js Project

First, ensure Node.js and npm are installed. Then, create a new project directory:

mkdir web-scraper
cd web-scraper
npm init -y
Enter fullscreen mode Exit fullscreen mode

Next, install the required packages:

npm install axios cheerio puppeteer
Enter fullscreen mode Exit fullscreen mode
  • Axios helps you fetch static HTML pages.
  • Cheerio parses and extracts data from HTML.
  • Puppeteer handles sites that rely on JavaScript rendering.

Step 2: Scraping Static Websites Using Cheerio

Let’s start with a simple example. Suppose you want to scrape article titles from a blog page.

Create a file named cheerioScraper.js and add the following code:

const axios = require('axios');
const cheerio = require('cheerio');

async function scrapeStaticSite() {
  try {
    const { data } = await axios.get('https://example.com/blog');
    const $ = cheerio.load(data);
    const titles = [];

    $('h2.article-title').each((i, element) => {
      titles.push($(element).text());
    });

    console.log('Article Titles:', titles);
  } catch (error) {
    console.error('Error fetching data:', error.message);
  }
}

scrapeStaticSite();
Enter fullscreen mode Exit fullscreen mode

This code fetches the HTML of a webpage, loads it into Cheerio, and extracts all article titles wrapped in <h2> tags.

Step 3: Scraping Dynamic Websites Using Puppeteer

Some websites use JavaScript to load content dynamically. For those, use Puppeteer.

Create a file named puppeteerScraper.js and add:

const puppeteer = require('puppeteer');

async function scrapeDynamicSite() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com/products', { waitUntil: 'networkidle2' });

  const data = await page.evaluate(() => {
    const items = [];
    document.querySelectorAll('.product').forEach(product => {
      const name = product.querySelector('.title').innerText;
      const price = product.querySelector('.price').innerText;
      items.push({ name, price });
    });
    return items;
  });

  console.log('Product Data:', data);
  await browser.close();
}

scrapeDynamicSite();
Enter fullscreen mode Exit fullscreen mode

This script launches a headless browser, loads the webpage, and extracts product names and prices from dynamically rendered elements.

Step 4: Saving the Scraped Data

Once you’ve extracted the data, you can save it to a JSON file for later use.

Add the following code after your scraping logic:

const fs = require('fs');
fs.writeFileSync('data.json', JSON.stringify(data, null, 2));
console.log('Data saved to data.json');
Enter fullscreen mode Exit fullscreen mode

This creates a file named data.json containing the scraped information in a readable format.

Step 5: Handle Common Scraping Challenges

Web scraping may face challenges like rate limits, CAPTCHAs, and website layout changes. To manage these:

  • Use delays or random intervals between requests.
  • Rotate user agents or proxy servers.
  • Keep your scraping code updated as site structures evolve.

Conclusion

By using Cheerio for quick scraping of static pages and Puppeteer for dynamic content, it is possible to create a truly reliable and rapid web scraper in Node.js.

Once you are set up to collect data, you can go beyond scraping, using it for automation and analytics or for research. And when dealing with a large scale data project you may need to hire Node.js developers to optimize speeds, handle proxies and scale the servers!

Top comments (0)