Puppeteer for Web Scraping: Scraping JavaScript Websites With Node.js (For Complete Beginners)

You try to scrape a website with Scrapy. Your spider runs. It extracts data. But something's wrong. The selectors you wrote match nothing. The data you wanted isn't there.

You check the HTML. It's empty. Just placeholder divs. No content.

Then you open the site in your browser. Everything's there. Products, prices, reviews. All visible. Perfect.

But your scraper sees an empty page.

The problem: JavaScript. The website loads data dynamically. JavaScript runs in the browser, fetches data from an API, populates the page. Your scraper makes an HTTP request. Gets the raw HTML. JavaScript never executes. The page stays empty.

Scrapy can't handle this. Scrapy is fast because it's just HTTP requests. It doesn't run JavaScript. For most sites, that's fine. For modern websites built with React, Vue, Angular, that's a nightmare.

You need a real browser. A browser that executes JavaScript. Renders the page. Shows you the final HTML with all data populated.

That's Puppeteer.

Puppeteer is a Node.js library that controls a real browser (Chrome/Chromium) programmatically. You tell it what to do. Click buttons. Wait for data to load. Extract the rendered HTML. It all works.

No more empty pages. No more missing data.

Let me show you how.

---

What Puppeteer Actually Is (And Why You Need It)

Puppeteer is a Node.js library that lets you control a browser (Chrome or Chromium) from your code.

Think of it like this:

Scrapy: Sends HTTP requests. Gets HTML. Extracts data. Fast. But can't run JavaScript.

Puppeteer: Opens a real browser. Executes JavaScript. Renders the page. Extracts data. Slower. But handles any website.

When to use Puppeteer:

• Website loads data with JavaScript (React, Vue, Angular)
• Need to click buttons or submit forms
• Need to wait for dynamic content to load
• Need screenshots of pages
• Need to test JavaScript interactions
• Website requires browser features (cookies, sessions, local storage)

When NOT to use Puppeteer:

• Static HTML sites (use Scrapy)
• Simple HTTP scraping (use requests library)
• High-volume scraping at scale (Puppeteer is slower)
• Don't need JavaScript execution

The trade-off:

Puppeteer is slower than Scrapy (opens real browser, executes JavaScript). But it works on any website, no matter how much JavaScript they use.

---

Installation: Getting Started

Puppeteer needs Node.js. Let's install everything.

Step 1: Install Node.js

Mac:

brew install node

Windows:

Download from nodejs.org

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install nodejs npm

Check it worked:

node --version
npm --version

You should see version numbers.

Step 2: Create a Project Folder

mkdir my-scraper
cd my-scraper

Step 3: Initialize Node Project

npm init -y

This creates package.json (tracks your dependencies).

Step 4: Install Puppeteer

npm install puppeteer

This downloads Puppeteer and installs Chrome/Chromium automatically.

Wait: This takes a few minutes. Puppeteer downloads a full browser. It's big. Be patient.

Check it worked:

ls node_modules

You should see a puppeteer folder.

---

Your First Script: The Hello World of Scraping

Let's write a simple script that opens a website and takes a screenshot.

Create the Script

touch scraper.js

Write the Code

// scraper.js

const puppeteer = require('puppeteer');

(async () => {
  // Launch browser
  const browser = await puppeteer.launch();

  // Open new page
  const page = await browser.newPage();

  // Go to website
  await page.goto('https://example.com');

  // Take screenshot
  await page.screenshot({path: 'screenshot.png'});

  console.log('Screenshot saved as screenshot.png');

  // Close browser
  await browser.close();
})();

Run the Script

node scraper.js

What happens:

1. Browser launches (you'll see a Chrome window open)
2. Opens example.com
3. Takes a screenshot
4. Saves to screenshot.png
5. Closes browser

Check your folder. screenshot.png is there. You just automated a browser.

---

Understanding Puppeteer Concepts

Let's break down what's happening.

Async/Await

All Puppeteer operations are asynchronous (they take time).

const page = await page.goto('https://example.com');

await means "wait for this to complete, then continue."

Without await, the script would try to do everything at once and fail.

Browser vs Page

Browser: The entire Chrome application. Can have multiple pages.

Page: A single tab in the browser.

const browser = await puppeteer.launch();  // Open Chrome
const page = await browser.newPage();      // Open a tab
await page.goto('https://example.com');   // Load website in tab

Common Operations

// Navigate to URL
await page.goto('https://example.com');

// Wait for element to load
await page.waitForSelector('.product');

// Click a button
await page.click('button.next-page');

// Type in input
await page.type('input.search', 'laptop');

// Get text content
const text = await page.$eval('.price', el => el.textContent);

// Take screenshot
await page.screenshot({path: 'page.png'});

// Close browser
await browser.close();

---

Scraping Data: The Real Work

Let's scrape actual data from a real website.

Example: Books Website

We'll scrape book titles and prices from a simple book website.

The Script

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Go to website
  await page.goto('https://books.toscrape.com');

  // Wait for books to load
  await page.waitForSelector('.product_pod');

  // Extract book data
  const books = await page.evaluate(() => {
    const bookElements = document.querySelectorAll('.product_pod');
    const booksArray = [];

    bookElements.forEach(book => {
      const title = book.querySelector('h3 a').getAttribute('title');
      const price = book.querySelector('.price_color').textContent;

      booksArray.push({
        title: title,
        price: price
      });
    });

    return booksArray;
  });

  console.log('Books found:');
  console.log(books);

  await browser.close();
})();

Run it

node scraper.js

Output:

Books found:
[
  { title: 'A Light in the Attic', price: '£51.77' },
  { title: 'Tango with Django', price: '£13.99' },
  { title: 'His Dark Materials', price: '£22.65' },
  ...
]

You just scraped data from a website.

What's Happening

const books = await page.evaluate(() => {
  // This code runs INSIDE the browser
  // You have access to DOM, window, document
  const bookElements = document.querySelectorAll('.product_pod');
  // Extract data
  // Return results
});

page.evaluate() lets you run JavaScript code inside the website. You have full access to the DOM. Extract whatever you want.

---

Handling Multiple Pages: Pagination

Most websites have multiple pages. Let's scrape all of them.

The Script

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const allBooks = [];
  let pageNumber = 1;
  let hasNextPage = true;

  while (hasNextPage) {
    // Build URL for current page
    const url = `https://books.toscrape.com/catalogue/page-${pageNumber}.html`;

    console.log(`Scraping page ${pageNumber}...`);

    // Go to page
    await page.goto(url);

    // Wait for content to load
    await page.waitForSelector('.product_pod');

    // Extract books
    const books = await page.evaluate(() => {
      const bookElements = document.querySelectorAll('.product_pod');
      const booksArray = [];

      bookElements.forEach(book => {
        const title = book.querySelector('h3 a').getAttribute('title');
        const price = book.querySelector('.price_color').textContent;

        booksArray.push({
          title: title,
          price: price
        });
      });

      return booksArray;
    });

    allBooks.push(...books);

    // Check if there's a next page
    const nextButton = await page.$('.next a');

    if (nextButton) {
      pageNumber++;
    } else {
      hasNextPage = false;
    }
  }

  console.log(`Total books scraped: ${allBooks.length}`);
  console.log(allBooks);

  await browser.close();
})();

What This Does

1. Loops through pages
2. Waits for content to load
3. Extracts book data
4. Checks if next page exists
5. Repeats until no more pages

---

Real Example: Scraping a Product Listing Page

Let's build a more realistic scraper. We'll scrape product data.

The Setup

We'll scrape a simple e-commerce site. Extract:

• Product name
• Price
• Rating
• Number of reviews

The Script

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: true  // Run without showing browser window
  });

  const page = await browser.newPage();

  // Set timeout
  page.setDefaultTimeout(10000);

  try {
    // Go to product page
    console.log('Loading product page...');
    await page.goto('https://example-shop.com/products');

    // Wait for products to load
    console.log('Waiting for products to load...');
    await page.waitForSelector('.product-item', {timeout: 5000});

    // Extract product data
    console.log('Extracting product data...');
    const products = await page.evaluate(() => {
      const productElements = document.querySelectorAll('.product-item');
      const productsArray = [];

      productElements.forEach(product => {
        const name = product.querySelector('.product-name')?.textContent?.trim() || 'N/A';
        const price = product.querySelector('.product-price')?.textContent?.trim() || 'N/A';
        const rating = product.querySelector('.product-rating')?.textContent?.trim() || 'N/A';
        const reviews = product.querySelector('.review-count')?.textContent?.trim() || '0';

        // Only add if we got a name
        if (name !== 'N/A') {
          productsArray.push({
            name: name,
            price: price,
            rating: rating,
            reviews: reviews
          });
        }
      });

      return productsArray;
    });

    // Display results
    console.log('\n=== Products Found ===\n');
    products.forEach((product, index) => {
      console.log(`${index + 1}. ${product.name}`);
      console.log(`   Price: ${product.price}`);
      console.log(`   Rating: ${product.rating}`);
      console.log(`   Reviews: ${product.reviews}`);
      console.log('');
    });

    console.log(`Total products: ${products.length}`);

  } catch (error) {
    console.error('Error:', error.message);
  } finally {
    await browser.close();
  }
})();

Run it

node scraper.js

What's Happening

1. Headless mode: Browser runs without GUI (faster, cleaner)
2. Wait for selector: Waits for products to load before extracting
3. Optional chaining: ?. handles missing elements gracefully
4. Error handling: Try-catch catches any errors
5. Finally block: Always closes browser, even if error occurs

---

Handling Dynamic Content: Waiting for Data

Sometimes data takes time to load. JavaScript makes API calls. You need to wait.

Example: Waiting for AJAX Content

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://example.com/dynamic-content');

  // Wait for specific element
  await page.waitForSelector('.dynamic-content');

  // Or wait for specific condition
  await page.waitForFunction(
    () => document.querySelectorAll('.item').length > 0,
    {timeout: 5000}
  );

  // Or wait a specific time
  await page.waitForTimeout(2000);  // Wait 2 seconds

  // Now extract data
  const data = await page.evaluate(() => {
    return document.querySelector('.dynamic-content').textContent;
  });

  console.log(data);

  await browser.close();
})();

Waiting Methods

Wait for element:

await page.waitForSelector('.element');

Wait for function:

await page.waitForFunction(() => someCondition);

Wait for navigation:

await page.waitForNavigation();
await page.click('a');  // Click link and wait

Wait fixed time (avoid if possible):

await page.waitForTimeout(2000);  // Wait 2 seconds

Best practice: Use waitForSelector or waitForFunction when possible. Fixed waits are unreliable.

---

Clicking Buttons and Interactions

Sometimes you need to click buttons, submit forms, scroll the page.

Clicking a Button

// Click a button
await page.click('button.load-more');

// Wait for new content to load
await page.waitForSelector('.new-content');

// Extract data
const data = await page.evaluate(() => {
  return document.querySelector('.new-content').textContent;
});

Filling Forms

// Type in search box
await page.type('input[name="search"]', 'laptop');

// Select dropdown
await page.select('select[name="category"]', 'electronics');

// Submit form
await page.click('button[type="submit"]');

// Wait for results
await page.waitForSelector('.results');

Scrolling

// Scroll to bottom (for infinite scroll sites)
await page.evaluate(() => {
  window.scrollBy(0, window.innerHeight);
});

// Or scroll to specific element
await page.evaluate(() => {
  document.querySelector('.target-element').scrollIntoView();
});

Real Example: Infinite Scroll

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://example.com/infinite-scroll');

  // Scroll and load multiple times
  for (let i = 0; i < 5; i++) {
    console.log(`Loading more content... ${i + 1}/5`);

    // Scroll to bottom
    await page.evaluate(() => {
      window.scrollBy(0, window.innerHeight);
    });

    // Wait for new content
    await page.waitForTimeout(1000);
  }

  // Extract all loaded content
  const allItems = await page.evaluate(() => {
    const items = document.querySelectorAll('.item');
    return Array.from(items).map(item => ({
      title: item.querySelector('.title').textContent,
      description: item.querySelector('.description').textContent
    }));
  });

  console.log(`Total items: ${allItems.length}`);
  console.log(allItems);

  await browser.close();
})();

---

Saving Data to File

Scraping data to console is fun, but you need to save it somewhere.

Save to JSON

const fs = require('fs');
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://books.toscrape.com');
  await page.waitForSelector('.product_pod');

  const books = await page.evaluate(() => {
    const bookElements = document.querySelectorAll('.product_pod');
    const booksArray = [];

    bookElements.forEach(book => {
      const title = book.querySelector('h3 a').getAttribute('title');
      const price = book.querySelector('.price_color').textContent;

      booksArray.push({title, price});
    });

    return booksArray;
  });

  // Save to JSON file
  fs.writeFileSync('books.json', JSON.stringify(books, null, 2));

  console.log('Saved to books.json');

  await browser.close();
})();

Save to CSV

const fs = require('fs');
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://books.toscrape.com');
  await page.waitForSelector('.product_pod');

  const books = await page.evaluate(() => {
    const bookElements = document.querySelectorAll('.product_pod');
    const booksArray = [];

    bookElements.forEach(book => {
      const title = book.querySelector('h3 a').getAttribute('title');
      const price = book.querySelector('.price_color').textContent;

      booksArray.push({title, price});
    });

    return booksArray;
  });

  // Convert to CSV
  const csv = [
    'Title,Price',
    ...books.map(b => `"${b.title}","${b.price}"`)
  ].join('\n');

  fs.writeFileSync('books.csv', csv);

  console.log('Saved to books.csv');

  await browser.close();
})();

---

Error Handling: Making Scripts Reliable

Real scraping has errors. Networks fail. Selectors change. Elements don't load.

Basic Error Handling

const puppeteer = require('puppeteer');

(async () => {
  let browser;

  try {
    browser = await puppeteer.launch();
    const page = await browser.newPage();

    // Set timeout for all operations
    page.setDefaultTimeout(10000);

    await page.goto('https://example.com', {
      waitUntil: 'networkidle2'  // Wait for network to be stable
    });

    await page.waitForSelector('.product', {timeout: 5000});

    const data = await page.evaluate(() => {
      const element = document.querySelector('.product');
      if (!element) {
        throw new Error('Product element not found');
      }
      return element.textContent;
    });

    console.log('Data:', data);

  } catch (error) {
    console.error('Error:', error.message);
  } finally {
    if (browser) {
      await browser.close();
    }
  }
})();

Retry Logic

async function scrapeWithRetry(url, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      console.log(`Attempt ${i + 1}/${maxRetries}`);

      const browser = await puppeteer.launch();
      const page = await browser.newPage();

      await page.goto(url, {waitUntil: 'networkidle2'});
      await page.waitForSelector('.content');

      const data = await page.evaluate(() => {
        return document.querySelector('.content').textContent;
      });

      await browser.close();
      return data;

    } catch (error) {
      console.error(`Attempt ${i + 1} failed:`, error.message);

      if (i === maxRetries - 1) {
        throw error;  // Final attempt failed
      }

      // Wait before retrying
      await new Promise(resolve => setTimeout(resolve, 2000));
    }
  }
}

// Usage
(async () => {
  try {
    const data = await scrapeWithRetry('https://example.com');
    console.log('Success:', data);
  } catch (error) {
    console.error('Failed after retries:', error.message);
  }
})();

---

Common Mistakes (And How to Fix Them)

Mistake 1: Forgetting to Close Browser

// Bad
await page.goto('https://example.com');
const data = await page.evaluate(() => {/*...*/});
console.log(data);
// Browser still running!

// Good
try {
  await page.goto('https://example.com');
  const data = await page.evaluate(() => {/*...*/});
  console.log(data);
} finally {
  await browser.close();
}

Browser consumes memory. Always close it.

Mistake 2: Not Waiting for Content

// Bad
await page.goto('https://example.com');
const data = await page.evaluate(() => {
  return document.querySelector('.product').textContent;  // Might not exist yet!
});

// Good
await page.goto('https://example.com');
await page.waitForSelector('.product');  // Wait first
const data = await page.evaluate(() => {
  return document.querySelector('.product').textContent;
});

Content loads asynchronously. Always wait.

Mistake 3: Accessing Elements That Don't Exist

// Bad
const price = document.querySelector('.price').textContent;  // Crashes if element missing

// Good
const price = document.querySelector('.price')?.textContent || 'N/A';  // Safe

Use optional chaining ?. to handle missing elements.

Mistake 4: Running Too Fast

// Bad
for (let i = 0; i < pages; i++) {
  await page.goto(url);
  // No delay, website thinks it's an attack
}

// Good
for (let i = 0; i < pages; i++) {
  await page.goto(url);
  await page.waitForTimeout(1000);  // 1 second delay
}

Add delays between requests. Be respectful.

Mistake 5: Forgetting About Headless Mode

// Development (see what's happening)
const browser = await puppeteer.launch({
  headless: false  // Show browser window
});

// Production (faster)
const browser = await puppeteer.launch({
  headless: true  // No GUI
});

Headless mode is faster. Use it in production.

---

Tips for Better Scraping

Tip 1: Set User Agent

Some websites block headless browsers. Disguise yourself:

await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36');

Tip 2: Set Viewport Size

Some websites act differently on mobile. Control viewport:

await page.setViewport({width: 1920, height: 1080});

Tip 3: Disable Images (Faster)

Images slow things down. Disable them:

await page.setRequestInterception(true);

page.on('request', request => {
  if (request.resourceType() === 'image') {
    request.abort();
  } else {
    request.continue();
  }
});

Tip 4: Add Logging

Know what's happening:

console.log('Step 1: Loading page...');
await page.goto(url);

console.log('Step 2: Waiting for content...');
await page.waitForSelector('.content');

console.log('Step 3: Extracting data...');
const data = await page.evaluate(() => {/*...*/});

console.log('Step 4: Done!');

Tip 5: Use Puppeteer Debugger

Debug interactively:

const browser = await puppeteer.launch({
  headless: false,
  devtools: true  // Opens DevTools automatically
});

---

Real Complete Example

Let's build a complete, production-ready scraper.

The Script

const puppeteer = require('puppeteer');
const fs = require('fs');

async function scrapeBooks() {
  let browser;

  try {
    console.log('Launching browser...');
    browser = await puppeteer.launch({
      headless: true
    });

    const page = await browser.newPage();
    page.setDefaultTimeout(10000);

    const allBooks = [];
    let currentPage = 1;
    let hasNextPage = true;

    while (hasNextPage && currentPage <= 3) {  // Limit to 3 pages
      try {
        const url = `https://books.toscrape.com/catalogue/page-${currentPage}.html`;

        console.log(`\nScraping page ${currentPage}...`);

        await page.goto(url, {waitUntil: 'networkidle2'});
        await page.waitForSelector('.product_pod');

        const books = await page.evaluate(() => {
          const bookElements = document.querySelectorAll('.product_pod');
          const booksArray = [];

          bookElements.forEach(book => {
            const title = book.querySelector('h3 a')?.getAttribute('title') || 'N/A';
            const price = book.querySelector('.price_color')?.textContent || 'N/A';
            const availability = book.querySelector('.instock')?.textContent?.trim() || 'Unknown';

            booksArray.push({
              title,
              price,
              availability
            });
          });

          return booksArray;
        });

        console.log(`Found ${books.length} books on page ${currentPage}`);
        allBooks.push(...books);

        // Check for next page button
        const nextButton = await page.$('li.next a');
        hasNextPage = !!nextButton;

        if (hasNextPage) {
          currentPage++;
          await page.waitForTimeout(1000);  // Be respectful
        }

      } catch (pageError) {
        console.error(`Error on page ${currentPage}:`, pageError.message);
        break;
      }
    }

    // Save to file
    const outputPath = 'books.json';
    fs.writeFileSync(outputPath, JSON.stringify(allBooks, null, 2));

    console.log(`\n=== Results ===`);
    console.log(`Total books scraped: ${allBooks.length}`);
    console.log(`Saved to ${outputPath}`);

    // Display sample
    console.log('\nFirst 5 books:');
    allBooks.slice(0, 5).forEach((book, index) => {
      console.log(`${index + 1}. ${book.title}`);
      console.log(`   Price: ${book.price}`);
      console.log(`   Availability: ${book.availability}\n`);
    });

  } catch (error) {
    console.error('Fatal error:', error);
  } finally {
    if (browser) {
      await browser.close();
      console.log('\nBrowser closed.');
    }
  }
}

// Run the scraper
scrapeBooks();

Run it

node scraper.js

Output

Launching browser...

Scraping page 1...
Found 20 books on page 1

Scraping page 2...
Found 20 books on page 2

Scraping page 3...
Found 20 books on page 3

=== Results ===
Total books scraped: 60
Saved to books.json

First 5 books:
1. A Light in the Attic
   Price: £51.77
   Availability: In stock

2. Tango with Django
   Price: £13.99
   Availability: In stock

Browser closed.

---

Next Steps: Going Further

You now know Puppeteer basics. Here are advanced topics:

Performance

• Use connection pooling (scrape multiple sites concurrently)
• Cache pages to avoid re-scraping
• Optimize selectors

Robustness

• Handle network failures gracefully
• Implement exponential backoff for retries
• Monitor scraper health

Ethics

• Check robots.txt (like Scrapy)
• Add delays between requests
• Respect rate limits
• Don't overload servers

Integration

• Save to databases (PostgreSQL, MongoDB)
• Use task queues (Bull, RabbitMQ)
• Deploy to cloud (AWS Lambda, Google Cloud Functions)
• Combine with Scrapy for large-scale scraping

---

Puppeteer vs Scrapy

Use Puppeteer When

• Website uses JavaScript (React, Vue, Angular)
• Need to click buttons or interact with page
• Need screenshots or PDFs
• Website requires browser features

Use Scrapy When

• Static HTML sites
• High-volume scraping at scale
• Simple HTTP scraping
• Already familiar with Python

The Best Approach

• Scrapy + Selenium/Puppeteer: Use Scrapy for the framework, add Puppeteer/Selenium middleware for JavaScript sites
• Or use Scrapy-Playwright (Scrapy middleware for browser automation)

---

Summary

Puppeteer controls a real browser from your code.

What it does:

• Opens Chrome/Chromium
• Executes JavaScript
• Waits for content to load
• Extracts data from rendered HTML
• Handles interactions (clicks, forms, scrolling)

Basic workflow:

1. Launch browser
2. Navigate to URL
3. Wait for content
4. Extract data
5. Close browser

Key concepts:

• Browser = Chrome application
• Page = individual tab
• await = wait for operation to complete
• page.evaluate() = run code in the website
• Error handling = try/catch/finally

When to use:

• JavaScript-heavy websites
• Dynamic content that needs JavaScript to render
• Need interactions like clicking buttons

Remember:

• Always close the browser (memory management)
• Always wait for content (async operations)
• Handle errors gracefully
• Be respectful (add delays, check robots.txt)

You can now scrape any website, even the ones that load data with JavaScript.

---

Next Steps:

1. Try scraping a simple website
2. Expand to multiple pages
3. Save data to files
4. Add error handling
5. Deploy as a scheduled job

Happy scraping!

---

Resources:

• Puppeteer documentation: https://pptr.dev
• Example website: https://books.toscrape.com
• Node.js documentation: https://nodejs.org