How to scrape JavaScript-rendered pages with Python

This blog was initially posted to Crawlbase Blog

Most websites use JavaScript to load their content so it’s harder to scrape data. If you try to use traditional tools like requests and BeautifulSoup you will miss out on information that only shows up after the page is fully loaded.

This article explores how to scrape JavaScript-rendered pages. We’ll go over static vs dynamic content, the challenges of scraping, and the tools available. We’ll be focusing on Pyppeteer, a powerful tool that lets you work with JavaScript content in Python.

We'll walk you through a hands-on example of using Pyppeteer to pull product details from a dynamic web page. You’ll also get tips on solving common issues and making your scraping more efficient with Crawlbase Smart Proxy.

Static vs. JavaScript-Rendered Pages

When it comes to web scraping, understanding the difference between static and JavaScript-rendered pages is key.

Static pages are straightforward. They load all their content when you visit the site. You can use simple tools like requests or BeautifulSoup to scrape data from these pages because the data is in the HTML already.

On the other hand, JavaScript-rendered pages load their content dynamically using JavaScript. This means the HTML you see initially may not have the data you want. JavaScript runs after the page loads and pulls in more data from the server. This makes scraping harder because the data may not be in the HTML when you first visit the page.

For instance, if you’re trying to scrape product details from an e-commerce site, you might find that the product listings only appear after the page finishes loading. In such cases, traditional scraping methods won't work, and you need a more advanced tool that can handle JavaScript execution.

Challenges of Scraping Dynamic Content

Scraping JavaScript-rendered pages has its own set of challenges. Here are the main ones:

Content Loading Times

JavaScript loads content asynchronously, so elements might not be available immediately. Scraping too soon can result in missing data.

Changing Element Selectors

Websites change their design frequently, which can change the HTML structure and selectors. If your scraper relies on these, it will break when changes happen.

Captchas and Anti-Scraping Measures

Many sites have CAPTCHAs and IP blocking to prevent scraping. Navigating these while adhering to terms of service can be tricky.

Handling Pagination

Dynamic pages often paginate data, so your scraper can’t navigate through multiple pages to collect all the data.

Resource Intensive

JavaScript rendering can be resource-intensive, so scraping takes longer, and server load increases.

Popular Tools for Scraping JavaScript-Rendered Pages in Python

When scraping JavaScript-rendered pages, the tool you choose matters. Here are three popular options: Selenium, Playwright, and Pyppeteer.

Selenium

Selenium is a popular web scraping tool. It can control web browsers and simulate user interactions. It’s perfect for scraping dynamic content. Selenium supports multiple languages, including Python.

Pros:

• Versatility: Works with many browsers (Chrome, Firefox, Safari).
• Robustness: Good for testing web applications.
• Wide Community Support: Many tutorials and resources are available.

Use Cases: Selenium is good for projects that need browser automation like logging in to websites or filling out forms.

Playwright

Playwright is a newer tool that’s gaining popularity for scraping dynamic content. It supports multiple languages, including Python, and is fast.

Pros:

• Cross-Browser Testing: Works with Chromium, Firefox, WebKit.
• Automatic Waiting: Reduces manual delays in scripts.
• Easy Setup: Simple and easy-to-use

Use Cases: Playwright is good for projects that need speed and reliability especially in testing web applications.

Pyppeteer

Pyppeteer is a Python port of Puppeteer, a popular Node.js library. It allows you to control headless Chrome browsers.

Pros:

• Headless Mode: Faster since no UI is rendered.
• Easy Navigation: Simplifies tasks like taking screenshots and generating PDFs.
• JavaScript Execution: Interacts with JavaScript-heavy pages efficiently.

Use Cases: Pyppeteer is good for scraping data from modern websites with heavy JavaScript usage, like e-commerce platforms.

In summary, each tool has its strengths and best-use scenarios. Depending on your project requirements, you can choose the one that fits your needs best.

Practical Example: Using Pyppeteer to Scrape Dynamic Content

In this section, we will go through a practical example of using Pyppeteer to scrape dynamic content from a JavaScript rendered web page. We will cover everything from setting up your environment to extracting product details.

Prerequisites

• Python 3.6+ installed on your system
• Basic understanding of Python and HTML

Setting Up the Python Environment

Before we dive into scraping, let's prepare our Python environment.

Create a new project directory:

mkdir pyppeteer_scraper
cd pyppeteer_scraper

Set up a virtual environment:

python -m venv venv
source venv/bin/activate  # For Windows, use venv\Scripts\activate

Install the necessary packages:

pip install pyppeteer

Note: If you run into issues, make sure you are using the correct Python version and check for any installation errors.

Inspecting a JavaScript-rendered Page for Selectors

Once you have your environment set up, you will want to find the selectors for the elements you want to scrape. Here’s how:

1. Open your browser and go to the target page. For this example, we will use the Walmart URL below, as Walmart also uses JavaScript rendering.

https://www.walmart.com/search?q=smartwatch

1. Right-click on the product title and select Inspect to open the Developer Tools.

1. Hover over elements in the HTML structure to see the associated selectors. For example, look for class names or unique attributes that are stable across page loads, such as data-* attributes.

Creating a Scraper Using Pyppeteer to Extract Product Details

Create the Scraper Script

Create a new Python file named scraper.py and start by adding the following imports:

import asyncio
from pyppeteer import launch

Define the Scraper Function

Next, we’ll define a function that will handle the scraping process. This function will launch a browser, navigate to the desired URL with custom User-Agent and headers, and extract the product details.

async def scrape_product_details(url):
    browser = await launch()
    page = await browser.newPage()

    # Set custom User-Agent and headers
    await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3')
    await page.setExtraHTTPHeaders({
        'Accept-Language': 'en-US,en;q=0.9',
        'Accept-Encoding': 'gzip, deflate, br',
    })

    await page.goto(url)

    await asyncio.sleep(10)  # Wait for 10 seconds

    titles = await page.evaluate('''() => {
        return Array.from(document.querySelectorAll('span[data-automation-id="product-title"]')).map(x => x.innerText);
    }''')

    prices = await page.evaluate('''() => {
        return Array.from(document.querySelectorAll('div[data-automation-id="product-price"] span.w_iUH7'))
            .map(x => x.innerText.replace('current price ', ''));
    }''')

    for title, price in zip(titles, prices):
        print(f'Title: {title}, Price: {price}')

    await browser.close()

Run the Scraper

Add this to the end of your scraper.py file so it runs when you run the script.

if __name__ == "__main__":
    url = 'https://www.walmart.com/search?q=smartwatch'  # Replace with your target URL
    asyncio.get_event_loop().run_until_complete(scrape_product_details(url))

Complete Script Example

Here’s what the complete scraper.py file should look like:

import asyncio
from pyppeteer import launch

async def scrape_product_details(url):
    browser = await launch()
    page = await browser.newPage()

    # Set custom User-Agent and headers
    await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3')
    await page.setExtraHTTPHeaders({
        'Accept-Language': 'en-US,en;q=0.9',
        'Accept-Encoding': 'gzip, deflate, br',
    })

    await page.goto(url)

    await asyncio.sleep(10)  # Wait for 10 seconds

    titles = await page.evaluate('''() => {
        return Array.from(document.querySelectorAll('span[data-automation-id="product-title"]')).map(x => x.innerText);
    }''')

    prices = await page.evaluate('''() => {
        return Array.from(document.querySelectorAll('div[data-automation-id="product-price"] span.w_iUH7'))
            .map(x => x.innerText.replace('current price ', ''));
    }''')

    for title, price in zip(titles, prices):
        print(f'Title: {title}, Price: {price}')

    await browser.close()

if __name__ == "__main__":
    url = 'https://www.walmart.com/search?q=smartwatch'
    asyncio.get_event_loop().run_until_complete(scrape_product_details(url))

When you run the script, it will open a headless browser, navigate to the specified URL, and print the product titles and prices to the console.

Common Pitfalls and Troubleshooting Tips

When scraping websites that use JavaScript to load content, you might run into some common issues. Here are a few tips to help you troubleshoot.

Handling Delays and Timeouts

Web scraping can be tricky because pages take time to load. If your scraper tries to access elements before they are loaded, it will error. Here are some ways to handle delays:

1. Use Fixed Wait Times: You can add a fixed wait before your scraper starts looking for elements. This gives the page time to load. For example, you can use a sleep function:

await asyncio.sleep(5)  # Wait for 5 seconds

1. Dynamic Waiting: Instead of relying solely on fixed waits, consider using dynamic waits like waitForSelector. This will pause the script until the given element appears on the page.

Resolving Errors Related to Page Loading and Rendering

You may run into errors due to page loading or rendering issues. Here are some common problems and how to fix them:

1. Check the URL: Make sure you are navigating to the correct URL. A typo will give you a “page not found” error.
2. Inspect Page Elements: Use your browser’s developer tools to inspect the elements you are scraping if the HTML structure has changed, update your selectors.
3. Network Issues: If your internet is slow or unstable, it will affect how the page loads. Try testing your scraper on a faster connection.
4. Handle JavaScript Errors: Some websites will block scrapers or have JavaScript errors that prevent the page from rendering. If your scraper is not working, check the console for JavaScript errors and adjust your scraping strategy.
5. Use Error Handling: Use try-except blocks in your code to catch and handle errors. This way, your scraper will keep running even if it hits a minor issue.

Optimizing with Crawlbase Smart Proxy

Crawlbase Smart Proxy helps you scrape faster by routing requests through multiple IP addresses so websites can’t block you. This service prevents IP blocks, accelerates data collection through simultaneous requests, and allows access to region-specific content.

To use Crawlbase Smart Proxy, sign up for an account and get your proxy credentials. Then, configure your scraper by routing through Crawlbase. Here’s a quick example in Python:

proxy_url = 'http://_USER_TOKEN_:@smartproxy.crawlbase.com:8012'

async with pyppeteer.launch({
    'args': [f'--proxy-server={proxy_url}']
}) as browser:
    page = await browser.newPage()
    await page.goto('https://example.com')

Replace _USER_TOKEN_ with your Crawlbase token. By integrating Crawlbase Smart Proxy into your workflow, you can improve efficiency and reduce the chances of interruptions from IP blocks, enabling more successful data extraction.

Final Thoughts

Scraping JavaScript-rendered pages can be challenging but fun. With the right tools like Pyppeteer, Selenium, or Playwright, you can do it. In this blog, we covered the differences between static and JavaScript-rendered pages, common challenges, and a practical example (Pyppeteer) to help you get started.

Use optimization techniques like Crawlbase Smart Proxy to scrape faster and avoid IP bans. As you start web scraping, remember to respect the terms of service of the websites you target. Follow best practices and you’ll get the data you need while keeping good relations with web services. Happy scraping!

Frequently Asked Questions

Q. What is a JavaScript-rendered page?

A JavaScript rendered page is a web page that loads content dynamically using JavaScript. Unlike static pages that show all content immediately, JavaScript rendered pages load data after the initial page load. This makes it harder to scrape because the content is not visible in the page source code right away.

Q. Why do I need a special tool to scrape JavaScript-rendered pages?

Special tools like Selenium, Pyppeteer, or Playwright are needed to scrape JavaScript rendered pages because they can simulate a real user’s browser. These tools allow you to wait for the page to fully load and render the content before extracting data. Without them, you might miss important information or get incomplete results.

Q. How can I avoid getting blocked while scraping?

To avoid getting blocked, you can use techniques like rotating user agents, adding delays between requests, and using proxies. Tools like Crawlbase Smart Proxy can help you manage your IPs, so websites won’t detect and block your scraping activities. Always remember to follow the website’s terms of service to keep your scraping ethical.