DEV Community: Crawlbase

Build a No-Code AI Scraper With Crawlbase in Minutes

Crawlbase — Wed, 07 Jan 2026 13:35:55 +0000

Web scraping just got simpler. Instead of wrestling with HTML selectors and brittle scripts, you can now describe what data you need and let AI handle the extraction. This guide shows you how to build an intelligent scraper using Crawlbase Web MCP—no coding required.

🛠️ What You'll Need

Before starting, gather these tools:

Cursor IDE — Download from the official Cursor website
Crawlbase account — Sign up and grab your API credentials
Crawlbase Web MCP — Follow the setup guide to configure it

⚙️ How It Works

The system relies on three components working together:

Crawlbase's Crawling API handles the heavy lifting—loading pages, managing proxies, and bypassing CAPTCHAs. Crawlbase Web MCP acts as a bridge, letting AI communicate with Crawlbase securely. Cursor's AI agents interpret your instructions, extract the data, and format it cleanly.

You describe what you want. The system does the rest.

🚀 Building Your Scraper

Step 1: Open Cursor

Launch Cursor IDE. This is where you'll give instructions to the AI agent.

Step 2: Write Your Prompt

Tell the agent exactly what you need. For example, to scrape eBay's Best Selling Products:

Crawl eBay's Best Selling Products page at https://www.ebay.com/str/bestsellingproducts as raw HTML. Extract the title, price, condition, seller info, and product URL for each product. Output the result in a valid JSON-formatted file named output.json.

Hit Approve when prompted.

Step 3: Let the AI Work

The agent takes over:

Loads the page through Crawlbase Web MCP
Parses the HTML
Extracts the specified fields
Generates the JSON file

No manual parsing. No selector maintenance.

Step 4: Review Results

Check the generated JSON file. For eBay, the agent typically extracts around ten products with all requested details, cleaned and ready to use.

✨ Best Practices

Be Specific

Vague prompts produce vague results. Instead of "Get data from this website," try: "Extract the product name, price, rating, and seller from each product card."

Define the Format

Specify your output structure upfront:

Output as JSON with keys: title (string), price (number), condition (string), url (string)

Handle Missing Data

Real pages are messy. Tell the agent what to do when fields don't exist:

If a field is missing, set it to null. If a product is out of stock, still include it but add availability: false.

💡 Why AI Scraping Works Better

✅ Zero Code — Describe the data you need. Skip the CSS selectors and XPath expressions.

✅ Adapts to Changes — When sites update their layouts, the agent adjusts. It interprets content rather than relying on fragile selectors.

✅ Smart Extraction — Prices get recognized regardless of format. Seller details get captured even when positioned differently across listings.

✅ Flexible Output — Want CSV instead of JSON? Just ask. Same prompt, different format.

✅ Complete Infrastructure — Crawlbase handles JavaScript rendering, proxy rotation, CAPTCHA bypassing, and session management in the background.

📊 Common Use Cases

Market Research — Monitor competitor pricing and inventory across multiple stores
Price Tracking — Automate regular price checks and get alerts when changes occur
Product Discovery — Identify trending items and bestsellers quickly
Data Collection — Build clean datasets without manual copy-paste
Content Aggregation — Compile catalogs from multiple sources efficiently

💰 Cost Comparison

Approach	Annual Cost
Traditional scraping	$8,000 - $25,000
AI-powered scraping	$600 - $4,000

Most teams save 70% to 90% while getting faster setup and zero maintenance.

🎯 Getting Started

The eBay example is a good starting point. Set up your Crawlbase account, enable Web MCP, open Cursor, and run the prompt. It takes a few minutes to see how much time this approach saves.

Once you've tested it, you can apply the same process to any site you need to scrape. The workflow stays consistent—only the prompt changes.

🤔 FAQ

What's the best web scraper for AI automation?
Crawlbase Web MCP combined with Claude or GPT-4 provides the most complete automation. It handles dynamic content while the LLM interprets and extracts data.

How do I make an AI web scraper?

Choose your tools (Crawlbase Web MCP + Claude/GPT API)
Load the page via Crawlbase
Extract HTML content
Send to LLM with extraction instructions
Parse the structured response
Loop through pages as needed

Can AI actually handle web scraping?
Yes. AI enhances scraping by understanding unstructured layouts, adapting to site changes, extracting semantic meaning, and handling variations automatically. It doesn't replace traditional scrapers but makes them more resilient and intelligent.

Found this helpful? Drop a ❤️ and let me know what you'd like to scrape next in the comments!

The Crawlbase Cyberweek Deals 2025

Crawlbase — Thu, 27 Nov 2025 12:46:42 +0000

Products Included in This CyberWeek Deal

1. Crawling API

Extract structured web data at scale. Fast, reliable, and pipeline-ready.

Scraping at scale without getting blocked is one of the biggest challenges for businesses and developers. Crawlbase's Crawling API addresses this by automatically handling CAPTCHA and browser fingerprinting, allowing you to focus on extracting data rather than battling anti-bot systems.

Key Features:

Automatic CAPTCHA bypass and proxy rotation
JavaScript rendering for dynamic websites
99.9% uptime with global proxy pool
Simple API integration
CyberWeek Deal: Get 15% OFF your next month’s billing
Coupon: CRAWL15
Validity: November 26 - December 2, 2025 (Only 7 Days!)

Get Started → - View Pricing →

2. Smart AI Proxy

Bypass restrictions and scale your requests seamlessly with intelligent proxy rotation.

Traditional proxies fail when websites become more sophisticated. Crawlbase's Smart AI Proxy adapts to anti-scraping measures in real-time, ensuring your requests always get through, even on the most challenging websites.

Key Features:

AI-powered request routing and retry logic
Residential, datacenter, and mobile IP pools
Automatic session management
Geo-targeting capabilities
CyberWeek Deal: Get 15% OFF your next month’s billing
Coupon: CRAWL15
Validity: November 26 - December 2, 2025 (Only 7 Days!)

Get Started → - View Pricing →

3. Crawler

Maximize web crawling with async, high-success automation; trusted for massive-scale projects.

Crawlbase Crawler is your asynchronous scraping solution that handles large-scale data extraction with zero infrastructure headaches.

Key Features:

Fully asynchronous crawling
Anonymous scraping with automatic IP rotation
Webhook notifications when jobs complete
JSON output with structured data
CyberWeek Deal: Get 15% OFF your next month’s billing
Coupon: CRAWL15
Validity: November 26 - December 2, 2025 (Only 7 Days!)

Get Started → - View Pricing →

4. Cloud Storage

Securely store and access all your crawled data in one scalable hub.

Crawlbase Cloud Storage is your best secured and scalable option to store scraped data. Unlike other web scraping providers that leave web-scraped data to you, Cloud Storage keeps everything under one roof, enabling you to scrape, store, search, and reuse data anytime with no extra setup.

Key Features:

Automatic data retention and backup
Fast retrieval via API or dashboard
Pay only for what you use
Enterprise-grade security and encryption
CyberWeek Deal: Get 15% OFF your next month’s billing
Coupon: CRAWL15
Validity: November 26 - December 2, 2025 (Only 7 Days!)

Get Started → - View Pricing →

5. LinkedIn Scraper

Automate professional data collection to fuel leads, insights, and growth.

LinkedIn is a goldmine for B2B leads, recruitment, and market intelligence, but scraping it can be challenging. Crawlbase's LinkedIn Scraper makes it simple to extract profiles, company data, and job postings without getting banned.

Key Features:

Extract profiles, companies, jobs, and posts
No LinkedIn account required
Bypass rate limits and anti-scraping measures
Structured JSON output
Real-time and batch scraping options
CyberWeek Deal: Get 15% OFF your next month’s billing
Coupon: CRAWL15
Validity: November 26 - December 2, 2025 (Only 7 Days!)

Get Started → - View Pricing →

Don't Miss Out – Act Now!

This 15% discount is our biggest offer of the year, and it's only available for 7 days. This is the perfect time to take advantage of these deals and scale your data extraction this season.

What happens after December 2?

The discount disappears. Prices return to normal. You'll have to wait another year for this deal.

Claim your 15% discount now before it's too late!

Get Started with Crawlbase

Crawlbase MCP Server for Real-Time AI Web Scraping

Crawlbase — Mon, 22 Sep 2025 11:02:52 +0000

Large Language Models (LLMs) like Claude, ChatGPT, and others are great at generating text, answering questions, and simulating intelligent behavior. But when it comes to real-time data from the web, they fall short, causing developers to be stuck with:

Manually pasting crawled/scraped results into prompts
Hallucinations from missing or outdated context
Code editors suggesting fixes without runtime awareness
Agents that break the moment your data updates

Why? Because LLMs are fundamentally disconnected from the live web.

The Crawlbase MCP Server is the missing link between artificial intelligence and real-world data. It empowers your AI tools and autonomous agents to securely and reliably fetch, parse, and act on live web information.

What you’ll learn in this AI web scraping guide

How to integrate the Crawlbase MCP Server with tools like Claude Desktop, Cursor, and Windsurf.

The LLM Bottleneck: Why AI Agents Struggle with Live Web Data

At the heart of every LLM, from Claude to ChatGPT, is a massive static training dataset. That means while these models can reason, respond, and predict, they can’t observe. They don’t have live access to the changing world around them.

That’s because:

LLMs are not browsers
They operate in secure, sandboxed environments that restrict outbound web access.
Their knowledge is frozen in time, and updates occur only during occasional retraining.

Why Model Context Protocol (MCP) Matters

To address this disconnect, you can utilize the Model Context Protocol (MCP), a standardized method that enables AI models and external tools to communicate effectively.

Think of it like USB for AI.

Just as USB made it easy to plug any device into any computer, MCP makes it easy for AI agents to integrate with any tool or data source, including live web sources.

MCP defines a consistent interface for LLMs to request and retrieve context from external systems, and that’s where the Crawlbase MCP Server comes in.

How MCP Unlocks Real-Time Web Access

By speaking the MCP protocol, the Crawlbase Server becomes a plug-and-play bridge between AI models and live web content. AI tools like Claude Desktop, Cursor, and Windsurf can now:

Request URLs or search queries
Get real-time, structured web data in return
Inject that data back into the model’s context window for reasoning and response

What is the Crawlbase MCP Server?

The Crawlbase Model Context Protocol (MCP) Server is the connective tissue between LLM agents and the real-time web.

Built on top of Crawlbase’s proven scraping infrastructure (used by over 70,000 developers), it enables AI tools like Claude Desktop, Cursor, and Windsurf to tap directly into fresh, structured web data — without encountering blocks, rate limits, or hallucinations. It plugs directly into tools that support the Model Context Protocol (MCP) and handles:

Real-time web scraping
JavaScript rendering
Proxy rotation and anti-bot evasion
Structured output for seamless LLM integration

How to Get Started with Crawlbase MCP Server

The Crawlbase MCP Server serves as your launchpad for real-time intelligence, enabling the development of AI agents, streamlining research, and enhancing productivity. Here’s how to get started:

Step 1: Get your Crawlbase Tokens

First off, secure your account with Crawlbase to get your first 1,000 requests free and extra 9,000 when you add your credit card. Sign up, go to your account documentation, and save a copy of your Crawling API Normal and JavaScript tokens.

Step 2: Integrate Crawlbase MCP Server for AI Web Scraping

Visit the GitHub Crawlbase repository page. You can find the code and documentation for the Crawlbase MCP Server here.

Run Context Commands

These are the special Crawlbase commands you can use to get and work with web content in different ways for use in LLM contexts once the MCP server is installed.

crawl - Crawl a URL and return HTML
crawl_markdown - Extract clean markdown from a URL
crawl_screenshot - Take a screenshot of a webpage

Crawlbase MCP Setup in Claude Desktop

Step 1: Open Claude Desktop → File → Settings → Developer → Edit Config

Step 2: Copy the Crawlbase MCP, then paste it in the `claude_desktop_config.json` file

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Be sure to replace your_token_here and your_js_token_here with your actual Crawlbase tokens in the configuration file.

Step 3: Save the config file and restart Claude Desktop.

If you return to the settings, Crawlbase MCP will appear under Local MCP servers.

Step 4: Utilizing the MCP

You’re now ready to use Crawlbase MCP. To begin, simply enter a prompt like:

“Crawl New York Times and return markdown”.

If a confirmation dialog for using Crawlbase MCP appears, be sure to grant permission when prompted.

Claude replies with the output, formatted in markdown.

Crawlbase MCP Setup in Cursor IDE

Step 1: Open Cursor IDE → File → Preferences → Cursor Settings → Tools and Integrations → Add Custom MCP

Step 2: Copy the Crawlbase MCP, then paste it in the `mcp.json` file

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Step 3: Once the config file is saved, an indicator will confirm that the Crawlbase MCP is active.

Note: Restart Cursor if you don’t see this indicator after saving the file.

Step 4: Utilize the Chat window to send commands to the Crawlbase MCP.

You're all set to start using Crawlbase MCP. Try entering something like:

“Crawl New York Times and save as markdown”

You may see a confirmation button; simply click it to proceed.

Below is the result generated from the prompt. As shown, Cursor created a markdown file and saved the output to it.

As you can see, Cursor delegates the live crawling task to the Crawlbase MCP server.

Crawlbase MCP Setup in WindSurf

Step 1: Open WindSurf IDE → File → Preferences → WindSurf Settings → General → MCP Servers → Manage MCPs → View raw config

Step 2: Copy the Crawlbase MCP, then paste it in the `mcp_config.json` file

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Be sure to replace your_token_here and your_js_token_here with your actual Crawlbase tokens in the configuration file.

Step 3: Save the config file and hit Refresh

The Crawlbase MCP should appear in the list of MCP servers.

Step 4: Use the Chat window to send commands to the Crawlbase MCP.

Now that everything’s set up, we’ll use the same prompt as before:

“Crawl New York Times and save as markdown”

Here’s what the prompt produced—Windsurf generated a markdown file and saved the results.

As shown once more, Windsurf hands off live crawling to the Crawlbase MCP server.

That's it, now your LLMs are capable of navigating and searching the internet without getting blocked.

Don’t let your agents work blind. Give them the power to see, learn, and respond with live data. Sign up on Crawlbase today and start building AI that’s truly connected to the world.

How to Crawl Apple App Store Data Without Getting Blocked

Crawlbase — Mon, 25 Aug 2025 13:05:13 +0000

This blog was initially posted to Crawlbase Blog

How to Scrape Apple App Store Data?

Our first step is to create an account with Crawlbase, which will enable us to utilize the Crawling API and serve as our platform for reliably fetching data from the App Store.

Creating a Crawlbase account

Sign up for a Crawlbase account and log in.
Once registered, you’ll receive 1,000 free requests. Add your billing details before using any of the free credits to get an extra 9,000 requests.
Go to your Account Docs and save your Normal Request token for this blog’s purpose.

Setting up the Environment

Next, ensure that Node.js is installed on your device, as it is the backbone of our scraping script, providing a fast JavaScript runtime and access to essential libraries.

Installing Node on Windows:

Go to the official Node.js website and download the Long-Term Support (LTS) version for Windows.
Launch the installer and follow the prompts. Leave the default options selected.
Verify installation by opening a new Command Prompt and running the following commands:

node -v
npm -v

For macOS:

Go to [https://nodejs.org](https://nodejs.org/) and download the macOS installer (LTS).
Follow the installation wizard.
Open the Terminal and confirm the installation:

node -v
npm -v

For Linux (Ubuntu/Debian):

Open your terminal to add the NodeSource repository and install Node.js:

curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
sudo apt-get install -y nodejs

Verify your installation:

node -v
npm -v

Fetch Script

Grab the script below and save it with a .js extension, any IDE or a coding environment you like will work. Once you've saved it, double-check that all the necessary dependencies are installed in your Node.js setup. After that, you should be all set.

import { CrawlingAPI } from 'crawlbase';

const CRAWLBASE_NORMAL_TOKEN = '<Normal requests token>';
const URL = 'https://apps.apple.com/us/app/google-authenticator/id388497605';

async function crawlAppStore() {
  const api = new CrawlingAPI({ token: CRAWLBASE_NORMAL_TOKEN });
  const options = {
    userAgent: 'Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/30.0',
  };

  const response = await api.get(URL, options);

  if (response.statusCode !== 200) {
    throw new Error(`Request failed with status code: ${response.statusCode}`);
  }

  return response.body;
}

IMPORTANT: Make sure to replace <Normal requests token> with your actual Crawlbase normal request token before running the script.

This script shows how to use Crawlbase’s Crawling API to retrieve HTML content from the Apple App Store without getting blocked. Note that the response hasn’t been scraped yet. We still need to remove unnecessary elements, clean the data, and produce a parsed, structured response.

Locating specific CSS selectors

Now that you understand how to send a simple API request using Node.js, let’s locate the data we need from our target URL so we can later write code to clean and parse it.

The first thing you’ll notice is the main section at the top. It’s usually where we’ll find the most important details and is typically well-structured, making it an ideal target for scraping.

Go ahead and open your target URL and locate each selector. For example, let’s search for the title:

Take note of the .app-header__title and do the same for subtitle, seller, category, stars, rating, and price. Once that's done, this section is complete.

The process is pretty much the same for the rest of the page. Here’s another example: if you want to include the customer average rating in the Ratings and Reviews section, right-click on the data and select Inspect:

You know the gist. It should now be a piece of cake for you to locate the remaining data you need.

Parsing the HTML in Node.js

Now that you’re an expert in extracting the CSS selectors, it is time to build the code to parse the HTML. This is where Cheerio comes in. It is a lightweight and powerful library that enables us to select relevant data from the HTML source code within Node.js.

Start by creating your project folder and run:

npm init -y
npm install crawlbase lodash casenator cheerio

Import the Required Libraries

Then in your .js file, import the required libraries for this project, including Cheerio:

import _ from 'lodash';
import { CrawlingAPI } from 'crawlbase';
import { toCamelCase } from 'casenator';
import * as cheerio from 'cheerio';

Don’t forget to set up the Crawling API as well as the target website:

const CRAWLBASE_NORMAL_TOKEN = '<Normal requests token>';
const URL = 'https://apps.apple.com/us/app/google-authenticator/id388497605';

Functions for Scraping Apple Store Data

This is where we’ll use the CSS selectors we’ve collected earlier. Let’s write the part of the code that pulls the bits of information from the App Store page.

function scrapePrimaryAppDetails($) {
  let title = $('.app-header__title').text().trim();
  const titleBadge = $('.badge--product-title').text().trim();
  title = title.replace(titleBadge, '').trim();
  const subtitle = $('.app-header__subtitle').text().trim();
  const seller = $('.app-header__identity').text().trim();
  let category = null;
  try {
    category = $('.product-header__list__item a.inline-list__item').text().trim().split('in')[1].trim();
  } catch {
    category = null;
  }
  const stars = $('.we-star-rating').attr('aria-label');
  const rating = $('.we-rating-count').text().trim().split('•')[1].trim();
  const price = $('.app-header__list__item--price').text().trim();

  return { title, subtitle, seller, category, stars, rating, price };
}

Just like that, it will extract the title, subtitle, seller, category, star rating, overall ratings, and price.

From this point, you can add more functions for each section of the page. You can add the Preview Image and Description, as well as user reviews, etc.

Combine Everything in One Function

Once the scraper is complete, we need to combine everything in one function and print the result:

function scrapeAppStore(html) {
  const $ = cheerio.load(html);
  return {
    primaryAppDetails: scrapePrimaryAppDetails($),
    appPreviewAndDescription: scrapeAppPreviewAndDescription($),
    ratingsAndReviews: { reviews: scrapeRatingsAndReviews($) },
    informationSection: scrapeInformationSection($),
    relatedAppsAndRecommendations: scrapeRelatedAppsAndRecommendations($),
  };
}

Complete Code to Scrape Apple App Store Data

import _ from 'lodash';
import { CrawlingAPI } from 'crawlbase';
import { toCamelCase } from 'casenator';
import * as cheerio from 'cheerio';

const CRAWLBASE_NORMAL_TOKEN = '<Normal requests token>';
const URL = 'https://apps.apple.com/us/app/google-authenticator/id388497605';

async function crawlAppStore() {
  const api = new CrawlingAPI({ token: CRAWLBASE_NORMAL_TOKEN });
  const options = {
    userAgent: 'Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/30.0',
  };

  const response = await api.get(URL, options);

  if (response.statusCode !== 200) {
    throw new Error(`Request failed with status code: ${response.statusCode}`);
  }
  return response.body;
}

function scrapePrimaryAppDetails($) {
  let title = $('.app-header__title').text().trim();
  const titleBadge = $('.badge--product-title').text().trim();
  title = title.replace(titleBadge, '').trim();
  const subtitle = $('.app-header__subtitle').text().trim();
  const seller = $('.app-header__identity').text().trim();
  let category = null;
  try {
    category = $('.product-header__list__item a.inline-list__item').text().trim().split('in')[1].trim();
  } catch {
    category = null;
  }
  const stars = $('.we-star-rating').attr('aria-label');
  const rating = $('.we-rating-count').text().trim().split('•')[1].trim();
  const price = $('.app-header__list__item--price').text().trim();

  return { title, subtitle, seller, category, stars, rating, price };
}

function scrapeAppPreviewAndDescription($) {
  const sources = $('source').toArray();
  const imageUrl =
    sources
      .map((element) => $(element).attr('srcset'))
      .filter((srcset) => srcset)
      .map((srcset) => srcset.split(',')[0].trim().split(' ')[0])
      .find((url) => url) || null;
  let appDescription = $('.section__description').text().trim();
  appDescription = appDescription.replace(/^Description\s*/, '');

  return { imageUrl, appDescription };
}

function scrapeRatingsAndReviews($) {
  const reviews = [];
  $('.we-customer-review').each((index, element) => {
    const stars = $(element).find('.we-star-rating').attr('aria-label');
    const reviewerName = $(element).find('.we-customer-review__user').text().trim();
    const reviewTitle = $(element).find('.we-customer-review__title').text().trim();
    const fullReviewText = $(element).find('.we-customer-review__body').text().trim();
    const reviewDate = $(element).find('.we-customer-review__date').attr('datetime');
    reviews.push({ stars, reviewerName, reviewTitle, fullReviewText, reviewDate });
  });

  return reviews;
}

function scrapeInformationSection($) {
  const information = {};
  $('dl.information-list dt').each((index, element) => {
    const key = $(element).text().trim();
    const value = $(element).next('dd').text().trim();
    if (key && value) {
      const camelKey = toCamelCase(key);
      if (camelKey === 'languages') {
        information[camelKey] = _.uniq(value.split(',').map((item) => item.trim())).sort();
      } else if (camelKey === 'compatibility') {
        information[camelKey] = _.uniq(
          value
            .split('\n')
            .map((item) => item.trim())
            .filter((item) => item),
        ).sort();
      } else {
        information[camelKey] = value;
      }
    }
  });

  return information;
}

function scrapeRelatedAppsAndRecommendations($) {
  function extractAppsFromSection(headlineText) {
    const results = [];
    $('h2.section__headline').each((index, element) => {
      const currentHeadlineText = $(element).text().trim();
      if (currentHeadlineText === headlineText) {
        const parent = $(element).parent();
        const nextSibling = parent.next();
        nextSibling.find('a.we-lockup--in-app-shelf').each((appIndex, appElement) => {
          const appTitle = $(appElement).find('.we-lockup__title').text().trim();
          const appUrl = $(appElement).attr('href');
          if (appTitle && appUrl) {
            results.push({
              title: appTitle,
              url: appUrl,
            });
          }
        });
      }
    });

    return results;
  }

  return {
    developerApps: extractAppsFromSection('More By This Developer'),
    relatedApps: extractAppsFromSection('You Might Also Like'),
  };
}

function scrapeAppStore(html) {
  const $ = cheerio.load(html);
  const data = {
    primaryAppDetails: {
      ...scrapePrimaryAppDetails($),
    },
    appPreviewAndDescription: {
      ...scrapeAppPreviewAndDescription($),
    },
    ratingsAndReviews: {
      reviews: scrapeRatingsAndReviews($),
    },
    informationSection: {
      ...scrapeInformationSection($),
    },
    relatedAppsAndRecommendations: {
      ...scrapeRelatedAppsAndRecommendations($),
    },
  };

  return data;
}

const html = await crawlAppStore();
const data = scrapeAppStore(html);
console.log(JSON.stringify(data, null, 2));

And when you run your script:

npm run crawl

You’ll see the output in this structure:

This organized structure provides a solid foundation for further analysis, reporting, or visualization, regardless of your end goal.

Check out the complete code in our GitHub repository for this blog.

Scrape Apple Store Data with Crawlbase

Scraping the Apple App Store can provide valuable insights into how apps are presented, user responses, and the performance of competitors. With Crawlbase and a solid HTML parser like Cheerio, you can automate the extraction of Apple data and turn it into something actionable.

For tracking reviews, comparing prices, or just exploring the app ecosystem, this setup can save you time and effort while delivering the data you need.

Start your next scraping project now with Crawlbase’s Smart Proxy and Crawling API to avoid getting blocked!

How to Scrape Data Behind Login Pages Using Python

Crawlbase — Tue, 24 Jun 2025 12:20:54 +0000

This blog was initially posted to Crawlbase Blog

In this article, we’ll show you a unique way to extract your session cookies from a logged-in session and pass them to an API, allowing it to log in to a website and extract the data you need. The process may sound complicated, but give it a chance and see how Crawlbase can simplify the entire process for you - see it in action.

Authenticated Scraping with Python Requests Library

Let’s put our extracted cookies into action. First, ensure that your Python environment is completely set up. Install the latest Python version, use any of your preferred IDE, and install the Python Requests module. Once your environment is set up, we can proceed with the exercise.

Say we want to scrape this Facebook Hashtag Music page, and our goal is to scrape data from protected web pages if you try to open this using Chrome Incognito mode (without logging in to your Facebook account), you’ll get the sign-in page:

We can try to scrape this page manually by using Python alone to see what will happen. Create a file and name it scraping_with_crawlbase.py, then copy and paste the code below.

import requests
from requests.exceptions import RequestException

TARGET_URL = "https://www.facebook.com/hashtag/music"
HEADERS = {
  'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
  'accept-language': 'en-US,en',
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36',
  'sec-fetch-mode': 'navigate',
  'Cookie': '<cookies-goes-here>'
}
OUTPUT_FILE_NAME = "output.html"

try:
    response = requests.get(TARGET_URL, headers=HEADERS)
    response.raise_for_status()

    html_content = response.text
    with open(OUTPUT_FILE_NAME, "w", encoding="utf-8") as file:
        file.write(response.text)

    print(f"\nPage successfully saved to '{OUTPUT_FILE_NAME}'\n")

except RequestException as error:
    print(f"\n Failed to fetch the page: {error}\n")

Make sure to replace <cookies-goes-here> with the actual cookies you extracted from your Facebook account earlier and run the code using the command below.

python scraping_with_crawlbase.py

After running the script, open the output.html file. You’ll notice that the content looks blank or incomplete. If you inspect it, you’ll see that it's mostly unexecuted JavaScript.

Why? Because the data you’re looking for is loaded dynamically with JavaScript, and requests alone can’t execute JavaScript like a browser does.

So, how do we resolve this issue? That’s what we’ll cover in the next section.

Scraping Behind Login Using Crawlbase

Now that we’ve seen the limitations of using Python’s requests library alone. Let’s use Crawlbase to handle problems like rendering JavaScript and working behind login walls. Here’s how you can do it:

Step 1: Prepare Your Script. Create or update your scraping_with_crawlbase.py file with the following code:

import json
import requests
from requests.exceptions import RequestException

API_TOKEN = "<Javascript requests token>"
TARGET_URL = "https://www.facebook.com/hashtag/music"
SCRAPER = "facebook-hashtag"
COOKIES = """
<cookies-goes-here>
"""
COUNTRY = "US"

API_ENDPOINT = "https://api.crawlbase.com/"

params = {
    "token": API_TOKEN,
    "url": TARGET_URL,
    "scraper": SCRAPER,
    "cookies": COOKIES,
    "country": COUNTRY
}

try:
    response = requests.get(API_ENDPOINT, params=params)
    response.raise_for_status()

    json_string_content = response.text
    json_data = json.loads(json_string_content)
    pretty_json = json.dumps(json_data, indent=2)
    print(pretty_json)

except RequestException as error:
    print(f"\nFailed to fetch the page: {error}\n")

Step 2: Replace <Javascript requests token> with your Crawlbase JavaScript token. If you don’t have an account yet, sign up to Crawlbase to claim your free API requests.
Step 3: Replace <cookies-goes-here> with the same cookies you extracted earlier from your logged-in Facebook session.

Make sure the cookies are properly formatted. Otherwise, Crawlbase might reject them. According to the cookies documentation, the correct format should look like this:

cookies: key1=value1; key2=value2; key3=value3

Step 4: Now run the script using:

python scraping_with_crawlbase.py

If everything's set up correctly, you’ll see a clean JSON output printed in your terminal. This is the actual content from the Facebook hashtag page successfully scraped.

{
  "original_status": 200,
  "pc_status": 200,
  "url": "https://www.facebook.com/hashtag/music",
  "domain_complexity": "standard",
  "body": {
    "hashtag": "",
    "posts": [
      {
        "userName": "Dave Moffatt Music",
        "text": "You\u2019ll get by with a smileYou can\u2019t win at everything but you can try! @eraserheads_official #nevada #music #withasmile #song",
        "url": "https://www.facebook.com/hashtag/music?__cft__[0]=AZWbgQE-_wYwW47AUbqqhzfqC6moiJrxFQs7glnpepq5ibId2fvbkZe1E3UoNwI-Ywj4gaQp3qbQjOMGmNVD1fu4Ofx-uPcDfWPJGhRCtKrHKV1G-rXqg2mxRSzd93AL281FwDSfjERvTMkdWK6bZI_cJC_CxDD63x_K5WycyUe1lnt5kBwyBOdIk4z2jfeFeRCZASbYvSLGQS9eQ4GQh-c2&__tn__=%2CO%2CP-R#?bee",
        "dateTime": "oSspoenrdt0iS27g8ie7lm4c2gt19779f1mpraaec87et108um8 b3,7 56g",
        "likesCount": "",
        "sharesCount": "",
        "commentsCount": "",
        "links": [
          {
            "link": "https://www.facebook.com/hashtag/nevada?__eep__=6&__cft__[0]=AZWbgQE-_wYwW47AUbqqhzfqC6moiJrxFQs7glnpepq5ibId2fvbkZe1E3UoNwI-Ywj4gaQp3qbQjOMGmNVD1fu4Ofx-uPcDfWPJGhRCtKrHKV1G-rXqg2mxRSzd93AL281FwDSfjERvTMkdWK6bZI_cJC_CxDD63x_K5WycyUe1lnt5kBwyBOdIk4z2jfeFeRCZASbYvSLGQS9eQ4GQh-c2&__tn__=*NK-R",
            "text": "#nevada"
          },
          {
            "link": "https://www.facebook.com/hashtag/music?__eep__=6&__cft__[0]=AZWbgQE-_wYwW47AUbqqhzfqC6moiJrxFQs7glnpepq5ibId2fvbkZe1E3UoNwI-Ywj4gaQp3qbQjOMGmNVD1fu4Ofx-uPcDfWPJGhRCtKrHKV1G-rXqg2mxRSzd93AL281FwDSfjERvTMkdWK6bZI_cJC_CxDD63x_K5WycyUe1lnt5kBwyBOdIk4z2jfeFeRCZASbYvSLGQS9eQ4GQh-c2&__tn__=*NK-R",
            "text": "#music"
          },
          {
            "link": "https://www.facebook.com/hashtag/withasmile?__eep__=6&__cft__[0]=AZWbgQE-_wYwW47AUbqqhzfqC6moiJrxFQs7glnpepq5ibId2fvbkZe1E3UoNwI-Ywj4gaQp3qbQjOMGmNVD1fu4Ofx-uPcDfWPJGhRCtKrHKV1G-rXqg2mxRSzd93AL281FwDSfjERvTMkdWK6bZI_cJC_CxDD63x_K5WycyUe1lnt5kBwyBOdIk4z2jfeFeRCZASbYvSLGQS9eQ4GQh-c2&__tn__=*NK-R",
            "text": "#withasmile"
          },
          {
            "link": "https://www.facebook.com/hashtag/song?__eep__=6&__cft__[0]=AZWbgQE-_wYwW47AUbqqhzfqC6moiJrxFQs7glnpepq5ibId2fvbkZe1E3UoNwI-Ywj4gaQp3qbQjOMGmNVD1fu4Ofx-uPcDfWPJGhRCtKrHKV1G-rXqg2mxRSzd93AL281FwDSfjERvTMkdWK6bZI_cJC_CxDD63x_K5WycyUe1lnt5kBwyBOdIk4z2jfeFeRCZASbYvSLGQS9eQ4GQh-c2&__tn__=*NK-R",
            "text": "#song"
          }
        ]
      }
      // Note: some results have been omitted for brevity.
    ]
  }
}

Bonus Step: The Crawlbase Facebook datascraper isn’t limited to scraping just hashtag pages. It also supports other types of Facebook content. So, if your target page falls into one of the categories below, you’re in luck:
- facebook-group
- facebook-page
- facebook-profile
- facebook-event

All you have to do is update two lines in your script to match the type of page you want to scrape:

TARGET_URL = "https://www.facebook.com/hashtag/music"
SCRAPER = "facebook-hashtag"

For example, if you want to scrape a private Facebook group, change it to something like:

TARGET_URL = "https://www.facebook.com/groups/examplegroup"
SCRAPER = "facebook-group"

Just swap in the correct URL and corresponding web scraper name and Crawlbase will take care of the rest.

How to Use Web Scraping for Price Intelligence

Crawlbase — Mon, 09 Jun 2025 10:43:56 +0000

This blog was initially posted to Crawlbase Blog

In this age, it is a necessity to have some automation when handling a wide-scope and complex task such as price intelligence. So, in this post, we’ll help you automate the process and guide you on how to build a pricing scraper with Python and Crawlbase that can make this task faster, easier, and more reliable.

Selecting Target Websites (Amazon and eBay)

Before you start writing code, it’s important to plan ahead: decide which websites you want to target, what data points you need to extract, and how you plan to use that data. Many developers naturally begin with popular sites like Amazon and eBay because they offer rich, diverse datasets that are valuable for applications like price intelligence.

For this coding exercise, we'll focus on the following SERP URLs as our targets:
Amazon
eBay

Set Up Your Workspace

At this point, you’ll need to choose your preferred programming language. For this example, we’ll use Python, as it’s widely considered one of the easiest languages to get started with. That said, you're free to use any other language; follow along and apply the same logic provided here.

Setup Your Coding Environment

Install Python 3 on your computer.
Create a root directory in our filesystem.
Under the root directory, create a file named requirements.txt, and add the following entries:

requests
tabulate
price_parser
pandas

Then run:

python -m pip install -r requirements.txt

Obtain Your API Credentials
1. Create an account at Crawlbase and log in
2. After registration, you will receive 1,000 free requests
3. Go to your account dashboard and copy your Crawling API Normal requests token

Sending Requests Using Crawlbase

Let’s start by building the scraper for the Amazon Search Engine Results Page. In the root directory of your project, create a file named amazon_serp_scraper.py, and follow along.

Add Import statements to your script.

import requests
import urllib.parse
from requests.exceptions import RequestException

Add function searches for products on Amazon using a given query, with optional parameters for the country code and top-level domain (such as .com or .co.uk), and returns a list of product dictionaries.

def get_products_from_amazon(query: str, country: str = None, top_level_domain: str = 'com') -> list[dict]:

Sets up your API token and Crawlbase endpoint to send the request.

API_TOKEN = "<Normal requests token>"
API_ENDPOINT = "https://api.crawlbase.com/"

Construct the parameters for the GET request by encoding the search query for the Amazon URL and specifying the Amazon SERP scraper along with an optional country parameter.

params = {
    "token": API_TOKEN,
    "url": f"https://www.amazon.{top_level_domain}/s?k={urllib.parse.quote_plus(query)}",
    "scraper": "amazon-serp",
    "country": country
}

Send the request to the Crawling API, then parse and return the product data.

 response = requests.get(API_ENDPOINT, params=params)
 response.raise_for_status()

 result = response.json()
 return result['body']['products']

To execute the program, insert the following code snippet to the end of the script:

if __name__ == "__main__":

    import json

    products = get_products_from_amazon("Apple iPhone 15 Pro Max 256GB", country="US", top_level_domain="co.uk")
    pretty_json = json.dumps(products, indent=2)
    print(pretty_json)

You can modify the values in the function call to suit your needs. For example, replace "Apple iPhone 15 Pro Max 256GB" with any other product name, change "US" to a different country code, or update "co.uk" to another Amazon domain such as "com" or "de".

Complete code example:

import requests
import urllib.parse
from requests.exceptions import RequestException

def get_products_from_amazon(query: str, country: str = None, top_level_domain: str = 'com') -> list[dict]:
    API_TOKEN = "<Normal requests token>"
    API_ENDPOINT = "https://api.crawlbase.com/"

    params = {
        "token": API_TOKEN,
        "url": f"https://www.amazon.{top_level_domain}/s?k={urllib.parse.quote_plus(query)}",
        "scraper": "amazon-serp",
        "country": country
    }

    response = requests.get(API_ENDPOINT, params=params)
    response.raise_for_status()

    result = response.json()

    return result['body']['products']

if __name__ == "__main__":

    import json

    products = get_products_from_amazon("Apple iPhone 15 Pro Max 256GB", country="US", top_level_domain="co.uk")
    pretty_json = json.dumps(products, indent=2)
    print(pretty_json)

Run the script from the terminal using the following command:

python amazon_serp_scraper.py

Once successful, you will see the following raw data output:

[
  {
    "name": "Apple iPhone 16 Pro Max 256GB: 5G Mobile phone with Apple Intelligence - Desert Titanium + Silicone Case with MagSafe - Black",
    "price": "\u00a31,138.00",
    "rawPrice": 1138.0,
    "currency": "\u00a3",
    "offer": "",
    "customerReview": "4.4 out of 5 stars",
    "customerReviewCount": "367",
    "shippingMessage": "",
    "asin": "B0DGTJ6Y1S",
    "image": "https://m.media-amazon.com/images/I/61EpQCNARNL._AC_UY218_.jpg",
    "url": "https://www.amazon.co.uk/Apple-iPhone-Pro-Max-256GB/dp/B0DGTJ6Y1S/ref=sr_1_1?dib=eyJ2IjoiMSJ9.ZChvrJybm7TlnZ2-2tQPcAJhiEM1rKrU7CkKwkwiDWAnDvRZyGd490ktAc-ukHTMrhCNjGZN-mUv_pB9jpM_b-kakh857EvjHVDsbaPlnqdaWdgP8h8JlYlqZZnnW7Y8aaJ_8IdO3jTMYnwEljkT641W-0jpmOwTktsR4YGToL3KgkE6J14jT_5xU3EZNFkl_L1IYUL72a1mwtfhDapB17WcNOKS6lxZeGSha2Sw1BA.ZYtrbxfYI-d4vIcsxiU9hG5ahBmcc5rtSNtz9VB-nc0&dib_tag=se&keywords=Apple+iPhone+15+Pro+Max+256GB&qid=1748790249&sr=8-1",
    "isPrime": false,
    "sponsoredAd": false,
    "couponInfo": "",
    "badgesInfo": [],
    "boughtInfo": ""
  },
  // Note: some results have been omitted for brevity.
  {
    "name": "Apple iPhone 11 Pro Max, 256GB, Midnight Green (Renewed)",
    "price": "\u00a3264.00",
    "rawPrice": 264.0,
    "currency": "\u00a3",
    "offer": "",
    "customerReview": "4.1 out of 5 stars",
    "customerReviewCount": "258",
    "shippingMessage": "",
    "asin": "B082BGTHP6",
    "image": "https://m.media-amazon.com/images/I/71g5LVVdbaL._AC_UY218_.jpg",
    "url": "https://www.amazon.co.uk/Apple-iPhone-256GB-Midnight-Renewed/dp/B082BGTHP6/ref=sr_1_16?dib=eyJ2IjoiMSJ9.ZChvrJybm7TlnZ2-2tQPcAJhiEM1rKrU7CkKwkwiDWAnDvRZyGd490ktAc-ukHTMrhCNjGZN-mUv_pB9jpM_b-kakh857EvjHVDsbaPlnqdaWdgP8h8JlYlqZZnnW7Y8aaJ_8IdO3jTMYnwEljkT641W-0jpmOwTktsR4YGToL3KgkE6J14jT_5xU3EZNFkl_L1IYUL72a1mwtfhDapB17WcNOKS6lxZeGSha2Sw1BA.ZYtrbxfYI-d4vIcsxiU9hG5ahBmcc5rtSNtz9VB-nc0&dib_tag=se&keywords=Apple+iPhone+15+Pro+Max+256GB&qid=1748790249&sr=8-16",
    "isPrime": false,
    "sponsoredAd": false,
    "couponInfo": "",
    "badgesInfo": [],
    "boughtInfo": ""
  }
]

Next, create a new file named ebay_serp_scraper.py for the eBay integration, and add the following code using the same approach we applied in the Amazon scraper.

import requests
import urllib.parse
from requests.exceptions import RequestException

def get_products_from_ebay(query: str, country: str = None, top_level_domain: str = 'com') -> list[dict]:
    API_TOKEN = "<Normal requests token>"
    API_ENDPOINT = "https://api.crawlbase.com/"

    params = {
        "token": API_TOKEN,
        "url": f"https://www.ebay.{top_level_domain}/sch/i.html?_nkw={urllib.parse.quote_plus(query)}",
        "scraper": "ebay-serp",
        "country": country
    }

    response = requests.get(API_ENDPOINT, params=params)
    response.raise_for_status()

    result = response.json()

    return result['body']['products']

if __name__ == "__main__":

    import json

    products = get_products_from_ebay("Apple iPhone 15 Pro Max 256GB", country="US", top_level_domain="co.uk")
    pretty_json = json.dumps(products, indent=2)
    print(pretty_json)

Once you run the script by running python ebay_serp_scraper.py in your terminal, you should see the raw output as shown below:

[
  {
    "title": "New listingiPhone 15 Pro Max - 256gb Blue Titanium",
    "subTitles": ["Pre-owned"],
    "price": {
      "current": {
        "from": "\u00a3513.93",
        "to": "\u00a3513.93"
      },
      "trendingPrice": null,
      "previousPrice": ""
    },
    "soldDate": "",
    "endDate": "",
    "bidsCount": 0,
    "hotness": "",
    "additionalHotness": [],
    "customerReviews": {
      "review": "",
      "count": 0,
      "link": ""
    },
    "shippingMessage": "Postage not specified",
    "image": "https://i.ebayimg.com/images/g/cQkAAOSww3toOdUj/s-l500.webp",
    "url": "https://www.ebay.co.uk/itm/146617243528?_skw=Apple+iPhone+15+Pro+Max+256GB&itmmeta=01JWP0SV257V8N99DP0GB2B2FK&hash=item2223119788:g:cQkAAOSww3toOdUj&itmprp=enc%3AAQAKAAAA4FkggFvd1GGDu0w3yXCmi1e4U9NgN8cbPDciCpZxd4y9M7arS1PIV503raP8NGkFrAGVGTcTJ93CtRBBBR%2BVdNPFCvqlvQFqv0p54lMQYaGfzNm2BDkKB8pOYhDKGV44h10dcGqP8Txe9bOa2%2BxUU4c03zlQjb5BBp5tZ6gfqvtXuJtaf6xW09Np944o3hzvtrVoM%2Fv9BcdTnUznCeLvNxWNQB9IG4%2BYEFyqvvcZce%2FbSdKA4hmdFerIaeAjWlWEpu7%2F1ZZV9ElPz9yibQZ6YcvruBvpIS3YYlvAypHn6yp8%7Ctkp%3ABk9SR5yx58DlZQ",
    "location": "United Kingdom",
    "time": {
      "timeLeft": "6d 23h left",
      "timeEnd": "(Sun, 08:00)"
    },
    "listingDate": "",
    "topRatedSeller": false,
    "sponsoredAd": false,
    "sellerInfo": "davidawilliams1 (2,001) 100%"
  },
  // Note: some results have been omitted for brevity.
  {
    "title": "Great Condition - Apple iPhone 15 Pro Max 256GB Blue Titanium 98% Battery -1047-",
    "subTitles": ["Pre-owned"],
    "price": {
      "current": {
        "from": "\u00a3709.00",
        "to": "\u00a3709.00"
      },
      "trendingPrice": null,
      "previousPrice": ""
    },
    "soldDate": "",
    "endDate": "",
    "bidsCount": 0,
    "hotness": "20 watchers",
    "additionalHotness": [],
    "customerReviews": {
      "review": "",
      "count": 19,
      "link": "https://www.ebay.co.uk/p/24062761146?iid=236096139018#UserReviews"
    },
    "shippingMessage": "Postage not specified",
    "image": "https://i.ebayimg.com/images/g/xIEAAeSw-NhoILdt/s-l140.webp",
    "url": "https://www.ebay.co.uk/itm/236096139018?_skw=Apple+iPhone+15+Pro+Max+256GB&epid=24062761146&itmmeta=01JWP0SV29B7VZRTR5336M14WJ&hash=item36f86d2f0a:g:xIEAAeSw-NhoILdt&itmprp=enc%3AAQAKAAABAFkggFvd1GGDu0w3yXCmi1d4wiok8NH9ED6XupF4lEB05soO%2BX9YmeNcvnhAT2xl8%2BMR4mTSpPbdLrOdQTzvZweJCy9AbPRRp%2F3fEqBcyd%2B5KAg80QbLwBoSqH0%2BeYox7n1qscvQrlMrB71D%2FNgjlJfHccWLZ%2FGngbtsV5ccazqhwfxQdUKd0i%2BgDc6vXrrD%2F5SrwBFP5B7By2Vao286t0uJFGHFt28Hit8Si6T2mAYp5VdmEJN4YPv0xAlEwjmiPZ94WWHtYHGzBfZgdg9pRZHngDYjfm0iVEkcLab9c3G5xbhlMKPjRvjgC8DcoRUnHD%2FJSfIkRTLH0PzFBsjmyTw%3D%7Ctkp%3ABFBMpLHnwOVl",
    "location": "United Kingdom",
    "time": {
      "timeLeft": "",
      "timeEnd": ""
    },
    "listingDate": "",
    "topRatedSeller": false,
    "sponsoredAd": true,
    "sellerInfo": "2mselectronics (728) 100%"
  }
]

Extracting and Structuring Price Data

While the current API output is clean and provides valuable information, not all of it is relevant to our specific goal in this context. In this section, we'll demonstrate how to further standardize and parse the data to extract only the specific data points you need.

Let’s create a new file named structured_consolidated_data.py and insert the following code:

from amazon_serp_scraper import get_products_from_amazon
from ebay_serp_scraper import get_products_from_ebay

def get_structured_consolidated_data(query: str, country: str = None, top_level_domain: str = 'com') -> list[dict]:
    products = []

    amazon_data = get_products_from_amazon(query, country=country, top_level_domain=top_level_domain)
    products.extend([{"product": item['name'], "price": item['price'], "url": item['url'], "source": "Amazon"} for item in amazon_data])

    ebay_data = get_products_from_ebay(query, country=country, top_level_domain=top_level_domain)
    products.extend([{"product": item['title'], "price": item['price']['current']['to'].strip(), "url": item['url'], "source": "Ebay"} for item in ebay_data])

    return products

This script pulls relevant product data from Amazon and eBay from the Crawlbase data scrapers. It extracts and combines key details such as source, product, price, and URL from each search result into a single, easy-to-use list.

Here is the complete code:

from amazon_serp_scraper import get_products_from_amazon
from ebay_serp_scraper import get_products_from_ebay

def get_structured_consolidated_data(query: str, country: str = None, top_level_domain: str = 'com') -> list[dict]:
    products = []

    amazon_data = get_products_from_amazon(query, country=country, top_level_domain=top_level_domain)
    products.extend([{"product": item['name'], "price": item['price'], "url": item['url'], "source": "Amazon"} for item in amazon_data])

    ebay_data = get_products_from_ebay(query, country=country, top_level_domain=top_level_domain)
    products.extend([{"product": item['title'], "price": item['price']['current']['to'].strip(), "url": item['url'], "source": "Ebay"} for item in ebay_data])

    return products

if __name__ == "__main__":

    import json

    products = get_structured_consolidated_data("Apple iPhone 15 Pro Max 256GB", country="US", top_level_domain="co.uk")

    pretty_json = json.dumps(products, indent=2)
    print(pretty_json)

Run the script from the terminal using the following command:

python structured_consolidated_data.py

Once you get a successful response, the consolidated output will be displayed in a structured format, showing the source field instead of raw data.

[
  {
    "name": "Apple iPhone 16 Pro Max 256GB: 5G Mobile phone with Apple Intelligence - Desert Titanium + Silicone Case with MagSafe - Black",
    "price": "\u00a31,138.00",
    "url": "https://www.amazon.co.uk/Apple-iPhone-Pro-Max-256GB/dp/B0DGTJ6Y1S/ref=sr_1_1?dib=eyJ2IjoiMSJ9.ZChvrJybm7TlnZ2-2tQPcAJhiEM1rKrU7CkKwkwiDWAnDvRZyGd490ktAc-ukHTMrhCNjGZN-mUv_pB9jpM_b-kakh857EvjHVDsbaPlnqdaWdgP8h8JlYlqZZnnW7Y8aaJ_8IdO3jTMYnwEljkT641W-0jpmOwTktsR4YGToL3KgkE6J14jT_5xU3EZNFkl_L1IYUL72a1mwtfhDapB17WcNOKS6lxZeGSha2Sw1BA.ZYtrbxfYI-d4vIcsxiU9hG5ahBmcc5rtSNtz9VB-nc0&dib_tag=se&keywords=Apple+iPhone+15+Pro+Max+256GB&qid=1748791075&sr=8-1",
    "source": "Amazon"
  },
  // Note: some results have been omitted for brevity.
  {
    "name": "Apple iPhone 15 Pro Max (256 GB) - White Titanium - Good Condition",
    "price": "\u00a3679.99",
    "url": "https://www.ebay.co.uk/itm/205475768012?_skw=Apple+iPhone+15+Pro+Max+256GB&itmmeta=01JWP14FHS44GRVT9KYP8AFKEW&hash=item2fd74f66cc:g:QkcAAOSwLKRoHd86&itmprp=enc%3AAQAKAAABAFkggFvd1GGDu0w3yXCmi1enej%2BIHaZBwUjnCYNkoIrYanJLRykGLG546KgFE4C%2BH%2FGVT3ptDyAFH87uYJ2y6Ih4qylSr70KgmTvf7QzxWJJb8UIuLl9GWlI4h4QLVbnS26iLFU08zLSz8kbcbyI5kILO9IRzzTpKec0Cxb4G8ujEojvnrdM8G3oP5ud4QwSccYRK7L8PnDvS7qECHgMXmshCmZh749EOMqeDYRFSCqmPYQ6etMUr0y38Wag%2BT%2BLOIkx8XxR3fTC4FbbMPGGUDdNpG1jLJ3e%2F6X9tZuQuDp4lprfyjTKWD564verk%2FxhORgHzaHvDhmmEE121dibsOU%3D%7Ctkp%3ABFBMgvmRweVl",
    "source": "Ebay"
  }
]

Make Use of the Scraped Data

Now that we've successfully parsed the relevant data for our needs, we can begin applying it to basic pricing intelligence. For example, we can write a script to calculate and show the average price of the "Apple iPhone 15 Pro Max 256GB" listed on Amazon and compare it to the average price on eBay.

Create a new file named market_average_price.py and insert the following code:

import pandas as pd
from price_parser import Price
from structured_consolidated_data import get_structured_consolidated_data

products = get_structured_consolidated_data("Apple iPhone 15 Pro Max 256GB", country="US", top_level_domain="co.uk")
products_with_prices = filter(lambda product: (product["price"] is not None and product["price"].strip() != ''), products)
sanitized_products = [product | {"price": float(Price.fromstring(product["price"]).amount) } for product in products_with_prices]

data = list(sanitized_products)
df = pd.DataFrame(data)

# Check data types and examine the source column
iphone_mask = df['name'].str.contains('iPhone', case=False, na=False)

# Calculate average prices for iPhone models by source
iphone_df = df[df['name'].str.contains('iPhone', case=False, na=False)]

# Group by source and calculate average prices
avg_prices = iphone_df.groupby('source')['price'].agg(['mean', 'count']).round(2)
avg_prices.columns = ['Average Price (£)', 'Number of Products']

print("\n\nAverage iPhone prices by source:")
print(avg_prices)

# Calculate the difference
amazon_avg = avg_prices.loc['Amazon', 'Average Price (£)']
ebay_avg = avg_prices.loc['Ebay', 'Average Price (£)']
difference = amazon_avg - ebay_avg

print("\n\nPrice comparison:")
print(f"Amazon average: £{amazon_avg}")
print(f"eBay average: £{ebay_avg}")
print(f"Difference: £{difference:.2f} (Amazon is {'higher' if difference > 0 else 'lower'} than eBay)\n\n")

To run the script, execute the following command in your terminal:

python market_average_price.py

Once successful, you’ll see output similar to the following:

Explore the full codebase on GitHub.

How to Unblock Amazon with Smart Proxy

Crawlbase — Mon, 09 Jun 2025 10:35:10 +0000

This blog was initially posted to Crawlbase Blog

This guide will show you how to unblock Amazon proxy issues and scrape data reliably using Crawlbase’s Smart Proxy. A complete guide to scraping Amazon effectively.

Setting Up Your Coding Environment

Before building your Amazon proxy unblocker, you'll need to set up a basic Python environment. Here's how to get started:

Install Python 3 on your computer
Install requests module, which makes it easy to send HTTP requests in Python.

python -m pip install requests

Note: You can write and run your code using any text editor, but using an IDE can speed things up. Tools like PyCharm or VS Code are great for writing Python code, especially for beginners, as they include helpful features like syntax highlighting, error checking, and debugging tools.

Obtaining Credentials

Sign up for a Crawlbase account and log in to receive your 5,000 free requests
Get your Smart Proxy Private token

Making Your First Successful Request

At this point, your coding environment should be ready. Let’s try sending your first request.

In this example code, we'll try to fetch the HTML content of this Amazon product details page. You are free to copy this code, but make sure to replace the Private_token with the actual token or authentication key obtained from your Crawlbase account.

import requests
from urllib3.exceptions import InsecureRequestWarning

requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

url_to_crawl = "https://www.amazon.com/Apple-iPhone-Silicone-Case-MagSafe/dp/B0CHX2XFLN"

crawlbase_normal_token = "<Normal requests token>"
crawlbase_smart_proxy_url = (
    f"https://{crawlbase_normal_token}:@smartproxy.crawlbase.com:8013"
)

try:
    response = requests.get(
        url=url_to_crawl,
        proxies={
            "http": crawlbase_smart_proxy_url,
            "https": crawlbase_smart_proxy_url
        },
        verify=False,
        timeout=30,
    )
    response.raise_for_status()

    print("Response Code:", response.status_code)
    print("Response Body:", response.text)

except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")

You may refer to our GitHub repository for the source code.

Key Things to Know

Smart Proxy URL: The format https://<TOKEN>:@smartproxy.crawlbase.com:8013 is how authentication is handled. Your token is used as the username in the proxy connection.
verify=False: This disables SSL verification on the client side, which is required here because SSL is handled by the proxy itself, as noted in the Smart Proxy documentation.

Once you run this code, you should see a 200 response and the full HTML of the Amazon product page similar to the image below.

Unblocking Amazon with Smart Proxy: A Practical Use Case

Now, let’s apply what you’ve learned into practice. We’ll show you how to extract a list of reviews from an Amazon product page and save the data into a CSV file.

Extracting Specific Data

We'll use the Data Scraper feature from Crawlbase called Amazon-product-details Scraper via the CrawlbaseAPI-Parameters header. This allows our code to automatically parse the Amazon page and return clean, structured JSON data.

import requests
import json
from urllib3.exceptions import InsecureRequestWarning

requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

url_to_crawl = "https://www.amazon.com/Apple-iPhone-Silicone-Case-MagSafe/dp/B0CHX2XFLN"
crawlbase_normal_token = "<Normal requests token>"
crawlbase_crawling_api_parameters = "scraper=amazon-product-details"
crawlbase_smart_proxy_url = (
    f"https://{crawlbase_normal_token}:@smartproxy.crawlbase.com:8013"
)

try:
    response = requests.get(
        url=url_to_crawl,
        headers={"CrawlbaseAPI-Parameters": crawlbase_crawling_api_parameters},
        proxies={"http": crawlbase_smart_proxy_url, "https": crawlbase_smart_proxy_url},
        verify=False,
        timeout=30,
    )
    response.raise_for_status()

    json_data = json.loads(response.text)
    product_reviews = json_data["body"]["reviews"]

    for review in product_reviews:
        # TODO save the values here in a CSV file
        # but console print for now
        print("--------------------")
        print("Author: ", review["reviewerName"])
        print("Rating: ", review["reviewRating"])
        print("  Date: ", review["reviewDate"])
        print("Review: ", review["reviewText"])

except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")

You may refer to our GitHub repository for the source code.

How This Works

CrawlbaseAPI-Parameters: The scraper=amazon-product-details parameter tells Crawlbase to analyze the product page and return structured JSON that includes reviews, ratings, product info, etc.
Print JSON response: We extract the list of reviews from json_data["body"]["reviews"] and loop through them. For each product review, we print the Author, Rating, Date, and Review text.

Compiling Extracted Data into CSV

Lastly, you could easily modify the code to save the reviews to a CSV file for later analysis. Here's an example of how to save the data.

import requests
import json
import csv # newly added code
from urllib3.exceptions import InsecureRequestWarning

requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

url_to_crawl = "https://www.amazon.com/Apple-iPhone-Silicone-Case-MagSafe/dp/B0CHX2XFLN"
crawlbase_normal_token = "<Normal requests token>"
crawlbase_crawling_api_parameters = "scraper=amazon-product-details"
crawlbase_smart_proxy_url = (
    f"https://{crawlbase_normal_token}:@smartproxy.crawlbase.com:8013"
)

try:
    response = requests.get(
        url=url_to_crawl,
        headers={"CrawlbaseAPI-Parameters": crawlbase_crawling_api_parameters},
        proxies={"http": crawlbase_smart_proxy_url, "https": crawlbase_smart_proxy_url},
        verify=False,
        timeout=30,
    )
    response.raise_for_status()

    json_data = json.loads(response.text)
    product_reviews = json_data["body"]["reviews"]

    # start of newly replaced code
    with open("product_reviews.csv", "w", newline="") as file:
        writer = csv.writer(file)
        writer.writerow(["Author", "Rating", "Date", "Review"])  # Header
        for review in product_reviews:
            writer.writerow(
                [
                    review["reviewerName"],
                    review["reviewRating"],
                    review["reviewDate"],
                    review["reviewText"],
                ]
            )
    # end of newly replaced code

except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")

You may refer to our GitHub repository for the source code.

This simple snippet writes to a new CSV file named product_reviews.csv.

This is a basic use case for how to interact with Amazon’s product pages, and you can adapt the script for different tasks, such as extracting other product details like prices, ASIN values, and descriptions.

We've published the complete code for this solution on GitHub. You can view it here.

How to Scrape Google Hotels in Python

Crawlbase — Fri, 18 Apr 2025 11:57:28 +0000

This blog was initially posted to Crawlbase Blog

When planning trips, most travellers turn to Google to find hotels. The platform shows hotel listings, prices, reviews, and availability all in one place. For businesses, analysts, or travel platforms, this data is gold. Scraping Google Hotels can help you track pricing trends, monitor competitors, and analyze market opportunities in the travel industry.

In this guide, we’ll show you how to scrape Google Hotels using Python and the Crawlbase Crawling API. With this method, you can collect hotel data at scale without worrying about blocks, CAPTCHAs, or IP bans. We’ll cover everything from setting up your environment to writing a complete scraper for hotel listings and individual hotel pages.

Let’s start.

Key Data to Extract from Google Hotels

When scraping Google Hotels, it’s important to know which data points matter most. These details are useful for price monitoring, competitive analysis, and building travel tools.

The image below shows some of the most valuable fields you can extract:

Setting Up Your Python Environment

Before scraping Google Hotels, you need to prepare your Python environment. This includes installing Python itself and a few essential libraries for sending requests and extracting data.

🐍 Install Python

If you haven’t installed Python yet, download and install the latest version from the official Python website. During installation, make sure to check the box that says "Add Python to PATH" — this will let you run Python from the command line.

To check if Python is installed, run this in your terminal or command prompt:

python --version

You should see the installed version number.

✅ Install Required Libraries

To scrape Google Hotels, we’ll use:

carwlbase – to send HTTP requests use Crawlbase Crawling API.
beautifulsoup4 – to parse and extract content from HTML.

Install them using pip:

pip install requests
pip install beautifulsoup4

📝 Create Your Python File

Create new files where you’ll write your scraping code, for example:

touch google_hotels_listing_scraper.py
touch google_hotel_details_scraper.py

Or just create them manually in your preferred code editor.

🔑 Get Your Crawlbase Token

If you haven’t already, sign up at Crawlbase and get your API token. You’ll need this token to authenticate your scraping requests.

from crawlbase import CrawlingAPI

# Replace CRAWLBSE_JS_TOKEN with your actual token.
crawling_api = CrawlingAPI({ 'token': 'CRAWLBASE_JS_TOKEN' })

Note: Crawlbase provides two types of tokens. A normal token for static sites and a JS Token for JS-rendered sites. For Google Hotels scraping, we need a JS token. See the documentation for more.

Now, your setup is complete. Next, we’ll inspect the HTML structure of Google Hotels and start writing the scraper.

Scraping Google Hotels Search Results

In this section, we’ll scrape hotel listings from Google Hotels using Python, BeautifulSoup, and the Crawlbase Crawling API. You’ll learn how to extract hotel details, handle pagination, and save data into a JSON file.

🧩 Inspecting the HTML for Selectors

First, open Google Hotels in your browser, search a location (e.g., “New York”), and inspect the page.

Here are some key CSS classes used in the hotel listings:

Hotel card:div.BcKagd
Hotel name: h2.BgYkof
Price: span.qQOQpe.prxS3d
Rating: span.KFi5wf.lA0BZ

We’ll use these selectors in our scraper.

🧪 Writing the Hotels Listings Scraper

Now, let’s write a function to extract hotel data using Crawlbase and BeautifulSoup.

from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
import json

crawling_api = CrawlingAPI({ 'token': 'CRAWLBASE_JS_TOKEN' })

def make_crawlbase_request(url):
    response = crawling_api.get(url)
    if response['headers']['pc_status'] == '200':
        return response['body'].decode('utf-8')
    return None

def parse_hotel_listings(html):
    soup = BeautifulSoup(html, "html.parser")
    hotel_data = []

    hotels = soup.find_all("div", class_="BcKagd")
    for hotel in hotels:
        name = hotel.find("h2", class_="BgYkof")
        price = hotel.find("span", class_="qQOQpe prxS3d")
        rating = hotel.find("span", class_="KFi5wf lA0BZ")
        link = hotel.find("a", class_="PVOOXe")

        hotel_data.append({
            "name": name.text if name else "N/A",
            "price": price.text if price else "N/A",
            "rating": rating.text if rating else "N/A",
            "link": "https://www.google.com" + link["href"] if link else "N/A"
        })

    return hotel_data

🔁 Handling Pagination

Google Hotels loads more results across multiple pages. Using the Crawlbase Crawling API, we can simulate button clicks with the css_click_selector parameter. We can also use the ajax_wait parameter to make sure the content is fully loaded after the click. This ensures the Crawling API returns the full HTML of the next page after the button is clicked and the content is rendered.

Let’s update our make_crawlbase_request function to include these parameters and add exception handling for better reliability:

def make_crawlbase_request(url, css_click_element=None):
    try:
        options = {}

        if css_click_element:
            options['css_click_selector'] = css_click_element
            options['ajax_wait'] = 'true'

        response = crawling_api.get(url, options)
        if response['headers'].get('pc_status') == '200':
            return response['body'].decode('utf-8')

        return response

    except Exception as e:
        print(f"Error during Crawlbase request: {e}")
        return {}

💾 Saving Data in a JSON File

Once you’ve collected all the hotel data, save it to a JSON file:

def save_to_json(data, filename="google_hotels.json"):
    with open(filename, "w", encoding="utf-8") as f:
        json.dump(data, f, ensure_ascii=False, indent=2)

✅ Complete Code Example

Here is the complete code that combines all the steps above:

from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
import json

crawling_api = CrawlingAPI({ 'token': 'CRAWLBASE_JS_TOKEN' })

def make_crawlbase_request(url, css_click_element=None):
    try:
        options = {}

        if css_click_element:
            options['css_click_selector'] = css_click_element
            options['ajax_wait'] = 'true'

        response = crawling_api.get(url, options)
        if response['headers'].get('pc_status') == '200':
            return response['body'].decode('utf-8')

        return response

    except Exception as e:
        print(f"Error during Crawlbase request: {e}")
        return {}

def parse_hotel_listings(html):
    soup = BeautifulSoup(html, "html.parser")
    hotel_data = []

    hotels = soup.find_all("div", class_="BcKagd")
    for hotel in hotels:
        name = hotel.find("h2", class_="BgYkof")
        price = hotel.find("span", class_="qQOQpe prxS3d")
        rating = hotel.find("span", class_="KFi5wf lA0BZ")
        link = hotel.find("a", class_="PVOOXe")

        hotel_data.append({
            "name": name.text if name else "N/A",
            "price": price.text if price else "N/A",
            "rating": rating.text if rating else "N/A",
            "link": "https://www.google.com" + link["href"] if link else "N/A"
        })

    return hotel_data

def save_to_json(data, filename="google_hotels.json"):
    with open(filename, "w", encoding="utf-8") as f:
        json.dump(data, f, ensure_ascii=False, indent=2)

def main():
    url = "https://www.google.com/travel/hotels/New-York?q=New+York&currency=USD"
    all_hotels = []
    max_pages = 2
    page_count = 0

    while page_count < max_pages:
        html = ''

        if page_count == 0:
            # For 1st page
            html = make_crawlbase_request(url)
        else:
            # For next pages
            html = make_crawlbase_request(url, 'button[jsname="OCpkoe"]')

        if not html:
            break

        hotels = parse_hotel_listings(html)
        all_hotels.extend(hotels)

        page_count += 1

    save_to_json(all_hotels)
    print(f"Scraped {len(all_hotels)} hotels and saved to google_hotels.json")

if __name__ == "__main__":
    main()

Example Output:

[
  {
    "name": "31 Street Broadway Hotel",
    "price": "$59",
    "rating": "2.5",
    "link": "https://www.google.com/travel/search?q=New%20York&qs=MihDaG9JeFBLSXpvWDR6SWZMQVJvTkwyY3ZNVEZ3ZDJnMU4yYzFOUkFCOAA&currency=USD&ved=2ahUKEwiY1rucg9CMAxUIAPkAHXyaE5EQyvcEegQIAxA-&ap=KigKEgm4tF8JXhxEQBF5jsg3iI5SwBISCfZ7hYTLm0RAEXmOyLfKcVLA&ts=CAESCgoCCAMKAggDEAAaXAo-EjwKCS9tLzAyXzI4NjIlMHg4OWMyNGZhNWQzM2YwODNiOjB4YzgwYjhmMDZlMTc3ZmU2MjoITmV3IFlvcmsSGhIUCgcI6Q8QBBgQEgcI6Q8QBBgRGAEyAhAAKgcKBToDVVNE"
  },
  {
    "name": "The One Boutique Hotel",
    "price": "$90",
    "rating": "3.3",
    "link": "https://www.google.com/travel/search?q=New%20York&qs=MidDaGtJZ0t6dDBjdkZ6dG1jQVJvTUwyY3ZNWEUxWW14eWF6a3pFQUU4AA&currency=USD&ved=2ahUKEwiY1rucg9CMAxUIAPkAHXyaE5EQyvcEegQIAxBV&ap=KigKEgm4tF8JXhxEQBF5jsg3iI5SwBISCfZ7hYTLm0RAEXmOyLfKcVLA&ts=CAESCgoCCAMKAggDEAAaXAo-EjwKCS9tLzAyXzI4NjIlMHg4OWMyNGZhNWQzM2YwODNiOjB4YzgwYjhmMDZlMTc3ZmU2MjoITmV3IFlvcmsSGhIUCgcI6Q8QBBgQEgcI6Q8QBBgRGAEyAhAAKgcKBToDVVNE"
  },
  {
    "name": "Ly New York Hotel",
    "price": "$153",
    "rating": "4.4",
    "link": "https://www.google.com/travel/search?q=New%20York&qs=MihDaG9JbU9UeXpldUN6cnlrQVJvTkwyY3ZNVEYyY0d3MGJuSXpZaEFCOAA&currency=USD&ved=2ahUKEwiY1rucg9CMAxUIAPkAHXyaE5EQyvcEegQIAxBu&ap=KigKEgm4tF8JXhxEQBF5jsg3iI5SwBISCfZ7hYTLm0RAEXmOyLfKcVLA&ts=CAESCgoCCAMKAggDEAAaXAo-EjwKCS9tLzAyXzI4NjIlMHg4OWMyNGZhNWQzM2YwODNiOjB4YzgwYjhmMDZlMTc3ZmU2MjoITmV3IFlvcmsSGhIUCgcI6Q8QBBgQEgcI6Q8QBBgRGAEyAhAAKgcKBToDVVNE"
  },
  {
    "name": "King Hotel Brooklyn Sunset Park",
    "price": "$75",
    "rating": "3.4",
    "link": "https://www.google.com/travel/search?q=New%20York&qs=MihDaG9JbllMLW1iTG5uLTNDQVJvTkwyY3ZNVEZ5ZDNKNWQyUXdiQkFCOAA&currency=USD&ved=2ahUKEwiY1rucg9CMAxUIAPkAHXyaE5EQyvcEegUIAxCJAQ&ap=KigKEgm4tF8JXhxEQBF5jsg3iI5SwBISCfZ7hYTLm0RAEXmOyLfKcVLA&ts=CAESCgoCCAMKAggDEAAaXAo-EjwKCS9tLzAyXzI4NjIlMHg4OWMyNGZhNWQzM2YwODNiOjB4YzgwYjhmMDZlMTc3ZmU2MjoITmV3IFlvcmsSGhIUCgcI6Q8QBBgQEgcI6Q8QBBgRGAEyAhAAKgcKBToDVVNE"
  },
  {
    "name": "Aman New York",
    "price": "$2,200",
    "rating": "4.4",
    "link": "https://www.google.com/travel/search?q=New%20York&qs=MidDaGtJc3Q3dF80YmhzWW9ZR2cwdlp5OHhNV1kyTW1Sd2VIbHNFQUU4AA&currency=USD&ved=2ahUKEwiY1rucg9CMAxUIAPkAHXyaE5EQyvcEegUIAxCiAQ&ap=KigKEgm4tF8JXhxEQBF5jsg3iI5SwBISCfZ7hYTLm0RAEXmOyLfKcVLA&ts=CAESCgoCCAMKAggDEAAaXAo-EjwKCS9tLzAyXzI4NjIlMHg4OWMyNGZhNWQzM2YwODNiOjB4YzgwYjhmMDZlMTc3ZmU2MjoITmV3IFlvcmsSGhIUCgcI6Q8QBBgQEgcI6Q8QBBgRGAEyAhAAKgcKBToDVVNE"
  },
  .... more
]

Automating Amazon Scraping Tutorial

Crawlbase — Fri, 11 Apr 2025 13:21:35 +0000

This blog was initially posted to Crawlbase Blog

Amazon remains one of the most valuable platforms in the world, with almost 640 billion US dollars in net sales revenue in 2024 alone. This speaks to the site’s popularity and relevance in the e-commerce space. Being a top marketplace and a credible data source, makes it a goldmine for data scraping.

However, extracting information manually from Amazon might be daunting, especially for non-technical professionals and businesses. That’s why we put together a practical handbook that covers scraping E-commerce product data, listing, prices, reviews, ratings, images, and ads from Amazon.

This blog will act as the ultimate guide for Amazon scraping automation, providing insights on extracting data from different sources from the world’s foremost e-commerce site.

Practical Amazon Scraping Tutorials

Extracting data from Amazon requires efficient techniques to bypass restrictions and get structured, valuable insights. Here, we break down five scraping tutorials to help you collect essential Amazon data for market research, competitor analysis, and business growth.

Scrape Amazon Search Pages

Amazon’s search pages contain valuable insights, including product rankings, pricing, and keyword trends. By scraping search results, you can:

Identify high-ranking products for specific keywords.
Analyze pricing strategies used by competitors.
Collect product availability and seller details for business intelligence.

How to do it:

You can use Crawlbase’s ‘amazon-serp’ scraper to extract search results without getting blocked. The Crawling API handles proxy rotation, CAPTCHAs, and rate limits, ensuring smooth data collection.

Scrape Amazon Best Sellers

Amazon’s best sellers list reveals top-performing products across categories. This data is crucial for:

Understanding market trends and consumer preferences.
Discovering profitable product niches before competitors.
Monitoring fluctuations in demand to adjust business strategies.

How to do it:

With Crawlbase’s built-in ‘amazon-best-sellers’ scraper, you can effortlessly extract bestseller rankings, product ASINs, ratings, and pricing trends.

Scrape Amazon Buy Box Data

The Amazon Buy Box is the most coveted space for sellers—over 80% of purchases happen here. Scraping Buy Box data allows you to:

Track winning sellers and their pricing strategies.
Monitor product availability and stock changes.
Identify key factors affecting Buy Box eligibility.

How to do it:

Scraping the Buy Box requires analyzing product pages and extracting elements like seller names, shipping details, and pricing updates. Crawlbase’s Crawling API automates this process while ensuring accurate data extraction.

Scrape Amazon PPC Ads

Amazon PPC (Pay-Per-Click) Ads influence sales and product visibility. Scraping PPC ads helps businesses:

Discover competitor bidding strategies.
Analyze ad placement effectiveness.
Optimize their ad campaigns for better ROI.

How to do it:

By targeting search result pages and extracting sponsored ad data, you can uncover keywords, ad copies, and competitor spending habits. Crawlbase’s API simplifies PPC ad scraping by handling dynamic content loading and JavaScript-heavy pages.

Scrape Amazon Product Pages & Reviews

Product pages and customer reviews are gold mines of information for e-commerce sellers and analysts. Scraping this data enables you to:

Extract detailed product descriptions, images, and specs.
Analyze customer sentiment and review patterns.
Identify product improvement opportunities based on feedback.

How to do it:

Crawlbase’s amazon-product-details and amazon-product-reviews scrapers extract structured data, including ratings, review counts, customer feedback, and more—all in a clean JSON format for easy analysis.

How Crawlbase Helps You Scrape Amazon Seamlessly

Crawlbase's Amazon Scraper delivers an unmatched 100% success rate, handling requests at a default speed of 20 per second. Proxyway research confirms that Crawlbase outperforms competitors with the highest success rate, ensuring flawless task completion. This reliability is especially crucial for Amazon scraping, where Crawlbase guarantees perfect accuracy and consistency, making it the go-to choice for users who demand dependable results.

Download Free Amazon Scraping Handbook

Scraping Amazon should be easy without blocks. With Crawlbase, you get a fully managed solution that handles all the complexities so you can focus on analyzing data.

Ready to master Amazon scraping? Download our free eBook now.

Ultimate Web Scraping Guide with Parsel in Python

Crawlbase — Fri, 11 Apr 2025 13:16:30 +0000

This blog was initially posted to Crawlbase Blog

Web scraping is a great way to get data from websites for research, business, and machine learning. If you’re working with HTML content, Python has many tools, but Parsel is the simplest and most flexible. It lets you extract data with XPath and CSS selectors in just a few lines of code.

In this guide, you’ll learn how to use Parsel in Python for web scraping, from setting up your environment to handling complex HTML structures and saving cleaned data. Whether you're new to web scraping or looking for a lightweight tool, Parsel can streamline your scraping workflow.

Setting Up Your Python Environment

Before you start web scraping with Parsel, you need to set up your Python environment. The good news is that it’s quick and easy. All you need is Python installed and a few essential libraries to get started.

Install Python

Make sure Python is installed on your system. You can download it from the official Python website. Once installed, open your terminal or command prompt and check the version:

python --version

Create a Virtual Environment

It’s a good practice to create a virtual environment so your dependencies stay organized:

python -m venv parsel_env
source parsel_env/bin/activate  # Use `parsel_env\Scripts\activate` on Windows

Install Parsel and Requests

Parsel is used to extract data, and Requests helps you fetch HTML content from web pages.

pip install parsel requests

That’s it! You’re now ready to scrape websites using Parsel in Python. In the next section, we’ll explore how XPath and CSS selectors work to target specific HTML elements.

Understanding XPath and CSS Selectors

To scrape data with Parsel in Python, you need to know how to find the right elements in the HTML. This is where XPath and CSS selectors come in. Both are powerful tools that help you locate and extract the exact data you need from a webpage.

What is XPath?

XPath stands for XML Path Language. It’s a way to navigate through HTML and XML documents. You can use it to select nodes, elements, and attributes in a web page.

Example:

selector.xpath('//h1/text()').get()

This XPath expression selects the text of the first <h1> tag on the page.

What is a CSS Selector?

CSS selectors are used in web design to style elements. In web scraping, they help target elements using class names, tags, or IDs.

Example:

selector.css('div.product-name::text').get()

This gets the text inside a <div> with the class product-name.

XPath vs. CSS Selectors

Parsel supports both methods, and you can use whichever one suits your scraping needs best. In the next section, we’ll put this into action and show you how to extract data using Parsel.

Extracting Data Using Parsel

Once you've learned the basics of XPath and CSS selectors, it's time to use Parsel in Python to start extracting data. This section will show how to parse HTML, select elements, and get the text or attributes you need from a webpage.

Parsing HTML Content

First, you need to load the HTML content into Parsel. You can use the Selector class from Parsel to do this.

from parsel import Selector

html = """
<html>
  <body>
    <h1>Web Scraping with Parsel</h1>
    <p class="info">This is a tutorial.</p>
  </body>
</html>
"""

selector = Selector(text=html)

Now the HTML is ready for data extraction.

Selecting Elements with XPath

You can use XPath to find specific elements. For example, if you want to get the text inside the <h1> tag:

title = selector.xpath('//h1/text()').get()
print(title)  # Output: Web Scraping with Parsel

XPath is very flexible and allows you to target almost any element in the HTML structure.

Selecting Elements with CSS Selectors

Parsel also supports CSS selectors. This method is shorter and easier to read, especially if you’re already familiar with CSS.

info = selector.css('p.info::text').get()
print(info)  # Output: This is a tutorial.

CSS selectors are great for selecting elements based on class names, IDs, or tags.

Extracting Text and Attributes

To get text, use ::text in CSS or /text() in XPath. To extract attributes like href or src, use the @ symbol in XPath or ::attr(attribute_name) in CSS.

XPath Example:

link = selector.xpath('//a/@href').get()

CSS Example:

link = selector.css('a::attr(href)').get()

These methods let you pull the exact data you need from links, images, and other elements.

Handling Complex HTML Structures

When scraping real websites, the HTML structure isn’t always simple. Pages often have deeply nested elements, dynamic content, or multiple elements with the same tag. Parsel in Python makes it easier to handle complex HTML structures with XPath and CSS selectors.

Navigating Nested Elements

You may need to go through several layers of tags to reach the data you want. XPath is beneficial for navigating nested elements.

html = """
<div class="product">
  <div class="details">
    <span class="name">Smartphone</span>
    <span class="price">$499</span>
  </div>
</div>
"""

from parsel import Selector
selector = Selector(text=html)

name = selector.xpath('//div[@class="details"]/span[@class="name"]/text()').get()
price = selector.xpath('//div[@class="details"]/span[@class="price"]/text()').get()

print(name)   # Output: Smartphone
print(price)  # Output: $499

This is helpful when the data is buried deep inside multiple <div> tags.

Handling Lists of Data

If the page contains a list of similar items, like products or articles, you can use .xpath() or .css() with .getall() to extract all items.

html = """
<ul>
  <li>Python</li>
  <li>Parsel</li>
  <li>Web Scraping</li>
</ul>
"""

selector = Selector(text=html)
topics = selector.css('ul li::text').getall()
print(topics)  # Output: ['Python', 'Parsel', 'Web Scraping']

Using getall() is great when you want to scrape multiple elements at once.

Conditional Selection

Sometimes, you only want data that matches specific conditions, like a certain class or attribute.

html = """
<a href="/blog" class="nav">Blog</a>
<a href="/contact" class="nav special">Contact</a>
"""

selector = Selector(text=html)
special_link = selector.xpath('//a[contains(@class, "special")]/@href').get()
print(special_link)  # Output: /contact

This is useful when you want to remove extra or unwanted content from your scrape.

With Parsel in Python, you can handle complex web pages and get clean, structured data. Next, we’ll see how to clean and format this data.

Cleaning and Structuring Extracted Data

Once you extract data with Parsel in Python, the next step is to clean and format it. Raw scraped data often has extra spaces, inconsistent formats, or duplicate entries. Cleaning and formatting your data makes it easier to analyze or store in a database.

Removing Extra Spaces and Characters

Text from web pages can include unnecessary white spaces or line breaks. You can clean it using Python string methods like .strip() and .replace().

raw_text = "\n   Product Name: Smartphone   \t"
clean_text = raw_text.strip()
print(clean_text)  # Output: Product Name: Smartphone

Standardizing Data Formats

It’s important to keep dates, prices, and other data in the same format. For example, if you're extracting prices:

price_text = "$499"
price = float(price_text.replace("$", ""))
print(price)  # Output: 499.0

This helps when performing calculations or storing values in databases.

Removing Duplicates

Sometimes, the same data appears multiple times on a page. You can use Python’s set() or check with conditions to remove duplicates:

items = ['Parsel', 'Python', 'Parsel']
unique_items = list(set(items))
print(unique_items)  # Output: ['Python', 'Parsel']

reating a Structured Format (List of Dictionaries)

Once cleaned, it's best to structure your data for easy saving. A common approach is using a list of dictionaries.

data = [
    {"name": "Smartphone", "price": 499},
    {"name": "Laptop", "price": 899}
]

This format is perfect for exporting to JSON, CSV or inserting into databases.

By cleaning and formatting your scraped data, you make it much more useful for real applications like data analysis, machine learning, or reporting. Next, we’ll see how to save this data in different formats.

How to Save Scraped Data (CSV, JSON, Database)

After cleaning and structuring your scraped data using Parsel in Python, the final step is to save it in a format that suits your project. The most common formats are CSV, JSON, and databases. Let’s explore how to save web-scraped data using each method.

Saving Data as CSV

CSV (Comma-Separated Values) is great for spreadsheets or importing into data tools like Excel or Google Sheets.

import csv

data = [
    {"name": "Smartphone", "price": 499},
    {"name": "Laptop", "price": 899}
]

with open("products.csv", mode="w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["name", "price"])
    writer.writeheader()
    writer.writerows(data)

Saving Data as JSON

JSON is commonly used when you want to work with structured data in web or API projects.

import json

with open("products.json", "w") as file:
    json.dump(data, file, indent=4)

Saving Data to a Database

Databases are ideal for handling large amounts of data and running queries. Here's how to insert scraped data into a SQLite database:

import sqlite3

conn = sqlite3.connect("products.db")
cursor = conn.cursor()

# Create table
cursor.execute("CREATE TABLE IF NOT EXISTS products (name TEXT, price REAL)")

# Insert data
for item in data:
    cursor.execute("INSERT INTO products (name, price) VALUES (?, ?)", (item["name"], item["price"]))

conn.commit()
conn.close()

By saving your scraped data in the right format, you can make it more accessible and ready for analysis, reporting, or machine learning.

Common Mistakes to Avoid with Parsel

When using Parsel for web scraping in Python, it’s easy to make small mistakes that can cause your scraper to break or collect the wrong data. Avoiding these common issues will help you build more reliable and accurate scrapers.

1. Not Checking the Website’s Structure

Before you write your XPath or CSS selectors, always inspect the HTML of the website. If the structure changes or is different from what you expect, your scraper won’t find the correct elements.

Tip: Use browser developer tools (right-click → Inspect) to check element paths.

2. Using the Wrong Selectors

Make sure you choose the correct XPath or CSS selector for the element you want. Even a small mistake can return no data or the wrong result.

Example:

✅ Correct: response.css('div.product-name::text')
❌ Incorrect: response.css('div.product-title::text') (if it doesn’t exist)

3. Not Handling Empty or Missing Data

Sometimes, a page might not have the element you're looking for. If your code doesn’t handle this, it may crash.

Fix:

name = selector.css('div.name::text').get(default='No Name')

4. Forgetting to Strip or Clean Data

Web content often includes extra spaces or newline characters. If you don’t clean the text, your final data might look messy.

Fix:

price = selector.css('span.price::text').get().strip()

5. Not Using a Delay Between Requests

Sending too many requests quickly can get your scraper blocked. Always add delays to act more like a human.

Fix:

import time
time.sleep(2)  # Wait 2 seconds between requests

Avoiding these mistakes will help you scrape cleaner, more accurate data with Parsel in Python and ensure your scripts run smoothly even as websites change. Keeping your scraper flexible and clean will save you time in the long run.

Final Thoughts

Parsel is a powerful tool for web scraping in Python. Using it, you can extract and structure data from websites. By mastering XPath and CSS selectors, you can target what you need from a page. Handling complex HTML and cleaning your data will give you reliable results.

With Parsel, you can automate data extraction for various use cases, whether for research or business insights. Just remember to follow best practices, and you’ll be scraping like a pro.

Frequently Asked Questions

Q. What is Parsel, and why should I use it for web scraping?

Parsel is a Python library that makes web scraping easy. It lets you extract data from websites by using XPath and CSS selectors to find the data you need. Parsel is lightweight, fast, and works well with other Python tools, so it’s a popular choice for scraping structured data from HTML pages.

Q. How do I handle dynamic websites with Parsel?

For websites that load content dynamically using JavaScript, Parsel might not be enough on its own. In these cases, consider combining Parsel with Selenium or Playwright to load JavaScript content before extracting data. These tools let you simulate browser interactions so you can scrape all the data you need.

Q. Can I save the scraped data using Parsel?

Yes, you can save the data extracted with Parsel in various formats like CSV, JSON or even directly into a database. After parsing and structuring the data, you can use Python’s built-in libraries like Pandas or JSON to store your results in the format you want for easy analysis.

Best Proxies for Web Scrapers in 2025

Crawlbase — Mon, 07 Apr 2025 16:09:18 +0000

This blog was initially posted to Crawlbase Blog

Data has become essential for competitive intelligence, market research, and business automation. Web scraping—automated data extraction from websites—has emerged as a crucial tool for businesses, researchers, and developers seeking valuable insights. As more companies turn to web scraping to maintain their edge, the challenges it presents are also changing. Websites are increasingly sophisticated in detecting bots, implementing stricter anti-scraping measures like CAPTCHAs, IP blocking, and rate-limiting.

This is where proxies come into play. Proxies serve as intermediaries between web scrapers and target websites, hiding real IP addresses, distributing requests across various locations, and ensuring seamless data collection. Without a dependable proxy, scrapers risk being blocked, which can disrupt data-gathering efforts and result in inaccurate or incomplete datasets.

This guide delves into the best proxies for web scraping in 2025, outlining their benefits, key features, and tips for selecting the right one for your specific scraping requirements.

What Makes a Good Proxy for Web Scraping?

The effectiveness of a proxy relies on several essential factors that influence its capability to bypass restrictions, ensure efficiency, and offer smooth access to the desired websites. Here are some considerations to keep in mind when choosing a proxy for web scraping in 2025:

1. Anonymity & Security

Websites continuously track incoming traffic to identify and prevent suspicious activities, like numerous requests from a single IP address. A reliable proxy provides anonymity by concealing the scraper’s identity and frequently changing IP addresses to evade detection. Residential and mobile proxies deliver the most significant level of anonymity because they utilize actual user IPs, making it more challenging for websites to identify and block them.

2. Speed & Performance

To scrape large amounts of data effectively, it's essential to use high-speed proxies to avoid slowdowns and request timeouts. Datacenter proxies are recognized for their speed, making them perfect for tasks that need quick responses. However, for extensive scraping projects that demand discretion, residential or ISP proxies offer a good mix of speed and reliability.

3. IP Rotation & Pool Size

A reliable proxy provider should include automatic IP rotation to help avoid blocks and spread requests over a wide range of IPs. The more extensive the proxy network, the higher the likelihood of steering clear of bans. Additionally, some providers offer session-based proxies, which keep the same IP for a specified time to evade detection by websites that monitor user sessions.

4. Geolocation Options

Specific web scraping tasks necessitate the use of proxies with location-specific IP addresses to access content that is restricted by region, including e-commerce prices, travel fares, and localized search engine results. A reliable proxy service offers a wide variety of IPs from different countries and cities, allowing scrapers to collect precise and location-specific data.

5. Pricing & Scalability

The ideal proxy balances cost and performance. Datacenter proxies are typically the most budget-friendly, whereas residential and mobile proxies are pricier because of their legitimacy and higher success rates. A basic proxy package may be enough for smaller projects, but for extensive scraping tasks, it's essential to opt for a scalable solution that offers flexible pricing options.

Types of Proxies for Web Scraping

Selecting the appropriate type of proxy is crucial for enhancing your web scraping approach. Various proxies fulfill different roles, and the ideal choice hinges on considerations like the security of the target website, the volume of scraping needed, and your budget. In the following sections, we will examine the four primary types of proxies utilized for web scraping in 2025.

1. Datacenter Proxies

Datacenter proxies are a popular and budget-friendly choice for web scraping. They are sourced from cloud servers instead of real users, offering fast connections that are perfect for extensive data gathering. However, because they aren't linked to actual residential or mobile devices, they can be more easily identified and blocked, particularly on sites with stringent anti-scraping protocols.

Pros:

High speed and low latency
Cost-effective compared to other proxies
Best for scraping less-secure websites

Cons:

Easily detected and blocked by advanced anti-bot systems
Lack of geo-diversity compared to residential proxies

2. Residential Proxies

Residential proxies utilize IP addresses that are assigned by Internet Service Providers (ISPs) to actual users. This gives them the appearance of being legitimate users instead of bots, which lowers the chances of detection and bans. These proxies are particularly useful for scraping sensitive websites that have strict anti-bot measures in place, although they tend to be more expensive.

Pros:

Difficult for websites to detect and block
Ideal for scraping dynamic and geo-restricted content
Suitable for large-scale operations with high success rates

Cons:

More expensive than datacenter proxies
Slower than datacenter proxies due to real-user routing

3. ISP (Static Residential) Proxies

ISP proxies, often referred to as static residential proxies, combine the benefits of both datacenter and residential proxies. They provide the fast performance of datacenter proxies while ensuring the genuine appearance of residential proxies, making them a dependable option for consistent and discreet web scraping. These proxies are particularly suited for situations that need ongoing sessions, like logging into accounts or scraping websites that monitor user activity over time.

Pros:

Combines speed and legitimacy
Stable connections with extended session support
Less likely to be flagged compared to datacenter proxies

Cons:

More expensive than datacenter proxies
Not as anonymous as rotating residential proxies

4. Mobile Proxies

Mobile proxies operate on 3G, 4G, and 5G networks, which makes them the most anonymous type of proxy available. Websites find it challenging to identify and block mobile IPs since thousands of genuine users share them. These proxies are ideal for scraping social media sites, ad networks, and mobile-centric websites, but their high price can make them impractical for extensive scraping unless absolutely needed.

Pros:

Extremely hard to detect and block
Best for scraping social media and mobile apps
Ideal for bypassing the strictest anti-bot defenses

Cons:

Expensive compared to other proxy types
Slower than datacenter proxies due to mobile network routing

Scraping Solutions with Built-in Proxies

Selecting the right proxy is essential for web scraping, but many tools and services come with integrated proxy management. This feature removes the hassle of manual proxy configuration. Such solutions streamline the scraping process, making them perfect for users who prefer effortless data extraction without the complications of IP rotation, bans, or intricate setups.

1. Scrapers with Optional Proxies

Some web scraping tools allow users to incorporate optional proxies, allowing them to choose to either use a proxy or scrape directly. This feature is advantageous for those targeting less-protected websites or managing their proxy pools. Well-known Python libraries such as Scrapy and Selenium support proxy integration, allowing developers to tailor their scraping approach according to the security level of the target site.

2. Ready-Made Proxy Scrapers

Some web scrapers are equipped with pre-configured proxies, making them perfect for users seeking a straightforward, plug-and-play solution. These tools automatically manage IP rotation, solve CAPTCHAs, and bypass anti-bot measures, minimizing the need for technical know-how. Services such as ScraperAPI and Smart Proxy by Crawlbase provide integrated proxy management, ensuring seamless and uninterrupted scraping.

3. Scraper APIs

For large-scale scraping projects, Scraper APIs provide a strong solution by managing proxy rotation, browser fingerprinting, and anti-bot measures seamlessly. These APIs employ AI-driven methods to imitate human behavior, which helps lower the chances of detection and bans. Crawlbase’s Crawling API stands out as a top solution that offers enterprise-level scraping with minimal setup.

How to Choose the Best Proxy for Your Scraping Needs

With a variety of proxy types to choose from, picking the right one hinges on your particular web scraping objectives. Whether you're gathering e-commerce prices, SEO information, travel rates, or social media posts, the proxy you select will significantly impact your success and efficiency.

1. Match the Proxy Type to Your Use Case

Different scraping scenarios require different proxy solutions. Below is a breakdown of the best proxy types based on common scraping use cases:

2. Evaluate Provider Features, Pricing, and Customer Support

After identifying the most suitable proxy type for your scraping requirements, it's crucial to assess proxy providers by considering several key factors:

Key Features to Consider:

Extensive IP Pool & Rotation: A larger pool of IPs reduces the chances of being blocked. Seek out providers that offer millions of IPs along with automatic rotation features.
Geolocation Coverage: For region-specific data needs, confirm that the provider supplies IP addresses from various countries and cities.
High Uptime & Speed: The effectiveness of scraping relies on fast, dependable proxies with minimal downtime.
Session Control: Certain scraping tasks may require sticky sessions, which maintain the same IP for a set period instead of rotating IPs with each request.

Final Thoughts

Web scraping remains a crucial tool for businesses and individuals seeking to gather valuable data from the Internet. However, the effectiveness of scraping largely depends on selecting the appropriate proxy to navigate restrictions, maintain anonymity, and ensure efficient data collection.

Understanding the various types of proxies—datacenter, residential, ISP, and mobile—can help you find the best fit for your specific scraping requirements, whether for SEO monitoring, e-commerce price tracking, travel fare aggregation, or social media data gathering. Assessing providers based on factors like IP rotation, speed, geolocation options, pricing, and customer support will enable you to make a well-informed choice.

Crawlbase’s Smart Proxy provides a sophisticated solution featuring automatic IP rotation, CAPTCHA bypassing, and impressive speed. Whether you're just starting or managing large-scale scraping operations, Smart Proxy guarantees that your web scraping activities are efficient, secure, and scalable.

Frequently Asked Questions (FAQs)

What is a Proxy Scraper?

A proxy scraper is a tool or script designed to automatically search for, collect, and verify proxy servers from different sources. It assists users in compiling a list of functional proxies that can be utilized for web scraping, maintaining anonymity, or circumventing geo-restrictions.

How to Use a Proxy Scraper?

Using a proxy scraper will depend on whether you're working with a custom script or a ready-made tool. If you're looking to create your proxy scraper, Python is a good choice. However, be aware that free proxies can be slow and may not always be reliable.

Can I Get a Proxy Scraper for Free?

You can indeed find free proxy scrapers, but they usually have their drawbacks. Most free proxy scrapers compile public proxy lists that tend to include slow, unreliable, or already blacklisted IP addresses. For more serious web scraping tasks, it's advisable to use a paid proxy scraper or a premium proxy service like Crawlbase's Smart Proxy to prevent bans and maintain stable connections.

Do I Need a Proxy List?

If you're handling your web scraping setup, you'll require a proxy list to rotate IPs and prevent detection. However, some Scraper APIs and proxy services take care of IP rotation for you, so you won't have to manage a proxy list on your own.

How to Bypass Cloudflare and Avoid Bot Detection

Crawlbase — Mon, 07 Apr 2025 15:54:06 +0000

This blog was initially posted to Crawlbase Blog

Cloudflare is a security tool that blocks bots and scrapers using IP tracking, JavaScript challenges and browser fingerprinting. This helps website owners but makes web scraping difficult by triggering CAPTCHAs and access restrictions.

If you need to scrape data from a Cloudflare protected site, you need to use smart techniques to bypass detection. In this guide, we will show you how Cloudflare detects bots, how to bypass it, and ethical scraping practices. We will also talk about how Crawlbase Smart Proxy makes it easy to access Cloudflare protected sites. Let’s get started!

Methods to Bypass Cloudflare Protection

Cloudflare has strong bot protection, but we can bypass it and stay undetected. Here are the ways:

1. Using Rotating Residential Proxies

Cloudflare tracks IP addresses and blocks suspicious ones. Rotating residential proxies helps you to avoid detection by switching between real user IPs. Residential proxies mimic real internet users so it’s hard for Cloudflare to block you.

2. Spoofing Headers and User-Agents

Browsers send headers like user-agent, referer, and cookies to identify themselves. Cloudflare checks these headers to detect bots. By rotating user-agents and setting headers to match real browsers, you reduce the chances of getting blocked.

3. Implementing Headless Browsers and AI-based Interactions

Headless browsers like Puppeteer and Selenium can simulate human-like browsing. To make requests more realistic, you can introduce AI-driven mouse movements, scrolling, and keystroke simulation. This will help bypass Cloudflare’s behavioral analysis.

4. Solving JavaScript Challenges and CAPTCHAs

Cloudflare’s JavaScript challenges and CAPTCHAs block bots that can’t execute scripts. Tools like Puppeteer and Playwright can render JavaScript, while CAPTCHA-solving services can solve challenges to keep access uninterrupted.

How to Integrate Crawlbase Smart Proxy in Your Scraper

The easiest way to avoid Cloudflare detection is by using Crawlbase Smart Proxy. It automatically rotates IPs, manages headers, and solves JavaScript challenges for seamless scraping. Below is a Python example of how to use it:

import requests
import time

# Crawlbase Smart Proxy setup
proxy_url = "http://_USER_TOKEN_@smartproxy.crawlbase.com:8012"
proxies = {"http": proxy_url, "https": proxy_url}

# Target URL protected by Cloudflare
url = "https://example.com/protected-page"

# Introduce a delay to mimic human behavior
time.sleep(2)  # Wait for 2 seconds before making the request

# Send request through Smart Proxy
response = requests.get(url, proxies=proxies, verify=False)

# Print response
print(response.text)

🔹 Note: Replace "_USER_TOKEN_" with your actual Crawlbase token, which you can obtain after signing up on Crawlbase.

By using Crawlbase Smart Proxy, you can efficiently bypass Cloudflare protection without worrying about IP blocks or CAPTCHAs, making your scraping process more reliable and efficient.

Final Thoughts

Cloudflare and bot detection is all about the right tools and strategies. Understanding how Cloudflare detects bots helps you choose the best approach whether it’s rotating residential proxies, spoofing headers or handling JavaScript challenges.

Crawlbase Smart Proxy makes it easy by automatically rotating IPs and solving CAPTCHAs so you can access protected websites smoothly and undetected. But always follow ethical scraping practices and respect website terms of service.

Frequently Asked Questions

Q. Can Cloudflare block web scraping completely?

Cloudflare has strong bot protection, but with the proper techniques - rotating proxies, spoofing headers, and solving JavaScript challenges, you can bypass its defenses and keep scraping undetected.

Q. How do I bypass Cloudflare bot protection?

The best way is to use a Smart Proxy service like Crawlbase Smart Proxy that automatically rotates IPs, solves CAPTCHAs and handles JavaScript challenges. Combine this with proper request headers and human-like browsing behavior for better success rates.

Q. Is it legal to bypass Cloudflare for web scraping?

Bypassing Cloudflare depends on the website’s terms of service. Always check legal guidelines and use ethical scraping practices to avoid legal issues or violating a site’s policy.

DEV Community: Crawlbase

Build a No-Code AI Scraper With Crawlbase in Minutes

🛠️ What You'll Need

⚙️ How It Works

🚀 Building Your Scraper

Step 1: Open Cursor

Step 2: Write Your Prompt

Step 3: Let the AI Work

Step 4: Review Results

✨ Best Practices

Be Specific

Define the Format

Handle Missing Data

💡 Why AI Scraping Works Better

📊 Common Use Cases

💰 Cost Comparison

🎯 Getting Started

🤔 FAQ

The Crawlbase Cyberweek Deals 2025

Products Included in This CyberWeek Deal

1. Crawling API

2. Smart AI Proxy

3. Crawler

4. Cloud Storage

5. LinkedIn Scraper

Don't Miss Out – Act Now!

Crawlbase MCP Server for Real-Time AI Web Scraping

What you’ll learn in this AI web scraping guide

The LLM Bottleneck: Why AI Agents Struggle with Live Web Data

Why Model Context Protocol (MCP) Matters

How MCP Unlocks Real-Time Web Access

What is the Crawlbase MCP Server?

How to Get Started with Crawlbase MCP Server

Step 1: Get your Crawlbase Tokens

Step 2: Integrate Crawlbase MCP Server for AI Web Scraping

Run Context Commands

Crawlbase MCP Setup in Claude Desktop

Step 1: Open Claude Desktop → File → Settings → Developer → Edit Config

Step 2: Copy the Crawlbase MCP, then paste it in the claude_desktop_config.json file

Step 3: Save the config file and restart Claude Desktop.

Step 4: Utilizing the MCP

Crawlbase MCP Setup in Cursor IDE

Step 1: Open Cursor IDE → File → Preferences → Cursor Settings → Tools and Integrations → Add Custom MCP

Step 2: Copy the Crawlbase MCP, then paste it in the mcp.json file

Step 3: Once the config file is saved, an indicator will confirm that the Crawlbase MCP is active.

Step 4: Utilize the Chat window to send commands to the Crawlbase MCP.

Crawlbase MCP Setup in WindSurf

Step 1: Open WindSurf IDE → File → Preferences → WindSurf Settings → General → MCP Servers → Manage MCPs → View raw config

Step 2: Copy the Crawlbase MCP, then paste it in the mcp_config.json file

Step 3: Save the config file and hit Refresh

Step 4: Use the Chat window to send commands to the Crawlbase MCP.

How to Crawl Apple App Store Data Without Getting Blocked

How to Scrape Apple App Store Data?

Creating a Crawlbase account

Setting up the Environment

Installing Node on Windows:

For macOS:

For Linux (Ubuntu/Debian):

Fetch Script

Locating specific CSS selectors

Parsing the HTML in Node.js

Import the Required Libraries

Functions for Scraping Apple Store Data

Combine Everything in One Function

Complete Code to Scrape Apple App Store Data

Scrape Apple Store Data with Crawlbase

How to Scrape Data Behind Login Pages Using Python

Authenticated Scraping with Python Requests Library

Scraping Behind Login Using Crawlbase

How to Use Web Scraping for Price Intelligence

Selecting Target Websites (Amazon and eBay)

Set Up Your Workspace

Setup Your Coding Environment

Sending Requests Using Crawlbase

Extracting and Structuring Price Data

Make Use of the Scraped Data

How to Unblock Amazon with Smart Proxy

Setting Up Your Coding Environment

Obtaining Credentials

Making Your First Successful Request

Step 2: Copy the Crawlbase MCP, then paste it in the `claude_desktop_config.json` file

Step 2: Copy the Crawlbase MCP, then paste it in the `mcp.json` file

Step 2: Copy the Crawlbase MCP, then paste it in the `mcp_config.json` file