DEV Community

gresiuseo
gresiuseo

Posted on

2scraper: How to Build Practical Web Data Pipelines Without Writing Every Scraper from Scratch

#2scraper #scraper

Web scraping is one of those engineering tasks that looks deceptively simple.

You open a website, inspect its HTML, find a few CSS selectors, write a Python script, and export the information into a CSV file.

The first version usually takes less than an hour.

Then the real problems begin.

The website changes its HTML structure. Product cards load dynamically. Pagination behaves differently from what you expected. Some fields are missing. A browser session gets blocked. A request returns an unexpected response. Your parser silently starts producing incomplete records.

Suddenly, a small scraping script becomes an application that needs maintenance, monitoring, testing, retries, browser management, and data validation.

This is exactly the kind of repetitive infrastructure work that 2scraper aims to reduce.

2scraper is an open-source collection of Python web scrapers, organized into separate repositories for individual websites. Instead of providing one generic scraping framework that tries to understand every website, it provides focused implementations designed around specific platforms and their data structures.

The repositories cover different categories, including:

  • E-commerce and product catalogs.
  • Social media and video platforms.
  • Classified advertisements.
  • Real estate listings.
  • Financial information.
  • Job listings.
  • Publishing platforms.
  • Marketplaces and auction websites.

Each scraper has its own documented capabilities, command-line interface, supported extraction modes, and output schema.

The idea is straightforward: start with an existing implementation, run it on your own infrastructure, inspect the extracted data, and adapt it to your application.

In this tutorial, we will explore how to use the collection in practical development scenarios.

We will:

  1. Understand the architecture of 2scraper.
  2. Select the right scraper for a project.
  3. Install and configure a real repository.
  4. Collect structured product data with Python.
  5. Work with JSON and CSV output.
  6. Validate scraping results programmatically.
  7. Handle incomplete runs and errors.
  8. Automate recurring extraction jobs.
  9. Integrate the output into a data pipeline.
  10. Decide when to use a local browser, direct HTTP requests, or a hosted browser.

Our main example will use the Amazon scraper, but the same general approach applies to other repositories, provided you follow their individual README instructions.

1. What Exactly Is 2scraper?

Start with the main GitHub organization:

https://github.com/2scraper

The organization contains multiple independent repositories.

For example:

Repository Main purpose
amazon-scraper Product listings, product pages, reviews
youtube-scraper Video metadata, comments, search
catawiki-scraper Auction lots, bids, sellers
stockx-scraper Sneaker prices and marketplace data
transfermarkt-scraper Football players, squads, transfers
google-play-scraper App listings and reviews
hackernews-scraper Stories, comments, user profiles
medium-scraper Articles, authors, publications
screener-scraper Financial and stock-market data

This repository-per-website approach is important.

Websites have different structures, different pagination mechanisms, and different ways of exposing information.

A universal scraper must make many assumptions. A site-specific scraper can instead implement the extraction logic around the actual website.

For example, an e-commerce scraper may need to understand:

  • Product identifiers.
  • Price containers.
  • Discount labels.
  • Availability indicators.
  • Product variations.
  • Seller information.
  • Search result pagination.

A social media scraper has completely different requirements.

Keeping these implementations separate makes the code easier to understand, test, and maintain.

Open source and local execution

The 2scraper repositories use the MIT license, according to the organization and repository documentation.

This is useful for developers who want to inspect the implementation, modify extraction logic, or integrate the code into their own systems.

The scrapers are designed to run on your own infrastructure.

That means you can:

  • Run them on a development computer.
  • Execute them on a Linux server.
  • Package them with Docker.
  • Schedule them with cron.
  • Integrate them into Python applications.
  • Store their output in your own database.

You are not required to send all collected data to a centralized scraping dashboard.

The individual repository is the source of truth for supported fields, required infrastructure, and available execution modes.

2. When Should You Use 2scraper?

Not every web scraping project needs an existing scraper.

If you need to extract five elements from a simple static website, a short BeautifulSoup script may be sufficient.

However, an existing site-specific implementation becomes much more attractive when the extraction process involves several interconnected problems.

Scenario A: Product price monitoring

Imagine that you operate an online store and want to monitor publicly displayed prices across several marketplaces.

You need to collect:

  • Product names.
  • Product identifiers.
  • Current prices.
  • Original prices.
  • Ratings.
  • Review counts.
  • Availability.
  • Product URLs.

Writing a scraper for every marketplace independently requires considerable work.

With 2scraper, you can inspect the available repositories and start from an implementation that already understands the target platform.

Scenario B: Competitor research

Suppose you want to compare products from several retailers.

Your application needs structured records rather than screenshots or raw HTML.

A scraper that produces predictable JSON or CSV output is much easier to integrate into a comparison engine.

Scenario C: Market research

You might want to analyze:

  • Used vehicle listings.
  • Real estate prices.
  • Auction activity.
  • Job offers.
  • Financial indicators.
  • Public product reviews.

The same principle applies: find the repository that already targets the platform and examine its extraction capabilities.

Scenario D: AI and data pipelines

Modern AI applications often need external information as input.

For example, you might want to collect product descriptions and reviews, normalize them, and then pass selected records to an LLM for classification or summarization.

A structured scraping output is a much better starting point than a collection of inconsistent HTML documents.

When should you avoid it?

There are also situations where 2scraper may not be the right solution.

  • The target website is not supported.
  • You need authenticated access that the repository does not implement.
  • You need a completely different data schema.
  • Your project requires an officially supported API with a formal SLA.
  • The repository's last live test is too old for your requirements.
  • The target's terms or applicable laws do not permit your intended collection.

In those cases, you may need a custom implementation or an official data API.

3. Choosing the Right Repository

Before cloning anything, inspect the repository directory.

Start here:

https://github.com/2scraper

For each target, check four things.

First: What data does it actually extract?

A repository might support product listings but not complete product details.

Another might support public comments but not private or authenticated content.

Do not assume that the presence of a website in the directory means every page on that website is supported.

Second: Which execution engines are supported?

Many repositories include:

  • Playwright.
  • Selenium.
  • Puppeteer.

Some targets also have a browserless HTTP implementation or use the website's own public data endpoints.

Playwright is generally the recommended starting point across the collection.

Third: Does the scraper require additional infrastructure?

Some repositories work with a normal local browser and no API credentials.

Others may require a residential proxy, CAPTCHA processing, or a hosted browser.

Fourth: How recent is the validation?

Look for test badges, dated measurements, sample output, and troubleshooting documentation.

A scraper that worked six months ago is not automatically guaranteed to work today.

Example: Amazon scraper

Repository:

https://github.com/2scraper/amazon-scraper

The documented implementation supports three main extraction modes:

Mode Extracted information
Listing Search results and best-seller products
Product Detailed product information and variations
Reviews Reviews displayed on the product page

It supports multiple Amazon marketplaces and exports structured data.

The repository also includes a comparison utility for analyzing differences between scraping runs.

This makes it a useful starting point for price-monitoring applications.

4. Installing Your First Scraper

Let's build a real project.

We will use Python and the Amazon scraper.

You need:

  • Python 3.9 or newer.
  • Git.
  • A terminal.
  • An internet connection.
  • Chromium installed through Playwright.

Start by cloning the repository:

git clone https://github.com/2scraper/amazon-scraper.git

cd amazon-scraper
Enter fullscreen mode Exit fullscreen mode

Create a virtual environment:

python -m venv .venv
Enter fullscreen mode Exit fullscreen mode

Activate it on Linux or macOS:

source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

On Windows:

.venv\Scripts\activate
Enter fullscreen mode Exit fullscreen mode

Install the core dependencies:

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Then install the Playwright engine:

pip install -r requirements-playwright.txt
Enter fullscreen mode Exit fullscreen mode

Install Chromium:

playwright install chromium
Enter fullscreen mode Exit fullscreen mode

The project now has the main dependencies required to execute the Playwright scraper.

Why use a virtual environment?

Python scraping projects often depend on several libraries with specific version requirements.

Playwright, Selenium, and Puppeteer-related packages can have conflicting dependencies.

Installing everything globally may cause one project to break another.

A dedicated virtual environment isolates the scraper's dependencies and makes deployment much easier.

If you need to run multiple browser engines, the repository documentation recommends separate environments rather than combining incompatible dependency sets.

5. Your First Scraping Task

Let's start with a small extraction job.

The Amazon scraper supports listing extraction using a URL.

For example, to collect wireless headphone listings:

python3 playwright_scraper.py \
  --url "https://www.amazon.com/s?k=wireless+headphones" \
  --pages 3
Enter fullscreen mode Exit fullscreen mode

This command:

  1. Opens the specified search page.
  2. Processes the requested pagination.
  3. Extracts product records.
  4. Removes duplicate products where applicable.
  5. Writes structured output.
  6. Generates run metadata.

The default output is JSON.

You can also request CSV:

python3 playwright_scraper.py \
  --url "https://www.amazon.com/s?k=wireless+headphones" \
  --pages 3 \
  --format csv \
  --out headphones.csv
Enter fullscreen mode Exit fullscreen mode

This is particularly useful when the next step is data analysis in Excel, pandas, or another reporting tool.

Keep the first run small

Do not start with hundreds of pages.

A good initial workflow is:

1 page
   |
   v
Inspect output
   |
   v
Validate fields
   |
   v
Run 3 pages
   |
   v
Check pagination
   |
   v
Increase collection size if appropriate
Enter fullscreen mode Exit fullscreen mode

This reduces unnecessary requests and helps you identify problems before they affect a large dataset.

6. Extracting Detailed Product Information

Search results and product detail pages are not the same thing.

A search result might contain:

  • Product title.
  • ASIN.
  • Current price.
  • Rating.
  • Review count.
  • Product image.

A product page can provide additional fields, such as:

  • Brand.
  • Seller.
  • Product description bullets.
  • Availability.
  • Images.
  • Variation information.

The Amazon scraper provides a separate product mode.

Example:

python3 playwright_scraper.py \
  --mode product \
  --url "https://www.amazon.com/dp/B07K5214NZ"
Enter fullscreen mode Exit fullscreen mode

Replace the example ASIN with a product you are authorized to analyze.

This distinction matters when designing a data pipeline.

For example, if your goal is to monitor the market and compare hundreds of products, collecting complete product pages for every item may be unnecessarily expensive.

A better approach is:

  1. Collect search listings.
  2. Identify products of interest.
  3. Fetch details only for selected products.
  4. Store the additional fields separately.

This is a practical way to reduce processing time and resource consumption.

7. Understanding the JSON Output

Structured output is one of the biggest advantages of using a prepared scraper.

Instead of receiving a raw HTML document and writing another parser, you receive records with defined fields.

A simplified product record may look like this:

{
  "source": "amazon.com",
  "sku": "B07K5214NZ",
  "title": "Example Wireless Headphones",
  "price": 49.99,
  "currency": "USD",
  "rating": 4.5,
  "review_count": 1250,
  "in_stock": true,
  "url": "https://www.amazon.com/dp/B07K5214NZ"
}
Enter fullscreen mode Exit fullscreen mode

This is an illustrative example of the data structure, not a copied live result.

The actual repository documents its complete schema and provides sample output files.

Some fields are specific to individual extraction modes.

For example, brand information is available in the product mode, while it may be null in listing mode because the search result itself does not expose a separate brand field.

That is an important distinction.

A missing field does not necessarily indicate a parser failure.

Sometimes the source page simply does not provide that information.

Load the output with Python

Create a separate analysis script:

import json

with open(
    "amazon_products.json",
    "r",
    encoding="utf-8"
) as file:

    products = json.load(file)

print("Products collected:", len(products))

for product in products[:5]:

    print(
        product.get("title"),
        product.get("price"),
        product.get("currency")
    )
Enter fullscreen mode Exit fullscreen mode

This is enough to begin building a basic price comparison application.

Calculate average product price 

prices = [
    item["price"]
    for item in products
    if item.get("price") is not None
]

if prices:

    average_price = sum(prices) / len(prices)

    print(
        f"Average price: {average_price:.2f}"
    )

else:

    print("No valid prices found")
Enter fullscreen mode Exit fullscreen mode

Notice the null check.

Real-world scraping data is rarely perfectly complete.

Your application should always handle missing values instead of assuming every product contains every field.

8. Validate Data Before Using It

A successful process exit does not automatically mean that the dataset is suitable for your application.

You should validate the output.

For example:

def validate_product(product):

    required_fields = [
        "sku",
        "title",
        "url"
    ]

    for field in required_fields:

        if not product.get(field):

            return False

    price = product.get("price")

    if price is not None and price < 0:

        return False

    return True


valid_products = [
    product
    for product in products
    if validate_product(product)
]

print(
    f"Valid records: {len(valid_products)}"
)

print(
    f"Rejected records: "
    f"{len(products) - len(valid_products)}"
)
Enter fullscreen mode Exit fullscreen mode

For a production pipeline, I would also validate:

  • Duplicate product identifiers.
  • Currency consistency.
  • URL validity.
  • Price types.
  • Timestamp formats.
  • Unexpected schema changes.

Data validation should happen before records enter your main database.

Otherwise, a small extraction problem can become a much larger analytics problem.

9. The Most Important File: Run Metadata

One of the most useful design decisions in the 2scraper collection is the use of metadata sidecar files.

A successful extraction can generate two files:

amazon_products.json
amazon_products.meta.json
Enter fullscreen mode Exit fullscreen mode

The first contains the extracted records.

The second describes how the run finished.

For example, metadata can contain:

{
  "source": "amazon.com",
  "mode": "listing",
  "status": "complete",
  "stop_reason": "completed",
  "pages_requested": 3,
  "pages_completed": 3,
  "pages_failed": [],
  "records": 77
}
Enter fullscreen mode Exit fullscreen mode

This is a simplified example of the documented metadata structure.

Why is this important?

Imagine your application runs every morning.

On Monday, it collects 500 products.

On Tuesday, the website changes its markup and the scraper collects only 12 products.

If your automation checks only whether a JSON file exists, it may assume that Tuesday's run was successful.

A metadata sidecar gives your application additional information about the extraction process.

You can distinguish between:

  • Complete runs.
  • Partial runs.
  • Failed runs.
  • Blocked sessions.
  • Genuine zero-result searches.
  • Pages that could not be parsed.

The repositories also use a separate last-attempt metadata file for attempts that do not produce a normal result file.

This helps preserve the last successful dataset instead of replacing it with an empty file after a failed run.

For automated data pipelines, this is extremely valuable.

10. Understanding Exit Codes

The collection uses a consistent exit-code convention across its scrapers.

Exit code Meaning
0 Complete
1 Application crash
2 Invalid command usage
3 Blocked
4 Genuine zero results
5 Content could not be obtained
6 Partial result

These distinctions are important.

Consider the following cases:

Exit code 4

The page loaded correctly, but no products matched the search.

This may be a valid result.

Exit code 3

The website refused the session before the scraper could collect the requested data.

This is not equivalent to an empty search.

Exit code 5

The page was not successfully fetched, a remote service failed, or the expected content could not be obtained.

Exit code 6

Some data was collected, but the run did not finish cleanly.

A production application should handle these cases differently.

For example, a genuine empty search might update a database, while a failed extraction should preserve the previous successful data.

11. Automating a Scraper from Another Python Application

You do not have to run the scraper manually.

A useful approach is to execute its command-line interface from a Python application.

The following example demonstrates how to run a scraper safely using subprocess.

import subprocess


def run_scraper(url, pages=2):

    command = [
        "python3",
        "playwright_scraper.py",
        "--url",
        url,
        "--pages",
        str(pages)
    ]

    result = subprocess.run(
        command,
        capture_output=True,
        text=True,
        timeout=300
    )

    print("Exit code:", result.returncode)

    print("Output:")
    print(result.stdout)

    if result.stderr:

        print("Errors:")
        print(result.stderr)

    return result.returncode


if __name__ == "__main__":

    code = run_scraper(
        "https://www.amazon.com/s?k=wireless+headphones",
        pages=2
    )

    if code == 0:

        print("Extraction completed")

    elif code == 6:

        print("Partial extraction")

    else:

        print("Extraction needs investigation")
Enter fullscreen mode Exit fullscreen mode

Using a list of command arguments instead of building a shell command string is preferable.

It avoids shell interpolation issues and makes it easier to handle URLs containing special characters.

Add scheduled execution

For a recurring data collection job, you can use:

  • Linux cron.
  • Windows Task Scheduler.
  • GitHub Actions.
  • Docker.
  • A dedicated job queue.

For example, a daily Linux cron entry might look like this:

0 6 * * * cd /opt/amazon-scraper && /opt/amazon-scraper/.venv/bin/python playwright_scraper.py --url "https://www.amazon.com/s?k=wireless+headphones" --pages 2
Enter fullscreen mode Exit fullscreen mode

This runs the collection every day at 06:00.

For a production deployment, add logging, monitoring, metadata validation, and alerting around the command.

12. Comparing Two Scraping Runs

Collecting data once is useful.

Collecting it repeatedly is where interesting applications begin.

Imagine that you run a product scraper every day.

You want to know:

  • Which products have changed prices?
  • Which products disappeared?
  • Which products are new?
  • Which products changed availability?
  • Which products received new ratings?

The Amazon repository includes diff_runs.py for comparing scraping runs.

Example:

python3 diff_runs.py \
  previous.json \
  current.json
Enter fullscreen mode Exit fullscreen mode

The exact supported options are available through:

python3 diff_runs.py --help
Enter fullscreen mode Exit fullscreen mode

The comparison tool is especially useful for price monitoring because it helps identify changes without requiring you to compare two large JSON files manually.

Why stable identifiers matter

Product titles are not reliable identifiers.

A title can change because of:

  • Updated descriptions.
  • Formatting changes.
  • Marketing text.
  • Different language versions.

The Amazon scraper uses ASIN as the product identifier.

This allows your application to compare records across different runs.

The same principle applies to other platforms.

Whenever possible, use the target website's stable identifier rather than a display name.

13. Choosing the Right Execution Engine

One of the interesting aspects of 2scraper is that repositories can provide several execution paths.

The available options depend on the individual project.

Playwright

Playwright is generally the recommended starting point.

It provides:

  • Browser automation.
  • Page navigation.
  • JavaScript execution.
  • Dynamic-content handling.
  • Chromium integration.
  • Remote browser connections through CDP.

It is particularly useful when the data is rendered dynamically.

Selenium

Selenium is a familiar choice for developers who already maintain browser automation projects.

Some repositories provide a Selenium implementation with the same site-specific extraction logic.

However, the supported features may differ.

Puppeteer

Puppeteer is another browser automation option.

Depending on the repository, the implementation may use Node Puppeteer or pyppeteer.

Check the repository's README before selecting this engine.

Direct HTTP requests

Sometimes a browser is unnecessary.

If the target exposes the required information in the initial HTML or through a documented public endpoint, a direct HTTP implementation can be significantly lighter.

You avoid the overhead of launching Chromium.

For example, a simple request-based workflow may look like this:

import requests

response = requests.get(
    "https://example.com/public-page",
    headers={
        "User-Agent": "MyResearchBot/1.0 (contact: dev@example.com)"
    },
    timeout=20
)

response.raise_for_status()

html = response.text

print(html[:500])
Enter fullscreen mode Exit fullscreen mode

This is only a generic HTTP example, not an implementation of any particular 2scraper repository.

Use the browserless path only when the site's public response actually contains the information you need and your use is permitted.

Which engine should you choose?

Requirement Recommended starting point
Dynamic JavaScript pages Playwright
Existing Selenium infrastructure Selenium
Existing Puppeteer application Puppeteer
Static HTML or accessible public endpoint Direct HTTP
Managed remote browser Playwright or Puppeteer over CDP

The most important rule is to use the execution mode tested and documented by the repository.

14. Understanding Proxies, CAPTCHA Processing, and Hosted Browsers

Some websites treat automated requests differently depending on the connection and browser environment.

This is where the optional infrastructure documented by individual 2scraper repositories becomes relevant.

The collection integrates with several 2Captcha products where a specific target requires them.

These products serve different purposes.

Residential proxies

A residential proxy provides an exit IP associated with a residential network.

This can be relevant when a website refuses datacenter IP ranges.

A scraper that works from a home connection may behave differently when executed on a cloud server.

However, a proxy is not a guarantee of access.

It also does not authorize activity that would otherwise violate a website's terms or applicable law.

CAPTCHA processing

Some sites present CAPTCHA challenges during normal page access.

Where a repository documents a CAPTCHA integration, it may use an external solving service.

Such processing should be explicitly enabled and capped.

Never assume that every CAPTCHA needs to be solved.

Some browser challenges clear automatically, while other failures may be caused by a rate limit, a network error, or a page-layout change.

Buying a CAPTCHA solution for every failed request is a poor debugging strategy.

Scraping Browser API

A hosted browser provides a remote browser session that can be controlled through the Chrome DevTools Protocol.

Instead of managing Chromium yourself, your application connects to the remote browser.

This can be useful when:

  • Your server cannot maintain browser infrastructure.
  • You need a managed browser environment.
  • A repository documents support for a remote CDP endpoint.
  • You want to centralize browser management.

The exact setup depends on the repository and its environment variables.

One important rule from the 2scraper documentation is that a hosted CDP browser should not be combined with local proxy, user-agent, or fingerprint configuration.

Choose one coherent browser configuration rather than mixing incompatible settings.

15. Building a Reliable Production Pipeline

A scraping script is not yet a complete data pipeline.

For a production application, I recommend separating the system into several components.

                 Scheduler
                     |
                     v
               Scraping Job
                     |
                     v
                2scraper
                     |
                     v
              JSON / CSV Output
                     |
                     v
              Metadata Validation
                     |
                     v
               Data Validation
                     |
                     v
                Database
                     |
                     v
              Analytics / Reports
Enter fullscreen mode Exit fullscreen mode

Each stage has a clear responsibility.

Scheduler

Starts extraction jobs at predefined intervals.

Scraping job

Executes the appropriate repository and records its exit status.

Output storage

Stores the extracted JSON or CSV data.

Metadata validation

Checks whether the run completed successfully.

Data validation

Checks the fields, identifiers, prices, and record counts.

Database

Stores normalized records for historical comparison.

Analytics

Calculates price changes, availability trends, and other useful information.

This architecture makes the scraper replaceable.

If a website changes its layout, you can update the extraction component without rewriting the analytics layer.

16. Testing and Maintenance

Websites change frequently.

A scraper should therefore be treated as software that needs testing, not as a script that can be forgotten after the first successful run.

The 2scraper repositories include offline test or smoke-test entry points, depending on the project.

These tests help verify parsing against saved, sanitized fixtures without requiring a live website request.

Some repositories also include scheduled GitHub Actions canaries that test the scraper against the live website.

A green canary is useful evidence that the last scheduled check succeeded, but it is not a guarantee that every query, marketplace, or page type works.

Recommended maintenance routine

After updating a repository:

git pull

python3 smoke_test.py
Enter fullscreen mode Exit fullscreen mode

If the repository uses pytest:

pytest tests/
Enter fullscreen mode Exit fullscreen mode

Check the README for the actual test command.

Then run a small live test using a limited number of pages.

Finally, compare the output schema against the version expected by your application.

Pin dependencies

For production deployments, avoid relying on an unchanging set of loosely specified dependencies.

Use a tested lock file when the repository provides one.

This makes builds more reproducible and reduces the risk of an unrelated package update breaking your scraping environment.

17. A Few Practical Mistakes to Avoid

After working with web scraping systems, several recurring mistakes become obvious.

Mistake 1: Assuming every run is complete

Always inspect exit codes and metadata.

A partial result is not the same as a complete dataset.

Mistake 2: Collecting too much data too early

Start with a small sample.

Validate the extraction before increasing the number of pages.

Mistake 3: Treating missing values as zero

A missing price is not a price of zero.

Keep null values distinct from real numeric values.

Mistake 4: Ignoring pagination

Some pages load additional products only after scrolling.

Check the actual number of records and whether all requested pages were processed.

Mistake 5: Using the wrong browser configuration

Do not combine incompatible proxy, fingerprint, and hosted-browser settings.

Follow the specific repository instructions.

Mistake 6: Assuming a CAPTCHA is the reason for every failure

A timeout, an HTTP 429 response, an invalid URL, and a page-layout change require different fixes.

Inspect the stop reason before changing infrastructure.

Mistake 7: Ignoring data provenance

Always record where the data came from and when it was collected.

Without timestamps and source identifiers, historical analysis quickly becomes unreliable.

18. Responsible Web Scraping

Before deploying any scraping workflow, consider the website's terms, applicable laws,

Top comments (0)