ScraperAPI Alternative: How to Build a Rotating Proxy Scraper with Python

If you are paying per-request for a managed scraper API and still managing proxy pools, retry logic, and output formatting on your own, an open-source self-hosted workflow can cut costs and give you full control. For developers and data teams who need high-volume web scraping with rotating proxies, automatic retries, and geographic targeting, the ScraperAPI Alternative repository provides a free Python-based starting point you can run locally or on your own infrastructure.

This article is for Python developers, data engineers, and technical founders who want to understand what a self-hosted scraper API replacement looks like, how to set it up, and when it makes sense compared to a managed service.

TL;DR

• Clone the repo, install dependencies, and run the CLI against a URL list with environment-configured proxies.
• The workflow handles rotating proxies, retry logic, response timing, and structured JSON or CSV export.
• Use it when you have your own proxy infrastructure, need predictable flat costs, or want to customize request behavior beyond what a managed API allows.
• Consider a managed service when you need anti-bot bypass, JavaScript rendering, or zero infrastructure maintenance.
• For a related tool that focuses on anti-bot protection, see the ZenRows Alternative repository.

Why Managed Scraper APIs Have Limits

Managed scraper APIs like ScraperAPI, ScrapingBee, and ZenRows solve a real problem: they bundle proxies, retries, and parsing so you can fetch data with a single HTTP call. That convenience comes with tradeoffs:

• Per-request pricing scales quickly when you need thousands or millions of pages.
• Rate limits and quotas can throttle large pipelines during peak usage.
• Limited customization over headers, fingerprinting, session handling, and retry policies.
• Geographic and proxy-type restrictions may not match your exact use case.
• Vendor lock-in makes migration expensive if pricing or terms change.

A self-hosted alternative makes sense when you already have proxy access, want to batch requests cheaply, or need to tweak behavior that a managed API abstracts away.

What the ScraperAPI Alternative Repository Provides

The ScraperAPI Alternative repository is a free, open-source Python project that demonstrates a local scraper API replacement. It is MIT-licensed and designed for developers who want structured output without subscription fees.

Verified capabilities from the repository:

• Rotating proxy pool support via environment-configured proxy lists.
• Automatic retry and failover with configurable attempt limits.
• Geographic targeting by selecting proxy countries.
• Python SDK and CLI for scripting and terminal use.
• Concurrent request support for faster batch extraction.
• JSON and CSV export with consistent field naming.

The extracted fields include:

url | status_code | content | proxy_country | proxy_type | attempts | response_time | timestamp

This is not a drop-in replacement for every ScraperAPI feature. It does not include built-in anti-bot bypass or headless browser rendering. If your targets use Cloudflare or reCAPTCHA, you may need the ZenRows Alternative repository instead, which focuses on anti-bot bypass and JavaScript rendering.

Setup and Installation

Prerequisites

• Python 3.11 or newer
• A proxy list or proxy service subscription (the repository does not provide free proxies)
• git and pip

Install

git clone https://github.com/data-scrape/scraperapi-alternative.git
cd scraperapi-alternative
pip install -r requirements.txt

Configure proxies

Create a .env file or export environment variables. Do not hardcode credentials in scripts.

export PROXY_LIST="http://user1:[e-mail]:8080,http://user2:[e-mail]:8080"
export MAX_RETRIES="3"
export TIMEOUT_SECONDS="30"
export OUTPUT_FORMAT="json"

Runnable Python Workflow

Below is a self-contained Python example that mirrors the repository's approach. It reads a list of target URLs, rotates through proxies, retries on failure, and writes structured results to a JSONL file. You can adapt it to the repository's CLI or import its modules once installed.

import os
import json
import time
import random
from datetime import datetime, timezone
from urllib.parse import urlparse
import requests

# Configuration from environment
PROXY_LIST = os.environ.get("PROXY_LIST", "").split(",")
MAX_RETRIES = int(os.environ.get("MAX_RETRIES", "3"))
TIMEOUT = int(os.environ.get("TIMEOUT_SECONDS", "30"))
OUTPUT_FILE = os.environ.get("OUTPUT_FILE", "scraping_results.jsonl")

# Target URLs to scrape (replace with your own list)
TARGET_URLS = [
    "https://httpbin.org/get",
    "https://httpbin.org/ip",
]

def pick_proxy():
    """Rotate through the proxy list randomly."""
    proxies = [p.strip() for p in PROXY_LIST if p.strip()]
    if not proxies:
        return None
    selected = random.choice(proxies)
    parsed = urlparse(selected)
    proxy_meta = {
        "http": selected,
        "https": selected,
    }
    country_hint = parsed.hostname.split(".")[0] if parsed.hostname else "unknown"
    return proxy_meta, country_hint

def fetch_url(target_url: str):
    """Fetch a single URL with retries and structured output."""
    attempts = 0
    last_error = None
    proxy_country = "direct"
    proxy_type = "none"

    while attempts < MAX_RETRIES:
        attempts += 1
        proxy_meta = None
        try:
            proxy_config = pick_proxy()
            if proxy_config:
                proxy_meta, proxy_country = proxy_config
                proxy_type = "rotating"
            else:
                proxy_country = "direct"
                proxy_type = "none"

            start = time.time()
            resp = requests.get(
                target_url,
                proxies=proxy_meta,
                timeout=TIMEOUT,
                headers={
                    "User-Agent": (
                        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                        "AppleWebKit/537.36 (KHTML, like Gecko) "
                        "Chrome/120.0.0.0 Safari/537.36"
                    )
                },
            )
            elapsed = round(time.time() - start, 3)

            return {
                "url": target_url,
                "status_code": resp.status_code,
                "content": resp.text[:2000],
                "proxy_country": proxy_country,
                "proxy_type": proxy_type,
                "attempts": attempts,
                "response_time": elapsed,
                "timestamp": datetime.now(timezone.utc).isoformat(),
            }
        except Exception as exc:
            last_error = str(exc)
            time.sleep(1 * attempts)  # exponential backoff feel

    # All retries exhausted
    return {
        "url": target_url,
        "status_code": None,
        "content": f"Failed after {attempts} attempts. Last error: {last_error}",
        "proxy_country": proxy_country,
        "proxy_type": proxy_type,
        "attempts": attempts,
        "response_time": None,
        "timestamp": datetime.now(timezone.utc).isoformat(),
    }

def main():
    results = []
    for url in TARGET_URLS:
        print(f"Scraping: {url}")
        result = fetch_url(url)
        results.append(result)
        # Polite pacing between requests
        time.sleep(random.uniform(1.0, 2.5))

    # Write JSONL output
    with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
        for r in results:
            f.write(json.dumps(r, ensure_ascii=False) + "\n")

    print(f"Done. Wrote {len(results)} records to {OUTPUT_FILE}")

if __name__ == "__main__":
    main()

How to run

# Set your proxies
export PROXY_LIST="http://user:[e-mail]:8080,http://user:[e-mail]:8080"

# Run the script
python scraper_workflow.py

Representative Output

Each line in scraping_results.jsonl is a JSON object:

{
  "url": "https://httpbin.org/get",
  "status_code": 200,
  "content": "{...}",
  "proxy_country": "us-proxy",
  "proxy_type": "rotating",
  "attempts": 1,
  "response_time": 1.234,
  "timestamp": "2026-08-21T10:00:00+00:00"
}

You can load this into pandas, a data warehouse, or a queue for downstream processing.

Use Cases

• SEO rank tracking: Scrape SERP pages from multiple countries to compare rankings.
• Price comparison: Monitor competitor pricing across regions without per-request fees.
• Content aggregation: Collect articles, listings, or directories on a scheduled basis.
• Data pipeline seeding: Produce structured JSONL files for ETL workflows.
• Migration testing: Run the open-source workflow alongside your current managed API to compare output and costs.

Comparison: Self-Hosted vs Managed Scraper

Dimension	Self-hosted (ScraperAPI Alternative)	Managed ScraperAPI/ZenRows
Best for	Teams with proxy infra and Python skills	Teams that want zero infrastructure
Setup model	Clone, install, configure proxies	Sign up, get API key, send HTTP requests
Data coverage	Any public URL you can request	Any public URL the platform supports
Output format	JSON/CSV via configurable export	JSON/JSONL/HTML via API parameters
Maintenance burden	You manage proxies, retries, and parsing	Provider manages proxies and anti-bot
Integration path	Python script, CLI, or custom wrapper	REST API call from any language
Quota/freshness	Limited by your proxy bandwidth and politeness	Limited by plan tier and rate limits
Pricing verification	Free (open source) + proxy costs	Verify current pricing on the provider's site

Limits, Compliance, and Maintenance

• Proxies are not included. You must bring your own proxy list or service. Free public proxies are unreliable and risky for production.
• No anti-bot bypass. If your target uses Cloudflare, reCAPTCHA, or advanced bot detection, this workflow may be blocked. For those cases, evaluate the ZenRows Alternative repository or a managed service.
• Rate limiting is your responsibility. Respect target-site terms, robots.txt directives where applicable, and applicable laws. Aggressive concurrent scraping can get your proxies banned.
• Page layout changes break parsing. If you extract fields from HTML rather than using the raw content approach above, monitor target pages for structural changes.
• Geographic accuracy depends on proxy quality. Verify that your proxies actually route through the advertised countries.
• Compliance varies by jurisdiction. Review local data-protection regulations and the target platform's terms of service before scraping at scale. For a detailed discussion of public web data compliance, see this Chinese-language public-web-data compliance guide.

FAQ

Is there an official ScraperAPI open-source project?
No. ScraperAPI is a commercial service. The ScraperAPI Alternative repository is an independent open-source project that demonstrates a similar workflow.

What data fields are returned?
The repository returns url, status_code, content, proxy_country, proxy_type, attempts, response_time, and timestamp. You can extend the script to parse additional fields.

How often should the workflow run?
It depends on your use case. Price monitors may run hourly; SEO trackers may run daily. Always add polite delays between requests and respect target-site rate limits.

What happens when a page layout changes?
If you parse HTML, you must update selectors when the layout changes. Using the raw-content approach shown above defers parsing to your downstream pipeline, which isolates the breakage.

Can this connect to n8n, a queue, or an AI agent?
Yes. The JSONL output is easy to ingest into n8n, Airflow, RabbitMQ, or an LLM context window. You can also wrap the script in a FastAPI service if you need an HTTP interface.

What should I verify before production use?
Test proxy reliability under load, confirm that your targets do not block your proxy IPs, measure actual response times, and review the legal context of the data you are collecting.

Does this replace ZenRows or ScrapingBee too?
Partially. It replaces the proxy-rotation and retry layer. It does not replace advanced anti-bot bypass or headless rendering. For anti-bot targets, compare with the ZenRows Alternative repository.

Next Steps

If you are ready to move from managed per-request pricing to a self-hosted Python scraper:

1. Clone the ScraperAPI Alternative repository.
2. Review the README, install dependencies, and configure your proxy list via environment variables.
3. Run a small batch, inspect the JSON output, and adapt the code to your target sites.
4. For anti-bot targets, also explore the ZenRows Alternative repository.
5. Browse the full data-scrape organization for more open-source scraping tools.

Start with a small test batch, measure reliability and cost, then scale your pipeline once you are confident in the results.