ZenRows Alternative: How to Extract Structured Web Data with Python

If you are looking for a ZenRows alternative that you can run on your own hardware and adapt to your own targets, the best starting point is an open-source Python scraper that handles request rotation, retry logic, and structured output parsing in one place. For developers, data engineers, and technical founders who want full control over how public web pages are fetched and normalized, the data-scrape/zenrows-alternative repository provides a free, MIT-licensed reference implementation you can extend.

This article is for anyone comparing managed scraping services against a self-hosted workflow. You will learn what the repository provides, how to set it up, how to write a runnable Python extraction pipeline, and when a managed service still makes sense.

TL;DR

• Clone the repository, install dependencies, and run the CLI against a list of public URLs.
• The workflow includes rotating user agents, optional proxy rotation, retries, and structured JSON or CSV export.
• Use it when you want predictable costs, full control over request behavior, and a scraper you can modify for your own targets.
• Do not expect any open-source script to guarantee bypass of modern anti-bot systems; always respect robots directives and platform terms.
• For related open-source scraping workflows, see the data-scrape/scraperapi-alternative and data-scrape/scrapingbee-alternative repositories.
• For a Chinese-language public-web-data reference on lawful data collection, see the public-web-data compliance guide.

Why Managed Scraping Services Have Limits

ZenRows, ScrapingBee, ScraperAPI, and similar services bundle proxies, retries, and parsing into a single API call. That convenience is valuable, but it comes with tradeoffs:

• Per-request or tiered pricing can become expensive at high volume.
• Quota and concurrency limits may throttle large pipelines.
• Limited control over headers, sessions, and retry policies.
• Migration friction increases when pricing, terms, or data formats change.

A self-hosted alternative is attractive when you already have proxy access, want flat infrastructure costs, or need to customize behavior that a managed API abstracts away.

What the Verified Repository Provides

The data-scrape/zenrows-alternative repository is an open-source Python project that demonstrates a local scraping workflow. It is not a managed service and does not promise unlimited coverage, but it gives you a working foundation for fetching and structuring public web pages.

Verified capabilities from the repository README and source:

• Rotating user agents to vary request signatures.
• Optional proxy rotation through an environment-configured proxy or a fetched proxy list.
• Retry loop with configurable attempts and timeout handling.
• HTML parsing with BeautifulSoup to extract title, text, links, images, and meta tags.
• Structured output as JSON or CSV with consistent field names.
• CLI and Python classes for scripting and batch use.

The extracted fields include:

url | status_code | title | text | html | links | images | meta_description | meta_keywords | scrape_time

This is a general-purpose scraping foundation. It is not a drop-in replacement for every ZenRows feature, and no open-source script can guarantee bypass of Cloudflare, reCAPTCHA, or other protections that platforms use to manage automated traffic. If you need those capabilities, evaluate managed providers carefully and verify their current offerings against your target sites.

Environment Setup

Prerequisites

• Python 3.11 or newer
• git and pip
• An optional proxy list or proxy service for rotation

Install the repository

git clone https://github.com/data-scrape/zenrows-alternative.git
cd zenrows-alternative
pip install -r requirements.txt

Configure environment variables

Create a .env file or export variables in your shell. Do not hardcode credentials or proxy URLs in scripts.

export TARGET_URL="https://example.com"
export PROXY_URL="http://user:[e-mail]:8080"
export MAX_RETRIES="3"
export TIMEOUT_SECONDS="30"
export OUTPUT_FILE="structured_output.json"

If you do not have a proxy, leave PROXY_URL empty. The script will fall back to direct requests, which is fine for small-scale testing against sites that allow it.

Runnable Python Workflow

The repository includes a ProxyScraper class that mirrors the workflow below: build a session, rotate user agents, optionally route through a proxy, retry on failure, parse the response with BeautifulSoup, and emit structured fields. The example below is self-contained and uses environment variables for anything that changes between environments.

import os
import json
import time
import random
from datetime import datetime, timezone
from urllib.parse import urljoin, urlparse
import requests
from bs4 import BeautifulSoup

# Configuration from environment
TARGET_URL = os.environ.get("TARGET_URL", "https://example.com")
PROXY_URL = os.environ.get("PROXY_URL", "")
MAX_RETRIES = int(os.environ.get("MAX_RETRIES", "3"))
TIMEOUT = int(os.environ.get("TIMEOUT_SECONDS", "30"))
OUTPUT_FILE = os.environ.get("OUTPUT_FILE", "structured_output.json")

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
]


def fetch_page(url: str, proxy_url: str = "") -> dict:
    """Fetch a public page with retries, header rotation, and optional proxy."""
    proxies = {"http": proxy_url, "https": proxy_url} if proxy_url else None
    session = requests.Session()
    result = {
        "url": url,
        "fetched_at": datetime.now(timezone.utc).isoformat(),
        "status_code": None,
        "title": None,
        "meta_description": None,
        "text": None,
        "links": [],
        "images": [],
        "error": None,
    }

    for attempt in range(1, MAX_RETRIES + 1):
        headers = {
            "User-Agent": random.choice(USER_AGENTS),
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.9",
            "Accept-Encoding": "gzip, deflate, br",
        }
        try:
            resp = session.get(
                url,
                headers=headers,
                proxies=proxies,
                timeout=TIMEOUT,
                allow_redirects=True,
            )
            result["status_code"] = resp.status_code
            soup = BeautifulSoup(resp.text, "html.parser")

            # Remove script and style elements before extracting text
            for tag in soup(["script", "style"]):
                tag.decompose()

            title_el = soup.find("title")
            result["title"] = title_el.get_text(strip=True) if title_el else None

            desc_el = soup.find("meta", attrs={"name": "description"}) or soup.find(
                "meta", attrs={"property": "og:description"}
            )
            result["meta_description"] = desc_el.get("content") if desc_el else None

            result["text"] = soup.get_text(separator=" ", strip=True)[:5000]

            base_url = resp.url
            links = [
                urljoin(base_url, a.get("href", ""))
                for a in soup.find_all("a", href=True)
            ]
            result["links"] = list(dict.fromkeys(links))[:100]

            images = [
                urljoin(base_url, img.get("src", ""))
                for img in soup.find_all("img", src=True)
            ]
            result["images"] = list(dict.fromkeys(images))[:100]

            return result

        except requests.exceptions.ProxyError:
            result["error"] = f"Proxy failed on attempt {attempt}"
            time.sleep(1)
        except requests.exceptions.Timeout:
            result["error"] = f"Timeout on attempt {attempt}"
            time.sleep(1)
        except requests.exceptions.RequestException as exc:
            result["error"] = f"Request failed on attempt {attempt}: {exc}"
            time.sleep(1)

    return result


def main():
    print(f"Fetching {TARGET_URL} ...")
    record = fetch_page(TARGET_URL, PROXY_URL)
    with open(OUTPUT_FILE, "w", encoding="utf-8") as fh:
        json.dump(record, fh, indent=2, ensure_ascii=False)
    print(f"Wrote structured output to {OUTPUT_FILE}")
    print(f"Status: {record['status_code']}, Title: {record['title']}")


if __name__ == "__main__":
    main()

Run the script after setting your environment variables:

python zenrows_workflow.py

Representative Output

The script writes a JSON file that separates raw HTML from extracted structure. Here is an example shape against a public documentation-style page:

{
  "url": "https://example.com/page",
  "fetched_at": "2026-08-24T02:00:00+00:00",
  "status_code": 200,
  "title": "Example Public Page",
  "meta_description": "A sample page used for structured extraction examples.",
  "text": "Example Public Page This is a sample paragraph with useful text content...",
  "links": [
    "https://example.com/about",
    "https://example.com/contact"
  ],
  "images": [
    "https://example.com/images/logo.png"
  ],
  "error": null
}

Use this structure as a contract for downstream tools. Your ingestion layer can expect url, status_code, title, text, links, and images on every successful fetch, with error populated only when every retry failed.

Build vs Managed: A Decision Checklist

Dimension	Self-hosted open-source workflow	Managed scraping service
Best for	Custom targets, predictable volume, full control	Fast time-to-data, no infra team
Setup model	Clone repo, install deps, configure proxies	Sign up, get API key
Data coverage	Any public URL you can reach	Provider-supported targets
Output format	JSON/CSV from your parser	Provider schema
Maintenance burden	You fix parsers when layouts change	Provider handles most target changes
Integration path	Python modules or CLI	REST API client
Proxy handling	Bring your own or fetched lists	Usually included
Compliance	Fully on you	Shared; verify policies

Use the self-hosted path when cost predictability and control matter more than convenience. Use a managed service when you need JavaScript rendering, anti-bot handling, or a team to maintain target-specific parsers.

Common Use Cases

• Price monitoring: Track public product pages and extract prices, titles, and availability.
• Content aggregation: Collect article headlines, summaries, and links for dashboards.
• SEO auditing: Extract meta descriptions, titles, and headings across a URL list.
• Lead research: Normalize public business pages into CRM-ready records.
• Competitive monitoring: Watch public feature pages and changelog updates.

Limit requests to public pages, honor robots.txt, and check the target site's terms before running at scale.

Compliance and Maintenance Notes

No scraping workflow is set-and-forget. Plan for the following operational concerns:

• Rate limiting: Space requests out, respect Retry-After headers, and start with a small test set.
• Robots directives: Check robots.txt for crawl rules and disallow paths.
• Terms of service: Public data is not automatically lawful data. Verify that your use case complies with the target platform's terms and applicable law.
• Layout drift: Page structures change. Keep your selectors simple and isolate parsing logic so one change does not break the whole pipeline.
• Privacy: Do not collect personal data, credentials, or non-public content. If a page requires login, it is outside the scope of this workflow.
• Proxy reliability: Free proxy lists are often slow or offline. For production workloads, use a reputable proxy provider and rotate credentials safely.

FAQ

Is there an official ZenRows API?
Yes. ZenRows is a commercial managed scraping service with its own API, pricing, and feature set. The repository discussed here is an independent open-source project, not an official integration.

What data fields does the open-source workflow return?
The repository extracts url, status_code, title, text, html, links, images, meta_description, meta_keywords, and scrape_time. You can extend the parser to return additional fields.

Can this bypass Cloudflare or CAPTCHA?
No open-source script can reliably bypass modern anti-bot protections, and doing so may violate the target site's terms. The repository includes header rotation and proxy support as request-management techniques, not guarantees.

How often should the workflow run?
Match the cadence to data freshness needs. Price monitors may run hourly; SEO audits may run weekly. Always stay within polite request rates and platform limits.

Can I connect this to a queue, database, or AI agent?
Yes. The JSON output is easy to feed into Celery, RabbitMQ, PostgreSQL, or an LLM context pipeline. Store the raw HTML only if you need it; otherwise keep the structured fields.

What should I verify before production use?
Confirm that your proxy provider allows your target sites, that your use case complies with platform terms, that your parsing selectors match the current page layout, and that your retry/backoff strategy will not overwhelm the target server.

Next Steps

Start with the data-scrape/zenrows-alternative repository to see the full CLI, configuration options, and source code. If you also want to compare rotating-proxy and headless-browser alternatives, explore the data-scrape/scraperapi-alternative and data-scrape/scrapingbee-alternative repositories on the same data-scrape profile.