DEV Community: Goodness E. Eboh

We benchmarked 4 web scraping APIs for AI agents against DataDome, Cloudflare, and Akamai

Goodness E. Eboh — Sun, 11 Oct 2026 17:14:55 +0000

This article was originally published on the Zenrows blog.

This post compares five web scraping APIs for AI agents using 800 benchmark requests against DataDome, Cloudflare, and Akamai. It then shows you how to connect the strongest performer to Claude Desktop, the OpenAI Agents SDK, and LangChain in a few lines of code.

The APIs are Zenrows, Firecrawl, Jina AI Reader, CrawlForge, and Bright Data. Every number below comes from a public benchmark repository with the raw per-request results, so you can rerun any of it yourself.

The short version

Provider	Protected-target success rate	Avg. response time
Zenrows	100.00%	5,215 ms
Jina	99.33%	1,257 ms
Firecrawl	66.67%	1,497 ms
CrawlForge	0.00%	867 ms

All four tested APIs scored 100 percent on an unprotected Wikipedia page. The protected targets are where they separated. Bright Data wasn't part of the test because it requires enterprise KYC.

Why test anti-bot access first

Scraping APIs for agents usually get compared on output format, MCP support, and JavaScript rendering. Those features matter once the agent has the page. The harder question is whether the agent gets the page at all when Cloudflare, DataDome, or Akamai protects the target.

Public docs, company blogs, and news sites rarely cause trouble. E-commerce pricing pages, job boards, and industry directories are different. They return challenge pages, empty responses, or partial content, and the agent has nothing useful to work with. That's why this comparison measures anti-bot success rate before anything else.

How the benchmark worked

Zenrows, Jina AI Reader, and CrawlForge ran on August 20, 2026, with Zenrows using Fetch in mode=auto.
Firecrawl ran on August 25, 2026, in proxy=auto mode, which starts in standard mode and retries in stealth mode.
Each provider got 50 requests per target at a fixed rate of two requests per second, for 800 requests in total.
The targets were JobLookup behind DataDome, Chrono24 behind Cloudflare, Reuters behind Akamai, and Wikipedia with no protection as a baseline.
A request counted as successful only if it returned an HTTP 200 status code and a valid page title or meaningful data.

Every figure was recomputed from the raw dataset on October 1, 2026, so each one traces back to a logged request.

Full results

Provider	Target	Protection	Success rate	Avg. response time (ms)
Zenrows	JobLookup	DataDome	100%	10,053
Zenrows	Chrono24	Cloudflare	100%	4,590
Zenrows	Reuters	Akamai	100%	4,103
Zenrows	Wikipedia	Unprotected	100%	2,115
Jina	JobLookup	DataDome	98%	1,029
Jina	Chrono24	Cloudflare	100%	1,417
Jina	Reuters	Akamai	100%	1,242
Jina	Wikipedia	Unprotected	100%	1,340
Firecrawl	JobLookup	DataDome	0%	1,299
Firecrawl	Chrono24	Cloudflare	100%	1,688
Firecrawl	Reuters	Akamai	100%	1,488
Firecrawl	Wikipedia	Unprotected	100%	1,512
CrawlForge	JobLookup	DataDome	0%	781
CrawlForge	Chrono24	Cloudflare	0%	709
CrawlForge	Reuters	Akamai	0%	728
CrawlForge	Wikipedia	Unprotected	100%	1,251

Zenrows returned a usable page on every request across all four targets. Jina came closest and failed once against JobLookup. Firecrawl hit a reCAPTCHA page on all 50 JobLookup requests and scored 100 percent on the other three targets. CrawlForge got 403s on JobLookup and Chrono24 and a 401 on Reuters, so it only succeeded on Wikipedia.

Provider	Avg. success rate (3 protected targets)	Avg. response time (4 targets)	Observed consumption per request (4 targets)
Zenrows	100.00%	5,215 ms	4.59 credits
Jina	99.33%	1,257 ms	24,922.58 tokens
Firecrawl	66.67%	1,497 ms	1 credit
CrawlForge	0.00%	867 ms	1 credit

Jina was the fastest API that returned content on the protected targets. CrawlForge's low average reflects how quickly it was turned away. Zenrows was slower on average, especially on the protected targets, where it was the only API to clear every request.

Consumption uses each provider's native unit, so the figures don't compare directly on cost. They're dashboard readings taken before and after the run, divided by the 200 requests each provider served, and the balances are in the repository. A low consumption figure says little about value when the requests behind it fail.

How Zenrows credits scale with protection

The 4.59-credit average for Zenrows blends four very different targets, because mode=auto escalates only as far as each target forces it to.

The unprotected Wikipedia page cost one credit per request. Reuters, behind Akamai, cost ten. Cloudflare-protected Chrono24 averaged 2.36 because not every request needed the full escalation. For an agent whose targets mix easy and hard pages, you pay for the protection each page puts up.

The five APIs

Zenrows

Zenrows fits agents that need protected and dynamic web sources. It scored 100 percent across the protected targets in this benchmark, and Zenrows publishes a 99.93 percent success rate across its wider target set. Zenrows Fetch handles Cloudflare, DataDome, Akamai, and PerimeterX, along with dynamic website support, in one API call.

It returns pages as LLM-ready Markdown, plain text, or PDF, and Zenrows Extract returns structured JSON without pre-written CSS selectors. The Agent Toolkit ships on every plan and includes the following:

An MCP server exposes scrape, extract, and more than 30 browser_* tools, with first-party setup guides for Claude Desktop, Claude Code, Cursor, Windsurf, VS Code, Zed, JetBrains IDEs, OpenCode, Google Antigravity, the Claude Agent SDK, the OpenAI Agents SDK, Google ADK, and Mastra. You can run it locally over npx or use the hosted endpoint, and the tool surface is identical either way.
A CLI gives terminal-native agents zenrows fetch and zenrows extract, and zenrows mcp config --client claude-code writes the MCP configuration for a client.
Agentic payments let an agent buy its own access. Zenrows publishes a machine-readable API catalogue with prices an agent can read without a login, and it settles through x402, MPP, or a card. An agent that hits a 402 challenge can complete the purchase and carry on.

Zenrows Batch adds async jobs of up to 100,000 URLs with concurrency, retries, and webhook delivery. The free tier gives you 5,000 credits a month with no card. Paid plans run from the Build tier at $16/month for 45,000 credits to the Scale tier at $458/month for five million credits, plus a custom Enterprise tier. Billing is pay per successful request, so blocked requests cost nothing.

Firecrawl

Firecrawl suits agents that access unprotected or lightly protected sources, and it averaged 66.67 percent across the protected targets here. Its output is built for LLMs, with Markdown, JSON Schema generation, and metadata extraction designed for RAG and chunking. It also offers /agent and /interact endpoints for multi-page exploration and stateful browser control.

All 50 of its JobLookup responses came back as a reCAPTCHA page, even with stealth retries. Its AI Extract feature needs a separate subscription on top of the base plan. Firecrawl has a free plan with 1,000 credits a month and paid plans from $19/month.

Jina AI Reader

Jina is a zero-setup gateway that converts URLs into LLM-ready Markdown when you prepend r.jina.ai/ to them. It includes image captioning for multimodal agents. It placed second in this benchmark at 99.33 percent across the protected targets.

Its token consumption varied with the amount of content returned and averaged 24,922.58 tokens per request, which makes billing harder to predict on large crawls. Jina offers 10 million free tokens, and paid usage starts at $50, at $0.050 per million tokens. It's a good match for quick lookups of documentation, news, and public web content.

CrawlForge

CrawlForge is MCP-native and ships 31 specialized tools an agent can discover without wrapper code, including fetch_url, deep_research, map_site, and scrape_template. It produces token-efficient Markdown, can run extraction schemas locally through Ollama at no cost, and charges a predictable one credit for a basic fetch and up to 10 for deep research.

It scored zero percent on all three protected targets and 100 percent on Wikipedia. CrawlForge has a free tier of 1,000 credits and paid plans from $19/month for 5,000 credits. It works best for agents on MCP-native hosts that mostly access unprotected sources.

Bright Data

Bright Data is the enterprise option for protected-site access. It runs a pool of more than 150 million IPs and holds SOC 2 Type II and ISO 27001 certifications. Its Web Unlocker, Browser API, and Residential Proxies products each have $499/month Scale plans, so the full stack costs more than a typical self-service tool. It fits enterprise teams with five-figure monthly data budgets, compliance requirements, and dedicated data engineering staff.

Connecting Zenrows to your agent

You can give an agent access to live web data through the Zenrows MCP server or through a framework integration. Each pattern below needs a Zenrows API key, which you can get from the signup page.

Claude Desktop and other MCP hosts

{
  "mcpServers": {
    "zenrows": {
      "command": "npx",
      "args": ["-y", "@zenrows/mcp"],
      "env": {
        "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY"
      }
    }
  }
}

Add that block to claude_desktop_config.json and restart Claude Desktop. Claude can then call the scrape and browser_* tools whenever a task needs them. To see how this server compares with the alternatives, read the test of seven MCP servers for web scraping.

OpenAI Agents SDK

import asyncio
import os

from agents import Agent, HostedMCPTool, Runner

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

agent = Agent(
    name="Web Research Assistant",
    instructions="Use live web data when you need current information.",
    tools=[
        HostedMCPTool(
            tool_config={
                "type": "mcp",
                "server_label": "zenrows",
                "server_description": "Web data access through Zenrows.",
                "server_url": "https://mcp.zenrows.com/mcp",
                "authorization": ZENROWS_API_KEY,
                "require_approval": "never",
            }
        )
    ],
)

async def main():
    result = await Runner.run(
        agent,
        "Visit https://news.ycombinator.com/ and summarize the three most recent posts.",
    )
    print(result.final_output)

if __name__ == "__main__":
    asyncio.run(main())

The OpenAI Agents SDK supports MCP natively through HostedMCPTool, so this connects the hosted Zenrows MCP server with no custom function-calling logic. If you'd prefer to register Fetch as a plain function tool, this guide to building a web-aware agent with the OpenAI Agents SDK covers it.

LangChain

from langchain_zenrows import ZenRowsUniversalScraper
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent

llm = ChatOpenAI(model="gpt-4o-mini")
zenrows_tool = ZenRowsUniversalScraper()

agent = create_react_agent(llm, [zenrows_tool])

result = agent.invoke({
    "messages": [
        {
            "role": "user",
            "content": "Scrape https://www.etsy.com/ in Markdown and return the four cheapest products in JSON."
        }
    ]
})

The langchain-zenrows package exposes Zenrows as a LangChain tool, so the agent calls it whenever it needs live web data. Set ZENROWS_API_KEY as an environment variable before you run it.

Everything at a glance

Criteria	Zenrows	Firecrawl	Jina AI Reader	CrawlForge	Bright Data
Best for	AI agents accessing protected and dynamic web sources	RAG, LLM data ingestion, and unprotected or lightly protected sources	Quick lookups of documentation, news, and public web content	MCP-native agents accessing primarily unprotected sources	Enterprise teams requiring protected web access
Protected-target success rate	100%	66.67%	99.33%	0%	Not tested (requires enterprise-level KYC)
Output formats	Markdown, plaintext, PDF, structured JSON with Extract	Markdown, JSON Schema, metadata	LLM-ready Markdown, visual descriptions	Markdown, extraction schemas	HTML, Markdown, structured data and other formats depending on product
MCP server	Yes	Yes	Yes, official and community	Yes	Yes
SDK support	Python, Node.js, Go	Python, Node.js, and others	HTTP API	No dedicated SDK	Multiple SDK and API integrations
Free tier	5,000 credits/month, no card	1,000 credits/month	10 million tokens	1,000 credits	5,000 credits/month
Entry price	$16/month	$19/month	$50	$19/month	Product-dependent, with Scale plans from $499/month
Success-billed only	Yes	Partial (only API-level failures are free)	No	No	Partial (Web Unlocker, SERP API, and Web Scraper API products only)
KYC required	No	No	No	No	Yes for Residential and Mobile IP networks
Benchmark source	Our benchmark	Our benchmark	Our benchmark	Our benchmark	Not tested

What the benchmark settles

Every API here handles output format and tool coverage well. Protected access decides whether an agent returns an answer or a challenge page, and that's where the five tools differ most.

Across 800 requests, Zenrows was the only API to return a usable page every time on DataDome, Cloudflare, Akamai, and the unprotected baseline. It charged one credit for the page that needed no escalation. Jina came closest and runs faster on open targets. Firecrawl and CrawlForge are reasonable choices for agents whose targets stay public.

Choose based on the targets your agent will hit. If any of them sit behind an anti-bot system, start with 5,000 free credits and run your own list through Fetch before you commit.

What's next

Clone the benchmark repository and swap in your own targets to see how each API handles them.
Build a full agent on top of Fetch with this guide to an AI lead generation agent with the OpenAI Agents SDK.
Compare a wider set of options in the roundup of AI web scraping tools.

FAQ

Can AI agents scrape websites protected by Cloudflare or DataDome?

They can when the API they call handles the protection for them. In this benchmark, two of the four tested APIs returned nothing usable from the DataDome target across 50 requests each. Zenrows returned a valid page on every request to the Cloudflare, DataDome, and Akamai targets.

What's the difference between a web scraping API and a scraping MCP server?

A web scraping API is the service that fetches the page. An MCP server hands that service to an agent as a callable tool. Zenrows offers both, so the same Fetch capability works over REST, through the CLI, or as an MCP tool your agent discovers automatically.

Does an AI agent need JavaScript rendering to scrape a page?

It depends on the target. Pages that build their content in the browser return little usable text without rendering, which is why a plain HTTP fetch often comes back empty. Zenrows' mode=auto decides per request, so the agent doesn't have to.

Can I use Zenrows with CrewAI?

You can. CrewAI and other agent frameworks connect to Zenrows through its MCP server, and LangChain agents can use the langchain-zenrows package.

How we scraped the 2026 FIFA World Cup data and what shocked us

Goodness E. Eboh — Tue, 08 Sep 2026 14:46:14 +0000

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/scrape-2026-fifa-world-cup-data-python

We built a Python pipeline over three sources (Sofascore, FIFA, and Wikipedia) to pull the full 2026 World Cup dataset. Each source delivered its data differently. Each needed a different extraction strategy. This is how we handled all three, and what the data revealed when we had it.

Full source on GitHub.

Before you start

Python 3.10 or later
Zenrows account for your API key
playwright, requests, beautifulsoup4, pandas, python-dotenv

pip install requests playwright beautifulsoup4 pandas python-dotenv

Three sources, three access patterns

No single source had everything. FIFA had team-level metrics. Sofascore had player-level statistics. Wikipedia had tournament metadata.

The access patterns were all different across the three sources.

Sofascore delivered data through a JavaScript-rendered JSON endpoint
FIFA delivered it through a backend API behind a per-session JWT
Wikipedia delivered server-rendered HTML

Standard requests handles the last one. It fails silently on the first two, returning an empty shell or a challenge page instead of data.

Retrieving Sofascore data from a JavaScript endpoint

Sofascore is a single-page application. The front end calls an internal API and JavaScript renders the result into the browser. requests alone returns the application shell with no data.

We used Zenrows Fetch with mode=auto. Because the endpoint returns JSON, there was no HTML parsing step. The response was already structured data.

import os
import time
from urllib.parse import urlencode

import requests

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

class SofascoreClient:
    BASE_URL = "https://www.sofascore.com/api/v1"

    def _get(self, endpoint: str, params: dict | None = None):
        base_url = f"{self.BASE_URL}/{endpoint.lstrip('/')}"
        target_url = f"{base_url}?{urlencode(params)}" if params else base_url

        retries = 3
        for attempt in range(retries):
            try:
                response = requests.get(
                    "https://api.zenrows.com/v1/",
                    params={
                        "apikey": ZENROWS_API_KEY,
                        "url": target_url,
                        "mode": "auto",  # handles JS rendering and proxy escalation per request
                    },
                    timeout=120,
                )
                response.raise_for_status()
                return response.json()
            except (
                requests.exceptions.ReadTimeout,
                requests.exceptions.ConnectionError,
                requests.exceptions.HTTPError,
            ) as e:
                print(f"Attempt {attempt + 1}/{retries} failed: {e}")
                if attempt == retries - 1:
                    raise
                wait = 2 ** attempt
                print(f"Retrying in {wait}s...")
                time.sleep(wait)

    def fetch_statistics(self, group, order, page=1, limit=20, accumulation="total"):
        endpoint = f"unique-tournament/{TOURNAMENT_ID}/season/{SEASON_ID}/statistics"
        params = {
            "group": group,
            "order": order,
            "page": page,
            "limit": limit,
            "accumulation": accumulation,
        }
        return self._get(endpoint, params=params)

Retrieving FIFA data from an authenticated API

FIFA's data came from a backend API called with a JWT bearer token issued for the current browsing session. Every visitor gets a temporary token; the front end attaches it to subsequent API calls. The endpoint couldn't be queried as a public URL.

We used Zenrows Browser Sessions with Playwright to follow the same authenticated flow a normal browser follows and intercepted the API responses.

import os
from playwright.async_api import async_playwright

class FIFAClient:
    def __init__(self):
        self.api_key = os.getenv("ZENROWS_API_KEY")
        self.connection_url = f"wss://browser.zenrows.com?apikey={self.api_key}"
        self.responses = {}

    async def _handle_response(self, response):
        if "gameday-prod.fifa.mangodev.co.uk" not in response.url:
            return
        self.responses[response.url] = await response.json()

    async def fetch_all(self):
        async with async_playwright() as p:
            browser = await p.chromium.connect_over_cdp(self.connection_url)
            context = browser.contexts[0]
            page = await context.new_page()

            page.on("response", self._handle_response)  # intercept all API responses
            await page.goto(self.URL)
            await self._remove_cookie_banner(page)

            for tab in self.TABS:
                await self._click_tab(page, tab)

            self._save_files()

Retrieving Wikipedia tournament metadata

Wikipedia is server-rendered. Fetch retrieved it with no extra capabilities. BeautifulSoup located the stadium summary table by its headers.

from bs4 import BeautifulSoup
import re

def normalize(self):
    with open(self.INPUT_FILE, encoding="utf-8") as f:
        soup = BeautifulSoup(f, "html.parser")

    target_table = None

    for table in soup.find_all("table", class_="wikitable"):
        headers = [th.get_text(" ", strip=True) for th in table.find_all("th")]
        if "Year" in headers and "Host" in headers and "Stadiums" in headers:
            target_table = table
            break

    records = []
    for row in target_table.find("tbody").find_all("tr")[1:]:
        cells = row.find_all(["th", "td"])
        if len(cells) < 4:
            continue
        year_match = re.search(r"\d{4}", cells[0].get_text())
        if not year_match:
            continue
        records.append({
            "year": int(year_match.group()),
            "hosts": [t.strip() for t in cells[1].stripped_strings if t.strip()],
            "cities": int(re.search(r"\d+", cells[2].get_text()).group()),
            "stadiums": int(re.search(r"\d+", cells[3].get_text()).group()),
        })

Normalizing three sources into one format

Three sources fetched three ways produce three shapes of data. Each had its own normalization layer. Sofascore returned duplicate players across statistic groups, so we keyed on player IDs to deduplicate.

def normalize_player(self, record):
    player = record.get("player", {})
    team = record.get("team", {})
    stats = {k: v for k, v in record.items() if k not in {"player", "team", "rating"}}
    return {
        "player_id": player.get("id"),
        "player": player.get("name"),
        "team": team.get("name"),
        "rating": record.get("rating"),
        "stats": stats,
    }

def normalize_dataset(self, dataset):
    with open(f"data/raw/sofascore/{dataset}.json") as f:
        data = json.load(f)

    players = {}
    for record in data:
        normalized = self.normalize_player(record)
        if normalized["player_id"]:
            players[normalized["player_id"]] = normalized  # keep one record per player

    return list(players.values())

What the data revealed

Ten findings from the dataset that challenged what we expected.

Dembélé scored six goals from just over two expected goals, close to three times his xG
The Golden Glove winner (Unai Simón) didn't appear in our top ten goalkeepers by rating
Bellingham, Rodri, and Tchouaméni dominated the defensive action rankings ahead of centre-backs
Japan recorded the highest shot conversion rate while taking the lowest share of shots from inside the box
Brian Brobbey recorded a 75% conversion rate without missing a single big chance
Michael Olise set a new World Cup record with seven assists and led the tournament in big chances created
More pressing volume didn't produce quicker ball recovery. Türkiye applied fewer pressures and recovered faster than France, England, and Argentina
Argentina reached the final with the most bookings and no VAR interventions against them
Spain led the tournament in defensive line breaks while holding around 90% passing accuracy
The 2026 three-country World Cup used 16 stadiums, fewer than Spain used alone in 1982

The full findings with charts are in the original article.

What's next

Zenrows Batch for large-scale collection across thousands of URLs
Zenrows Browser Sessions for any authenticated scraping workflow
Web data for LLM fine-tuning for feeding scraped data into training pipelines

How to Build a RAG Pipeline with Live Web Data Using ZenRows and LlamaIndex

Goodness E. Eboh — Mon, 13 Jul 2026 12:42:04 +0000

How do you make sure the documents entering your VectorStoreIndex contain actual content when building a LlamaIndex RAG (retrieval augmented generation) pipeline? This question comes up because the built-in SimpleWebPageReader struggles with many of the dynamic web pages it encounters.

Instead of valid page content, the reader can return Cloudflare challenge pages, empty React shells, or content with a poor signal-to-noise ratio relative to the tokens spent. This tutorial directly addresses this issue. It shows how replacing the built-in web reader with one that returns clean LlamaIndex documents from live URLs improves the quality of data available to agents, query engines, and vector indexes in RAG systems.

Why LlamaIndex web scraping loaders fail on real websites

There are three major failure modes that LlamaIndex's built-in web loaders show against modern web infrastructure.

The first is the Cloudflare/WAF (Web Application Firewall) block. When the SimpleWebPageReader makes a standard HTTP request using its underlying libraries like requests or urllib, it lacks capabilities such as browser fingerprints, TLS fingerprints, and human-like behavioral patterns. Security layers on websites protected by WAFs use these signals to identify automated requests and cause the reader to receive HTTP error pages or challenge screens instead.

The second failure mode is SPA (Single Page Application) empty shells. Many modern websites use frameworks like React, Angular, or Vue. When they are scraped, the initial HTML response is usually just an empty template shell containing an initialization script block. Since the reader does not execute JavaScript, it only sees the initial HTML shell.

The third is token bloat. Even for modern unprotected static sites, built-in readers pull raw HTML or raw text, which usually include navigation bars, headers, footers, sidebars, cookie banners, and tracking scripts. This unnecessary data, when indexed, costs tokens that would have been better spent pulling in more relevant content.

Using the SimpleWebPageReader to load data from an anti-bot challenge page shows exactly how output failures can occur.

from llama_index.readers.web import SimpleWebPageReader
documents = SimpleWebPageReader(html_to_text=True).load_data(
    ["https://www.scrapingcourse.com/cloudflare-challenge"]
)

print(documents[0].text[:200])

The reader was only able to pull the challenge page.

If a document like this is added to the VectorStoreIndex, LLM (Large Language Model) outputs would have no information related to the content that was meant to be pulled. At the same time, tokens are still consumed during both retrieval and generation. To avoid these failure modes, use a reader designed to access protected websites, execute JavaScript, and return cleaner content for indexing. The ZenRowsWebReader is one such reader.

Setting up `ZenRowsWebReader`

The ZenRowsWebReader is supported in LlamaIndex and can return valid Document objects from live pages that are ingestible by AI applications. It can access websites protected by WAF challenges, render JavaScript, and output markdown, all of which improve document quality. Here is how you can set it up.

To follow along, first generate a ZenRows API key from the ZenRows Dashboard and store it as the ZENROWS_API_KEY environment variable.

Next, import the reader and configure it with the following parameters to handle live web pages.

When reading sites protected by advanced anti-bot mechanisms like Cloudflare, enable the premium_proxy parameter to use residential IPs. Residential IPs are less likely to be identified as automated traffic by anti-bot systems.

from llama_index.readers.web import ZenRowsWebReader

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

Lastly, use the ZenRowsWebReader against the same website that the SimpleWebPageReader failed to scrape.

from llama_index.readers.web import ZenRowsWebReader
from dotenv import load_dotenv
import os

load_dotenv()

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

documents = reader.load_data([
    "https://www.scrapingcourse.com/cloudflare-challenge"
])

print(documents[0].text[:500])

Running it, the output shows that the reader successfully accessed the website, scraped its content, and returned it in Markdown.

Building a RAG pipeline with live web data

Having resolved the data quality issue with scraping web content from live pages using ZenRowsWebReader, the next step is to include the reader in a RAG pipeline. While the complete code is a single file, the implementation consists of four stages: configuring the pipeline to use OpenAI models, ingesting live web data with the reader, creating the vector index, and querying it to generate responses.

Step 1: Configure the OpenAI model and embedding model

Import all dependencies, load the API keys, and configure the OpenAI model and embedding model.

from dotenv import load_dotenv
import os

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

Settings.llm = OpenAI(
    model="gpt-4o-mini",
    api_key=OPENAI_API_KEY
)

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_key=OPENAI_API_KEY
)

This configuration uses gpt-4o-mini for fast, cost-effective response generation and text-embedding-3-small to create semantic embeddings that provide a good balance of retrieval quality, latency, and cost. Both models can be replaced with other supported OpenAI models depending on your specific requirements.

Step 2: Load live web data into LlamaIndex documents

Just like in the previous section, import the ZenRowsWebReader and configure it to load data from the target URLs into LlamaIndex Document objects.

from llama_index.readers.web import ZenRowsWebReader

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

documents = reader.load_data([
    "https://www.scrapingcourse.com/cloudflare-challenge",
    "https://www.scrapingcourse.com/javascript-rendering"
])

print(f"Loaded {len(documents)} document(s)")

Step 3: Create the vector index

Import VectorStoreIndex and pass the Document objects into it to create the index.

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents)

print("Index created successfully")

This example creates an in-memory VectorStoreIndex for simplicity. LlamaIndex also supports persisting indexes to disk or external vector stores.

Step 4: Query the index

Use the query engine to retrieve relevant content from the index and pass it to the OpenAI model for response generation.

query_engine = index.as_query_engine()

response = query_engine.query(
    "Summarize the information available across the indexed pages."
)

print("\nAnswer:\n")
print(response.response)

The result is that the LLM retrieved valid indexed data from the documents and generated output that matched the scraped content.

While this pipeline delivers valid data, websites frequently change layouts and content. So it’s important to have a layer that updates the index with fresh Document objects.

Keeping your knowledge base current

One-off loads deliver valid content, but they can quickly become outdated. Periodic refreshes allow current web content to be re-scraped from live pages as source content changes. But if the logic isn't right, it can cause the same content to be re-ingested repeatedly, increasing storage costs and reducing retrieval quality through duplicate content.

You can achieve efficient refreshes by generating hashes from the scraped page content and comparing them against previously stored values. This logic runs after loading documents with the ZenRowsWebReader and before passing the updated documents to the VectorStoreIndex.from_documents().

In addition to the existing reader setup, this implementation uses the json and hashlib modules to persist and compare content hashes.

import json
import hashlib

To set up the logic, create a hash store. In production, this is usually a database rather than a local file. Then create a set of functions that store hashes of scraped documents and compare them against newly scraped content.

HASH_STORE = "content_hashes.json"

def load_hashes():
    if os.path.exists(HASH_STORE):
        with open(HASH_STORE, "r") as f:
            return json.load(f)
    return {}

def save_hashes(hashes):
    with open(HASH_STORE, "w") as f:
        json.dump(hashes, f, indent=4)

def get_updated_documents(urls):
    documents = reader.load_data(urls)

    if not documents:
        print("No documents returned")
        return []

    stored_hashes = load_hashes()
    updated_documents = []

    for doc in documents:
        if not doc.text.strip():
            source_url = doc.metadata.get("source_url", "unknown")
            print(f"Skipping empty content: {source_url}")
            continue

        source_url = doc.metadata.get("source_url", "unknown")

        content_hash = hashlib.md5(
            doc.text.encode("utf-8")
        ).hexdigest()

        previous_hash = stored_hashes.get(source_url)

        if previous_hash == content_hash:
            print(f"No changes detected: {source_url}")
            continue

        print(f"Updated content detected: {source_url}")

        stored_hashes[source_url] = content_hash
        updated_documents.append(doc)

    save_hashes(stored_hashes)
    return updated_documents

documents = get_updated_documents([
    "https://www.scrapingcourse.com/cloudflare-challenge",
    "https://www.scrapingcourse.com/javascript-rendering"
])

During each refresh cycle, the function compares the latest scraped content against the previously stored hashes. Unchanged pages are skipped, while pages with updated content are returned for re-indexing.

At this point, the pipeline is fully functional. But even with clean web data and a working retrieval pipeline, you can still encounter issues with answer quality when querying.

Troubleshooting incomplete answers

Incomplete answers can result from retrieval configuration in LlamaIndex or from incomplete page content returned by the web reader. Here is how to troubleshoot both.

Insufficient retrieval context

During retrieval, LlamaIndex returns only a subset of the highest-scoring chunks from the vector index to provide relevant context for the LLM. By default, this naive top-k retrieval matches query embeddings with chunk embeddings. Since live webpages are broken down into multiple markdown chunks, important context can end up distributed across multiple document chunks.

The fix

Increase the top-k parameter to 10 so the query engine pulls more relevant chunks into the retrieval context, improving the likelihood of retrieving related information.

query_engine = index.as_query_engine(
    similarity_top_k=10
)

Missing dynamic JavaScript content

If your final output indicates missing information or returns placeholder strings such as "Enable JavaScript to see products," the scraper may be capturing the page before JavaScript-rendered elements have finished loading.

The fix

In the ZenRows reader parameters, verify that js_render=True is enabled and that a wait=2000 configuration has been added. These settings allow dynamic elements time to load so the target elements can fully render before the page is converted into a LlamaIndex document.

Conclusion

You have now built a RAG pipeline that uses the ZenRowsWebReader to ingest live web data into LlamaIndex. It can retrieve content from protected and JavaScript-driven websites, keep indexed content up to date through periodic refreshes, and generate responses from documents that standard web loaders often fail to retrieve.

DEV Community: Goodness E. Eboh

We benchmarked 4 web scraping APIs for AI agents against DataDome, Cloudflare, and Akamai

The short version

Why test anti-bot access first

How the benchmark worked

Full results

How Zenrows credits scale with protection

The five APIs

Zenrows

Firecrawl

Jina AI Reader

CrawlForge

Bright Data

Connecting Zenrows to your agent

Claude Desktop and other MCP hosts

OpenAI Agents SDK

LangChain

Everything at a glance

What the benchmark settles

What's next

FAQ

How we scraped the 2026 FIFA World Cup data and what shocked us

Before you start

Three sources, three access patterns

Retrieving Sofascore data from a JavaScript endpoint

Retrieving FIFA data from an authenticated API

Retrieving Wikipedia tournament metadata

Normalizing three sources into one format

What the data revealed

What's next

How to Build a RAG Pipeline with Live Web Data Using ZenRows and LlamaIndex

Why LlamaIndex web scraping loaders fail on real websites

Setting up ZenRowsWebReader

Building a RAG pipeline with live web data

Step 1: Configure the OpenAI model and embedding model

Step 2: Load live web data into LlamaIndex documents

Step 3: Create the vector index

Step 4: Query the index

Keeping your knowledge base current

Troubleshooting incomplete answers

Insufficient retrieval context

The fix

Missing dynamic JavaScript content

The fix

Conclusion

Setting up `ZenRowsWebReader`