DEV Community: ZenRows

How to Build a Stateful Web Research Agent with ZenRows and LangGraph

Praise James — Tue, 21 Jul 2026 19:07:55 +0000

Imagine a LangGraph research agent monitoring technology news for live web data. The scrape node retrieves an article, the extraction node pulls the key facts, and the synthesis node turns them into a briefing. This workflow looks reliable until the scrape node hits a protected site. Instead of article content, it returns a challenge page, and LangGraph stores that response in state as if it were valid content. That is the problem this tutorial solves.

In this tutorial, you'll build a stateful web research agent with LangGraph and ZenRows. The agent will scrape news articles into graph state, retry failed retrievals through a conditional branch, extract structured fields, and generate a briefing from multiple sources. ZenRows retrieves valid content from protected, JavaScript-rendered sites, reducing how often the retry branch needs to run. By the end, you'll have a three-node research graph that routes retrieval failures into a retry branch before bad data reaches downstream nodes.

Why a Failed Web Scraping Node is Worse Inside a Graph Than in a Script

A failed scraping node corrupts every downstream step in a LangGraph workflow. In a traditional scraping script, a failed request fails at one spot. You read the traceback, fix the error, and rerun. A LangGraph node reads from a shared state object and writes its output back into that same state, so when a scraping node retrieves a challenge page instead of the expected content, the graph does not stop.

Downstream nodes read the corrupted state as valid input, run their LLM (Large Language Model) calls and tool actions, and produce output that amplifies the original mistake. By the time the failure surfaces, tracing it back is difficult because the chain of causation now spans several nodes. Without a retry branch, the graph checkpoints the bad node as successful, and one scraping failure becomes a total workflow failure.

This tutorial adds a fallback conditional edge that catches bad output before it propagates and makes the agent reliable in production. ZenRows provides reliable web access that reduces how often the retry branch fires, since premium proxies and JavaScript rendering handle most bot-protected pages on the first call.

Adding ZenRows as a Scraping Node

The simplest way to add web scraping to a LangGraph agent is to wrap a ZenRows retrieval call inside a graph node and write the returned Markdown into state. This section builds that retrieval layer. You'll define the graph state, create a ZenRows scraping node, and compile a graph that stores the returned Markdown in the results field.

Prerequisites

To follow along, you'll need to create a ZenRows account to get your API key and an OpenAI account with billing enabled for LLM calls.

Step 1: Install the required packages

Install LangGraph, OpenAI, ZenRows LangChain integration, and Python dotenv.

pip install langgraph langchain-openai langchain-zenrows python-dotenv

Step 2: Add your ZenRows and OpenAI API keys

Create a .env file in your project root so LangGraph can read both your ZenRows key for retrieval and your OpenAI key for the later LLM nodes.

ZENROWS_API_KEY=<YOUR_ZENROWS_API_KEY>
OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>

Step 3: Define the graph state

Every LangGraph node in this tutorial reads from and writes to the same state object. Define a ResearchState TypedDict with one field per data point the graph needs to track across nodes.

Create a state.py file:

from typing import TypedDict


class ResearchState(TypedDict):
    urls: list[str]
    results: list[str]
    extracted: list[dict]
    briefing: str
    current_url: str
    status: str
    retry_count: int

Here's what each field in the ResearchState schema does:

urls stores the URLs the agent needs to research.
results stores the Markdown ZenRows returns.
extracted stores structured article data.
briefing stores the final synthesized report.
current_url tracks the URL currently being processed.
status records whether retrieval succeeded.
retry_count tracks how many times the graph has retried a failed retrieval.

Keeping retrieval metadata in state makes debugging easier because every node can see exactly what happened during the previous step.

Step 4: Set up the ZenRows scraping node

This node uses ZenRowsUniversalScraper from langchain-zenrows as a LangChain tool inside LangGraph. You'll activate the premium_proxy and js_render parameters to access dynamic contentdynamic content. The node retrieves a page with ZenRows, converts it to Markdown, and writes the result back into state.

Create a nodes.py file:

from dotenv import load_dotenv
from langchain_zenrows import ZenRowsUniversalScraper

from state import ResearchState

# load environment variables
load_dotenv()

# initialize the scraper
scraper = ZenRowsUniversalScraper()

def scrape_node(state: ResearchState) -> ResearchState:
    # retrieve the current page
    content = scraper.invoke({
        "url": state["current_url"],
        "js_render": True,
        "premium_proxy": True,
        "response_type": "markdown",
    })

    # write the result into graph state
    return {
        **state,
        "results": state["results"] + [content],
        "status": "ok",
    }

The node always returns status: "ok". It has no way yet to tell a real article apart from a blocked page.

Step 5: Build the graph

Create a graph.py file for the first version of the workflow. Register the scrape node as the entry point, and connect it straight to END to verify the retrieval step before adding more branches.

from langgraph.graph import END
from langgraph.graph import StateGraph

from nodes import scrape_node
from state import ResearchState

# create the graph object
graph = StateGraph(ResearchState)

# register nodes
graph.add_node("scrape", scrape_node)

# define execution flow
graph.set_entry_point("scrape")
graph.add_edge("scrape", END)

# compile the graph
app = graph.compile()

At this point, the graph contains one node and one transition.

Step 6: Run the graph

Invoke the graph with a target URL.

result = app.invoke({
    "urls": [],
    "results": [],
    "current_url": "https://www.scrapingcourse.com/ecommerce/",
    "status": "",
})
print(result)

Then run it:

python graph.py

State after execution:

{
    'urls': [],
    'results': [
        "[Skip to navigation]...\n\n# Shop\n\nShowing 1-16 of 188 results...\n\n- **Abominable Hoodie** $69.00..."
    ],
    'current_url': 'https://www.scrapingcourse.com/ecommerce/',
    'status': 'ok'
}

The output is truncated for brevity. Congratulations! The retrieval node works. Next, you'll add retrieval validation and a retry branch so the graph can recover when a target page returns blocked content instead of usable data.

Adding a Retry Branch for Failed Retrievals in AI Agents

A retry branch prevents bad retrievals from corrupting the entire graph. The retrieval node you built earlier assumes every request succeeds. However, a page can return a CAPTCHA, a Cloudflare challenge, or an empty response even when the request itself completes successfully. From LangGraph's perspective, that response is still data. Unless you validate the retrieval result, the graph has no way to distinguish an article from a challenge page. The solution is to make retrieval quality part of the graph's routing logic.

Step 1: Detect blocked pages

Define a list of phrases that signal a blocked page. Then create a function in nodes.py that checks for them.

BLOCK_SIGNALS = [
    "Checking your browser",
    "Enable JavaScript and cookies to continue",
    "Verify you are human",
]



def is_blocked(content: str) -> bool:
    # empty responses are unusable
    if not content.strip():
        return True

    # common challenge page signals
    return any(signal in content for signal in BLOCK_SIGNALS)

is_blocked() doesn't yet change what the scrape node does with that information.

Step 2: Update the scrape node

Update the scrape node to call is_blocked() and store the result as a status, rather than assuming every retrieval succeeded.

def scrape_node(state: ResearchState) -> ResearchState:
    # scrape the current URL 
    content = scraper.invoke(
        {
            "url": state["current_url"],
            "js_render": True,
            "premium_proxy": True,
            "response_type": "markdown",
        }
    )

    status = "blocked" if is_blocked(content) else "ok"

    return {
        **state,
        "results": state["results"] + [content],
        "status": status,
    }

The only change to the scrape node is the status field. It gives the graph enough information to make routing decisions later.

Step 3: Create a retry node

Create a dedicated retry node in nodes.py. The node will retry with a different proxy origin instead of repeating the identical request. If the second retrieval still returns blocked content, the node logs the URL before advancing so you can identify which source introduced low-quality data into the workflow.

def retry_node(state: ResearchState) -> ResearchState:
    # retry the failed retrieval with a different proxy origin
    content = scraper.invoke(
        {
            "url": state["current_url"],
            "js_render": True,
            "premium_proxy": True,
            "proxy_country": "us",
            "response_type": "markdown",
        }
    )
    status = "blocked" if is_blocked(content) else "ok"

    if status == "blocked":
        print(
            f"Retry failed for {state['current_url']}. "
            "Advancing with blocked content."
        )

    existing = list(state["results"])
    if existing:
        existing[-1] = content
    else:
        existing.append(content)

    return {
        **state,
        "results": existing,
        "status": status,
        "retry_count": state["retry_count"] + 1,
    }

This graph uses a single retry attempt. The retry node replaces the last retrieval, or appends if there's nothing yet to replace, so the graph never ends up with two conflicting results for the same URL.

Step 4: Add a routing function

Define a check_retrieval function that decides whether the graph should retry or continue.

def check_retrieval(state: ResearchState) -> str:
    if state["status"] == "blocked" and state["retry_count"] < 1:
        return "retry"

    return "continue"

Step 5: Register the conditional edge

Update the graph.py file. Register the retry node, then add a conditional edge that routes the scrape node's output through check_retrieval so the graph can branch based on the previous step's decision.

from langgraph.graph import StateGraph
from langgraph.graph import END

from state import ResearchState


from nodes import (
    scrape_node,
    retry_node,
    check_retrieval,
)

graph = StateGraph(ResearchState)

graph.add_node("scrape", scrape_node)
graph.add_node("retry", retry_node)

graph.set_entry_point("scrape")

# route to retry or end based on what check_retrieval returns
graph.add_conditional_edges(
    "scrape",
    check_retrieval,
    {
        "retry": "retry",
        "continue": END,
    },
)

graph.add_edge("retry", END)

app = graph.compile()

The graph now contains two possible paths. Instead of treating retrieval quality as an implementation detail inside a function, the graph treats it as part of the execution flow.

Step 6: Test the retry path

Run the graph against a protected target.

result = app.invoke(
    {
        "urls": [],
        "results": [],
        "current_url": "https://www.scrapingcourse.com/cloudflare-challenge",
        "status": "",
        "retry_count": 0,
    }
)
print(result)

Run it:

python graph.py

Cloudflare's challenge page may not trigger on every request because ZenRows resolves most challenges automatically. The example code block below shows what a blocked retrieval looks like when it occurs, so you can see what is_blocked() checks for.

State after scrape_node blocked retrieval:

#Illustrative blocked response, not a ZenRows result
{
    "urls": [],
    "results": [
        "Enable JavaScript and cookies to continue"
    ],
    "current_url": "https://www.scrapingcourse.com/cloudflare-challenge",
    "status": "blocked",
    "retry_count": 0
}

When the retry node runs and fails again, the graph updates its state before advancing. You will get this output:

Retry failed for https://www.scrapingcourse.com/cloudflare-challenge. Advancing with blocked content.

{
    "urls": [],
    "results": [
        "Enable JavaScript and cookies to continue"
    ],
    "current_url": "https://www.scrapingcourse.com/cloudflare-challenge",
    "status": "blocked",
    "retry_count": 1
}

Here's what running the graph against the target actually returned. ZenRows resolved the challenge on the first attempt, so status came back "ok" and the retry node never had to run.

State after scrape_node successful retrieval:

{
    'urls': [],
    'results': [
        '[![](https://www.scrapingcourse.com/assets/images/logo.svg) Scraping Course](http://www.scrapingcourse.com/)\n\n# Cloudflare Challenge\n\n![](https://www.scrapingcourse.com/assets/images/challenge.svg)\n\n## You bypassed the Cloudflare challenge! :D'
    ],
    'current_url': 'https://www.scrapingcourse.com/cloudflare-challenge',
    'status': 'ok',
    'retry_count': 0
}

The graph now distinguishes between a successful request and a successful retrieval. Next, you'll connect this retrieval logic to extraction and synthesis nodes so the graph can turn raw content into a briefing.

Building the Full Three-Node Research Graph

The graph you've built so far can retrieve content and recover from blocked pages. In this section, you'll build a news monitoring workflow with three nodes:

Scrape retrieves article content with ZenRows Universal Scraper API.
Extract uses an AI model to extract structured data from each article.
Synthesize combines those fields into a briefing.

The execution flow looks like this:

Step 1: Update `nodes.py`

Add the model and JSON imports to nodes.py and initialize the model. This tutorial uses gpt-4o-mini. You can swap it with your preferred LLM.

import json

from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_zenrows import ZenRowsUniversalScraper

from state import ResearchState

load_dotenv()

# initialize the scraper and model
scraper = ZenRowsUniversalScraper()
llm = ChatOpenAI(model="gpt-4o-mini")

Step 2: Update the ZenRows scraping node

The first version of the graph scraped a single URL. A research workflow needs to process a list of sources. You'll update scrape_node to loop through urls and keep each result in state.

def scrape_node(state: ResearchState) -> ResearchState:
    results = []

    for url in state["urls"]:
        content = scraper.invoke(
            {
                "url": url,
                "js_render": True,
                "premium_proxy": True,
                "response_type": "markdown",
            }
        )

        results.append(content)

    status = (
        "blocked"
        if any(is_blocked(content) for content in results)
        else "ok"
    )

    return {
        **state,
        "results": results,
        "status": status,
    }

The graph now works through the full urls list, and current_url only matters to the single-page version you built earlier.

Step 3: Update the retry node

Update the retry node to loop through each result and retry only the ones is_blocked() flags, instead of retrying the entire batch.

def retry_node(state: ResearchState) -> ResearchState:
    results = list(state["results"])

    for index, content in enumerate(results):
        if not is_blocked(content):
            continue

        retried_content = scraper.invoke(
            {
                "url": state["urls"][index],
                "js_render": True,
                "premium_proxy": True,
                "proxy_country": "us",
                "response_type": "markdown",
            }
        )

        # log failed retries before advancing
        if is_blocked(retried_content):
            print(
                f"Retry failed for {state['urls'][index]}. "
                "Advancing with blocked content."
            )

        results[index] = retried_content

    status = (
        "blocked"
        if any(is_blocked(content) for content in results)
        else "ok"
    )

    return {
        **state,
        "results": results,
        "status": status,
        "retry_count": state["retry_count"] + 1,
    }

The retry node now retries selectively.

Step 4: Create the extraction node

The extraction node converts Markdown into structured data. For each article, extract_node asks the model to return a fixed set of fields in structured JSON format. LLMs sometimes wrap their JSON response in markdown fences even when told not to. The cleaning step strips those out before parsing. Add this node to the same nodes.py file.

def extract_node(state: ResearchState) -> ResearchState:
    extracted = []

    for i, content in enumerate(state["results"]):
        response = llm.invoke(
            f"""
Extract the following fields from this article.

Return valid JSON only. No markdown fences, no preamble.

{{
  "headline": "...",
  "source": "...",
  "date": "...",
  "summary": "..."
}}

Article:

{content[:4000]}
"""
        )

        cleaned = (
            response.content
            .replace("```

json", "")
            .replace("

```", "")
            .strip()
        )

        try:
            extracted.append(json.loads(cleaned))
        except json.JSONDecodeError:
            print(
                f"JSON parse failed for article {i}. "
                f"Raw response:\n{cleaned[:300]}"
            )
            extracted.append({
                "headline": "Parse error",
                "source": state["urls"][i] if i < len(state["urls"]) else "unknown",
                "date": "",
                "summary": "Extraction failed. The model returned malformed JSON.",
            })

    return {
        **state,
        "extracted": extracted,
    }

Each article becomes a dictionary of structured fields for the synthesis node to read. The prompt only sends the model the first 4,000 characters of each article, since large news pages are full of navigation menus, related links, and footer content that add tokens without improving extraction quality. A cleaning step strips markdown fences before parsing, and any response that still fails to parse becomes a placeholder naming the source URL instead of crashing the run.

Step 5: Create the synthesis node

The synthesis node extracts the structured fields from each article and compiles them into a briefing. Create it in the same nodes.py.

def synthesize_node(state: ResearchState) -> ResearchState:
    response = llm.invoke(
        f"""
Write a concise news briefing.

Requirements:
- One bullet per story
- Preserve the key facts
- Keep each bullet under 40 words

Articles:

{json.dumps(state["extracted"], indent=2)}
"""
    )

    return {
        **state,
        "briefing": response.content,
    }

The scraping, extraction, and synthesis nodes are independent, so the agent is easier to inspect.

Step 6: Assemble the graph

Now update graph.py to build your complete LangGraph web research agent. Register the new nodes and connect them to the retrieval workflow.

from langgraph.graph import StateGraph
from langgraph.graph import END

from state import ResearchState


from nodes import (
    scrape_node,
    retry_node,
    check_retrieval,
    extract_node,
    synthesize_node,
)

graph = StateGraph(ResearchState)

# register nodes
graph.add_node("scrape", scrape_node)
graph.add_node("retry", retry_node)
graph.add_node("extract", extract_node)
graph.add_node("synthesize", synthesize_node)

# define entry point 
graph.set_entry_point("scrape")

# route blocked retrievals through retry branch
graph.add_conditional_edges(
    "scrape",
    check_retrieval,
    {
        "retry": "retry",
        "continue": "extract",
    },
)

# continue graph execution
graph.add_edge("retry", "extract")
graph.add_edge("extract", "synthesize")
graph.add_edge("synthesize", END)

# compile graph
app = graph.compile()

Retrieval failures remain isolated from the rest of the workflow. The graph only advances to extraction after the retrieval phase finishes.

Step 7: Run the research agent

Test the graph against three technology-focused news sources: CNET, Ars Technica, and TechCrunch.

result = app.invoke(
    {
        "urls": [
            "https://www.cnet.com",
            "https://arstechnica.com",
            "https://techcrunch.com",
        ],
        "results": [],
        "extracted": [],
        "briefing": "",
        "current_url": "",
        "status": "",
        "retry_count": 0,
    }
)

print("\nExtracted fields:")
print(result["extracted"])

print("\nFinal briefing:")
print(result["briefing"])

print("\nStatus:", result["status"])
Run it:
python graph.py

Let's look at what the state contains after the extraction and synthesis stages.

State after extraction:

[{'headline': 'Wearables Have a Huge Weakness. What Can We Do?', 'source': 'CNET', 'date': '2026', 'summary': 'This article discusses the significant weaknesses of wearable technology and explores potential solutions to address these issues.'}, {'headline': 'How a massive password-cracking operation spilled credentials for thousands of orgs', 'source': 'Ars Technica', 'date': '2026-06-16', 'summary': 'A password-cracking operation has compromised the credentials of thousands of organizations, including major companies like Oracle, Lenovo, and FedEx.'}, {'headline': 'The slowtech revolution is here to kill your phone addiction and rescue your attention span', 'source': 'TechCrunch', 'date': '2026-06-17', 'summary': "The article discusses the emerging slowtech movement which aims to combat smartphone addiction and improve users' attention spans."}]

State after synthesis:

- A CNET article highlights the significant weaknesses of wearable technology and suggests potential solutions to address these vulnerabilities.
- Ars Technica reports a major password-cracking operation has compromised credentials of thousands of organizations, including Oracle, Lenovo, and FedEx.
- TechCrunch discusses the slowtech movement, aimed at reducing smartphone addiction and enhancing users' attention spans.

Your output will differ depending on the articles available when you run the agent. However, the state transition should remain consistent. Congratulations! You now have a stateful LangGraph research agent that retrieves web content with ZenRows, routes around retrieval failures, gathers information, and produces a final briefing from multiple websites.

Wrapping Up

This step-by-step tutorial covered how to build a three-node LangGraph web research agent with ZenRows that handles its own failures. The pattern you built here is the foundation of any production research or market monitoring agent that depends on live web data.

Now, you know:

How to add a ZenRows scraping node to a LangGraph StateGraph
How to detect a blocked retrieval and route it through a retry branch with add_conditional_edges
How to build a multi-node graph that scrapes, extracts, and synthesizes real news sources into a briefing

The retry branch you built catches bad data before it spreads. ZenRows handles the access layer, so the graph only processes valid data. Try ZenRows for free and see how your research agent performs on protected pages.

How to Build a Competitor Intelligence Agent with CrewAI and ZenRows

Benny — Thu, 16 Jul 2026 21:45:31 +0000

A competitor intelligence agent enables real-time pricing visibility, automated positioning tracking, and structured competitor briefs for brands without manual research. At the center of these workflows are a researcher agent responsible for gathering data and an analysis agent responsible for generating a summary and a downstream report.

However, one of the things that makes the researcher agent's output trustworthy is its retrieval layer, since a poor retrieval-layer output can make the analysis agent's recommendation questionable. If the researcher agent searches the webpage and retrieves a challenge page, an empty response, or blocked content, every downstream conclusion becomes less trustworthy. With a 99.93% success rate on protected websites, ZenRows provides the reliable retrieval layer that makes these workflows practical in production.

This tutorial shows why a CrewAI researcher agent can fail on protected competitor pages. It starts with a custom ZenRows-based tool setup, then later shows the MCP server as an alternative approach.

Prerequisites

This tutorial works best with Python 3.10 or newer. If you are on an older Python version, create a dedicated virtual environment with a current Python installation to avoid dependency conflicts.

Python 3.10 to 3.13. CrewAI requires this range.
ZenRows API key. Create an account at zenrows.com and copy your key from the dashboard. This key authenticates every scrape the researcher agent runs for data extraction.
Anthropic API key. The crew uses Claude to drive both agents. Generate a key in the Anthropic Console.
Install dependencies and the required packages using pip install "crewai[anthropic]" crewai-tools zenrows python-dotenv.
Create a .env file and save your API keys there.

Why the built-in ScrapeWebsiteTool fails on competitor pages

The problem, as established earlier, starts before the analysis. A CrewAI workflow depends primarily on the information collected, because a competitor intelligence agent relies on a chain of steps. The researcher AI agent collects information, the analysis agent interprets it, and the final output is built from what the researcher agent retrieved. This means the scraping layer becomes the foundation of the entire workflow. If the researcher agent receives incomplete content, every other agent working with it will work with the wrong data.

CrewAI is an open-source framework that enables faster prototyping of AI-driven data extraction pipelines by orchestrating autonomous multi-agent systems. One of its features is CrewAI's built-in ScrapeWebsiteTool, which, though designed to extract and read website URL content, makes standard HTTP requests without any JavaScript rendering. This is a problem because many competitor pages use Cloudflare, DataDome, or similar anti-bot systems that can block standard HTML requests.

Browsers load the page structure first, then JavaScript fetches and inserts pricing tables and other dynamic content. A standard request using just CrewAI's built-in ScrapeWebsiteTool would receive the page shell but not the data your researcher agent needs. Likewise, it would return a challenge page or a Cloudflare 1020 error instead of the competitor page if that specified website has an anti-bot protection layer. The agent, however, doesn't validate the output. It just takes the incomplete output as the actual content.

Let's set up the ScrapeWebsiteTool for making HTTP requests and target a protected site. Run it yourself and see the challenge it faces.

from crewai_tools import ScrapeWebsiteTool

# Cloudflare-protected site
tool = ScrapeWebsiteTool(
    website_url="https://www.yelp.com"
)

text = tool.run()
print(text)

The output below shows that ScrapeWebsiteTool returned an exception and could not retrieve the relevant results from Yelp because it was behind Cloudflare's protection. The analyst agent will work with bad data. This is one example of the limitation of CrewAI's ScrapeWebsiteTool, not proof that it always returns an empty page.

((venv) ) benny@Mac competitor_intelligence % python scrapertool.py
The following text is scraped website content:

yelp.com Please enable JS and disable any ad blocker
((venv) ) benny@Mac competitor_intelligence %

ZenRows solves the retrieval-layer issue by successfully fetching pages from protected sites. ZenRows is an AI web data infrastructure that combines JavaScript rendering and premium proxy rotation to retrieve data from protected sites. Other recommendations, according to CrewAI's docs, are Scrapfly and Firecrawl. ZenRows is the right fit for this workflow because it is purpose-built for retrieving data from heavily protected targets and maintains a 99.93% success rate across supported domains. Below is a script using ZenRows on the same target (Yelp) to retrieve data.

import os
import requests
from dotenv import load_dotenv

load_dotenv()

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

# Same protected competitor URL used with ScrapeWebsiteTool
url = "https://www.yelp.com"

params = {
    "url": url,
    "apikey": ZENROWS_API_KEY,
    "js_render": "true",
    "premium_proxy": "true"
}

response = requests.get(
    "https://api.zenrows.com/v1/",
    params=params,
    timeout=180
)

print("status code:", response.status_code)
print("bytes returned:", len(response.text))
print(response.text[:1000])

ZenRows was able to reach the site and successfully retrieved the actual page HTML.

((venv) ) benny@Mac competitor_intelligence % python zenrowss.py

status code: 200
bytes returned: 501630

<html lang="en-US" prefix="og: http://ogp.me/ns#" style="margin: 0; padding: 0; border: 0; font-size: 100%; font: inherit; vertical-align: baseline;" data-cookbook-theme="light" class="">
<head>
<script>document.documentElement.className=document.documentElement.className.replace(/no-js/,"js");</script>
<meta http-equiv="Content-Language" content="en-US">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="mask-icon" sizes="any" href="https://s3-media0.fl.yelpcdn.com/assets/srv0/yelp_large_assets/b2bb2fb0ec9c/assets/img/logos/yelp_burst.svg" content="#FF1A1A">
<link rel="icon" href="https://s3-media0.fl.yelpcdn.com/assets/srv0/yelp_large_assets/594b78d06612/assets/img/logos/favicon.ico">

<script>
window.ga = window.ga || function() {
    (ga.q = ga.q || []).push(arguments)
};
ga.l = +new Date;
window.ygaPageStartTime = new Date().getTime();
</script>

<script>
window.yelp = window.yelp || {};
window.yelp.cookieTypePreferencesHeader = '["ANALYTICS"...';
</script>

...

((venv) ) benny@Mac competitor_intelligence %

The difference is visible in the response itself. CrewAI's built-in ScrapeWebsiteTool completed the request but returned a Cloudflare challenge page instead of the competitor's website. From CrewAI's perspective, the task did not fail because content was returned. The analyst CrewAI agent will then analyze the data and draw a false conclusion if you use only ScrapeWebsiteTool. In the next section, you will learn how to build a ZenRows tool for reliable web scraping and to avoid that failure.

Building the ZenRows custom tool for CrewAI web scraping

CrewAI allows you to create a custom tool by turning a Python function into something an agent can call. The function receives the input, performs a calculation, and returns the result to the agent. To add a custom web scraping tool to CrewAI, decorate a Python function with @tool. This will return a scraped page as a string and pass the function into our agent's tools list. You can learn more about this from the official CrewAI documentation.

from crewai.tools import tool

@tool("Scrape competitor page")
def scrape_page(url: str) -> str:
    """Describe what the tool does and when the agent should call it."""
    # call your scraper here
    return "page content as a string"

Now that the tool wrapper exists, you will use it with ZenRows and let the researcher call it instead of making normal HTTP requests. This ensures the agent gets the right extracted data. In the code below, js_render runs the page in a headless browser so JavaScript can load, premium_proxy routes the request through residential proxies, and response_type="markdown" returns clean markdown format for the agent. Using js_render and premium_proxy can increase your credit cost per request. Read more about how to navigate resources from the official ZenRows documentation. The tool created is then added to the researcher agent's tool list.

import os

from crewai import Agent
from crewai.tools import tool
from zenrows import ZenRowsClient

# retries=2 retries failed requests (429 and 5xx) with exponential back-off
client = ZenRowsClient(os.environ["ZENROWS_API_KEY"], retries=2)


@tool("Scrape competitor page as Markdown")
def scrape_page_markdown(url: str) -> str:
    """Fetch the full content of a single competitor web page and return it as
    Markdown. Pass one URL as a string. Use this to read pricing pages, product
    pages, and positioning copy from sites that block standard scrapers."""
    response = client.get(
        url,
        params={
            "js_render": "true",
            "premium_proxy": "true",
            "response_type": "markdown",
        },
    )
    response.raise_for_status()
    return response.text

researcher = Agent(
    role="Competitor Pricing Researcher",
    goal=(
        "Scrape each competitor pricing page and extract product names, "
        "prices, and positioning claims"
    ),
    backstory=(
        "You read competitor pricing pages and report exactly what is on the "
        "page. You never invent numbers."
    ),
    tools=[scrape_page_markdown],
    verbose=True,
)

The researcher can now open the competitor page on the web and return the content as Markdown. In the next section, you will extend the tool by adding an analyst agent before running the whole crew.

Building the competitor intelligence crew

To build a competitor intelligence crew, use two agents: a researcher and an analyst. The researcher uses ZenRows to scrape the competitor pricing page, and the analyst turns the researcher's output into a structured brief. Crew.kickoff() will run them in sequence and return the comparison to you. Let's break everything down.

1 . The researcher agent scrapes target pages and pulls product information. The ZenRows tool appears in the tools list to ensure a page's protection layer doesn't limit web scraping tasks.

from crewai import Agent

from zenrows_tools import scrape_page_markdown

researcher = Agent(
    role="Competitor Pricing Researcher",
    goal=(
        "Scrape each competitor pricing page and extract every plan name, "
        "its price, and any positioning claim on the page"
    ),
    backstory=(
        "You read competitor pricing pages and report exactly what is on the "
        "page. You never invent numbers or fill gaps with assumptions."
    ),
    tools=[scrape_page_markdown],
    verbose=True,
)

2 . The Analyst agent compares and writes the detailed comparison report. It doesn't use the ZenRows tool because it doesn't need to access any webpage. It only works with whatever the researcher gives it. This also ensures you spend fewer tokens and that the analyst uses only the output from your data collection.

from crewai import Agent

from zenrows_tools import scrape_page_markdown

researcher = Agent(
    role="Competitor Pricing Researcher",
    goal=(
        "Scrape each competitor pricing page and extract every plan name, "
        "its price, and any positioning claim on the page"
    ),
    backstory=(
        "You read competitor pricing pages and report exactly what is on the "
        "page. You never invent numbers or fill gaps with assumptions."
    ),
    tools=[scrape_page_markdown],
    verbose=True,
)

Each agent has a task, and the analyst's task uses the research task as context so CrewAI can pass the scraped output forward. The crew holds both the agent and the task and runs everything sequentially. CrewAI uses an LLM to drive agents and print a report from the scraped pages. The code below shows how that sequence works in practice.

import os
from dotenv import load_dotenv

load_dotenv()

from crewai import Agent, Task, Crew, Process, LLM
from crewai.tools import tool
from zenrows import ZenRowsClient

# -----------------------------
# ZenRows Client
# -----------------------------
zenrows = ZenRowsClient(os.environ["ZENROWS_API_KEY"], retries=2)

# -----------------------------
# Scraping Tool (wait added so client-side content renders)
# -----------------------------
@tool("Scrape page as Markdown")
def scrape_page_markdown(url: str) -> str:
    """Scrape a page and return it as Markdown. Return only visible scraped content."""
    response = zenrows.get(
        url,
        params={
            "js_render": True,
            "premium_proxy": True,
            "response_type": "markdown",
            "wait": 3000,  # wait 3s for client-side content to load. requires js_render
        },
    )
    response.raise_for_status()
    return response.text

# -----------------------------
# LLM
# -----------------------------
llm = LLM(model="anthropic/claude-sonnet-4-6", temperature=0.1)

# -----------------------------
# TARGETS (product pages, not category pages)
# Swap these for two comparable products. Confirm they resolve first.
# -----------------------------
TARGETS = [
    "https://www.ikea.com/us/en/p/markus-office-chair-vissle-dark-gray-90289172/",
    "https://www.amazon.com/dp/B00FS3VJAO",  # comparable office chair
]

# -----------------------------
# RESEARCHER (STRICT SCRAPER)
# -----------------------------
researcher = Agent(
    role="Retail Product Data Extractor",
    goal=(
        "Extract ONLY visible product name and price from retail product pages. "
        "No inference, no assumptions."
    ),
    backstory=(
        "You are a deterministic extraction system. "
        "If data is not visible in scraped content, it does not exist."
    ),
    tools=[scrape_page_markdown],
    llm=llm,
    verbose=True,
)

# -----------------------------
# ANALYST (STRICT GROUNDED)
# -----------------------------
analyst = Agent(
    role="Retail Pricing Analyst",
    goal=(
        "Compare product pricing across two retailers using ONLY scraped data. "
        "No external knowledge or assumptions."
    ),
    backstory=(
        "You ONLY use provided scraped outputs. "
        "Missing fields must be marked as 'NOT FOUND IN SCRAPE'."
    ),
    llm=llm,
    verbose=True,
)

# -----------------------------
# TASK 1: SCRAPING
# -----------------------------
research_task = Task(
    description=(
        "Scrape each product page below and extract the product name and its "
        "current price. Use only what is visible on the page. If a value is not "
        "present, write NOT FOUND IN SCRAPE.\n\n"
        + "\n".join(TARGETS)
    ),
    expected_output="Product name and current price for each URL, from scraped content only.",
    agent=researcher,
)

# -----------------------------
# TASK 2: ANALYSIS
# -----------------------------
analysis_task = Task(
    description=(
        "Compare the two products using only the researcher's scraped data. "
        "Do not add outside knowledge. Mark anything missing as NOT FOUND IN SCRAPE.\n\n"
        "Produce:\n"
        "- a table of product name and price per retailer\n"
        "- which product is cheaper\n"
        "- one or two insights drawn from the data"
    ),
    expected_output="A short pricing comparison: a table and a brief takeaway.",
    agent=analyst,
    context=[research_task],
)

# -----------------------------
# CREW
# -----------------------------
crew = Crew(
    agents=[researcher, analyst],
    tasks=[research_task, analysis_task],
    process=Process.sequential,
    verbose=True,
)

# -----------------------------
# RUN
# -----------------------------
if __name__ == "__main__":
    result = crew.kickoff()
    print("\n\n===== FINAL OUTPUT =====\n")
    print(result)

Here is part of the output of the code below (it was a long report). The report is generated entirely from the scraped content returned by ZenRows. A standard scraper would not work reliably here because the retrieval step would likely return a challenge page, incomplete HTML, or a JavaScript shell instead of the underlying content, leaving nothing for the analyst agent to work with.

From the output below, you will notice that the researcher agent remains grounded in the scraped data. For example, it does not attempt to convert currencies or fill in missing information that is not present in the source. Once you have gotten to this point, you have two options. The first is to pass the full page content to downstream agents and let them perform their own analysis, as we did in the script above. The second option is to extract structured fields immediately after retrieval and store them in a database or comparison table. This is often the better choice for use cases where you care about specific attributes rather than the full page content. ZenRows AutoParse is designed for this workflow, returning structured data that can be inserted directly into downstream systems without requiring a second extraction step.

The code is available in this GitHub repository.

╭────────────────────────────────────────────────────────────────── Crew Completion ──────────────────────────────────────────────────────────────────╮
│                                                                                                                                              │
│  Crew Execution Completed                                                                                                                   │
│  Name: crew                                                                                                                                 │
│  ID: 6972714b-a13f-4f66-8ffe-47c908b5ed01                                                                                                    │
│                                                                                                                                              │
│  Final Output: ## Pricing Comparison: Office Chairs                                                                                         │
│                                                                                                                                              │
│  ### Product & Price Table                                                                                                                  │
│                                                                                                                                              │
│  | Retailer | Product Name | Scraped Price |                                                                                                │
│  |----------|--------------|---------------|                                                                                                │
│  | **IKEA** | MARKUS Office Chair | $299.99 USD |                                                                                          │
│  | **Amazon** | BestOffice Ergonomic Office Chair (Mid-Back, Black) | BOB 290.15 |                                                       │
│                                                                                                                                              │
│  ---                                                                                                                                         │
│                                                                                                                                              │
│  ### Which Product Is Cheaper?                                                                                                              │
│                                                                                                                                              │
│  **Cannot be determined.** The Amazon listing returned a price denominated in **BOB (Bolivian Boliviano)**, while the IKEA listing         │
│  returned a price in **USD**. These two currencies cannot be compared without a conversion rate, and applying one would require             │
│  external knowledge outside the scraped data. No valid apples-to-apples price comparison is possible from the data as returned.             │
│                                                                                                                                              │
│  ---                                                                                                                                         │
│                                                                                                                                              │
│  ### Insights from the Scraped Data                                                                                                         │
│                                                                                                                                              │
│  1. **Currency anomaly on Amazon:** The scraper returned `BOB290.15` for the Amazon listing — BOB is the currency code for the             │
│     Bolivian Boliviano. This is flagged in the source data as a likely geo-detection or display error. The true USD price for this          │
│     listing is **NOT FOUND IN SCRAPE** and cannot be assumed.                                                                               │
│                                                                                                                                              │
│  2. **Product detail disparity:** IKEA's scraped data returned only a short product name (`MARKUS`), while Amazon's scrape returned         │
│     a lengthy, keyword-stuffed title. This reflects a structural difference in how each retailer formats product data—not a reflection      │
│     of product quality or completeness.                                                                                                     │
│                                                                                                                                              │
│  ---                                                                                                                                         │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Returning structured data with autoparse

Instead of returning a full markdown document, you can pass autoparse=True so that ZenRows returns structured output. Autoparsing reads the page's structured markup and hands you a JSON string for the analyst to use. Keep in mind that Autoparse is experimental, domain-limited, and only works on pages with a parser.

To autoparse, modify the tool using the code below.

@tool("Scrape competitor page as structured JSON")
def scrape_page_structured(url: str) -> str:
    """Fetch a single competitor product or pricing page and return auto-parsed
    structured data as a JSON string. Pass one URL as a string. Use this when
    you need discrete fields such as product name and price rather than prose."""
    response = zenrows.get(
        url,
        params={
            "js_render": True,
            "premium_proxy": True,
            "autoparse": True,  # replaces response_type; autoparse returns JSON
        },
    )
    response.raise_for_status()
    return response.text

To switch the tool to structured output, replace "response_type": "markdown" with "autoparse": True. Autoparse cannot be combined with response_type, so it goes in its place, not alongside it.

You can keep js_render and premium_proxy on here, because these pricing pages build their content client-side and sit behind anti-bot protection, so without them, autoparse parses an empty shell. Autoparse itself adds no cost; you only pay for js_render and premium_proxy, so this request's cost stays the same as the Markdown version.

Use the ZenRows MCP server instead of a custom tool

MCP is the right choice when the crew is part of a larger multi-agent system and tool maintenance overhead matters; the custom @tool is right when you need tight control over parameters or when Node.js is not available. The MCP exposes the same scraping capability through an agent tool, but shifts the same integration from code to server configuration. CrewAI can mount any MCP server as an agent tool. CrewAI's MCPServerAdapter starts the server, exposes its tools, and stops the server when the block exits. Everything that uses those tools, the agents, the tasks, and kickoff(), has to live inside the with block, because the tools are only connected while the server is running. The server is open source; here is the GitHub repository. You can read more about it via the Official ZenRows MCP documentation.

The MCP server runs on Node.js, so to use it, you'll need to install Node 20.6 or later. ZenRows also offers a hosted server that you can connect to via URL if you want to avoid the Node dependency, which can simplify deployment.

pip install "crewai-tools[mcp]"

Since ZenRows hosted is the recommended way for any app calling an LLM API directly, let's explore that route. The MCP server is free.

import os
from typing import Any

from dotenv import load_dotenv

load_dotenv()

from crewai import Agent, Task, Crew, Process, LLM
from crewai.tools import BaseTool
from crewai_tools import MCPServerAdapter

# Claude drives every agent. ANTHROPIC_API_KEY is read from the environment.
llm = LLM(model="anthropic/claude-sonnet-4-6", temperature=0.1)

# Product pages to compare. Confirm they resolve first.
TARGETS = [
    "https://www.ikea.com/us/en/p/markus-office-chair-vissle-dark-gray-90289172/",
    "https://www.amazon.com/dp/B00FS3VJAO",  # comparable office chair
]

# Hosted ZenRows MCP server. No Node, no npx, just your API key.
SERVER_PARAMS = {
    "url": "https://mcp.zenrows.com/mcp",
    "transport": "streamable-http",
    "headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
}


class NullStrippedTool(BaseTool):
    """Wraps an MCP tool and removes None arguments before the call.

    The MCP adapter fills unset optional fields with null, which the ZenRows
    server rejects. Stripping None here keeps only the values that were set.
    """

    inner: Any = None

    def _run(self, *args: Any, **kwargs: Any) -> Any:
        clean = {k: v for k, v in kwargs.items() if v is not None}
        return self.inner._run(*args, **clean)


def wrap(tools):
    return [
        NullStrippedTool(
            name=t.name,
            description=t.description,
            args_schema=t.args_schema,
            inner=t,
        )
        for t in tools
    ]


def build_and_run():
    # The MCP tools are only live inside this block.
    with MCPServerAdapter(SERVER_PARAMS, "scrape") as mcp_tools:
        scrape_tools = wrap(mcp_tools)

        researcher = Agent(
            role="Retail Product Data Extractor",
            goal=(
                "Extract ONLY the visible product name and current price from "
                "each product page. No inference, no assumptions."
            ),
            backstory=(
                "You are a deterministic extraction system. If a value is not "
                "visible in the scraped content, it does not exist."
            ),
            tools=scrape_tools,
            llm=llm,
            verbose=True,
        )

        analyst = Agent(
            role="Retail Pricing Analyst",
            goal=(
                "Compare product pricing across two retailers using ONLY "
                "scraped data. No external knowledge or assumptions."
            ),
            backstory=(
                "You ONLY use provided scraped outputs. Missing fields must be "
                "marked as 'NOT FOUND IN SCRAPE'."
            ),
            llm=llm,
            verbose=True,
        )

        research_task = Task(
            description=(
                "Scrape each product page below and extract the product name "
                "and its current price. Use only what is visible on the page. "
                "If a value is not present, write NOT FOUND IN SCRAPE.\n\n"
                "When you call the scrape tool, enable JavaScript rendering and "
                "premium proxy, and wait for the page content to load, since "
                "these pages render product data client-side.\n\n"
                + "\n".join(TARGETS)
            ),
            expected_output=(
                "Product name and current price for each URL, from scraped "
                "content only."
            ),
            agent=researcher,
        )

        analysis_task = Task(
            description=(
                "Compare the two products using only the researcher's scraped "
                "data. Do not add outside knowledge. Mark anything missing as "
                "NOT FOUND IN SCRAPE.\n\n"
                "Produce:\n"
                "- a table of product name and price per retailer\n"
                "- which product is cheaper\n"
                "- one or two insights drawn from the data"
            ),
            expected_output="A short pricing comparison: a table and a brief takeaway.",
            agent=analyst,
            context=[research_task],
        )

        crew = Crew(
            agents=[researcher, analyst],
            tasks=[research_task, analysis_task],
            process=Process.sequential,
            verbose=True,
        )

        result = crew.kickoff()
        print("\n\n===== FINAL OUTPUT =====\n")
        print(result)


if __name__ == "__main__":
    build_and_run()

You can find every code snippet in this tutorial in this GitHub repository.

Wrapping up

You now have a production-ready competitor intelligence workflow built with CrewAI and ZenRows, rather than a scrapers that fail when competitors place their pricing pages behind Cloudflare, DataDome, or other anti-bot systems.

The current system works by having a researcher agent retrieve data directly from competitors' websites using ZenRows, which has a documented 99.93% success rate, while the analyst agent uses that data to generate a report. Whether you choose the custom @tool for total control via Integration or the ZenRows MCP server for easier tool management, the architecture remains the same: reliable retrieval, grounded analysis, and actionable intelligence generated from live web data.

Try ZenRows for free and get your comprehensive competitor analysis today!

How to Build a RAG Pipeline with Live Web Data Using ZenRows and LlamaIndex

Goodness E. Eboh — Mon, 13 Jul 2026 12:42:04 +0000

How do you make sure the documents entering your VectorStoreIndex contain actual content when building a LlamaIndex RAG (retrieval augmented generation) pipeline? This question comes up because the built-in SimpleWebPageReader struggles with many of the dynamic web pages it encounters.

Instead of valid page content, the reader can return Cloudflare challenge pages, empty React shells, or content with a poor signal-to-noise ratio relative to the tokens spent. This tutorial directly addresses this issue. It shows how replacing the built-in web reader with one that returns clean LlamaIndex documents from live URLs improves the quality of data available to agents, query engines, and vector indexes in RAG systems.

Why LlamaIndex web scraping loaders fail on real websites

There are three major failure modes that LlamaIndex's built-in web loaders show against modern web infrastructure.

The first is the Cloudflare/WAF (Web Application Firewall) block. When the SimpleWebPageReader makes a standard HTTP request using its underlying libraries like requests or urllib, it lacks capabilities such as browser fingerprints, TLS fingerprints, and human-like behavioral patterns. Security layers on websites protected by WAFs use these signals to identify automated requests and cause the reader to receive HTTP error pages or challenge screens instead.

The second failure mode is SPA (Single Page Application) empty shells. Many modern websites use frameworks like React, Angular, or Vue. When they are scraped, the initial HTML response is usually just an empty template shell containing an initialization script block. Since the reader does not execute JavaScript, it only sees the initial HTML shell.

The third is token bloat. Even for modern unprotected static sites, built-in readers pull raw HTML or raw text, which usually include navigation bars, headers, footers, sidebars, cookie banners, and tracking scripts. This unnecessary data, when indexed, costs tokens that would have been better spent pulling in more relevant content.

Using the SimpleWebPageReader to load data from an anti-bot challenge page shows exactly how output failures can occur.

from llama_index.readers.web import SimpleWebPageReader
documents = SimpleWebPageReader(html_to_text=True).load_data(
    ["https://www.scrapingcourse.com/cloudflare-challenge"]
)

print(documents[0].text[:200])

The reader was only able to pull the challenge page.

If a document like this is added to the VectorStoreIndex, LLM (Large Language Model) outputs would have no information related to the content that was meant to be pulled. At the same time, tokens are still consumed during both retrieval and generation. To avoid these failure modes, use a reader designed to access protected websites, execute JavaScript, and return cleaner content for indexing. The ZenRowsWebReader is one such reader.

Setting up `ZenRowsWebReader`

The ZenRowsWebReader is supported in LlamaIndex and can return valid Document objects from live pages that are ingestible by AI applications. It can access websites protected by WAF challenges, render JavaScript, and output markdown, all of which improve document quality. Here is how you can set it up.

To follow along, first generate a ZenRows API key from the ZenRows Dashboard and store it as the ZENROWS_API_KEY environment variable.

Next, import the reader and configure it with the following parameters to handle live web pages.

When reading sites protected by advanced anti-bot mechanisms like Cloudflare, enable the premium_proxy parameter to use residential IPs. Residential IPs are less likely to be identified as automated traffic by anti-bot systems.

from llama_index.readers.web import ZenRowsWebReader

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

Lastly, use the ZenRowsWebReader against the same website that the SimpleWebPageReader failed to scrape.

from llama_index.readers.web import ZenRowsWebReader
from dotenv import load_dotenv
import os

load_dotenv()

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

documents = reader.load_data([
    "https://www.scrapingcourse.com/cloudflare-challenge"
])

print(documents[0].text[:500])

Running it, the output shows that the reader successfully accessed the website, scraped its content, and returned it in Markdown.

Building a RAG pipeline with live web data

Having resolved the data quality issue with scraping web content from live pages using ZenRowsWebReader, the next step is to include the reader in a RAG pipeline. While the complete code is a single file, the implementation consists of four stages: configuring the pipeline to use OpenAI models, ingesting live web data with the reader, creating the vector index, and querying it to generate responses.

Step 1: Configure the OpenAI model and embedding model

Import all dependencies, load the API keys, and configure the OpenAI model and embedding model.

from dotenv import load_dotenv
import os

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

Settings.llm = OpenAI(
    model="gpt-4o-mini",
    api_key=OPENAI_API_KEY
)

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_key=OPENAI_API_KEY
)

This configuration uses gpt-4o-mini for fast, cost-effective response generation and text-embedding-3-small to create semantic embeddings that provide a good balance of retrieval quality, latency, and cost. Both models can be replaced with other supported OpenAI models depending on your specific requirements.

Step 2: Load live web data into LlamaIndex documents

Just like in the previous section, import the ZenRowsWebReader and configure it to load data from the target URLs into LlamaIndex Document objects.

from llama_index.readers.web import ZenRowsWebReader

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

reader = ZenRowsWebReader(
    api_key=ZENROWS_API_KEY,
    js_render=True,
    premium_proxy=True,
    response_type="markdown",
    wait=2000
)

documents = reader.load_data([
    "https://www.scrapingcourse.com/cloudflare-challenge",
    "https://www.scrapingcourse.com/javascript-rendering"
])

print(f"Loaded {len(documents)} document(s)")

Step 3: Create the vector index

Import VectorStoreIndex and pass the Document objects into it to create the index.

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents)

print("Index created successfully")

This example creates an in-memory VectorStoreIndex for simplicity. LlamaIndex also supports persisting indexes to disk or external vector stores.

Step 4: Query the index

Use the query engine to retrieve relevant content from the index and pass it to the OpenAI model for response generation.

query_engine = index.as_query_engine()

response = query_engine.query(
    "Summarize the information available across the indexed pages."
)

print("\nAnswer:\n")
print(response.response)

The result is that the LLM retrieved valid indexed data from the documents and generated output that matched the scraped content.

While this pipeline delivers valid data, websites frequently change layouts and content. So it’s important to have a layer that updates the index with fresh Document objects.

Keeping your knowledge base current

One-off loads deliver valid content, but they can quickly become outdated. Periodic refreshes allow current web content to be re-scraped from live pages as source content changes. But if the logic isn't right, it can cause the same content to be re-ingested repeatedly, increasing storage costs and reducing retrieval quality through duplicate content.

You can achieve efficient refreshes by generating hashes from the scraped page content and comparing them against previously stored values. This logic runs after loading documents with the ZenRowsWebReader and before passing the updated documents to the VectorStoreIndex.from_documents().

In addition to the existing reader setup, this implementation uses the json and hashlib modules to persist and compare content hashes.

import json
import hashlib

To set up the logic, create a hash store. In production, this is usually a database rather than a local file. Then create a set of functions that store hashes of scraped documents and compare them against newly scraped content.

HASH_STORE = "content_hashes.json"

def load_hashes():
    if os.path.exists(HASH_STORE):
        with open(HASH_STORE, "r") as f:
            return json.load(f)
    return {}

def save_hashes(hashes):
    with open(HASH_STORE, "w") as f:
        json.dump(hashes, f, indent=4)

def get_updated_documents(urls):
    documents = reader.load_data(urls)

    if not documents:
        print("No documents returned")
        return []

    stored_hashes = load_hashes()
    updated_documents = []

    for doc in documents:
        if not doc.text.strip():
            source_url = doc.metadata.get("source_url", "unknown")
            print(f"Skipping empty content: {source_url}")
            continue

        source_url = doc.metadata.get("source_url", "unknown")

        content_hash = hashlib.md5(
            doc.text.encode("utf-8")
        ).hexdigest()

        previous_hash = stored_hashes.get(source_url)

        if previous_hash == content_hash:
            print(f"No changes detected: {source_url}")
            continue

        print(f"Updated content detected: {source_url}")

        stored_hashes[source_url] = content_hash
        updated_documents.append(doc)

    save_hashes(stored_hashes)
    return updated_documents

documents = get_updated_documents([
    "https://www.scrapingcourse.com/cloudflare-challenge",
    "https://www.scrapingcourse.com/javascript-rendering"
])

During each refresh cycle, the function compares the latest scraped content against the previously stored hashes. Unchanged pages are skipped, while pages with updated content are returned for re-indexing.

At this point, the pipeline is fully functional. But even with clean web data and a working retrieval pipeline, you can still encounter issues with answer quality when querying.

Troubleshooting incomplete answers

Incomplete answers can result from retrieval configuration in LlamaIndex or from incomplete page content returned by the web reader. Here is how to troubleshoot both.

Insufficient retrieval context

During retrieval, LlamaIndex returns only a subset of the highest-scoring chunks from the vector index to provide relevant context for the LLM. By default, this naive top-k retrieval matches query embeddings with chunk embeddings. Since live webpages are broken down into multiple markdown chunks, important context can end up distributed across multiple document chunks.

The fix

Increase the top-k parameter to 10 so the query engine pulls more relevant chunks into the retrieval context, improving the likelihood of retrieving related information.

query_engine = index.as_query_engine(
    similarity_top_k=10
)

Missing dynamic JavaScript content

If your final output indicates missing information or returns placeholder strings such as "Enable JavaScript to see products," the scraper may be capturing the page before JavaScript-rendered elements have finished loading.

The fix

In the ZenRows reader parameters, verify that js_render=True is enabled and that a wait=2000 configuration has been added. These settings allow dynamic elements time to load so the target elements can fully render before the page is converted into a LlamaIndex document.

Conclusion

You have now built a RAG pipeline that uses the ZenRowsWebReader to ingest live web data into LlamaIndex. It can retrieve content from protected and JavaScript-driven websites, keep indexed content up to date through periodic refreshes, and generate responses from documents that standard web loaders often fail to retrieve.

How to Build a Market Research Agent with ZenRows and LangChain

Humna Ghufran — Mon, 13 Jul 2026 12:40:57 +0000

ZenRows gives LangChain agents reliable access to the real web. The langchain-zenrows package exposes a single LangChain tool class, ZenRowsUniversalScraper, that handles anti-bot bypass, JavaScript rendering, and clean Markdown output behind one scraper API call. Standard web scraping tools make plain HTTP requests with no JS execution and no anti-bot bypass, so when your agent hits a protected web page it gets back a Cloudflare challenge page or nothing at all.

This tutorial builds a working market research agent that scrapes real web data and gives your large language model access to LangChain real-time web data including from bot-protected pages, returning structured data your LLM can reason over.

By the end, you will have a working LangChain web scraping setup that handles data extraction from any target without infrastructure headaches.

Why LangChain's Default Web Scraping Tools Fail on Real Web Pages

LangChain ships with built-in web scrapers like WebBaseLoader. These work fine on simple, static web pages. They use standard HTTP requests, they do not render dynamic content, and they have no mechanism for bypassing anti-bot systems.

Most sites with commercially useful web data such as product pages, pricing pages, and competitor sites are protected. When your LangChain agent sends a plain HTTP request to these pages, the server often recognizes it as a bot and returns a challenge page instead of the actual web content.

Your agent then reads the challenge page as if it were real scraped data. That leads to incorrect inputs and ultimately produces unreliable output. This is the core limitation of standard web scraping tools in AI agent workflows.

Here is what a standard LangChain web request returns against a bot-protected page:

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://www.scrapingcourse.com/antibot-challenge")
docs = loader.load()
print(docs[0].page_content)

Output:

Just a moment...Enable JavaScript and cookies to continue

The agent receives that text, has nothing useful to work with, and either hallucinates a response or tells you it could not access the page. Neither is useful in production.

This is the core problem. Your agent works perfectly in local tests against clean pages, then breaks the moment it hits a real target. The fix is to replace the retrieval layer.

Setting Up ZenRows as the LangChain Scraper Agent Tool

To add ZenRows to a LangChain agent, install langchain-zenrows, pass ZenRowsUniversalScraper to create_react_agent, and invoke with your prompt. This is the core step in integrating web scraping into your LangChain workflow. The full LangChain integration reference is in the ZenRows LangChain docs.

Installation

Start by installing the required libraries:

pip install langgraph langchain-openai langchain-zenrows

Agent Setup

from langchain_zenrows import ZenRowsUniversalScraper
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
import os

# Set these in your environment before running
# export ZENROWS_API_KEY=your_key_here
# export OPENAI_API_KEY=your_key_here

def scraper():
    llm = ChatOpenAI(model="gpt-4o-mini")
    zenrows_tool = ZenRowsUniversalScraper()

    # The agent reads the URL from the prompt and passes it to ZenRowsUniversalScraper automatically
    agent = create_react_agent(llm, [zenrows_tool])

    try:
        result = agent.invoke(
            {
                "messages": [{"role": "user", "content": "What is the major highlight on https://www.scrapingcourse.com/antibot-challenge"}]
            }
        )
        for message in result["messages"]:
            print(f"{message.content}")
    except NameError:
        print("Agent not available.")
    except Exception as e:
        print(f"Error running agent: {e}")

scraper()

This is a setup test. The agent scrapes the same antibot challenge page from the previous section to confirm ZenRows is working. Once this runs cleanly, you are ready to point the agent at any target.

Note: The page refers to its feature as the "Antibot challenge", which matches the URL slug antibot-challenge.

Output:

The major highlight on https://www.scrapingcourse.com/antibot-challenge is that you bypassed the Antibot challenge.

A few things to note about this setup:

ZenRowsUniversalScraper reads your API key from the ZENROWS_API_KEY environment variable automatically; you do not need to pass it in code.
Anti-bot bypass activates on demand when the agent calls the scraping tool against a protected page.
You do not need to configure headless browsers, proxy pools, or browser sessions for the agent use case.
The create_react_agent function comes from langgraph.prebuilt. It wires the large language model and the agent tools together so the agent can decide when to call ZenRowsUniversalScraper based on your prompt.

Building the LangChain Scraper Agent for Market Research

The agent below visits a product catalog web page, scrapes it with autoparse=True enabled in the tool configuration, and returns the four cheapest products as structured data. Each step maps to something the agent does: it receives the prompt, calls ZenRowsUniversalScraper with the source URL, gets scraped data back, and the LLM formats it as JSON. This is LangChain web scraping doing real data extraction work end to end.

Here is the complete market research agent:

from langchain_zenrows import ZenRowsUniversalScraper
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
import os

# Set these in your environment before running
# export ZENROWS_API_KEY=your_key_here
# export OPENAI_API_KEY=your_key_here

def market_research_agent():
    llm = ChatOpenAI(model="gpt-4o-mini")
    zenrows_tool = ZenRowsUniversalScraper(autoparse=True)
    # autoparse=True tells ZenRowsUniversalScraper to extract structured data automatically

    agent = create_react_agent(llm, [zenrows_tool])

    try:
        result = agent.invoke(
            {
                "messages": [{"role": "user", "content": "Visit https://www.scrapingcourse.com/ecommerce/ and scrape the page. Return the 4 cheapest products as JSON with title, price, and url fields."}]
            }
        )
        for message in result["messages"]:
            print(f"{message.content}")
    except NameError:
        print("Agent not available.")
    except Exception as e:
        print(f"Error running agent: {e}")

market_research_agent()

When you run this, the agent works through four steps:

It reads the prompt and identifies the target URL.
It calls ZenRowsUniversalScraper on the product catalog page with autoparse=True already set in the tool configuration, which extracts structured data automatically.
The LLM reads that data and formats it as JSON.
It prints the results.

Output from https://www.scrapingcourse.com/ecommerce/:

[
  {
    "title": "Affirm Water Bottle",
    "price": "$7.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/affirm-water-bottle/"
  },
  {
    "title": "Arcadio Gym Short",
    "price": "$20.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/arcadio-gym-short/"
  },
  {
    "title": "Argus All-Weather Tank",
    "price": "$22.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/argus-all-weather-tank/"
  },
  {
    "title": "Aero Daily Fitness Tee",
    "price": "$24.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/aero-daily-fitness-tee/"
  }
]

You can adapt this same pattern to any product category page. Change the URL in the prompt, adjust the number of products, or ask for different fields like ratings or review counts. The agent handles the rest.

Output Formats and When to Use Each

ZenRowsUniversalScraper gives you two ways to control output. Set response_type to choose between markdown and plaintext. Use autoparse=True when you want structured extraction instead of raw page content.

These are not interchangeable options in the same parameter. You can find the full parameter reference in the ZenRows API docs.

Format	What it returns	Best for
`response_type="markdown"`	Page content converted to clean Markdown	Text-heavy pages feeding into RAG pipelines or agent reasoning
`autoparse=True`	Structured extraction from the page. Output fields vary by page type. See the ZenRows API docs for supported page types.	E-commerce and catalog pages where you need product name, price, URL, and other key details
`response_type="plaintext"`	Plain text with no formatting or markup	Situations where token efficiency matters and structure is less important

markdown is a safe starting point for most agent workflows because it preserves readability without adding raw HTML noise.

Use autoparse=True specifically for pages where the data is structured and repeatable, like product listings, job boards, or real estate pages. Use response_type="plaintext" when you are feeding content into a pipeline that does not need formatting and you want to keep token usage low.

Extending to Multiple URLs

Once your agent works on a single page, you can scale it to a list of competitor or research URLs.

research_urls = [
    "https://www.scrapingcourse.com/ecommerce/",
    "https://www.scrapingcourse.com/ecommerce/page/2/",
    "https://www.scrapingcourse.com/ecommerce/page/3/"
]

all_results = []

for url in research_urls:
    result = agent.invoke(
        {
            "messages": [{"role": "user", "content": f"Scrape {url} and return the 4 cheapest products with title, price, and url."}]
        }
    )
    all_results.append(result)

One thing worth knowing is the session_id API parameter. You can pass it directly in the ZenRowsUniversalScraper scraping tool configuration to keep the same IP address across multiple requests to the same site. This works similarly to how proxy pools maintain consistent browser sessions during data collection. Check the ZenRows API docs for the exact usage.

For larger-scale data collection jobs, the same loop pattern works. You can pull a list of competitor URLs from a file, a database, or a previous web scrape, and run the LangChain agent against each one.

What to Do Next

Your LangChain scraper agent now has a retrieval layer that works on real web pages, including the ones that block standard web scrapers. The web scraping integration here removes infrastructure headaches around IP rotation, browser sessions, and anti-bot measures. That retrieval layer is what makes the agent dependable in production, where ZenRows holds a 99.93% success rate against protected targets.

ZenRows works as the scraper API layer for a wide range of LLM applications and RAG applications beyond market research. The same ZenRowsUniversalScraper LangChain tool works for:

Review summarization
Sentiment analysis on competitor content
Price monitoring across multiple web pages
Real-time data monitoring for content changes

If you want to see another full example, the ZenRows real estate AI agent tutorial follows the same pattern applied to a different domain.

If you want to extend this further, you can combine session_id with a longer multi-page loop to scrape paginated results. You can also use the css_extractor API parameter to pull specific elements when the autoparse parameter does not cover your exact data extraction needs.

🔥 How to Scrape Dynamic Web Pages in Python 🚀

ZenRows — Wed, 11 Oct 2023 11:47:29 +0000

Have you gotten poor results while scraping dynamic web page content? It's not just you. Crawling dynamic data is a challenging undertaking (to say the least) for standard scrapers. That's because JavaScript runs in the background when an HTTP request is made.

Scraping dynamic websites requires rendering the entire page in a browser and extracting the target information.

Join us in this step-by-step tutorial to learn all you need about dynamic web scraping with Python — the dos and don'ts, the challenges and solutions, and everything in between.

Let's dive right in!

What Is a Dynamic Website? 🤔

A dynamic website is one that doesn't have all its content directly in its static HTML. It uses server-side or client-side to display data, sometimes based on the user's actions (e.g., clicking, scrolling, etc.).

Put simply, these websites display different content or layout with every server request. This helps with loading time as there's no need to reload the same information each time the user wants to view “new” content.

How to identify them? One way is by disabling JavaScript in the command palette on your browser. If the website is dynamic, the content will disappear.

Let's use Saleor React Storefront as an example. Here's what its front page looks like:

Take notice of the titles, images and artist's name.

Now, let's disable JavaScript using the steps below:

Inspect the page: Right-click and select “Inspect” to open the DevTools window.
Navigate to the command palette: CTRL/CMD + SHIFT + P.
Search for “JavaScript.”
Click on Disable JavaScript.
Hit refresh.

What's the result? See below:

You see it for yourself! Disabling JavaScript removes all dynamic web content.

Alternatives to Dynamic Web Scraping With Python ❄

So, you want to scrape dynamic websites with Python…

Since libraries such as Beautiful Soup or Requests don't automatically fetch dynamic content, you're left with two options to complete the task:

Feed the content to a standard library.
Execute the page's internal JavaScript while scraping.

However, not all dynamic pages are the same. Some render content through JS APIs that can be accessed by inspecting the “Network” tab. Others store the JS-rendered content as JSON somewhere in the DOM (Document Object Model).

The good news is we can parse the JSON string to extract the necessary data in both cases.

Keep in mind that there are situations in which these solutions are inapplicable. For such websites, you can use headless browsers to render the page and extract your needed data.

The alternatives to crawling dynamic web pages with Python are:

Manually locating the data and parsing JSON string.
Using headless browsers to execute the page's internal JavaScript (e.g., Selenium or Pyppeteer, an unofficial Python port of Puppeteer).

What Is the Easiest Way to Scrape a Dynamic Website in Python? 🤔

It's true, headless browsers can be slow and performance-intensive. However, they lift all restrictions on web scraping. That is if you don't count anti-bot detection. And you shouldn't because we've already told you how to bypass such protections.

Manually locating data and parsing JSON strings presumes that accessing the JSON version of the dynamic data is possible. Unfortunately, that's not always the case, especially when it comes to high-level Single-page applications (SPAs).

Not to mention that mimicking an API request is not scalable. They often require cookies and authentications alongside other restrictions that can easily block you out.

The best way to scrape dynamic web pages in Python depends on your goals and resources. If you have access to the website's JSON and are looking to extract a single page's data, you may not need a headless browser.

However, barring this tiny portion of cases, most of the time using Beautiful Soup and Selenium is your best and easiest option.

Time to get our hands dirty! Get ready to write some code and see precisely how to scrape a dynamic website in Python!

Prerequisites 🛠

To follow this tutorial, you'll need to meet some requirements. We'll use the following tools:

Python 3: The latest version of Python will work best. At the time of writing, that is 3.11.2.
Selenium
Webdriver Manager: This will ensure that the browser's and the driver's versions match. You don't have to manually download the WebDriver for this purpose.

pip install selenium webdriver-manager

You now have everything you need. Let's go!

Method #1: Dynamic Web Scraping With Python Using Beautiful Soup 😋

Beautiful Soup is arguably the most popular Python library for crawling HTML data.

To extract information with it, we need our target page's HTML string. However, dynamic content is not directly present in a website's static HTML. This means that Beautiful Soup can't access JavaScript-generated data.

Here's a solution: it's possible to extract data from XHR requests if the website loads content using an AJAX request.

Method #2: Scraping Dynamic Web Pages in Python Using Selenium ✂

To understand how Selenium helps you scrape dynamic websites, first, we need to inspect how regular libraries, such as Requests, interact with them.

We'll use Angular as our target website:

Let's try scraping it with Requests and see the result. Before that, we have to install the Requests library that can be executed using the pip command.

pip install requests

Here's what our code looks like:

import requests 

url = 'https://angular.io/' 

response = requests.get(url) 

html = response.text 

print(html)

As you can see, only the following HTML was extracted:

<noscript> 
    <div class="background-sky hero"></div> 
    <section id="intro" style="text-shadow: 1px 1px #1976d2;"> 
        <div class="hero-logo"></div> 
        <div class="homepage-container"> 
            <div class="hero-headline">The modern web<br>developer's platform</div> 
        </div> 
    </section> 
    <h2 style="color: red; margin-top: 40px; position: relative; text-align: center; text-shadow: 1px 1px #fafafa; border-top: none;"> 
        <b><i>This website requires JavaScript.</i></b> 
    </h2> 
</noscript>

Though, inspecting the website shows more content than what was retrieved.

This is what happened when we disabled JavaScript on the page:

That's precisely what Requests was able to return. The library perceives no errors as it parses data from the website's static HTML, which is exactly what it was created to do.

In this case, aiming for the same result as what's displayed on the website is impossible. Can you guess why? That's right, it's because this is a dynamic web page.

To access the entire content and extract our target data, we must render the JavaScript.

It's time to make it right with Selenium dynamic web scraping.

We'll use the following script to quickly crawl our target website:

from selenium import webdriver 
from selenium.webdriver.chrome.service import Service as ChromeService 
from webdriver_manager.chrome import ChromeDriverManager 

url = 'https://angular.io/' 

driver = webdriver.Chrome(service=ChromeService( 
    ChromeDriverManager().install())) 

driver.get(url) 

print(driver.page_source)

There you have it! The page's complete HTML, including the dynamic web content.

Congratulations! You've just scraped your first dynamic website.

Selecting Elements in Selenium 👈

There are different ways to access elements in Selenium. We discuss this matter in depth in our web scraping with Selenium in Python guide.

Still, we'll explain this with an example. Let's select only the H2s on our target website:

Before we get to that, we need to inspect the website and identify the location of the elements we want to extract.

We can see that the class="text-container" is common for those headlines. We copy that and map the H2s to get elements using Chrome Driver.

Paste this code:

from selenium import webdriver 
from selenium.webdriver.common.by import By 
from selenium.webdriver.chrome.service import Service as ChromeService 
from webdriver_manager.chrome import ChromeDriverManager 

# instantiate options 
options = webdriver.ChromeOptions() 

# run browser in headless mode 
options.headless = True 

# instantiate driver 
driver = webdriver.Chrome(service=ChromeService( 
    ChromeDriverManager().install()), options=options) 

# load website 
url = 'https://angular.io/' 

# get the entire website content 
driver.get(url) 

# select elements by class name 
elements = driver.find_elements(By.CLASS_NAME, 'text-container') 
for title in elements: 
    # select H2s, within element, by tag name 
    heading = title.find_element(By.TAG_NAME, 'h2').text 
    # print H2s 
    print(heading)

You'll get the following:

"DEVELOP ACROSS ALL PLATFORMS" 
"SPEED & PERFORMANCE" 
"INCREDIBLE TOOLING" 
"LOVED BY MILLIONS"

Nice and easy! You can now scrape dynamic sites with Selenium effortlessly.

How to Scrape Infinite Scroll Web Pages With Selenium ♾

Some dynamic pages load more content as users scroll down to the bottom of the page. These are known as “Infine scroll websites.” Crawling them is a bit more challenging. To do so, we need to instruct our spider to scroll to the bottom, wait for all new content to load and only then begin scraping.

Understand this with an example. Let's use Scraping Club's sample page.

This script will scroll through the first 20 results and extract their title:

from selenium import webdriver 
from selenium.webdriver.common.by import By 
from selenium.webdriver.chrome.service import Service as ChromeService 
from webdriver_manager.chrome import ChromeDriverManager 
import time 

options = webdriver.ChromeOptions() 
options.headless = True 
driver = webdriver.Chrome(service=ChromeService( 
    ChromeDriverManager().install()), options=options) 

# load target website 
url = 'https://scrapingclub.com/exercise/list_infinite_scroll/' 

# get website content 
driver.get(url) 

# instantiate items 
items = [] 

# instantiate height of webpage 
last_height = driver.execute_script('return document.body.scrollHeight') 

# set target count 
itemTargetCount = 20 

# scroll to bottom of webpage 
while itemTargetCount > len(items): 
    driver.execute_script('window.scrollTo(0, document.body.scrollHeight);') 

    # wait for content to load 
    time.sleep(1) 

    new_height = driver.execute_script('return document.body.scrollHeight') 

    if new_height == last_height: 
        break 

    last_height == new_height 

    # select elements by XPath 
    elements = driver.find_elements(By.XPATH, "//div/h4/a") 
    h4_texts = [element.text for element in elements] 

    items.extend(h4_texts) 

    # print title 
    print(h4_texts)

Remark: It's important to set a target count for infinite scroll pages so you can end your script at some point.

In the previous example, we used yet another selector: By.XPath. It will locate elements based on an XPath instead of classes and IDs, as seen before. Inspect the page, right-click on a <div> containing the elements you want to scrape and select Copy Path.

Your result should look like this:

['Short Dress', 'Patterned Slacks', 'Short Chiffon Dress', 'Off-the-shoulder Dress', ...]

And there you have it, the H4s of the first 20 products!

Remark: Using Selenium for dynamic web scraping can get tricky with continuous Selenium updates. Do well to go through the latest changes.

Conclusion 🤝

Dynamic web pages are everywhere. Thus, there's a high enough chance you'll encounter them in your data extraction efforts. Remember that getting familiar with their structure will help you identify the best approach for retrieving your target information.

All methods we explored in this article come with their own faults and disadvantages. So, take a look at what ZenRows has to offer. The solution allows you to scrape dynamic websites using a simple API call. Try it for free today and save yourself time and resources.

Thanks for reading! This article was originally published on ZenRows: Dynamic Web Pages Scraping with Python: Guide to Scrape All Content.

If you liked this guide, please ❤️ like the article and subscribe for more! ⬇⬇

DEV Community: ZenRows

How to Build a Stateful Web Research Agent with ZenRows and LangGraph

Why a Failed Web Scraping Node is Worse Inside a Graph Than in a Script

Adding ZenRows as a Scraping Node

Prerequisites

Step 1: Install the required packages

Step 2: Add your ZenRows and OpenAI API keys

Step 3: Define the graph state

Step 4: Set up the ZenRows scraping node

Step 5: Build the graph

Step 6: Run the graph

Adding a Retry Branch for Failed Retrievals in AI Agents

Step 1: Detect blocked pages

Step 2: Update the scrape node

Step 3: Create a retry node

Step 4: Add a routing function

Step 5: Register the conditional edge

Step 6: Test the retry path

Building the Full Three-Node Research Graph

Step 1: Update nodes.py

Step 2: Update the ZenRows scraping node

Step 3: Update the retry node

Step 4: Create the extraction node

Step 5: Create the synthesis node

Step 6: Assemble the graph

Step 7: Run the research agent

Wrapping Up

How to Build a Competitor Intelligence Agent with CrewAI and ZenRows

Prerequisites

Why the built-in ScrapeWebsiteTool fails on competitor pages

Building the ZenRows custom tool for CrewAI web scraping

Building the competitor intelligence crew

Returning structured data with autoparse

Use the ZenRows MCP server instead of a custom tool

Wrapping up

How to Build a RAG Pipeline with Live Web Data Using ZenRows and LlamaIndex

Why LlamaIndex web scraping loaders fail on real websites

Setting up ZenRowsWebReader

Building a RAG pipeline with live web data

Step 1: Configure the OpenAI model and embedding model

Step 2: Load live web data into LlamaIndex documents

Step 3: Create the vector index

Step 4: Query the index

Keeping your knowledge base current

Troubleshooting incomplete answers

Insufficient retrieval context

The fix

Missing dynamic JavaScript content

The fix

Conclusion

How to Build a Market Research Agent with ZenRows and LangChain

Why LangChain's Default Web Scraping Tools Fail on Real Web Pages

Setting Up ZenRows as the LangChain Scraper Agent Tool

Installation

Agent Setup

Building the LangChain Scraper Agent for Market Research

Output Formats and When to Use Each

Extending to Multiple URLs

What to Do Next

🔥 How to Scrape Dynamic Web Pages in Python 🚀

What Is a Dynamic Website? 🤔

Alternatives to Dynamic Web Scraping With Python ❄

What Is the Easiest Way to Scrape a Dynamic Website in Python? 🤔

Prerequisites 🛠

Method #1: Dynamic Web Scraping With Python Using Beautiful Soup 😋

Method #2: Scraping Dynamic Web Pages in Python Using Selenium ✂

Selecting Elements in Selenium 👈

How to Scrape Infinite Scroll Web Pages With Selenium ♾

Conclusion 🤝

Step 1: Update `nodes.py`

Setting up `ZenRowsWebReader`