A competitor intelligence agent enables real-time pricing visibility, automated positioning tracking, and structured competitor briefs for brands without manual research. At the center of these workflows are a researcher agent responsible for gathering data and an analysis agent responsible for generating a summary and a downstream report.
However, one of the things that makes the researcher agent's output trustworthy is its retrieval layer, since a poor retrieval-layer output can make the analysis agent's recommendation questionable. If the researcher agent searches the webpage and retrieves a challenge page, an empty response, or blocked content, every downstream conclusion becomes less trustworthy. With a 99.93% success rate on protected websites, ZenRows provides the reliable retrieval layer that makes these workflows practical in production.
This tutorial shows why a CrewAI researcher agent can fail on protected competitor pages. It starts with a custom ZenRows-based tool setup, then later shows the MCP server as an alternative approach.
Prerequisites
This tutorial works best with Python 3.10 or newer. If you are on an older Python version, create a dedicated virtual environment with a current Python installation to avoid dependency conflicts.
- Python 3.10 to 3.13. CrewAI requires this range.
- ZenRows API key. Create an account at zenrows.com and copy your key from the dashboard. This key authenticates every scrape the researcher agent runs for data extraction.
- Anthropic API key. The crew uses Claude to drive both agents. Generate a key in the Anthropic Console.
- Install dependencies and the required packages using
pip install "crewai[anthropic]" crewai-tools zenrows python-dotenv. - Create a .env file and save your API keys there.
Why the built-in ScrapeWebsiteTool fails on competitor pages
The problem, as established earlier, starts before the analysis. A CrewAI workflow depends primarily on the information collected, because a competitor intelligence agent relies on a chain of steps. The researcher AI agent collects information, the analysis agent interprets it, and the final output is built from what the researcher agent retrieved. This means the scraping layer becomes the foundation of the entire workflow. If the researcher agent receives incomplete content, every other agent working with it will work with the wrong data.
CrewAI is an open-source framework that enables faster prototyping of AI-driven data extraction pipelines by orchestrating autonomous multi-agent systems. One of its features is CrewAI's built-in ScrapeWebsiteTool, which, though designed to extract and read website URL content, makes standard HTTP requests without any JavaScript rendering. This is a problem because many competitor pages use Cloudflare, DataDome, or similar anti-bot systems that can block standard HTML requests.
Browsers load the page structure first, then JavaScript fetches and inserts pricing tables and other dynamic content. A standard request using just CrewAI's built-in ScrapeWebsiteTool would receive the page shell but not the data your researcher agent needs. Likewise, it would return a challenge page or a Cloudflare 1020 error instead of the competitor page if that specified website has an anti-bot protection layer. The agent, however, doesn't validate the output. It just takes the incomplete output as the actual content.
Let's set up the ScrapeWebsiteTool for making HTTP requests and target a protected site. Run it yourself and see the challenge it faces.
from crewai_tools import ScrapeWebsiteTool
# Cloudflare-protected site
tool = ScrapeWebsiteTool(
website_url="https://www.yelp.com"
)
text = tool.run()
print(text)
The output below shows that ScrapeWebsiteTool returned an exception and could not retrieve the relevant results from Yelp because it was behind Cloudflare's protection. The analyst agent will work with bad data. This is one example of the limitation of CrewAI's ScrapeWebsiteTool, not proof that it always returns an empty page.
((venv) ) benny@Mac competitor_intelligence % python scrapertool.py
The following text is scraped website content:
yelp.com Please enable JS and disable any ad blocker
((venv) ) benny@Mac competitor_intelligence %
ZenRows solves the retrieval-layer issue by successfully fetching pages from protected sites. ZenRows is an AI web data infrastructure that combines JavaScript rendering and premium proxy rotation to retrieve data from protected sites. Other recommendations, according to CrewAI's docs, are Scrapfly and Firecrawl. ZenRows is the right fit for this workflow because it is purpose-built for retrieving data from heavily protected targets and maintains a 99.93% success rate across supported domains. Below is a script using ZenRows on the same target (Yelp) to retrieve data.
import os
import requests
from dotenv import load_dotenv
load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
# Same protected competitor URL used with ScrapeWebsiteTool
url = "https://www.yelp.com"
params = {
"url": url,
"apikey": ZENROWS_API_KEY,
"js_render": "true",
"premium_proxy": "true"
}
response = requests.get(
"https://api.zenrows.com/v1/",
params=params,
timeout=180
)
print("status code:", response.status_code)
print("bytes returned:", len(response.text))
print(response.text[:1000])
ZenRows was able to reach the site and successfully retrieved the actual page HTML.
((venv) ) benny@Mac competitor_intelligence % python zenrowss.py
status code: 200
bytes returned: 501630
<html lang="en-US" prefix="og: http://ogp.me/ns#" style="margin: 0; padding: 0; border: 0; font-size: 100%; font: inherit; vertical-align: baseline;" data-cookbook-theme="light" class="">
<head>
<script>document.documentElement.className=document.documentElement.className.replace(/no-js/,"js");</script>
<meta http-equiv="Content-Language" content="en-US">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="mask-icon" sizes="any" href="https://s3-media0.fl.yelpcdn.com/assets/srv0/yelp_large_assets/b2bb2fb0ec9c/assets/img/logos/yelp_burst.svg" content="#FF1A1A">
<link rel="icon" href="https://s3-media0.fl.yelpcdn.com/assets/srv0/yelp_large_assets/594b78d06612/assets/img/logos/favicon.ico">
<script>
window.ga = window.ga || function() {
(ga.q = ga.q || []).push(arguments)
};
ga.l = +new Date;
window.ygaPageStartTime = new Date().getTime();
</script>
<script>
window.yelp = window.yelp || {};
window.yelp.cookieTypePreferencesHeader = '["ANALYTICS"...';
</script>
...
((venv) ) benny@Mac competitor_intelligence %
The difference is visible in the response itself. CrewAI's built-in ScrapeWebsiteTool completed the request but returned a Cloudflare challenge page instead of the competitor's website. From CrewAI's perspective, the task did not fail because content was returned. The analyst CrewAI agent will then analyze the data and draw a false conclusion if you use only ScrapeWebsiteTool. In the next section, you will learn how to build a ZenRows tool for reliable web scraping and to avoid that failure.
Building the ZenRows custom tool for CrewAI web scraping
CrewAI allows you to create a custom tool by turning a Python function into something an agent can call. The function receives the input, performs a calculation, and returns the result to the agent. To add a custom web scraping tool to CrewAI, decorate a Python function with @tool. This will return a scraped page as a string and pass the function into our agent's tools list. You can learn more about this from the official CrewAI documentation.
from crewai.tools import tool
@tool("Scrape competitor page")
def scrape_page(url: str) -> str:
"""Describe what the tool does and when the agent should call it."""
# call your scraper here
return "page content as a string"
Now that the tool wrapper exists, you will use it with ZenRows and let the researcher call it instead of making normal HTTP requests. This ensures the agent gets the right extracted data. In the code below, js_render runs the page in a headless browser so JavaScript can load, premium_proxy routes the request through residential proxies, and response_type="markdown" returns clean markdown format for the agent. Using js_render and premium_proxy can increase your credit cost per request. Read more about how to navigate resources from the official ZenRows documentation. The tool created is then added to the researcher agent's tool list.
import os
from crewai import Agent
from crewai.tools import tool
from zenrows import ZenRowsClient
# retries=2 retries failed requests (429 and 5xx) with exponential back-off
client = ZenRowsClient(os.environ["ZENROWS_API_KEY"], retries=2)
@tool("Scrape competitor page as Markdown")
def scrape_page_markdown(url: str) -> str:
"""Fetch the full content of a single competitor web page and return it as
Markdown. Pass one URL as a string. Use this to read pricing pages, product
pages, and positioning copy from sites that block standard scrapers."""
response = client.get(
url,
params={
"js_render": "true",
"premium_proxy": "true",
"response_type": "markdown",
},
)
response.raise_for_status()
return response.text
researcher = Agent(
role="Competitor Pricing Researcher",
goal=(
"Scrape each competitor pricing page and extract product names, "
"prices, and positioning claims"
),
backstory=(
"You read competitor pricing pages and report exactly what is on the "
"page. You never invent numbers."
),
tools=[scrape_page_markdown],
verbose=True,
)
The researcher can now open the competitor page on the web and return the content as Markdown. In the next section, you will extend the tool by adding an analyst agent before running the whole crew.
Building the competitor intelligence crew
To build a competitor intelligence crew, use two agents: a researcher and an analyst. The researcher uses ZenRows to scrape the competitor pricing page, and the analyst turns the researcher's output into a structured brief. Crew.kickoff() will run them in sequence and return the comparison to you. Let's break everything down.
1 . The researcher agent scrapes target pages and pulls product information. The ZenRows tool appears in the tools list to ensure a page's protection layer doesn't limit web scraping tasks.
from crewai import Agent
from zenrows_tools import scrape_page_markdown
researcher = Agent(
role="Competitor Pricing Researcher",
goal=(
"Scrape each competitor pricing page and extract every plan name, "
"its price, and any positioning claim on the page"
),
backstory=(
"You read competitor pricing pages and report exactly what is on the "
"page. You never invent numbers or fill gaps with assumptions."
),
tools=[scrape_page_markdown],
verbose=True,
)
2 . The Analyst agent compares and writes the detailed comparison report. It doesn't use the ZenRows tool because it doesn't need to access any webpage. It only works with whatever the researcher gives it. This also ensures you spend fewer tokens and that the analyst uses only the output from your data collection.
from crewai import Agent
from zenrows_tools import scrape_page_markdown
researcher = Agent(
role="Competitor Pricing Researcher",
goal=(
"Scrape each competitor pricing page and extract every plan name, "
"its price, and any positioning claim on the page"
),
backstory=(
"You read competitor pricing pages and report exactly what is on the "
"page. You never invent numbers or fill gaps with assumptions."
),
tools=[scrape_page_markdown],
verbose=True,
)
Each agent has a task, and the analyst's task uses the research task as context so CrewAI can pass the scraped output forward. The crew holds both the agent and the task and runs everything sequentially. CrewAI uses an LLM to drive agents and print a report from the scraped pages. The code below shows how that sequence works in practice.
import os
from dotenv import load_dotenv
load_dotenv()
from crewai import Agent, Task, Crew, Process, LLM
from crewai.tools import tool
from zenrows import ZenRowsClient
# -----------------------------
# ZenRows Client
# -----------------------------
zenrows = ZenRowsClient(os.environ["ZENROWS_API_KEY"], retries=2)
# -----------------------------
# Scraping Tool (wait added so client-side content renders)
# -----------------------------
@tool("Scrape page as Markdown")
def scrape_page_markdown(url: str) -> str:
"""Scrape a page and return it as Markdown. Return only visible scraped content."""
response = zenrows.get(
url,
params={
"js_render": True,
"premium_proxy": True,
"response_type": "markdown",
"wait": 3000, # wait 3s for client-side content to load. requires js_render
},
)
response.raise_for_status()
return response.text
# -----------------------------
# LLM
# -----------------------------
llm = LLM(model="anthropic/claude-sonnet-4-6", temperature=0.1)
# -----------------------------
# TARGETS (product pages, not category pages)
# Swap these for two comparable products. Confirm they resolve first.
# -----------------------------
TARGETS = [
"https://www.ikea.com/us/en/p/markus-office-chair-vissle-dark-gray-90289172/",
"https://www.amazon.com/dp/B00FS3VJAO", # comparable office chair
]
# -----------------------------
# RESEARCHER (STRICT SCRAPER)
# -----------------------------
researcher = Agent(
role="Retail Product Data Extractor",
goal=(
"Extract ONLY visible product name and price from retail product pages. "
"No inference, no assumptions."
),
backstory=(
"You are a deterministic extraction system. "
"If data is not visible in scraped content, it does not exist."
),
tools=[scrape_page_markdown],
llm=llm,
verbose=True,
)
# -----------------------------
# ANALYST (STRICT GROUNDED)
# -----------------------------
analyst = Agent(
role="Retail Pricing Analyst",
goal=(
"Compare product pricing across two retailers using ONLY scraped data. "
"No external knowledge or assumptions."
),
backstory=(
"You ONLY use provided scraped outputs. "
"Missing fields must be marked as 'NOT FOUND IN SCRAPE'."
),
llm=llm,
verbose=True,
)
# -----------------------------
# TASK 1: SCRAPING
# -----------------------------
research_task = Task(
description=(
"Scrape each product page below and extract the product name and its "
"current price. Use only what is visible on the page. If a value is not "
"present, write NOT FOUND IN SCRAPE.\n\n"
+ "\n".join(TARGETS)
),
expected_output="Product name and current price for each URL, from scraped content only.",
agent=researcher,
)
# -----------------------------
# TASK 2: ANALYSIS
# -----------------------------
analysis_task = Task(
description=(
"Compare the two products using only the researcher's scraped data. "
"Do not add outside knowledge. Mark anything missing as NOT FOUND IN SCRAPE.\n\n"
"Produce:\n"
"- a table of product name and price per retailer\n"
"- which product is cheaper\n"
"- one or two insights drawn from the data"
),
expected_output="A short pricing comparison: a table and a brief takeaway.",
agent=analyst,
context=[research_task],
)
# -----------------------------
# CREW
# -----------------------------
crew = Crew(
agents=[researcher, analyst],
tasks=[research_task, analysis_task],
process=Process.sequential,
verbose=True,
)
# -----------------------------
# RUN
# -----------------------------
if __name__ == "__main__":
result = crew.kickoff()
print("\n\n===== FINAL OUTPUT =====\n")
print(result)
Here is part of the output of the code below (it was a long report). The report is generated entirely from the scraped content returned by ZenRows. A standard scraper would not work reliably here because the retrieval step would likely return a challenge page, incomplete HTML, or a JavaScript shell instead of the underlying content, leaving nothing for the analyst agent to work with.
From the output below, you will notice that the researcher agent remains grounded in the scraped data. For example, it does not attempt to convert currencies or fill in missing information that is not present in the source. Once you have gotten to this point, you have two options. The first is to pass the full page content to downstream agents and let them perform their own analysis, as we did in the script above. The second option is to extract structured fields immediately after retrieval and store them in a database or comparison table. This is often the better choice for use cases where you care about specific attributes rather than the full page content. ZenRows AutoParse is designed for this workflow, returning structured data that can be inserted directly into downstream systems without requiring a second extraction step.
The code is available in this GitHub repository.
╭────────────────────────────────────────────────────────────────── Crew Completion ──────────────────────────────────────────────────────────────────╮
│ │
│ Crew Execution Completed │
│ Name: crew │
│ ID: 6972714b-a13f-4f66-8ffe-47c908b5ed01 │
│ │
│ Final Output: ## Pricing Comparison: Office Chairs │
│ │
│ ### Product & Price Table │
│ │
│ | Retailer | Product Name | Scraped Price | │
│ |----------|--------------|---------------| │
│ | **IKEA** | MARKUS Office Chair | $299.99 USD | │
│ | **Amazon** | BestOffice Ergonomic Office Chair (Mid-Back, Black) | BOB 290.15 | │
│ │
│ --- │
│ │
│ ### Which Product Is Cheaper? │
│ │
│ **Cannot be determined.** The Amazon listing returned a price denominated in **BOB (Bolivian Boliviano)**, while the IKEA listing │
│ returned a price in **USD**. These two currencies cannot be compared without a conversion rate, and applying one would require │
│ external knowledge outside the scraped data. No valid apples-to-apples price comparison is possible from the data as returned. │
│ │
│ --- │
│ │
│ ### Insights from the Scraped Data │
│ │
│ 1. **Currency anomaly on Amazon:** The scraper returned `BOB290.15` for the Amazon listing — BOB is the currency code for the │
│ Bolivian Boliviano. This is flagged in the source data as a likely geo-detection or display error. The true USD price for this │
│ listing is **NOT FOUND IN SCRAPE** and cannot be assumed. │
│ │
│ 2. **Product detail disparity:** IKEA's scraped data returned only a short product name (`MARKUS`), while Amazon's scrape returned │
│ a lengthy, keyword-stuffed title. This reflects a structural difference in how each retailer formats product data—not a reflection │
│ of product quality or completeness. │
│ │
│ --- │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Returning structured data with autoparse
Instead of returning a full markdown document, you can pass autoparse=True so that ZenRows returns structured output. Autoparsing reads the page's structured markup and hands you a JSON string for the analyst to use. Keep in mind that Autoparse is experimental, domain-limited, and only works on pages with a parser.
To autoparse, modify the tool using the code below.
@tool("Scrape competitor page as structured JSON")
def scrape_page_structured(url: str) -> str:
"""Fetch a single competitor product or pricing page and return auto-parsed
structured data as a JSON string. Pass one URL as a string. Use this when
you need discrete fields such as product name and price rather than prose."""
response = zenrows.get(
url,
params={
"js_render": True,
"premium_proxy": True,
"autoparse": True, # replaces response_type; autoparse returns JSON
},
)
response.raise_for_status()
return response.text
To switch the tool to structured output, replace "response_type": "markdown" with "autoparse": True. Autoparse cannot be combined with response_type, so it goes in its place, not alongside it.
You can keep js_render and premium_proxy on here, because these pricing pages build their content client-side and sit behind anti-bot protection, so without them, autoparse parses an empty shell. Autoparse itself adds no cost; you only pay for js_render and premium_proxy, so this request's cost stays the same as the Markdown version.
Use the ZenRows MCP server instead of a custom tool
MCP is the right choice when the crew is part of a larger multi-agent system and tool maintenance overhead matters; the custom @tool is right when you need tight control over parameters or when Node.js is not available. The MCP exposes the same scraping capability through an agent tool, but shifts the same integration from code to server configuration. CrewAI can mount any MCP server as an agent tool. CrewAI's MCPServerAdapter starts the server, exposes its tools, and stops the server when the block exits. Everything that uses those tools, the agents, the tasks, and kickoff(), has to live inside the with block, because the tools are only connected while the server is running. The server is open source; here is the GitHub repository. You can read more about it via the Official ZenRows MCP documentation.
The MCP server runs on Node.js, so to use it, you'll need to install Node 20.6 or later. ZenRows also offers a hosted server that you can connect to via URL if you want to avoid the Node dependency, which can simplify deployment.
pip install "crewai-tools[mcp]"
Since ZenRows hosted is the recommended way for any app calling an LLM API directly, let's explore that route. The MCP server is free.
import os
from typing import Any
from dotenv import load_dotenv
load_dotenv()
from crewai import Agent, Task, Crew, Process, LLM
from crewai.tools import BaseTool
from crewai_tools import MCPServerAdapter
# Claude drives every agent. ANTHROPIC_API_KEY is read from the environment.
llm = LLM(model="anthropic/claude-sonnet-4-6", temperature=0.1)
# Product pages to compare. Confirm they resolve first.
TARGETS = [
"https://www.ikea.com/us/en/p/markus-office-chair-vissle-dark-gray-90289172/",
"https://www.amazon.com/dp/B00FS3VJAO", # comparable office chair
]
# Hosted ZenRows MCP server. No Node, no npx, just your API key.
SERVER_PARAMS = {
"url": "https://mcp.zenrows.com/mcp",
"transport": "streamable-http",
"headers": {"Authorization": f"Bearer {os.environ['ZENROWS_API_KEY']}"},
}
class NullStrippedTool(BaseTool):
"""Wraps an MCP tool and removes None arguments before the call.
The MCP adapter fills unset optional fields with null, which the ZenRows
server rejects. Stripping None here keeps only the values that were set.
"""
inner: Any = None
def _run(self, *args: Any, **kwargs: Any) -> Any:
clean = {k: v for k, v in kwargs.items() if v is not None}
return self.inner._run(*args, **clean)
def wrap(tools):
return [
NullStrippedTool(
name=t.name,
description=t.description,
args_schema=t.args_schema,
inner=t,
)
for t in tools
]
def build_and_run():
# The MCP tools are only live inside this block.
with MCPServerAdapter(SERVER_PARAMS, "scrape") as mcp_tools:
scrape_tools = wrap(mcp_tools)
researcher = Agent(
role="Retail Product Data Extractor",
goal=(
"Extract ONLY the visible product name and current price from "
"each product page. No inference, no assumptions."
),
backstory=(
"You are a deterministic extraction system. If a value is not "
"visible in the scraped content, it does not exist."
),
tools=scrape_tools,
llm=llm,
verbose=True,
)
analyst = Agent(
role="Retail Pricing Analyst",
goal=(
"Compare product pricing across two retailers using ONLY "
"scraped data. No external knowledge or assumptions."
),
backstory=(
"You ONLY use provided scraped outputs. Missing fields must be "
"marked as 'NOT FOUND IN SCRAPE'."
),
llm=llm,
verbose=True,
)
research_task = Task(
description=(
"Scrape each product page below and extract the product name "
"and its current price. Use only what is visible on the page. "
"If a value is not present, write NOT FOUND IN SCRAPE.\n\n"
"When you call the scrape tool, enable JavaScript rendering and "
"premium proxy, and wait for the page content to load, since "
"these pages render product data client-side.\n\n"
+ "\n".join(TARGETS)
),
expected_output=(
"Product name and current price for each URL, from scraped "
"content only."
),
agent=researcher,
)
analysis_task = Task(
description=(
"Compare the two products using only the researcher's scraped "
"data. Do not add outside knowledge. Mark anything missing as "
"NOT FOUND IN SCRAPE.\n\n"
"Produce:\n"
"- a table of product name and price per retailer\n"
"- which product is cheaper\n"
"- one or two insights drawn from the data"
),
expected_output="A short pricing comparison: a table and a brief takeaway.",
agent=analyst,
context=[research_task],
)
crew = Crew(
agents=[researcher, analyst],
tasks=[research_task, analysis_task],
process=Process.sequential,
verbose=True,
)
result = crew.kickoff()
print("\n\n===== FINAL OUTPUT =====\n")
print(result)
if __name__ == "__main__":
build_and_run()
You can find every code snippet in this tutorial in this GitHub repository.
Wrapping up
You now have a production-ready competitor intelligence workflow built with CrewAI and ZenRows, rather than a scrapers that fail when competitors place their pricing pages behind Cloudflare, DataDome, or other anti-bot systems.
The current system works by having a researcher agent retrieve data directly from competitors' websites using ZenRows, which has a documented 99.93% success rate, while the analyst agent uses that data to generate a report. Whether you choose the custom @tool for total control via Integration or the ZenRows MCP server for easier tool management, the architecture remains the same: reliable retrieval, grounded analysis, and actionable intelligence generated from live web data.
Try ZenRows for free and get your comprehensive competitor analysis today!
Top comments (0)