How to Build a Market Research Agent with ZenRows and LangChain

ZenRows gives LangChain agents reliable access to the real web. The langchain-zenrows package exposes a single LangChain tool class, ZenRowsUniversalScraper, that handles anti-bot bypass, JavaScript rendering, and clean Markdown output behind one scraper API call. Standard web scraping tools make plain HTTP requests with no JS execution and no anti-bot bypass, so when your agent hits a protected web page it gets back a Cloudflare challenge page or nothing at all.

This tutorial builds a working market research agent that scrapes real web data and gives your large language model access to LangChain real-time web data including from bot-protected pages, returning structured data your LLM can reason over.

By the end, you will have a working LangChain web scraping setup that handles data extraction from any target without infrastructure headaches.

Why LangChain's Default Web Scraping Tools Fail on Real Web Pages

LangChain ships with built-in web scrapers like WebBaseLoader. These work fine on simple, static web pages. They use standard HTTP requests, they do not render dynamic content, and they have no mechanism for bypassing anti-bot systems.

Most sites with commercially useful web data such as product pages, pricing pages, and competitor sites are protected. When your LangChain agent sends a plain HTTP request to these pages, the server often recognizes it as a bot and returns a challenge page instead of the actual web content.

Your agent then reads the challenge page as if it were real scraped data. That leads to incorrect inputs and ultimately produces unreliable output. This is the core limitation of standard web scraping tools in AI agent workflows.

Here is what a standard LangChain web request returns against a bot-protected page:

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://www.scrapingcourse.com/antibot-challenge")
docs = loader.load()
print(docs[0].page_content)

Output:

Just a moment...Enable JavaScript and cookies to continue

The agent receives that text, has nothing useful to work with, and either hallucinates a response or tells you it could not access the page. Neither is useful in production.

This is the core problem. Your agent works perfectly in local tests against clean pages, then breaks the moment it hits a real target. The fix is to replace the retrieval layer.

Setting Up ZenRows as the LangChain Scraper Agent Tool

To add ZenRows to a LangChain agent, install langchain-zenrows, pass ZenRowsUniversalScraper to create_react_agent, and invoke with your prompt. This is the core step in integrating web scraping into your LangChain workflow. The full LangChain integration reference is in the ZenRows LangChain docs.

Installation

Start by installing the required libraries:

pip install langgraph langchain-openai langchain-zenrows

Agent Setup

from langchain_zenrows import ZenRowsUniversalScraper
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
import os

# Set these in your environment before running
# export ZENROWS_API_KEY=your_key_here
# export OPENAI_API_KEY=your_key_here

def scraper():
    llm = ChatOpenAI(model="gpt-4o-mini")
    zenrows_tool = ZenRowsUniversalScraper()

    # The agent reads the URL from the prompt and passes it to ZenRowsUniversalScraper automatically
    agent = create_react_agent(llm, [zenrows_tool])

    try:
        result = agent.invoke(
            {
                "messages": [{"role": "user", "content": "What is the major highlight on https://www.scrapingcourse.com/antibot-challenge"}]
            }
        )
        for message in result["messages"]:
            print(f"{message.content}")
    except NameError:
        print("Agent not available.")
    except Exception as e:
        print(f"Error running agent: {e}")

scraper()

This is a setup test. The agent scrapes the same antibot challenge page from the previous section to confirm ZenRows is working. Once this runs cleanly, you are ready to point the agent at any target.

Note: The page refers to its feature as the "Antibot challenge", which matches the URL slug antibot-challenge.

Output:

The major highlight on https://www.scrapingcourse.com/antibot-challenge is that you bypassed the Antibot challenge.

A few things to note about this setup:

• ZenRowsUniversalScraper reads your API key from the ZENROWS_API_KEY environment variable automatically; you do not need to pass it in code.
• Anti-bot bypass activates on demand when the agent calls the scraping tool against a protected page.
• You do not need to configure headless browsers, proxy pools, or browser sessions for the agent use case.
• The create_react_agent function comes from langgraph.prebuilt. It wires the large language model and the agent tools together so the agent can decide when to call ZenRowsUniversalScraper based on your prompt.

Building the LangChain Scraper Agent for Market Research

The agent below visits a product catalog web page, scrapes it with autoparse=True enabled in the tool configuration, and returns the four cheapest products as structured data. Each step maps to something the agent does: it receives the prompt, calls ZenRowsUniversalScraper with the source URL, gets scraped data back, and the LLM formats it as JSON. This is LangChain web scraping doing real data extraction work end to end.

Here is the complete market research agent:

from langchain_zenrows import ZenRowsUniversalScraper
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
import os

# Set these in your environment before running
# export ZENROWS_API_KEY=your_key_here
# export OPENAI_API_KEY=your_key_here

def market_research_agent():
    llm = ChatOpenAI(model="gpt-4o-mini")
    zenrows_tool = ZenRowsUniversalScraper(autoparse=True)
    # autoparse=True tells ZenRowsUniversalScraper to extract structured data automatically

    agent = create_react_agent(llm, [zenrows_tool])

    try:
        result = agent.invoke(
            {
                "messages": [{"role": "user", "content": "Visit https://www.scrapingcourse.com/ecommerce/ and scrape the page. Return the 4 cheapest products as JSON with title, price, and url fields."}]
            }
        )
        for message in result["messages"]:
            print(f"{message.content}")
    except NameError:
        print("Agent not available.")
    except Exception as e:
        print(f"Error running agent: {e}")

market_research_agent()

When you run this, the agent works through four steps:

1. It reads the prompt and identifies the target URL.
2. It calls ZenRowsUniversalScraper on the product catalog page with autoparse=True already set in the tool configuration, which extracts structured data automatically.
3. The LLM reads that data and formats it as JSON.
4. It prints the results.

Output from https://www.scrapingcourse.com/ecommerce/:

[
  {
    "title": "Affirm Water Bottle",
    "price": "$7.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/affirm-water-bottle/"
  },
  {
    "title": "Arcadio Gym Short",
    "price": "$20.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/arcadio-gym-short/"
  },
  {
    "title": "Argus All-Weather Tank",
    "price": "$22.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/argus-all-weather-tank/"
  },
  {
    "title": "Aero Daily Fitness Tee",
    "price": "$24.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/aero-daily-fitness-tee/"
  }
]

You can adapt this same pattern to any product category page. Change the URL in the prompt, adjust the number of products, or ask for different fields like ratings or review counts. The agent handles the rest.

Output Formats and When to Use Each

ZenRowsUniversalScraper gives you two ways to control output. Set response_type to choose between markdown and plaintext. Use autoparse=True when you want structured extraction instead of raw page content.

These are not interchangeable options in the same parameter. You can find the full parameter reference in the ZenRows API docs.

Format	What it returns	Best for
response_type="markdown"	Page content converted to clean Markdown	Text-heavy pages feeding into RAG pipelines or agent reasoning
autoparse=True	Structured extraction from the page. Output fields vary by page type. See the ZenRows API docs for supported page types.	E-commerce and catalog pages where you need product name, price, URL, and other key details
response_type="plaintext"	Plain text with no formatting or markup	Situations where token efficiency matters and structure is less important

markdown is a safe starting point for most agent workflows because it preserves readability without adding raw HTML noise.

Use autoparse=True specifically for pages where the data is structured and repeatable, like product listings, job boards, or real estate pages. Use response_type="plaintext" when you are feeding content into a pipeline that does not need formatting and you want to keep token usage low.

Extending to Multiple URLs

Once your agent works on a single page, you can scale it to a list of competitor or research URLs.

research_urls = [
    "https://www.scrapingcourse.com/ecommerce/",
    "https://www.scrapingcourse.com/ecommerce/page/2/",
    "https://www.scrapingcourse.com/ecommerce/page/3/"
]

all_results = []

for url in research_urls:
    result = agent.invoke(
        {
            "messages": [{"role": "user", "content": f"Scrape {url} and return the 4 cheapest products with title, price, and url."}]
        }
    )
    all_results.append(result)

One thing worth knowing is the session_id API parameter. You can pass it directly in the ZenRowsUniversalScraper scraping tool configuration to keep the same IP address across multiple requests to the same site. This works similarly to how proxy pools maintain consistent browser sessions during data collection. Check the ZenRows API docs for the exact usage.

For larger-scale data collection jobs, the same loop pattern works. You can pull a list of competitor URLs from a file, a database, or a previous web scrape, and run the LangChain agent against each one.

What to Do Next

Your LangChain scraper agent now has a retrieval layer that works on real web pages, including the ones that block standard web scrapers. The web scraping integration here removes infrastructure headaches around IP rotation, browser sessions, and anti-bot measures. That retrieval layer is what makes the agent dependable in production, where ZenRows holds a 99.93% success rate against protected targets.

ZenRows works as the scraper API layer for a wide range of LLM applications and RAG applications beyond market research. The same ZenRowsUniversalScraper LangChain tool works for:

• Review summarization
• Sentiment analysis on competitor content
• Price monitoring across multiple web pages
• Real-time data monitoring for content changes

If you want to see another full example, the ZenRows real estate AI agent tutorial follows the same pattern applied to a different domain.

If you want to extend this further, you can combine session_id with a longer multi-page loop to scrape paginated results. You can also use the css_extractor API parameter to pull specific elements when the autoparse parameter does not cover your exact data extraction needs.

Comments

[Luis Cruzy]

I appreciate how this tutorial highlights the limitations of standard web scraping tools in LangChain and provides a solution using ZenRows. One thing that caught my attention was the use of the ZenRowsUniversalScraper tool, which seems to handle anti-bot bypass and JavaScript rendering seamlessly. I'm curious to know if there are any specific use cases where this scraper might not work as expected, such as with highly dynamic or interactive web pages. Additionally, have you considered exploring other potential applications of this integration, such as data mining or automated content generation?