DEV Community: Humna Ghufran

How to Build a Market Research Agent with ZenRows and LangChain

Humna Ghufran — Mon, 13 Jul 2026 12:40:57 +0000

ZenRows gives LangChain agents reliable access to the real web. The langchain-zenrows package exposes a single LangChain tool class, ZenRowsUniversalScraper, that handles anti-bot bypass, JavaScript rendering, and clean Markdown output behind one scraper API call. Standard web scraping tools make plain HTTP requests with no JS execution and no anti-bot bypass, so when your agent hits a protected web page it gets back a Cloudflare challenge page or nothing at all.

This tutorial builds a working market research agent that scrapes real web data and gives your large language model access to LangChain real-time web data including from bot-protected pages, returning structured data your LLM can reason over.

By the end, you will have a working LangChain web scraping setup that handles data extraction from any target without infrastructure headaches.

Why LangChain's Default Web Scraping Tools Fail on Real Web Pages

LangChain ships with built-in web scrapers like WebBaseLoader. These work fine on simple, static web pages. They use standard HTTP requests, they do not render dynamic content, and they have no mechanism for bypassing anti-bot systems.

Most sites with commercially useful web data such as product pages, pricing pages, and competitor sites are protected. When your LangChain agent sends a plain HTTP request to these pages, the server often recognizes it as a bot and returns a challenge page instead of the actual web content.

Your agent then reads the challenge page as if it were real scraped data. That leads to incorrect inputs and ultimately produces unreliable output. This is the core limitation of standard web scraping tools in AI agent workflows.

Here is what a standard LangChain web request returns against a bot-protected page:

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://www.scrapingcourse.com/antibot-challenge")
docs = loader.load()
print(docs[0].page_content)

Output:

Just a moment...Enable JavaScript and cookies to continue

The agent receives that text, has nothing useful to work with, and either hallucinates a response or tells you it could not access the page. Neither is useful in production.

This is the core problem. Your agent works perfectly in local tests against clean pages, then breaks the moment it hits a real target. The fix is to replace the retrieval layer.

Setting Up ZenRows as the LangChain Scraper Agent Tool

To add ZenRows to a LangChain agent, install langchain-zenrows, pass ZenRowsUniversalScraper to create_react_agent, and invoke with your prompt. This is the core step in integrating web scraping into your LangChain workflow. The full LangChain integration reference is in the ZenRows LangChain docs.

Installation

Start by installing the required libraries:

pip install langgraph langchain-openai langchain-zenrows

Agent Setup

from langchain_zenrows import ZenRowsUniversalScraper
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
import os

# Set these in your environment before running
# export ZENROWS_API_KEY=your_key_here
# export OPENAI_API_KEY=your_key_here

def scraper():
    llm = ChatOpenAI(model="gpt-4o-mini")
    zenrows_tool = ZenRowsUniversalScraper()

    # The agent reads the URL from the prompt and passes it to ZenRowsUniversalScraper automatically
    agent = create_react_agent(llm, [zenrows_tool])

    try:
        result = agent.invoke(
            {
                "messages": [{"role": "user", "content": "What is the major highlight on https://www.scrapingcourse.com/antibot-challenge"}]
            }
        )
        for message in result["messages"]:
            print(f"{message.content}")
    except NameError:
        print("Agent not available.")
    except Exception as e:
        print(f"Error running agent: {e}")

scraper()

This is a setup test. The agent scrapes the same antibot challenge page from the previous section to confirm ZenRows is working. Once this runs cleanly, you are ready to point the agent at any target.

Note: The page refers to its feature as the "Antibot challenge", which matches the URL slug antibot-challenge.

Output:

The major highlight on https://www.scrapingcourse.com/antibot-challenge is that you bypassed the Antibot challenge.

A few things to note about this setup:

ZenRowsUniversalScraper reads your API key from the ZENROWS_API_KEY environment variable automatically; you do not need to pass it in code.
Anti-bot bypass activates on demand when the agent calls the scraping tool against a protected page.
You do not need to configure headless browsers, proxy pools, or browser sessions for the agent use case.
The create_react_agent function comes from langgraph.prebuilt. It wires the large language model and the agent tools together so the agent can decide when to call ZenRowsUniversalScraper based on your prompt.

Building the LangChain Scraper Agent for Market Research

The agent below visits a product catalog web page, scrapes it with autoparse=True enabled in the tool configuration, and returns the four cheapest products as structured data. Each step maps to something the agent does: it receives the prompt, calls ZenRowsUniversalScraper with the source URL, gets scraped data back, and the LLM formats it as JSON. This is LangChain web scraping doing real data extraction work end to end.

Here is the complete market research agent:

from langchain_zenrows import ZenRowsUniversalScraper
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
import os

# Set these in your environment before running
# export ZENROWS_API_KEY=your_key_here
# export OPENAI_API_KEY=your_key_here

def market_research_agent():
    llm = ChatOpenAI(model="gpt-4o-mini")
    zenrows_tool = ZenRowsUniversalScraper(autoparse=True)
    # autoparse=True tells ZenRowsUniversalScraper to extract structured data automatically

    agent = create_react_agent(llm, [zenrows_tool])

    try:
        result = agent.invoke(
            {
                "messages": [{"role": "user", "content": "Visit https://www.scrapingcourse.com/ecommerce/ and scrape the page. Return the 4 cheapest products as JSON with title, price, and url fields."}]
            }
        )
        for message in result["messages"]:
            print(f"{message.content}")
    except NameError:
        print("Agent not available.")
    except Exception as e:
        print(f"Error running agent: {e}")

market_research_agent()

When you run this, the agent works through four steps:

It reads the prompt and identifies the target URL.
It calls ZenRowsUniversalScraper on the product catalog page with autoparse=True already set in the tool configuration, which extracts structured data automatically.
The LLM reads that data and formats it as JSON.
It prints the results.

Output from https://www.scrapingcourse.com/ecommerce/:

[
  {
    "title": "Affirm Water Bottle",
    "price": "$7.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/affirm-water-bottle/"
  },
  {
    "title": "Arcadio Gym Short",
    "price": "$20.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/arcadio-gym-short/"
  },
  {
    "title": "Argus All-Weather Tank",
    "price": "$22.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/argus-all-weather-tank/"
  },
  {
    "title": "Aero Daily Fitness Tee",
    "price": "$24.00",
    "url": "https://www.scrapingcourse.com/ecommerce/product/aero-daily-fitness-tee/"
  }
]

You can adapt this same pattern to any product category page. Change the URL in the prompt, adjust the number of products, or ask for different fields like ratings or review counts. The agent handles the rest.

Output Formats and When to Use Each

ZenRowsUniversalScraper gives you two ways to control output. Set response_type to choose between markdown and plaintext. Use autoparse=True when you want structured extraction instead of raw page content.

These are not interchangeable options in the same parameter. You can find the full parameter reference in the ZenRows API docs.

Format	What it returns	Best for
`response_type="markdown"`	Page content converted to clean Markdown	Text-heavy pages feeding into RAG pipelines or agent reasoning
`autoparse=True`	Structured extraction from the page. Output fields vary by page type. See the ZenRows API docs for supported page types.	E-commerce and catalog pages where you need product name, price, URL, and other key details
`response_type="plaintext"`	Plain text with no formatting or markup	Situations where token efficiency matters and structure is less important

markdown is a safe starting point for most agent workflows because it preserves readability without adding raw HTML noise.

Use autoparse=True specifically for pages where the data is structured and repeatable, like product listings, job boards, or real estate pages. Use response_type="plaintext" when you are feeding content into a pipeline that does not need formatting and you want to keep token usage low.

Extending to Multiple URLs

Once your agent works on a single page, you can scale it to a list of competitor or research URLs.

research_urls = [
    "https://www.scrapingcourse.com/ecommerce/",
    "https://www.scrapingcourse.com/ecommerce/page/2/",
    "https://www.scrapingcourse.com/ecommerce/page/3/"
]

all_results = []

for url in research_urls:
    result = agent.invoke(
        {
            "messages": [{"role": "user", "content": f"Scrape {url} and return the 4 cheapest products with title, price, and url."}]
        }
    )
    all_results.append(result)

One thing worth knowing is the session_id API parameter. You can pass it directly in the ZenRowsUniversalScraper scraping tool configuration to keep the same IP address across multiple requests to the same site. This works similarly to how proxy pools maintain consistent browser sessions during data collection. Check the ZenRows API docs for the exact usage.

For larger-scale data collection jobs, the same loop pattern works. You can pull a list of competitor URLs from a file, a database, or a previous web scrape, and run the LangChain agent against each one.

What to Do Next

Your LangChain scraper agent now has a retrieval layer that works on real web pages, including the ones that block standard web scrapers. The web scraping integration here removes infrastructure headaches around IP rotation, browser sessions, and anti-bot measures. That retrieval layer is what makes the agent dependable in production, where ZenRows holds a 99.93% success rate against protected targets.

ZenRows works as the scraper API layer for a wide range of LLM applications and RAG applications beyond market research. The same ZenRowsUniversalScraper LangChain tool works for:

Review summarization
Sentiment analysis on competitor content
Price monitoring across multiple web pages
Real-time data monitoring for content changes

If you want to see another full example, the ZenRows real estate AI agent tutorial follows the same pattern applied to a different domain.

If you want to extend this further, you can combine session_id with a longer multi-page loop to scrape paginated results. You can also use the css_extractor API parameter to pull specific elements when the autoparse parameter does not cover your exact data extraction needs.

How to turn an AI prototype into a production system

Humna Ghufran — Fri, 22 May 2026 15:14:44 +0000

AI tools have changed how fast software gets off the ground. Today, a single developer can go from an idea to a working AI prototype in days, sometimes hours. In a controlled study on GitHub Copilot, developers finished a coding task 55.8% faster with AI help. That’s why prototypes are everywhere right now.

But these tools also hide decisions that production systems cannot afford to ignore. A prototype can “work” while the hard parts stay undefined, like:

Who can log in?
Where data is allowed to live?
Which services exist and how do they talk?
What is the deployment model?
What does it costs to run?
Who owns each part when something breaks?

In many AI-generated prototypes, authentication, data boundaries, infra topology, deployment costs and ownership stay implicit or missing entirely.

AI builders optimize for instant output, but production demands explicit responsibility. If those choices stay hidden, teams hit the same wall: the app runs, but it is hard to review, hard to secure, expensive to deploy and risky to hand off.

In this article, I’ll decode hidden decisions step by step. I’ll show how to take a fragile AI prototype and make it reviewable, ownable and deployable. You’ll also see how Bit Cloud speeds this up by surfacing scope, infrastructure and delivery artifacts early.

What does production mean in this walkthrough?

Production is when a system becomes transferable and operable without guesswork. The checklist below is what that looks like in practice.

Production means:

Clear scope and boundaries: One defined job, explicit dependencies and clear “out of scope.”
Known infrastructure and deployment model: Where it runs, how it ships, what it relies on and what costs it creates.
Reviewable code structure: Components and responsibilities are readable without starting from the UI.
Tests that express behavior: Tests document intent and boundaries so changes stay safe.
Documentation that supports handoff: Decisions, assumptions and ops details are written down for a clean transfer.

Starting artifact: Useful output without explicit responsibility

As a concrete starting point, I used a simple frontend checkout prototype built in Replit. The application renders a centered checkout card with a single item, a fixed price and a “Pay Now” action.

The flow completes visually and responds to user input, confirming that the basic interaction works.

What’s missing:

This prototype behaves like a demo that processes a request, not a system with explicit responsibility. The plan mentions a database, an API route and server logic, but it does not define the production decisions that make checkout safe and ownable.

Key questions remain unanswered, like who is authorized to pay or view an order, what prevents duplicate charges, what happens on timeouts or partial failures and how the system proves what occurred after the click. Infrastructure and operations are also still implicit, which means deployment, observability and cost drivers are not yet visible.

Reframing the prototype as a system

Instead of treating the prototype as something to be polished, Bit Cloud treats it as something to be structured. The goal is not better output, but making system responsibility explicit as early as possible.

Bit Cloud approaches the prototype as a system-in-waiting. The first step is decomposition. The generated application is broken down into explicit components, execution flows and ownership boundaries.

What was previously implied by UI behavior is converted into defined responsibilities: where authentication lives, how state is managed, which components own business logic and how external services are integrated.

Hope AI is Bit Cloud’s system intelligence layer, used at this stage not as a prompt engine but as a restructuring tool. Once boundaries and flows are defined inside Bit Cloud, Hope AI regenerates or reorganizes parts of the system to align with that architecture. The output reflects architectural intent rather than raw generated code. Code, tests and documentation are created in the context of the system design managed in Bit Cloud, not in isolation.

At this point, the prototype stops being a single blob of behavior and starts becoming a system that can be reviewed, reasoned about and safely extended. The rest of the walkthrough builds on this foundation.

Within Bit Cloud, Hope AI acts as the engine that turns architectural decisions into structured software. At this stage, it is not used as a prompt engine but as a restructuring tool. Once boundaries and flows are defined, Hope AI regenerates or reorganizes parts of the system to match that structure, producing artifacts that reflect architectural intent rather than raw output. Code, tests and documentation are created as part of the Bit Cloud system design, not in isolation.

Making scope and infrastructure explicit

Once the prototype is reframed as a system, the next step is to surface what was previously hidden: scope, infrastructure and cost. This is where most AI-generated prototypes either gain clarity or accumulate risk. Until these decisions are explicit, progress is based on assumptions rather than engineering judgment.

Defining what services actually exist

The first change is moving from implied behavior to explicit services. Even in a simple checkout flow, this forces clarity. The system is no longer “one app,” but a set of responsibilities with clear ownership.

This decomposition replaces guesswork with structure.

Making data flow visible

With services defined, data flow can be traced end to end. User input moves through validation, business logic and persistence before producing a response. This makes it clear where state is created, where it is read and where consistency matters. It also exposes failure points that prototypes typically hide, such as partial updates or retry behavior.

Surfacing infrastructure and cost early

Once the system shape is known, infrastructure can no longer remain abstract. The deployment model becomes explicit: what runs as a service, what requires storage and what must scale independently. Compute usage, storage requirements, external API calls and environment separation can all be estimated and discussed early, before they become expensive constraints.

From V1 Alpha to something deployable

At this stage, the system has moved beyond a prototype and into a V1 Alpha. This does not mean the product is finished. It means the system is now structured in a way that allows it to be deployed, reviewed and extended without ambiguity.

The V1 Alpha contains concrete engineering artifacts that did not exist in the original prototype. V1 Alpha includes:

Structured code with explicit boundaries

The codebase is organized around defined components and responsibilities. UI logic, application logic, data access and integrations are separated so changes can be made deliberately rather than inferred from behavior.

Tests that express expected behavior

Basic tests exist to describe how the system should behave under normal conditions. These tests do not aim for full coverage, but they establish intent and provide a safety net for future changes.

Documentation for ownership and handoff

Key decisions, assumptions and system boundaries are documented. This includes what the system does, what it does not do and where responsibility lies. Another engineer can now review or take over the system without reverse-engineering intent.

A clear deployment path

The system can be deployed outside the prototype environment. The runtime, dependencies and environment configuration are defined well enough to support real deployment, even if further hardening is required.

What is intentionally deferred

Not everything is solved at this stage and that is by design. Performance optimization, advanced security hardening, observability and scale testing are intentionally deferred. These concerns depend on real usage patterns and are expensive to guess prematurely. The V1 Alpha exists to reduce uncertainty, not to optimize prematurely.

What changed from the original prototype
Compared to the initial prototype, the most important change is not visual. It is structural. The original prototype produced behavior without making decisions visible.

The V1 Alpha makes those decisions explicit. Ownership is clear. Flows are traceable. Assumptions are documented. The system can now be reasoned about as a system, not just interacted with as a UI.

Wrapping up

Turning a prototype into a production system is not about replacing speed with process. It is about applying just enough structure at the right moment. When responsibility is explicit, progress becomes predictable and production stops being a leap of faith.

If you’re holding a prototype that “works,” but still can’t be reviewed, owned or deployed with confidence, Bit Cloud helps you make the transition without guesswork. If you want a clearer path from experimentation to deployment, start with Bit Cloud today!