DEV Community

Zenrows profile image Tiioluwani
Tiioluwani for Zenrows

Posted on Originally published at zenrows.com

Build a Web Research Agent With Google ADK and Zenrows

#ai #tutorial #agents #python

This article was originally published on the Zenrows blog.

Your research agent might be reading fake pages. When a target sits behind Cloudflare, a plain requests call returns a 403 and a "Just a moment..." challenge page in place of the content. Unless the tool checks for that, the agent can treat the challenge markup as the answer and report on a page it never saw.

In this tutorial, you'll see how that happens and how to fix it in a Google Agent Development Kit (ADK) agent. You'll give the agent a web tool that routes each fetch through Zenrows Fetch and returns the page as Markdown, then connect a second agent to the hosted Zenrows MCP server with no function code. You'll need Python 3.10 or later, a Google AI Studio API key, and a Zenrows API key. The free plan includes 5,000 credits a month. The complete code is on GitHub.

What You'll Build

The same Cloudflare-protected URL fetched with plain requests returns a 403 challenge page, while the Zenrows fetch function returns the page heading as Markdown

  • You'll register a function tool, fetch_protected_page, on a Gemini-powered ADK agent.
  • You'll write a comparison script that fetches one URL with plain HTTP and with Zenrows.
  • You'll add a second agent that loads the Zenrows MCP tools through ADK's MCPToolset.

1. Set Up the Project 

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

The pinned requirements.txt installs google-adk 2.9.2 with its MCP extra.

google-adk[mcp]==2.9.2
mcp==1.30.0
python-dotenv==1.2.3
requests==2.34.2
pytest==9.1.1
ruff==0.16.8
-e .
Enter fullscreen mode Exit fullscreen mode

Copy the environment template and add your keys. The .gitignore excludes .env.

cp .env.example .env
Enter fullscreen mode Exit fullscreen mode 
GOOGLE_API_KEY=your_google_api_key
ZENROWS_API_KEY=your_zenrows_api_key
Enter fullscreen mode Exit fullscreen mode

Here's the project layout.

google-adk-zenrows-research-agent/
├── scripts/
│   ├── compare_fetch.py
│   └── list_mcp_tools.py
├── tests/
│   ├── test_agent.py
│   ├── test_mcp_agent.py
│   └── test_web.py
├── zenrows_mcp_agent/
│   ├── __init__.py
│   └── agent.py
├── zenrows_research_agent/
│   ├── __init__.py
│   ├── agent.py
│   └── web.py
├── .env.example
├── .gitignore
├── LICENSE
├── pyproject.toml
├── README.md
└── requirements.txt
Enter fullscreen mode Exit fullscreen mode

2. Write the Fetch Function

Create zenrows_research_agent/web.py.

"""Web retrieval tools used by the Google ADK agent."""
import os
import requests
ZENROWS_ENDPOINT = "https://api.zenrows.com/v1/"
DEFAULT_TIMEOUT = 180
MAX_TOOL_CONTENT = 12_000
def fetch_with_http(url: str) -> dict:
    """Fetch a public URL with a standard HTTP request for comparison."""
    try:
        # send the request without a protected-site access service.
        response = requests.get(
            url,
            headers={"User-Agent": "google-adk-research-agent/0.1"},
            timeout=20,
        )
    except requests.RequestException as exc:
        return {"status": "error", "url": url, "error": str(exc)}
    return {
        "status": "success" if response.ok else "blocked",
        "url": url,
        "status_code": response.status_code,
        "content": response.text[:2_000],
    }
def fetch_protected_page(url: str) -> dict:
    """Fetch a protected or JavaScript-rendered page through Zenrows.
    Args:
        url: The complete public URL the research agent needs to read.
    Returns:
        A dictionary containing the request status, source URL, and page content.
    """
    api_key = os.getenv("ZENROWS_API_KEY")
    if not api_key:
        return {
            "status": "error",
            "url": url,
            "error": "ZENROWS_API_KEY is not configured.",
        }
    # let Adaptive Stealth Mode select the request configuration.
    params = {
        "apikey": api_key,
        "url": url,
        "mode": "auto",
        "response_type": "markdown",
        "original_status": "true",
    }
    try:
        # give Adaptive Stealth Mode enough time to retrieve the page.
        response = requests.get(
            ZENROWS_ENDPOINT,
            params=params,
            timeout=DEFAULT_TIMEOUT,
        )
        response.raise_for_status()
    except requests.RequestException as exc:
        # remove the API key from the error text before it reaches the model.
        error = str(exc).replace(api_key, "[REDACTED]")
        return {"status": "error", "url": url, "error": error}
    # do not return a successful result without page content.
    if not response.text.strip():
        return {
            "status": "error",
            "url": url,
            "error": "Zenrows returned an empty response.",
        }
    return {
        "status": "success",
        "url": url,
        "content": response.text[:MAX_TOOL_CONTENT],
    }
Enter fullscreen mode Exit fullscreen mode

fetch_protected_page sends the URL to Zenrows with mode=auto, which turns on Adaptive Stealth Mode. It starts with the lightest configuration likely to work and escalates only when the target needs it. A few safeguards keep the agent honest.

  • response_type=markdown returns readable text in place of raw HTML.
  • original_status=true passes through the target's status code, so an error page from the target counts as an error.
  • The 180-second timeout gives mode=auto room to finish on protected targets.
  • Content is capped at 12,000 characters, and an empty response is reported as an error.
  • The API key is redacted from any error text before it reaches the model. fetch_with_http is a plain request used only for the comparison. ## 3. Register the Function as an ADK Tool Create zenrows_research_agent/agent.py. 
"""Google ADK agent configuration."""
from pathlib import Path
from dotenv import load_dotenv
from google.adk.agents import Agent
from .web import fetch_protected_page
# load the API keys stored in the project environment file.
load_dotenv(Path(__file__).resolve().parents[1] / ".env")
# register the Zenrows retrieval function as an ADK tool.
root_agent = Agent(
    model="gemini-3.6-flash",
    name="web_research_agent",
    description="Researches public web pages that ordinary HTTP tools cannot read.",
    instruction=(
        "You are a web research agent. Use fetch_protected_page whenever a task "
        "requires content from a supplied URL. Base the answer only on returned "
        "page content, mention the source URL, and state clearly when retrieval "
        "fails or the page does not contain the requested information."
    ),
    tools=[fetch_protected_page],
)
Enter fullscreen mode Exit fullscreen mode

ADK builds the tool schema from the function's name, type hints, and docstring, so the model only has to supply a URL.

4. Compare Plain HTTP With Zenrows

Create scripts/compare_fetch.py.

"""Compare standard HTTP access with Zenrows access."""
import json
from dotenv import load_dotenv
from zenrows_research_agent.web import fetch_protected_page, fetch_with_http
TARGET_URL = "https://www.scrapingcourse.com/cloudflare-challenge"
def summarize(result: dict) -> dict:
    """Reduce a fetch result to fields that are safe to print."""
    return {
        "status": result["status"],
        "url": result["url"],
        "status_code": result.get("status_code"),
        "content_preview": result.get("content", "")[:300],
        "error": result.get("error"),
    }
if __name__ == "__main__":
    # load the Zenrows API key from the project environment.
    load_dotenv()
    # retrieve the same URL through both access methods.
    comparison = {
        "standard_http": summarize(fetch_with_http(TARGET_URL)),
        "zenrows": summarize(fetch_protected_page(TARGET_URL)),
    }
    # print the comparison without exposing the API key.
    print(json.dumps(comparison, indent=2))
Enter fullscreen mode Exit fullscreen mode 
python scripts/compare_fetch.py
Enter fullscreen mode Exit fullscreen mode 
{
  "standard_http": {
    "status": "blocked",
    "url": "https://www.scrapingcourse.com/cloudflare-challenge",
    "status_code": 403,
    "content_preview": "<!DOCTYPE html><html lang=\"en-US\"><head><title>Just a moment...</title><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"><meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\"><meta name=\"robots\" content=\"noindex,nofollow\"><meta name=\"viewport\" content=\"width=device-width,initial-scal",
    "error": null
  },
  "zenrows": {
    "status": "success",
    "url": "https://www.scrapingcourse.com/cloudflare-challenge",
    "status_code": null,
    "content_preview": "[![](https://www.scrapingcourse.com/assets/images/logo.svg) Scraping Course](http://www.scrapingcourse.com/)\n\n# Cloudflare Challenge\n\n![](https://www.scrapingcourse.com/assets/images/challenge.svg)\n\n## You bypassed the Cloudflare challenge! :D",
    "error": null
  }
}
Enter fullscreen mode Exit fullscreen mode

The plain request got a 403 and the challenge page. Zenrows returned the page heading and the verification message. The Zenrows status_code reads null because the function doesn't return it.

5. Run the Agent 

adk run zenrows_research_agent
Enter fullscreen mode Exit fullscreen mode

Submit this task.

Fetch https://www.scrapingcourse.com/cloudflare-challenge and report the page title, whether the challenge was passed, and the visible verification text. Cite the source URL.
Enter fullscreen mode Exit fullscreen mode 
[web_research_agent]: Based on the content retrieved from [https://www.scrapingcourse.com/cloudflare-challenge](https://www.scrapingcourse.com/cloudflare-challenge):
* **Page Title / Main Heading:** Cloudflare Challenge
* **Challenge Status:** Passed / Bypassed
* **Visible Verification Text:** "You bypassed the Cloudflare challenge! :D"
* **Source URL:** `https://www.scrapingcourse.com/cloudflare-challenge`
Enter fullscreen mode Exit fullscreen mode

Run adk web if you'd prefer to test in ADK's browser UI.

6. Connect the Zenrows MCP Server

Create zenrows_mcp_agent/agent.py.

"""Google ADK agent that uses the hosted Zenrows MCP server."""
import os
from pathlib import Path
from dotenv import load_dotenv
from google.adk.agents import Agent
from google.adk.tools.mcp_tool import MCPToolset, StreamableHTTPConnectionParams
# load the API keys stored in the project environment file.
load_dotenv(Path(__file__).resolve().parents[1] / ".env")
# connect to the hosted Zenrows MCP server with the API key as a bearer token.
zenrows_toolset = MCPToolset(
    connection_params=StreamableHTTPConnectionParams(
        url="https://mcp.zenrows.com/mcp",
        headers={"Authorization": f"Bearer {os.getenv('ZENROWS_API_KEY', '')}"},
    )
)
# the agent chooses between the Zenrows scrape and browser tools on its own.
root_agent = Agent(
    model="gemini-3.6-flash",
    name="web_research_mcp_agent",
    description="Researches public web pages through the Zenrows MCP server.",
    instruction=(
        "You are a web research agent. Use the Zenrows tools whenever a task "
        "requires content from a supplied URL. Base the answer only on returned "
        "page content, mention the source URL, and state clearly when retrieval "
        "fails or the page does not contain the requested information."
    ),
    tools=[zenrows_toolset],
)
Enter fullscreen mode Exit fullscreen mode

MCPToolset connects to https://mcp.zenrows.com/mcp with your API key as a bearer token, and ADK loads the server's tools. Check which ones arrived with scripts/list_mcp_tools.py.

"""Check which Zenrows MCP tools Google ADK loads for the MCP agent."""
import asyncio
from zenrows_mcp_agent.agent import zenrows_toolset
RESEARCH_TOOLS = ["scrape", "extract", "crawl_create", "batch_create", "browser_navigate"]
async def main() -> None:
    # ask the hosted Zenrows MCP server which tools it exposes.
    tools = await zenrows_toolset.get_tools()
    await zenrows_toolset.close()
    names = {tool.name for tool in tools}
    # confirm the tools a research agent relies on are loaded.
    print(f"Loaded {len(names)} Zenrows tools")
    for name in RESEARCH_TOOLS:
        print(f"{name}: {'available' if name in names else 'not loaded'}")
if __name__ == "__main__":
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode 
python scripts/list_mcp_tools.py
adk run zenrows_mcp_agent
Enter fullscreen mode Exit fullscreen mode 
Loaded 62 Zenrows tools
scrape: available
extract: available
crawl_create: available
batch_create: available
browser_navigate: available
Enter fullscreen mode Exit fullscreen mode

ADK prints experimental-feature warnings before that output.

Function Tool or MCP Server

Function tool Zenrows MCP
What the agent calls fetch_protected_page, one typed function scrape, extract, Crawl, Batch, and browser tools
Request settings Fixed in code (mode=auto, Markdown) Chosen by the MCP server per task
Code you maintain The function and its tests One MCPToolset block
Pick it when You want one fixed, typed request You want the agent to choose among scraping, crawling, and browser actions

Wrapping Up

Your ADK agent now reads protected pages as Markdown through a typed function, and a second agent reaches the full Zenrows toolset over MCP. Change the URL and the prompt to point it at your own research tasks.

What's Next

Top comments (0)