Integrating Zenrows into smolagents for production web access

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/zenrows-smolagents

This guide shows you how to replace smolagents' VisitWebpageTool with a Zenrows @tool that returns full Markdown from JavaScript-rendered and protected pages. You need Python 3.10+, a Zenrows API key, and a Hugging Face token.

All code is in the GitHub repository.

Before you start

• Python 3.10 or later
• Zenrows account for your API key
• Hugging Face account for the model access token

Why replace VisitWebpageTool

VisitWebpageTool fetches with Python's requests library. On protected pages, the TLS handshake exposes the client signature and the target returns a challenge page instead of content. The agent receives that and reasons over it without knowing it's bad input.

Running VisitWebpageTool against a protected Walmart product page returned this:

Robot or human?
===============

Activate and hold the button to confirm that you're human. Thank You

That response goes straight into the model. Research tasks, summarization, and classification all run on it and still produce output.

Step 1: Install packages

pip install smolagents requests python-dotenv

Step 2: Add your API key

# .env
ZENROWS_API_KEY=your_zenrows_api_key_here
HF_TOKEN=your_hugging_face_token_here

Step 3: Create the Zenrows fetch tool

Create zenrows_tool.py. The @tool decorator registers fetch_page with your CodeAgent. mode=auto lets Zenrows pick the retrieval strategy per target — you don't configure browser rendering or proxies manually.

import os

import requests
from dotenv import load_dotenv
from smolagents import tool

load_dotenv()

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

@tool
def fetch_page(url: str) -> str:
    """
    Fetches webpage content and returns it as clean Markdown, including
    JavaScript-rendered and protected pages.

    Use this tool whenever you need to read a specific URL and retrieve
    webpage content for research, summarization, or analysis.

    Args:
        url: The webpage URL to fetch.
    """
    try:
        response = requests.get(
            "https://api.zenrows.com/v1/",
            params={
                "url": url,
                "apikey": ZENROWS_API_KEY,
                "mode": "auto",
                "response_type": "markdown",
            },
            timeout=30,
        )
        response.raise_for_status()
        return response.text

    except requests.RequestException as exc:
        raise RuntimeError(f"Failed to retrieve content from {url}") from exc

Step 4: Test the fetch tool before wiring it to an agent

# add to the bottom of zenrows_tool.py
if __name__ == "__main__":
    result = fetch_page("https://www.walmart.com/ip/AirPods-Pro-3/17835006350")
    print(result[1500:2500])

python zenrows_tool.py

You should see Markdown from the product page instead of a bot check.

Write a docstring the model will actually use

smolagents builds the tool description from the function signature and docstring at runtime. A vague docstring means the agent skips the tool or writes its own fetch code.

Vague:

@tool
def fetch_page(url: str) -> str:
    """
    Fetches webpage content.

    Args:
        url: The webpage URL.
    """

Specific (what's in zenrows_tool.py):

@tool
def fetch_page(url: str) -> str:
    """
    Fetches webpage content and returns it as clean Markdown, including
    JavaScript-rendered and protected pages.

    Use this tool whenever you need to read a specific URL and retrieve
    webpage content for research, summarization, or analysis.

    Args:
        url: The webpage URL to fetch.
    """

The specific version tells the model what the tool returns, when to call it, and what page types it handles. That's what routes retrieval tasks to fetch_page instead of fallback behavior.

Step 5: Initialize the CodeAgent

Create agent.py and register fetch_page:

import os

from dotenv import load_dotenv
from smolagents import CodeAgent, InferenceClientModel

from zenrows_tool import fetch_page

load_dotenv()

model = InferenceClientModel(
    model_id="Qwen/Qwen2.5-7B-Instruct",
    token=os.getenv("HF_TOKEN"),
)

agent = CodeAgent(
    tools=[fetch_page],
    model=model,
)

response = agent.run(
    """
    Go to https://techcrunch.com/2026/07/23/amd-takes-on-nvidia-with-its-helios-ai-rack-scale-system/

    Read the article and identify:
    - the company involved,
    - the main announcement,
    - the news category,
    - why it matters.
    """,
    max_steps=8,
)

print(response)

Step 6: Run the agent

python agent.py

The run finished in three steps. The agent called fetch_page once and structured its output from the returned content:

{'company': 'AMD',
 'main_announcement': 'AMD has launched the Helios AI rack-scale system, challenging Nvidia in the AI rack-scale system market.',
 'news_category': 'AI',
 'why_it_matters': 'AMD is challenging Nvidia in the AI rack-scale system market, which is a significant move as Nvidia has historically dominated this market.'}

Handling concurrent Hugging Face Spaces

Shared Spaces route all outbound requests through a shared IP pool. Sites enforcing IP-based rate limits see it as one source and respond with blocks or challenges.

fetch_page routes through Zenrows instead of directly from your Space to the target. mode=auto is already in the params block — no additional proxy config needed:

params={
    "url": url,
    "apikey": ZENROWS_API_KEY,
    "mode": "auto",         # Zenrows picks the retrieval strategy per target
    "response_type": "markdown",
}

What's next

The same @tool pattern applies across frameworks:

• OpenAI Agents SDK version uses @function_tool
• AG2 version uses a typed Tool registered on two agents
• LlamaIndex RAG pipeline for indexing live web content

Full project: GitHub repository