Build a web-aware agent with OpenAI Agents SDK and Zenrows

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/web-aware-agent-openai-agents-sdk-zenrows

---

This guide shows you how to register Zenrows Fetch as a @function_tool next to WebSearchTool in a single OpenAI Agents SDK agent, so the model routes between discovery and live-page retrieval on its own each turn. You need Python 3.9+, an OpenAI API key, and a Zenrows API key.

All the code is on GitHub.

Before you start

• Python 3.9 or later
• An OpenAI API key from your OpenAI developer dashboard
• A Zenrows API key from your Zenrows dashboard

Install dependencies:

python3 -m pip install openai-agents python-dotenv

Add both keys to a .env file and add it to .gitignore:

OPENAI_API_KEY=your_openai_api_key_here
ZENROWS_API_KEY=your_zenrows_api_key_here

Load them in your script:

from dotenv import load_dotenv

load_dotenv()

Set up a baseline agent with the built-in search

WebSearchTool is a hosted tool that runs on OpenAI's infrastructure. Enabling it takes one line.

import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool

load_dotenv()

agent = Agent(
    name="Product Researcher",
    instructions=(
        "You research products and web pages. "
        "Report exactly what you find, and state plainly when "
        "specific information is missing from your results."
    ),
    tools=[WebSearchTool()],
)

async def main():
    result = await Runner.run(
        agent,
        "Which laptops did Apple release most recently, and where can I buy them?",
    )
    print(result.final_output)

asyncio.run(main())

This works for broad discovery queries. The agent returns relevant pages with source links. The gap appears on live or protected targets.

Where the built-in search stops

Issue 1: indexed snippets, not live pages

Built-in search returns OpenAI's cached copy of a page, not the current version. For documentation, that's usually fine. For prices, stock levels, or anything that changes frequently, the cached copy lags.

Issue 2: JavaScript-rendered content

Many pages ship an HTML shell and fill in prices, inventory, and reviews via JavaScript at runtime. The search index stores the shell. The data you want was never captured.

import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool

load_dotenv()

agent = Agent(
    name="Product Researcher",
    instructions=(
        "You retrieve product details from web pages. "
        "Report exactly what you find, and say so plainly if content is missing."
    ),
    tools=[WebSearchTool()],
)

async def main():
    result = await Runner.run(
        agent,
        "What is the current price and stock status of "
        "https://www.amazon.com/dp/B0GR1JKMBV/ref=fs_a_mbt2_us1?th=1",
    )
    print(result.final_output)

asyncio.run(main())

The agent finds the page. It returns the product title and a clear statement that the price wasn't accessible:

I visited the Amazon product page for ASIN B0GR1JKMBV...
The page displays: "To see product details, add this item to your cart."
Therefore I could not retrieve the current price or availability.

Add Zenrows Fetch as a @function_tool

import os
import requests
from agents import function_tool

@function_tool
def fetch_page_content(url: str) -> str:
    """Fetch the full, current content of a specific web page as Markdown.

    Use this when you need the complete content of a known URL rather than
    a search result summary, such as live prices, stock levels, or data
    that loads via JavaScript after the page opens.

    Args:
        url: The full URL of the page to retrieve.
    """
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "url": url,
            "apikey": os.getenv("ZENROWS_API_KEY"),
            # mode=auto lets Zenrows pick the right settings per site
            "mode": "auto",
            "response_type": "markdown",
        },
        timeout=90,
    )

    # return the error as a string so the agent can react instead of crashing
    if response.status_code != 200:
        return f"Zenrows returned {response.status_code} for {url}"

    return response.text

Four things to note:

• @function_tool builds the schema from the function signature automatically. No JSON spec to write.
• mode=auto activates Adaptive Stealth Mode: JavaScript rendering and premium proxy escalation only when the target actually needs them.
• response_type=markdown strips markup so the model reads clean text.
• The docstring is what the model reads every turn to decide whether to call this tool. The phrase "rather than a search result summary" is what routes it away from built-in search when you have a specific URL.

Run both tools in a single agent loop

import asyncio
import os
import requests
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool, function_tool

load_dotenv()

@function_tool
def fetch_page_content(url: str) -> str:
    """Fetch the full, current content of a specific web page as Markdown.

    Use this when you need the complete content of a known URL rather than
    a search result summary, such as live prices, stock levels, or data
    that loads via JavaScript after the page opens.

    Args:
        url: The full URL of the page to retrieve.
    """
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "url": url,
            "apikey": os.getenv("ZENROWS_API_KEY"),
            "mode": "auto",
            "response_type": "markdown",
        },
        timeout=90,
    )

    if response.status_code != 200:
        return f"Zenrows returned {response.status_code} for {url}"

    return response.text

# the model routes between both tools from their descriptions alone
agent = Agent(
    name="Web-Aware Researcher",
    instructions=(
        "You research products on the web. "
        "Use web search to find the product page URL. "
        "Then call fetch_page_content on that URL and report the price "
        "and stock status from the fetched page content. "
        "State which tool each figure came from."
    ),
    tools=[WebSearchTool(), fetch_page_content],
)

def print_trace(items):
    for item in items:
        raw = getattr(item, "raw_item", None)
        name = getattr(raw, "name", None) or getattr(raw, "type", "")
        print(f"[{item.type}] {name}")

        if item.type == "tool_call_item" and name == "fetch_page_content":
            print("  args:", getattr(raw, "arguments", ""))

        if item.type == "tool_call_output_item":
            out = str(getattr(item, "output", ""))
            print(f"  output: {len(out)} chars")
            print(f"  body: {out[:500]}")

async def main():
    result = await Runner.run(
        agent,
        "Find the PriceOye page for the iPhone 17 Pro "
        "and report its current price and stock status.",
    )

    print_trace(result.new_items)
    print("\n---\n")
    print(result.final_output)

asyncio.run(main())

The trace shows the two-step routing. Turn one: web_search_call for discovery. Turn two: fetch_page_content with the URL it found.

[tool_call_item] web_search_call
[message_output_item] message
[tool_call_item] fetch_page_content
  args: {"url":"https://priceoye.pk/mobiles/apple/apple-iphone-17-pro"}
[tool_call_output_item]

---

Here are the details for the iPhone 17 Pro from its PriceOye product page:

Current Price: Rs 471,999
Stock Status: Only 1 left in stock

Price and stock status are directly extracted from the fetched page content
(functions.fetch_page_content tool).

No conditionals. No orchestration code. The routing lives entirely in the tool descriptions.

When to use each tool

Use WebSearchTool for discovery: finding pages, open-ended research, answering questions from indexed public content.

Use Zenrows as the retrieval layer when you have a specific URL and need the live contents — protected targets, JavaScript-rendered pages, anything where freshness matters.

For structured field extraction instead of full-page Markdown, use Zenrows Extract. For high-volume or scheduled retrieval across many URLs, use Zenrows Batch.

The same pattern carries over to multi-agent setups: building a web research multi-agent system with AG2 and Zenrows. And to smolagents, under a @tool decorator: Zenrows smolagents guide.

Conclusion

You now have an agent with two retrieval paths. WebSearchTool handles discovery. Zenrows reads the full live page once the URL is known, including JavaScript-rendered and anti-bot-defended targets. The model routes between them from the tool descriptions alone.

What's next

• Zenrows Batch for high-volume or scheduled retrieval
• Zenrows MCP server if you prefer ready-made tools over a custom wrapper
• OpenAI Agents SDK integration guide for the full reference