Your model provider probably includes a web search tool. It's one line of code, and for a stable fact, it works. It goes wrong on a page that changes or blocks bots, and it rarely warns you. This guide builds the other kind of agent: one tool that fetches a page through Decodo's Web Scraping API, and a short loop that runs the tool.
TL;DR
- Use a fetch tool when the answer changes faster than an index can follow, and leave stable facts to search.
- Define one tool, fetch_page(url), that calls Decodo's Web Scraping API with markdown: true on the premium pool, and returns truncated text instead of HTML.
- Send the question and the tool definition together, then check whether the response contains a function call rather than an answer.
- Run fetch_page when the model asks, append the result to the conversation, and repeat until the model answers or the turns run out.
Do you need a fetch tool?
One question for each outcome, run both ways:
| Question | Built-in web_search result | Live fetch result | Result |
|---|---|---|---|
| Price of Sharp Objects on books.toscrape.com | £47.82 | £47.82 | Both correct |
| Latest Python version on python.org | 3.14.6, 4 runs out of 4 | 3.14.7 | Search stale |
| Latest Node.js LTS on nodejs.org | 24.18.0 or 24.18.1 across 3 runs | 24.19.0 | Search stale |
| Ranked restaurants on a page that 403s a plain request | 1 real, 2 hallucinated, missed the top entry | the correct top 6 | Search hallucinated |
For a stable fact, a one-line search tool answers correctly, and a fetch loop is wasted work. In the rest, search answered incorrectly without warning, twice from a stale index and once from invented entries.
A plain requests.get() handles the first 3 rows at low volume. The API adds the proxy rotation that the last row needs.
What you'll need
- Python 3.11 or newer. The libraries still run on 3.10, but 3.10 loses support in October 2026.
- A Decodo account with Web Scraping API credentials. The free plan is enough to follow along.
- An OpenAI API key from the platform dashboard. The script sets the model in one constant, so you can use another OpenAI model with function calling.
Install both libraries:
pip install requests openai
Activate a plan, and the Playground tab shows your Basic authentication token:
The script reads DECODO_TOKEN directly, and OpenAI() reads OPENAI_API_KEY automatically, so set both first:
export DECODO_TOKEN="your_decodo_token"
export OPENAI_API_KEY="your_openai_key"
Run these in the same terminal session you'll start the script from. They don't carry over once you close that session.
Step 1: Define the fetch tool
The agent gets one tool. The tool takes a URL, fetches the page through the API, and returns text the model can read.
Three request-body fields matter here. target: universal is the generic template for any URL. markdown: true converts the HTML before it reaches you, and the parameters reference recommends Markdown for LLM input. proxy_pool has to be premium for the conversion.
Here's the tool:
import os
import requests
API_URL = "https://scraper-api.decodo.com/v2/scrape"
DECODO_TOKEN = os.environ["DECODO_TOKEN"]
MAX_CHARS = 8000
def fetch_page(url):
response = requests.post(
API_URL,
json={
"target": "universal",
"url": url,
"proxy_pool": "premium",
"markdown": True,
},
headers={"Authorization": f"Basic {DECODO_TOKEN}"},
timeout=120,
)
response.raise_for_status()
data = response.json()
if "results" not in data:
return f"Fetch failed: {data.get('message')}"
return data["results"][0]["content"][:MAX_CHARS]
Markdown alone didn't shrink python.org's downloads page enough. In one fetch, the HTML measured 49,413 tokens on OpenAI's o200k_base tokenizer. The Markdown measured 22,542, roughly half. MAX_CHARS cuts that remainder by another 89%, and the first 8,000 characters measure 2,464 tokens.
The results check handles one specific failure. When the API accepts the request but can't scrape the target, it still returns HTTP 200, with a message and no results key, so raise_for_status() passes. Indexing into results then raises KeyError, which nothing catches, ending the run.
Returning the message as a string lets the model read it and try a different URL. A bad DECODO_TOKEN still raises, because the model can't fix it.
Step 2. Set up the agent loop
The model sees only the tools you put in the request. You describe the tool once, in the format the Responses API expects, then send it with every request:
import json
from openai import OpenAI
client = OpenAI()
MODEL = "gpt-5.4-mini"
MAX_TURNS = 5
TOOLS = [
{
"type": "function",
"name": "fetch_page",
"description": (
"Fetch a web page and return its text as Markdown. "
"Use this whenever the answer depends on what a page says now."
),
"parameters": {
"type": "object",
"properties": {
"url": {"type": "string", "description": "The full URL to fetch."}
},
"required": ["url"],
"additionalProperties": False,
},
}
]
The tool's description tells the model when to call it. The parameters field is a JSON schema for the arguments, and the model fills them from the conversation. The name field comes back as call.name.
Nothing in TOOLS runs fetch_page. The Responses API returns the name and the arguments, and your code runs the tool:
def run(question):
messages = [{"role": "user", "content": question}]
for _ in range(MAX_TURNS):
response = client.chat.completions.create(
model=MODEL,
messages=messages,
tools=TOOLS
)
message = response.choices[0].message
messages.append(message)
if not message.tool_calls:
return message.content
for call in message.tool_calls:
args = json.loads(call.function.arguments)
print(f"[tool call] {call.function.name}({args})")
if call.function.name == "fetch_page":
result = fetch_page(args["url"])
else:
result = f"No tool named {call.function.name}."
print(f"[tool result] {len(result)} characters of Markdown")
messages.append({
"role": "tool",
"tool_call_id": call.id,
"name": call.function.name,
"content": result,
})
return f"Stopped at the {MAX_TURNS}-turn limit without a final answer."
MAX_TURNS stops the loop after 5 turns. Using the tool costs at least 2 turns, one to ask for the page and one to answer. Count them before running a long list of questions.
Each result carries call.call_id, matching the result to its own call. call.arguments is a JSON string, not a dict.
The else branch matters because a model can ask for a tool you declared and never implemented. With search_web declared and no branch for it, the model called the tool, read the fallback string, then answered that it couldn't use the tool. Raising would have ended the run.
Step 3. Run a real query against a live page
Pick a question whose answer changes faster than anything can index it. Omit the URL. The Hacker News front page reorders through the day, so training data and search indexes lag behind it. The entry point takes the question from the command line:
import sys
if __name__ == "__main__":
default_question = "What is the top story on Hacker News right now?"
question = " ".join(sys.argv[1:]) or default_question
print(f"[question] {question}")
print(f"[answer] {run(question)}")
With no argument, 1 run printed:
[question] What is the top story on Hacker News right now?
[tool call] fetch_page({'url': 'https://news.ycombinator.com/'})
[tool result] 8000 characters of Markdown
[answer] The current top story on Hacker News is **“My server is a phone now”** with **169 points** and **71 comments**.
The question named no URL, so the model picked where to look. The title came from this line of the Markdown:
[My server is a phone now](https://seg6.space/posts/phone-server/) ([seg6.space](from?site=seg6.space))
When asked the same question with the built-in search tool, the model named 2 different stories across 3 runs, and neither was on the page. Twice, the model said it couldn't see the live front page, then answered.
The full script
Save the script as agent.py, then run python agent.py:
import json
import os
import sys
import requests
from openai import OpenAI
API_URL = "https://scraper-api.decodo.com/v2/scrape"
DECODO_TOKEN = os.environ["DECODO_TOKEN"]
MAX_CHARS = 8000
MAX_TURNS = 5
client = OpenAI()
MODEL = "gpt-5.4-mini"
def fetch_page(url):
response = requests.post(
API_URL,
json={
"target": "universal",
"url": url,
"proxy_pool": "premium",
"markdown": True,
},
headers={"Authorization": f"Basic {DECODO_TOKEN}"},
timeout=120,
)
response.raise_for_status()
data = response.json()
if "results" not in data:
return f"Fetch failed: {data.get('message')}"
return data["results"][0]["content"][:MAX_CHARS]
TOOLS = [
{
"type": "function",
"name": "fetch_page",
"description": (
"Fetch a web page and return its text as Markdown. "
"Use this whenever the answer depends on what a page says now."
),
"parameters": {
"type": "object",
"properties": {
"url": {"type": "string", "description": "The full URL to fetch."}
},
"required": ["url"],
"additionalProperties": False,
},
}
]
def run(question):
messages = [{"role": "user", "content": question}]
for _ in range(MAX_TURNS):
response = client.responses.create(model=MODEL, input=messages, tools=TOOLS)
messages += response.output
calls = [item for item in response.output if item.type == "function_call"]
if not calls:
return response.output_text
for call in calls:
args = json.loads(call.arguments)
print(f"[tool call] {call.name}({args})")
if call.name == "fetch_page":
result = fetch_page(args["url"])
else:
result = f"No tool named {call.name}."
print(f"[tool result] {len(result)} characters of Markdown")
messages.append(
{
"type": "function_call_output",
"call_id": call.call_id,
"output": result,
}
)
return f"Stopped at the {MAX_TURNS}-turn limit without a final answer."
if __name__ == "__main__":
default_question = "What is the top story on Hacker News right now?"
question = " ".join(sys.argv[1:]) or default_question
print(f"[question] {question}")
print(f"[answer] {run(question)}")
Before running the script against any site, check the target's robots.txt and terms of service. urllib.robotparser reads robots.txt. The terms of service are a separate check. Copyright and privacy law can apply.
Common pitfalls
- Sending raw HTML. The conversation carries every earlier tool result, so one large page is re-sent on every later turn. Misspelling markdown sends raw HTML, because the API validates values but not key names.
- Truncating above the answer. MAX_CHARS cuts blindly at 8,000 characters, and one fetch kept 23 of the 30 front-page stories. Raise the cap when the fact you want sits lower.
- Assuming the page is server-rendered. JavaScript rendering is important; that's why headless stays off by default. Without it, a client-side rendered page comes back nearly empty, and the call still succeeds. quotes.toscrape.com/js/ returned 172 characters until headless was set to html. The [tool result] line shows the count.
- Trusting the URL the model returns. That widens the attack surface for indirect prompt injection. The fetched page enters the model's context, so its text can influence the next fetch and the answer you get. An allowlist in the fetch_page branch limits where the agent goes, not what it concludes.
- Leaving the loop unbounded. The loop exits only on a response with no function call, and a model that keeps calling never sends one. With MAX_TURNS set to 1, the run returns Stopped at the 1-turn limit without a final answer.
These all return success, except the unbounded loop. Watch the character count, not the status code.
Where to take it from here
Add a search tool for stable facts. Decodo's Fast Search API is one option, with its own token.
A second tool is another entry in TOOLS and another branch next to fetch_page. The loop already runs tools as many times as the model asks, one at a time, within the MAX_TURNS limit.
To skip writing the loop, use Decodo's MCP server, which exposes a fetch tool your client can load directly.
Conclusion
Search wins on stable facts. A live fetch wins on pages that change or block bots. Neither reliably tells you when it's wrong. The loop is the easy part. The rest is knowing when to trust the answer.
Top comments (0)