DEV Community: Decodo

How To Build an AI Agent That Browses the Web?

Decodo — Tue, 25 Aug 2026 13:17:19 +0000

Your model provider probably includes a web search tool. It's one line of code, and for a stable fact, it works. It goes wrong on a page that changes or blocks bots, and it rarely warns you. This guide builds the other kind of agent: one tool that fetches a page through Decodo's Web Scraping API, and a short loop that runs the tool.

TL;DR

Use a fetch tool when the answer changes faster than an index can follow, and leave stable facts to search.
Define one tool, fetch_page(url), that calls Decodo's Web Scraping API with markdown: true on the premium pool, and returns truncated text instead of HTML.
Send the question and the tool definition together, then check whether the response contains a function call rather than an answer.
Run fetch_page when the model asks, append the result to the conversation, and repeat until the model answers or the turns run out.

Do you need a fetch tool?

One question for each outcome, run both ways:

Question	Built-in web_search result	Live fetch result	Result
Price of Sharp Objects on books.toscrape.com	£47.82	£47.82	Both correct
Latest Python version on python.org	3.14.6, 4 runs out of 4	3.14.7	Search stale
Latest Node.js LTS on nodejs.org	24.18.0 or 24.18.1 across 3 runs	24.19.0	Search stale
Ranked restaurants on a page that 403s a plain request	1 real, 2 hallucinated, missed the top entry	the correct top 6	Search hallucinated

For a stable fact, a one-line search tool answers correctly, and a fetch loop is wasted work. In the rest, search answered incorrectly without warning, twice from a stale index and once from invented entries.

A plain requests.get() handles the first 3 rows at low volume. The API adds the proxy rotation that the last row needs.

What you'll need

Python 3.11 or newer. The libraries still run on 3.10, but 3.10 loses support in October 2026.
A Decodo account with Web Scraping API credentials. The free plan is enough to follow along.
An OpenAI API key from the platform dashboard. The script sets the model in one constant, so you can use another OpenAI model with function calling.

Install both libraries:

pip install requests openai

Activate a plan, and the Playground tab shows your Basic authentication token:

The script reads DECODO_TOKEN directly, and OpenAI() reads OPENAI_API_KEY automatically, so set both first:

export DECODO_TOKEN="your_decodo_token"
export OPENAI_API_KEY="your_openai_key"

Run these in the same terminal session you'll start the script from. They don't carry over once you close that session.

Step 1: Define the fetch tool

The agent gets one tool. The tool takes a URL, fetches the page through the API, and returns text the model can read.

Three request-body fields matter here. target: universal is the generic template for any URL. markdown: true converts the HTML before it reaches you, and the parameters reference recommends Markdown for LLM input. proxy_pool has to be premium for the conversion.

Here's the tool:

import os
import requests

API_URL = "https://scraper-api.decodo.com/v2/scrape"
DECODO_TOKEN = os.environ["DECODO_TOKEN"]
MAX_CHARS = 8000

def fetch_page(url):
    response = requests.post(
        API_URL,
        json={
            "target": "universal",
            "url": url,
            "proxy_pool": "premium",
            "markdown": True,
        },
        headers={"Authorization": f"Basic {DECODO_TOKEN}"},
        timeout=120,
    )
    response.raise_for_status()

    data = response.json()
    if "results" not in data:
        return f"Fetch failed: {data.get('message')}"

    return data["results"][0]["content"][:MAX_CHARS]

Markdown alone didn't shrink python.org's downloads page enough. In one fetch, the HTML measured 49,413 tokens on OpenAI's o200k_base tokenizer. The Markdown measured 22,542, roughly half. MAX_CHARS cuts that remainder by another 89%, and the first 8,000 characters measure 2,464 tokens.

The results check handles one specific failure. When the API accepts the request but can't scrape the target, it still returns HTTP 200, with a message and no results key, so raise_for_status() passes. Indexing into results then raises KeyError, which nothing catches, ending the run.

Returning the message as a string lets the model read it and try a different URL. A bad DECODO_TOKEN still raises, because the model can't fix it.

Step 2. Set up the agent loop

The model sees only the tools you put in the request. You describe the tool once, in the format the Responses API expects, then send it with every request:

import json
from openai import OpenAI

client = OpenAI()
MODEL = "gpt-5.4-mini"
MAX_TURNS = 5

TOOLS = [
    {
        "type": "function",
        "name": "fetch_page",
        "description": (
            "Fetch a web page and return its text as Markdown. "
            "Use this whenever the answer depends on what a page says now."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "url": {"type": "string", "description": "The full URL to fetch."}
            },
            "required": ["url"],
            "additionalProperties": False,
        },
    }
]

The tool's description tells the model when to call it. The parameters field is a JSON schema for the arguments, and the model fills them from the conversation. The name field comes back as call.name.

Nothing in TOOLS runs fetch_page. The Responses API returns the name and the arguments, and your code runs the tool:

def run(question):
    messages = [{"role": "user", "content": question}]

    for _ in range(MAX_TURNS):
        response = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            tools=TOOLS
        )
        message = response.choices[0].message
        messages.append(message)

        if not message.tool_calls:
            return message.content

        for call in message.tool_calls:
            args = json.loads(call.function.arguments)
            print(f"[tool call] {call.function.name}({args})")

            if call.function.name == "fetch_page":
                result = fetch_page(args["url"])
            else:
                result = f"No tool named {call.function.name}."

            print(f"[tool result] {len(result)} characters of Markdown")
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "name": call.function.name,
                "content": result,
            })

    return f"Stopped at the {MAX_TURNS}-turn limit without a final answer."

MAX_TURNS stops the loop after 5 turns. Using the tool costs at least 2 turns, one to ask for the page and one to answer. Count them before running a long list of questions.

Each result carries call.call_id, matching the result to its own call. call.arguments is a JSON string, not a dict.

The else branch matters because a model can ask for a tool you declared and never implemented. With search_web declared and no branch for it, the model called the tool, read the fallback string, then answered that it couldn't use the tool. Raising would have ended the run.

Step 3. Run a real query against a live page

Pick a question whose answer changes faster than anything can index it. Omit the URL. The Hacker News front page reorders through the day, so training data and search indexes lag behind it. The entry point takes the question from the command line:

import sys

if __name__ == "__main__":
    default_question = "What is the top story on Hacker News right now?"
    question = " ".join(sys.argv[1:]) or default_question
    print(f"[question] {question}")
    print(f"[answer] {run(question)}")

With no argument, 1 run printed:

[question] What is the top story on Hacker News right now?
[tool call] fetch_page({'url': 'https://news.ycombinator.com/'})
[tool result] 8000 characters of Markdown
[answer] The current top story on Hacker News is **“My server is a phone now”** with **169 points** and **71 comments**.

The question named no URL, so the model picked where to look. The title came from this line of the Markdown:

[My server is a phone now](https://seg6.space/posts/phone-server/) ([seg6.space](from?site=seg6.space))

When asked the same question with the built-in search tool, the model named 2 different stories across 3 runs, and neither was on the page. Twice, the model said it couldn't see the live front page, then answered.

The full script

Save the script as agent.py, then run python agent.py:

import json
import os
import sys
import requests
from openai import OpenAI

API_URL = "https://scraper-api.decodo.com/v2/scrape"
DECODO_TOKEN = os.environ["DECODO_TOKEN"]
MAX_CHARS = 8000
MAX_TURNS = 5

client = OpenAI()
MODEL = "gpt-5.4-mini"

def fetch_page(url):
    response = requests.post(
        API_URL,
        json={
            "target": "universal",
            "url": url,
            "proxy_pool": "premium",
            "markdown": True,
        },
        headers={"Authorization": f"Basic {DECODO_TOKEN}"},
        timeout=120,
    )
    response.raise_for_status()

    data = response.json()
    if "results" not in data:
        return f"Fetch failed: {data.get('message')}"

    return data["results"][0]["content"][:MAX_CHARS]

TOOLS = [
    {
        "type": "function",
        "name": "fetch_page",
        "description": (
            "Fetch a web page and return its text as Markdown. "
            "Use this whenever the answer depends on what a page says now."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "url": {"type": "string", "description": "The full URL to fetch."}
            },
            "required": ["url"],
            "additionalProperties": False,
        },
    }
]

def run(question):
    messages = [{"role": "user", "content": question}]

    for _ in range(MAX_TURNS):
        response = client.responses.create(model=MODEL, input=messages, tools=TOOLS)
        messages += response.output
        calls = [item for item in response.output if item.type == "function_call"]

        if not calls:
            return response.output_text

        for call in calls:
            args = json.loads(call.arguments)
            print(f"[tool call] {call.name}({args})")

            if call.name == "fetch_page":
                result = fetch_page(args["url"])
            else:
                result = f"No tool named {call.name}."

            print(f"[tool result] {len(result)} characters of Markdown")
            messages.append(
                {
                    "type": "function_call_output",
                    "call_id": call.call_id,
                    "output": result,
                }
            )

    return f"Stopped at the {MAX_TURNS}-turn limit without a final answer."

if __name__ == "__main__":
    default_question = "What is the top story on Hacker News right now?"
    question = " ".join(sys.argv[1:]) or default_question
    print(f"[question] {question}")
    print(f"[answer] {run(question)}")

Before running the script against any site, check the target's robots.txt and terms of service. urllib.robotparser reads robots.txt. The terms of service are a separate check. Copyright and privacy law can apply.

Common pitfalls

Sending raw HTML. The conversation carries every earlier tool result, so one large page is re-sent on every later turn. Misspelling markdown sends raw HTML, because the API validates values but not key names.
Truncating above the answer. MAX_CHARS cuts blindly at 8,000 characters, and one fetch kept 23 of the 30 front-page stories. Raise the cap when the fact you want sits lower.
Assuming the page is server-rendered. JavaScript rendering is important; that's why headless stays off by default. Without it, a client-side rendered page comes back nearly empty, and the call still succeeds. quotes.toscrape.com/js/ returned 172 characters until headless was set to html. The [tool result] line shows the count.
Trusting the URL the model returns. That widens the attack surface for indirect prompt injection. The fetched page enters the model's context, so its text can influence the next fetch and the answer you get. An allowlist in the fetch_page branch limits where the agent goes, not what it concludes.
Leaving the loop unbounded. The loop exits only on a response with no function call, and a model that keeps calling never sends one. With MAX_TURNS set to 1, the run returns Stopped at the 1-turn limit without a final answer.

These all return success, except the unbounded loop. Watch the character count, not the status code.

Where to take it from here

Add a search tool for stable facts. Decodo's Fast Search API is one option, with its own token.

A second tool is another entry in TOOLS and another branch next to fetch_page. The loop already runs tools as many times as the model asks, one at a time, within the MAX_TURNS limit.

To skip writing the loop, use Decodo's MCP server, which exposes a fetch tool your client can load directly.

Conclusion

Search wins on stable facts. A live fetch wins on pages that change or block bots. Neither reliably tells you when it's wrong. The loop is the easy part. The rest is knowing when to trust the answer.

Build Your First Web Scraper With the Decodo API in 15 Minutes

Decodo — Fri, 21 Aug 2026 10:49:19 +0000

The Decodo Web Scraping API fetches a page and returns the HTML. You send one POST request with the target URL, and the API handles proxy rotation, unblocking, and retries. In about 15 minutes, you'll have a Python script that calls the API and prints the titles and prices from a live page. No cost included, as the free plan covers it.

TL;DR

Copy the Basic authentication token from the Playground tab under Web Scraping API in the Decodo dashboard.
Send a JSON body with your target URL to https://scraper-api.decodo.com/v2/scrape using HTTP Basic auth, and set proxy_pool yourself – standard is cheaper and enough for static pages, while premium is for protected sites.
Read the page from results[0]["content"], which is a string of HTML by default, not a parsed object.
A failed scrape still returns HTTP 200, with "status": "failed" and no results key, so check for results instead of the HTTP status code.

What you'll need

Python 3.8 or newer
A Decodo account with Web Scraping API credentials
The Requests and Beautiful Soup libraries

Install both:

pip install requests beautifulsoup4

*The prices and measurements here are our own, taken in August 2026. The outputs are from real runs, so you can compare them with yours.

Step 1. Get your API credentials

Sign up and pick the free Web Scraping API plan, which gives you $1 of credit. The standard pool costs $0.50 per 1K, so $1 covers 2,000 requests, far more than this build needs.

The same $1 buys 2K standard requests or 667 with premium and JavaScript.

The dashboard tracks what's left of that credit. Open Web Scraping API in the sidebar. Its Pricing tab has the current rates and a Start for free button. Once you start, head to the Playground and find your Basic authentication token. The gear icon beside the token field opens Authentication settings, where you also find your username and password. You only need the token, so simply copy it by clicking on it.

Copy the token into TOKEN.

The token isn't a separate API key. It's the base64 encoding of your username and password, the value that HTTP Basic authentication sends anyway. If you have that pair, auth=(USERNAME, PASSWORD) in Requests builds the same header.

A placeholder constant is fine for a first run. Move the token into an environment variable before you share or commit the script.

Step 2. Make your first request

POST to https://scraper-api.decodo.com/v2/scrape. The body needs a URL, plus 2 more fields worth setting. target picks the template, and proxy_pool picks the IP pool. Use universal for a plain page fetch.

Here is the request on its own:

import requests

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = "YOUR_DECODO_TOKEN"

payload = {
    "target": "universal",
    "url": "https://books.toscrape.com/",
    "proxy_pool": "standard"
}

response = requests.post(
    API_URL,
    json=payload,
    headers={"Authorization": f"Basic {TOKEN}"},
    timeout=60
)
response.raise_for_status()

result = response.json()["results"][0]
print(result["status_code"])
print(result["content"][:120])

If you don't set proxy_pool, the API routes the request through the premium pool. On the free plan, premium costs 2x the standard rate, which cuts that credit from 2K requests to 1K. Standard is enough for a static sandbox page, and premium is for pages with bot protection. The parameters reference documents the rest of the fields.

The Playground pre-selects Premium. That's the pool you get when you don't set proxy_pool in the payload.

A successful call returns a results list. Each entry contains content and status_code, plus other fields. So a single call gives you two status codes. response.status_code comes from the API, and result["status_code"] comes from the target site.

The request makes two hops, and each one has its own status code. A 200 from the API can carry a 404 from the target.

The target's status code comes first:

200
<!DOCTYPE html>
<!--[if lt IE 7]>      <html lang="en-us" class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]

A failed scrape doesn't return a results list. The API still returns HTTP 200, but the body contains "status": "failed", an error code such as 613, and a human-readable message. So raise_for_status() passes, and Python raises a KeyError when you index into results. The full script guards against it.

Step 3. Parse the response and pull out data

content is a string of HTML by default, not a dictionary. If you call .get() on it, Python raises an AttributeError. Pass the string to Beautiful Soup, and you can search it with CSS selectors:

from bs4 import BeautifulSoup

soup = BeautifulSoup(result["content"], "html.parser")

print("Page title:", soup.title.get_text(strip=True))
print()

for book in soup.select("article.product_pod")[:5]:
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").get_text(strip=True)
    print(f"{price}  {title}")

The site truncates the link text on longer titles, and the markup shows "A Light in the …" rather than the full name. So the code takes the title from the title attribute instead. These selectors match books.toscrape.com as of August 2026.

That prints:

Page title: All products | Books to Scrape - Sandbox

£51.77  A Light in the Attic
£53.74  Tipping the Velvet
£50.10  Soumission
£47.82  Sharp Objects
£54.23  Sapiens: A Brief History of Humankind

The full script

Save this as scraper.py and paste your token into TOKEN:

import requests
from bs4 import BeautifulSoup

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = "YOUR_DECODO_TOKEN"

payload = {
    "target": "universal",
    "url": "https://books.toscrape.com/",
    "proxy_pool": "standard"
}

response = requests.post(
    API_URL,
    json=payload,
    headers={"Authorization": f"Basic {TOKEN}"},
    timeout=60
)
response.raise_for_status()

data = response.json()
if "results" not in data:
    raise SystemExit(f"Scrape failed: {data.get('message')}")

result = data["results"][0]
if result["status_code"] != 200:
    raise SystemExit(f"Target returned {result['status_code']}")

soup = BeautifulSoup(result["content"], "html.parser")

print("Page title:", soup.title.get_text(strip=True))
print()

for book in soup.select("article.product_pod")[:5]:
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").get_text(strip=True)
    print(f"{price}  {title}")

That status check is deliberately strict. Decodo counts 4xx as a successful retrieval when content comes back, so loosen it if you scrape pages that return 404 or 403 with a usable body.

Move from the sandbox to a commercial site

The sandbox page accepts all requests, so it doesn't show what the API is capable of. A direct request to a site like Tripadvisor listing page will get you a 403. Through the API, standard doesn't return the page, but premium does, so the payload below uses it.

Check what a live site allows before you point a scraper at it:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://www.tripadvisor.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://www.tripadvisor.com/Restaurants-g187147-Paris.html"))

That returns True for this path. robots.txt isn't the terms of service. Read those too before you scrape a commercial site.

The full script already handles the request. Three edits point it at Tripadvisor. First, add import re alongside the other imports at the top. Then change the payload:

payload = {
    "target": "universal",
    "url": "https://www.tripadvisor.com/Restaurants-g187147-Paris.html",
    "proxy_pool": "premium"
}

Next, replace everything from the soup = BeautifulSoup(...) line to the end of the file:

soup = BeautifulSoup(result["content"], "html.parser")
names = [a.get_text(strip=True) for a in soup.select("a[href*='/Restaurant_Review']")]
ranked = [n for n in names if re.match(r"^\d+\.\s", n)]

for name in ranked[:5]:
    print(name)

The requests.post call and both raise SystemExit guards in between stay exactly as they are.

One run prints:

1. Au Bourguignon Du Marais
2. Bistro L'Olivier
3. Le Poulbot
4. L'Oiseau Blanc Restaurant
5. Chez Pippo

The class names on the Tripadvisor page are non-semantic build hashes rather than named CSS classes. One of them is .biGQs, and a hash like that changes on the next release. The href pattern is more stable, and the numeric prefix separates ranked entries from the review snippets that link to the same page. Our notes on picking selectors that survive a redesign cover the wider trade-offs.

Premium doesn't return this page every time, so re-run before you start debugging. Rankings and markup both change, so treat the output as a sample rather than a fixture. This selector matches Tripadvisor.

Common pitfalls

A bad token stops the script at raise_for_status() with 401 Client Error: Unauthorized, which doesn't say why. Add print(response.text) above that line to see the reason:

401 Incorrect username or password – the token decodes correctly but doesn't match an account. A partial copy causes this.
401 Username invalid. – the token contains a character outside the base64 alphabet, such as a curly quote from a formatted document.
400 Authorization header is required – no credential reaches the API at all.

The API checks the token before the body, so fix auth errors first.

The rest look different:

ValueError: Invalid header value – the token includes a newline. Requests raises this before sending, so there's no response to inspect.
ModuleNotFoundError: No module named 'bs4' – install beautifulsoup4 but import bs4.
An empty list from select() – the selector doesn't match the markup. Check the class names on the live page.
KeyError: 'results' – the scrape itself fails, and the reason is in message.
429 Client Error: Too Many Requests – too much concurrency. The body reads Requests for this page type are temporarily limited.

How to extend the script

Reuse a single requests.Session() when you loop over a list of URLs. It cuts the time for 5 sequential calls from 11.2 to 7.3 seconds. On the free plan, the documented limit is 10 requests per second. When the list gets long, move to the async /v2/task/batch endpoint.

For a target that renders client-side, add "headless": "html" so the API runs JavaScript first. On the free plan, that takes standard from $0.50 to $0.75 per 1K. If the output goes to a model, set "markdown": true to get Markdown instead of HTML. On the sandbox page, that setting cut 51K characters to 10K. It needs the premium pool though, at $1.00 per 1K.

Some targets have a dedicated template rather than the generic one. If you pass an Amazon URL to universal, the API returns The provided 'url' is not supported for the selected target. Scraping Templates lists which sites have one.

There's enough credit left to point this at something you need. If something here doesn't work for your target, say so in the comments.

Conclusion

The request never changes: one POST, one token, one URL. The target decides everything else. The sandbox took the standard pool and a class-name selector. Tripadvisor needed premium, an href pattern, and a re-run when the page didn't come back.

Two habits carry over to whatever you scrape next. Check for results rather than the HTTP status, because a failed scrape still returns 200. And prefer selectors that survive a redesign, because the class names on a commercial site are build hashes.