DEV Community: Decodo

Build a Live Competitor Content Tracker With Python

Decodo — Tue, 29 Sep 2026 06:52:43 +0000

A competitor content tracker is a scheduled job that fetches a list of competitor pages, compares each one against the copy it stored last time, and reports only the differences worth reading. Many teams build the naive version first. A cron entry curls the page, difflib compares the HTML, and your digest fills with changes nobody made. So the week a competitor cuts its price, that change looks like all the rest. This guide builds the version that stays quiet unless the change matters.

TL;DR

A scheduled Python job stores the extracted body of each page and stays quiet while that body is unchanged.
Decodo's web scraping solutions handle the fetch. They render JavaScript, can get past blocks that refuse a plain GET, and post each finished task ID to a callback URL when you supply one.
Claude reads the diff and nothing else, then returns a change category plus a one-line summary. The category lets the digest route a launch differently from a nav tweak.

What you'll need

Python 3.10 or newer, with requests, trafilatura, pydantic, and anthropic 1.8 or newer installed
A Decodo account with Web Scraping API access, with its Basic header value in DECODO_TOKEN. This job spends 1 scrape per tracked page per check.
An Anthropic API key in ANTHROPIC_API_KEY, on an account with credit
A Slack incoming webhook URL in SLACK_WEBHOOK, or any URL that answers a POST with 200 while you test
Any scheduler you already run, such as cron, a systemd timer, or GitHub Actions

Why checking competitors by hand stops working

Checking by hand is website change monitoring done with a bookmark folder and good intentions. It survives 2 competitors and falls apart as the list grows, because every check depends on a human remembering to look. A launch or a price cut gets spotted a week late, by someone who wasn't looking for it.

Live here means scheduled and near real time. For marketing pages, hourly is the tightest interval worth running, and the alert arrives when the scrape finishes. Nothing fires at the instant a competitor's CMS saves a page, and marketing content doesn't need it to fire.

Rule out the cheap paths first. If you only need to know whether a competitor published something, their /feed.xml or sitemap.xml with lastmod answers that in one small request. A conditional GET settles the unchanged case for the cost of a 304, anywhere If-None-Match is honored. This tracker is for pages with no feed, where the change is an edit rather than a new URL.

changedetection.io is the incumbent there. It self-hosts under Docker, and it ships change summaries of its own. Deploy it if you want a monitoring product. Build this if you want the signal inside a pipeline you already run, routed by rules you own.

Getting a change signal you can trust

Before the diff is worth reading, you have to handle 2 failure modes.

The fetch fails without notice. I requested 4 pages with a browser User-Agent and no proxy, then requested the same 4 through Decodo:

Target	Plain GET	Through Decodo
`g2.com/products/slack/reviews`	`403`	`403`
`crunchbase.com/organization/stripe`	`403`	`200`
`similarweb.com/website/stripe.com`	`connect timeout`	`200`
`vercel.com/pricing`	`200`	`200`

Only 1 of the 4 answered a plain request. Decodo recovered 2 of the 3 that refused, but G2 refused both attempts, which is where the fetch layer stops. The failure that hurts a tracker is the quiet one. A tracker that stores an empty body reports a huge change on the next good pull, and that false positive has a real fetch failure underneath it.

The diff fires on nothing. I pulled each of 5 pages twice as raw HTML and twice as extracted Markdown, seconds apart, which is not long enough for anyone to have edited them. Raw HTML changed on 4 of the 5, by 2 lines at the low end and 10 at the high end, while all 5 extracted bodies came back unchanged. The noisiest page, and the 1 in 5 that never moved:

Page	Changed lines, raw HTML	Changed lines, extracted
`vercel.com/blog`	`10`	`0`
`posthog.com/blog`	`0`	`0`

None of that churn came from the copy. It was per-request identity every time, mostly tracing IDs and A/B assignment tokens regenerated on each render. Every extracted body ran 138 to 936 lines, so the zeros mean stable extraction and not an empty response. If you test on the 1 page in 5 that happens to be clean, raw HTML diffing looks fine until you add the other 4.

Which extractor you use decides how much of the remaining noise survives. I put one pair of captures of Linear's pricing page through both Decodo's Markdown conversion and through trafilatura, which drops navigation, footers, and sidebars:

Extractor	Body	Changed lines	Of those, nav links
`markdown: true`	`4,192 chars`	`28`	`7`
`trafilatura`	`1,943 chars`	`19`	`0`

A quarter of that first diff was navigation. Use markdown: true when you want no extra dependency, and a main-content extractor when the diff is what you act on. The 2026 WCXB benchmark scores the current field.

Decodo's web scraping solutions do the fetching half no extractor can do. With headless: "html", JavaScript renders before the markup reaches you, so a page that would arrive as an empty shell arrives as a real body. The geo parameter pins the request to a country, which matters when a competitor prices by region and when a rotating proxy pool would otherwise move the page under you. markdown: true is there when you would rather not add trafilatura.

A POST to /v3/task returns a task ID immediately. With callback_url set, Decodo posts to your endpoint at the moment the scrape finishes:

curl -X POST https://scraper-api.decodo.com/v3/task \
  -H "Authorization: $DECODO_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://competitor.example/pricing","markdown":true,
       "headless":"html","callback_url":"https://you.example/hook",
       "passthrough":"pricing"}'

The callback carries the task ID, not the page body, so you still fetch the body from /v3/task/{task_id}/results, where it stays retrievable for 24 hours. passthrough comes back untouched and tells your handler which page landed. The script below polls instead, because polling needs no public URL.

You schedule the repeat runs yourself. Point cron, a systemd timer, or a GitHub Actions schedule at the POST that submits the task. Decodo's walkthrough on how to schedule scraping tasks covers the cron and GitHub Actions routes.

From diff to signal

Getting the page and reading the diff sit on opposite sides of a seam, and 3 outcomes never leave the fetch side:

The diff is clean enough to act on and still too long to read. To get a real change without waiting a month, I seeded the snapshot store from an archived copy of Linear's pricing page and ran the tracker against the live one. That run produced 19 changed lines, and the ones that matter sit among the rest. A 10-line excerpt:

-- Linear Agent (beta)
+- Linear Agent
-- Code Intelligence (beta)
+- Code Intelligence
-Trusted by more than 33,000 companies
+Trusted by more than 40,000 companies
+Coding sessions**
+Loops**
-Data warehouse sync
+** Requires AI credits

Only the footnote that puts 2 plan features behind AI credits changes what a sales team says on a call, and it lands after 18 lines that mostly shuffle beta labels. Claude sorts through those lines, reading the diff and never the page.

The 3 blocks below are one file, tracker.py, in that order. Half of it is the 2 API calls. Decodo returns the rendered page, Claude turns a diff into a fixed shape, and nothing else touches a network:

import difflib, hashlib, os, re, sqlite3, time, requests, trafilatura
from typing import Literal
import anthropic
from pydantic import BaseModel

DECODO   = "https://scraper-api.decodo.com/v3/task"
HEADERS  = {"Authorization": os.environ["DECODO_TOKEN"],   # "Basic <base64 user:pass>"
            "Content-Type": "application/json"}
VOLATILE = re.compile(r"^\s*[\d,.]+\s*$")          # lines that are only a live counter
PRICING_PAGES = {"https://competitor.example/pricing"}   # list it in the run loop too
ACTION   = {"new_content": "notify", "pricing_change": "page",
            "positioning": "notify", "cosmetic": "drop"}

class Change(BaseModel):
    category: Literal["new_content", "pricing_change", "positioning", "cosmetic"]
    summary: str
    worth_paging: bool

PROMPT = """Below is a unified diff of one competitor page's extracted body,
previous against current. Name the single most important change as
new_content (a post or page that did not exist), pricing_change (a number,
tier, limit, or plan name moved), positioning (headline or product
description rewritten), or cosmetic (nav, link order, asset paths, counters).
Summarize it in 25 words or fewer. Set worth_paging only for a pricing_change
or a named product launch.

<diff>
{diff}
</diff>"""

def fetch(url):
    """Submit an async task, then poll for the rendered HTML."""
    body = {"url": url, "headless": "html", "proxy_pool": "premium"}
    r = requests.post(DECODO, json=body, headers=HEADERS, timeout=60)
    r.raise_for_status()                       # 401 means DECODO_TOKEN lost its "Basic " prefix
    task = r.json()["id"]
    for _ in range(60):
        r = requests.get(f"{DECODO}/{task}/results", headers=HEADERS, timeout=60)
        if r.status_code == 200:                   # 204 means still running
            res = r.json()["results"][0]
            return res["status_code"], res["content"]
        time.sleep(2)
    return None, ""

def body_of(html):
    """Main content only. Nav, footer and sidebars never reach the diff."""
    text = trafilatura.extract(html, include_links=True, include_tables=True) or ""
    return [l.strip() for l in text.splitlines()
            if l.strip() and not VOLATILE.match(l)]

def classify(diff):
    r = anthropic.Anthropic().messages.parse(
        model="claude-sonnet-5", max_tokens=4096,
        output_config={"effort": "low"},
        messages=[{"role": "user", "content": PROMPT.format(diff=diff[:12000])}],
        output_format=Change,
    )
    return None if r.stop_reason == "refusal" else r.parsed_output

That status_code == 200 check is load-bearing, and I added it only after a crash. A pending task answers /v3/task/{id}/results with 204 No Content and an empty body, and requests counts any 2xx as success, so a loop written on r.ok calls .json() on nothing.

Volume and stakes decide the model, and Anthropic's models overview lists the tradeoffs.

The other half is storage and a branch. It stops a failed fetch, a first sighting, and an unchanged page from ever looking like a change:

def check(conn, url, slack):
    status, html = fetch(url)
    lines = body_of(html) if status == 200 else []
    body  = "\n".join(lines)
    prev  = conn.execute("SELECT body, sha FROM pages WHERE url=?", (url,)).fetchone()

    # A body that vanished, or halved, is a fetch problem and not an edit.
    if not lines or (prev and len(body) < len(prev[0]) // 2):
        conn.execute("INSERT INTO failures VALUES(?,1) ON CONFLICT(url) "
                     "DO UPDATE SET n=n+1", (url,)); conn.commit()
        return "fetch_failed"                       # no snapshot written, no diff

    sha  = hashlib.sha256(body.encode()).hexdigest()
    keep = ("INSERT INTO pages VALUES(?,?,?) ON CONFLICT(url) "
            "DO UPDATE SET body=?, sha=?", (url, body, sha, body, sha))

    if prev is None:
        conn.execute(*keep); conn.commit()
        return "first_seen"                         # seed the store, stay silent
    if prev[1] == sha:
        return "unchanged"                          # stay silent

    diff = "\n".join(difflib.unified_diff(
        prev[0].splitlines(), lines, "previous", "current", lineterm="", n=1))
    change = classify(diff)
    if change is None:
        return "dropped"
    if url in PRICING_PAGES and repriced(prev[0], body):
        change.category, change.worth_paging = "pricing_change", True
    if ACTION[change.category] != "drop":
        sent = requests.post(slack, json={"text": f"*{change.category}* on <{url}>\n"
                                                  f"{change.summary}"}, timeout=15)
        sent.raise_for_status()                     # a dead webhook must not look like delivery
    conn.execute(*keep); conn.commit()              # advance only once the change is delivered
    return change.category

Two runs against an unchanged page print first_seen and then unchanged, with nothing sent anywhere. A forced 403 and a nav-only shell both returned fetch_failed, and the stored snapshot survived both.

To reach the alert path without waiting for a competitor, edit the stored body in tracker.db and set that row's sha to any other value, then run the tracker once more. The comparison is hash against hash, so an edited body on its own changes nothing. What you edit becomes the previous side of the diff, so write the older price in and let the live fetch supply the new one.

A failed fetch and an unchanged page look identical from a distance. check returns fetch_failed before it reaches the snapshot, so a block never overwrites a good body and never counts as a change. The snapshot also advances only once the alert is delivered, so a dead webhook leaves the change to be found again next run instead of losing it. The half-length check is there because a nav-only shell extracts to a handful of characters and would otherwise read as a total rewrite.

Pointed at the seeded Linear diff, the classifier returned this:

category    : new_content
summary     : Linear Agent and Code Intelligence exit beta; new features 'Coding sessions' and 'Loops' added, requiring AI credits.
worth_paging: False

That is 19 diff lines reduced to 3 facts worth knowing. I ran the same classifier over 3 more archived-against-live diffs, and 1 is worth your attention.

A Supabase pricing diff carried 52 changed lines with a dollar figure on them, but the model called it positioning and declined to page. That reads like a miss, so I checked. All 33 prices on the page appear in both versions, and the compute table had been reformatted rather than repriced, so the model was right and a currency regex would have paged on nothing.

That is the limit of diffing text. Diff prose when you want to know what changed, and compare extracted structure when you need to know whether a specific number did:

class Tier(BaseModel):
    name: str
    price: str

class Tiers(BaseModel):
    tiers: list[Tier]

TIERS = ("List every named plan or compute tier on this pricing page with its price "
         "exactly as written. Omit anything that has no price.\n\n")

def repriced(previous, current):
    """Which named tiers changed price. Empty means no price moved."""
    def tiers_of(body):
        r = anthropic.Anthropic().messages.parse(
            model="claude-sonnet-5", max_tokens=8000,
            output_config={"effort": "low"},
            messages=[{"role": "user", "content": TIERS + body[:20000]}],
            output_format=Tiers,
        )
        return {t.name: t.price for t in r.parsed_output.tiers}
    a, b = tiers_of(previous), tiers_of(current)
    return {k: (a[k], b[k]) for k in a.keys() & b.keys() if a[k] != b[k]}

if __name__ == "__main__":
    conn = sqlite3.connect("tracker.db")
    conn.execute("CREATE TABLE IF NOT EXISTS pages(url TEXT PRIMARY KEY, body TEXT, sha TEXT)")
    conn.execute("CREATE TABLE IF NOT EXISTS failures(url TEXT PRIMARY KEY, n INT)")
    for url in ["https://competitor.example/pricing"]:
        print(url, "->", check(conn, url, os.environ["SLACK_WEBHOOK"]))

On that Supabase pair, repriced returns nothing, which is right. Edit one tier from $110 to $129 in the stored body and it returns {'Large': ('$110', '$129')}, so the empty result above is a finding and not a silent failure. The PRICING_PAGES branch runs it only where a number is the point, and a moved price then overrides whatever the prose diff concluded. It costs 2 extra model calls on a page that changed, and it turns the question you least want to get wrong into a comparison.

3 ways to get this wrong

Diffing raw HTML instead of extracted text. Extract first, normalize second, and hash what's left, so neither the tracing ids nor the counter lines reach the comparison.
Scheduling faster than the content moves. An hourly check spends 168 requests a week on a blog that publishes weekly, when a weekly check would have caught the same post. Match the interval to the page, so pricing and changelog pages earn hourly checks, a blog index daily, and an about page weekly.
Sending the whole page instead of the diff. Counted with count_tokens on one run, Linear's extracted body came to 1,031 input tokens against 590 for the diff, so sending the page cost 1.7 times as much to learn the same thing. That 1.7x grows with the page, because a blog index keeps getting longer while the number of new posts per check stays flat. The tier check is the exception, and it does send the whole body, because a diff cannot show you the price of a tier that did not move.

Final thoughts

Getting the page and reading the diff are 2 different problems. Decodo's web scraping solutions own the first, and Claude owns the second without ever seeing a page it doesn't need. Merging them puts a parser bug in the same channel as a repositioned headline. Tracking several competitors is a loop over a URL list, and logging each change's category alongside the snapshot turns the store into a record of where a competitor puts its effort. Point the tracker at the one page you keep meaning to check, seed the store on the first run, and let the second run tell you whether anything moved.

How to Build an MCP Server for AI-powered Web Scraping

Decodo — Mon, 28 Sep 2026 07:28:18 +0000

An MCP server makes a tool available over a standard protocol, so MCP-compatible clients can call it without a custom integration. For scraping, that tool takes a URL and returns content a model can read. The quickest version wraps requests.get() in a tool decorator and works on static pages. Client-rendered pages are harder: the response can arrive as HTTP 200 with content still inside the page's JavaScript. A reliable tool says when content is missing, rather than letting the model answer from a page it never read. This guide builds one that handles both, in about 90 lines of Python.

TL;DR

With rendering on, our Web Scraping API cleared 4 or 5 of the 5 protected pages (3 passes), and 2 or 3 without it (5 passes), so headless is the parameter worth setting. Plain httpx cleared 0, a TLS-impersonating client 1.
Rendering raised one page from 174 characters to 1,567. Both calls reported success, so the server checks content length, not just the status.
markdown: true turned 652,316 characters of Zillow HTML into 10,010.
On mcp 2.x the import is from mcp.server.mcpserver_ import MCPServer_. Version 2.0 renamed FastMCP and moved its module, so the mcp.server.fastmcp path in older tutorials no longer exists.

What your agent needs to read a live page

Agents already manage context, call tools, and write code. Reading a live page is the capability worth adding next.

A custom fetch-and-parse tool has 2 problems, and only the first is about scraping. It can break on sites you didn't test. It also lives inside one agent's codebase, so you rebuild it for the next MCP client you adopt.

An MCP server solves the second problem by design. You write the tool once, and any MCP client can call it, including Claude Code, Cursor, and Windsurf.

The first problem is the backend's job.

What you'll need

Python 3.10 or newer
A Web Scraping API token from the free plan: run printf 'user:pass' | base64 with your credentials, or [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes('user:pass')) in PowerShell
A virtual environment: run python3 -m venv .venv, then .venv/bin/pip install "mcp>=2,<3" httpx (Windows: python -m venv .venv, then .venv\Scripts\pip.exe and .venv\Scripts\python.exe)

What a scraping backend handles

A plain HTTP client faces 3 failure modes on the live web.

Pages that load their content in the browser. The quotes on quotes.toscrape.com/js/ exist only inside a var data = [...] array until JavaScript runs, so a client that can't run it sees none of them. The page contains 0 quote elements with rendering off, and 10 with it on. The response is still HTTP 200 and still full of valid markup, so neither the status nor the size of the HTML tells you the content is missing.
Sites that refuse unfamiliar clients. We tested 5 search pages – jobs, retail, and property – 5 times each. Plain httpx with a Chrome user agent retrieved nothing on any pass. It advertises Chrome while sending Python's TLS signature, and anti-bot systems fingerprint the signature, not the header. curl_cffi, which impersonates Chrome's TLS, retrieved 1 of 5.
Markup that varies by site. A parser written for one site's markup often won't work on the next one, and raw HTML uses far more context than it needs.

On those same 5 pages, our Web Scraping API retrieved 4 or 5 with rendering on, averaging 4.7 over 3 passes, against 2 or 3 on default settings, averaging 2.4 over 5 passes. Rendering recovered the sites that the default never cleared.

One POST endpoint covers all 3. You send a URL against the universal target, add headless: "html" when a page needs JavaScript, and set geo for locale or device_type for device. We route requests through our standard and premium proxy pools, so most of the blocking problem is handled on our side. Our docs list the current country coverage. No scraper clears every page every time, which is why the tool reports failures instead of hiding them.

With markdown: true set in the Web Scraping API, the HTML becomes Markdown server-side, so a Python server needs no HTML-parsing dependency.

Markdown output is a fraction of the HTML's size:

Page	Raw HTML	Markdown
Zillow, Austin listings	652,316 chars	10,010 chars
Wikipedia, Web scraping	235,439 chars	63,166 chars
Hacker News front page	34,275 chars	10,046 chars

Every character you don't send is context the model keeps for reasoning.

Building the server

The server's only job is turning a tool call into an API call and formatting the response. One tool does that, and render_js goes in the signature with rendering off by default, so it stays off unless a page needs it.

Save this as server.py:

import os

import httpx
from mcp.server.mcpserver import MCPServer
from mcp.types import ToolAnnotations

API = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = os.environ.get("DECODO_TOKEN")
if not TOKEN:
    raise SystemExit("DECODO_TOKEN is unset. Set it to your Decodo user:password in base64.")

ALLOWED = [h for h in os.environ.get("ALLOWED_HOSTS", "").split(",") if h]
MAX_CHARS = 40_000

client = httpx.AsyncClient(timeout=90.0, limits=httpx.Limits(max_connections=10))
server = MCPServer(name="web-scraper", version="1.0.0")


def permitted(url: str) -> bool:
    """Allow everything when ALLOWED_HOSTS is unset, otherwise match the host exactly."""
    if not ALLOWED:
        return True
    try:
        return httpx.URL(url).host in ALLOWED
    except httpx.InvalidURL:
        return False


@server.tool(
    description=(
        "Fetch a live web page and return it as Markdown. Set render_js=true only when a page "
        "builds its content in the browser, because rendering is slower and costs more. "
        "country takes an ISO 3166-1 alpha-2 code such as US, DE or JP."
    ),
    annotations=ToolAnnotations(read_only_hint=True, open_world_hint=True),
)
async def scrape_page(url: str, render_js: bool = False, country: str | None = None) -> str:
    """Scrape one URL through the Decodo Web Scraping API and return Markdown."""
    if not permitted(url):
        return f"Refused: {url} is not in ALLOWED_HOSTS."

    payload = {"url": url, "target": "universal", "markdown": True}
    if render_js:
        payload["headless"] = "html"
    if country:
        payload["geo"] = country

    try:
        r = await client.post(API, json=payload, headers={"Authorization": f"Basic {TOKEN}"})
    except httpx.RequestError as exc:
        return f"Scrape failed: could not reach Decodo ({type(exc).__name__})."
    try:
        body = r.json()
    except ValueError:
        return f"Scrape failed: Decodo returned HTTP {r.status_code} and a non-JSON body."
    if not isinstance(body, dict):
        return f"Scrape failed: Decodo returned HTTP {r.status_code} and an unexpected body."

    if r.status_code != 200:
        errors = [e for e in (body.get("errors") or []) if isinstance(e, dict)]
        named = "; ".join(e.get("message", "") for e in errors)
        return f"Scrape failed ({r.status_code}): {named or body.get('message', 'bad request')}"

    results = body.get("results")
    if not isinstance(results, list) or not results or not isinstance(results[0], dict):
        return f"Scrape failed: {body.get('message', 'the target could not be scraped')}"
    result = results[0]

    landed = result.get("url") or url
    if not permitted(landed):
        return f"Refused: {url} redirected to {landed}, which is not in ALLOWED_HOSTS."

    status = result.get("status_code")
    content = str(result.get("content") or "")

    if len(content) < 500:
        if status != 200:
            return f"Scrape failed: {url} returned HTTP {status}."
        retry = "" if render_js else " If the page renders in a browser, retry with render_js=true."
        return f"{content}\n\n[Only {len(content)} characters came back.{retry}]"

    flag = "" if status == 200 else f"\n\n[{url} returned HTTP {status}, so this page may be partial.]"
    if len(content) > MAX_CHARS:
        return f"{content[:MAX_CHARS]}{flag}\n\n[Truncated at {MAX_CHARS} of {len(content)} chars.]"
    return content + flag


if __name__ == "__main__":
    server.run(transport="stdio")

MCP 2.0 changed that import line, renaming FastMCP to MCPServer. The migration guide documents the rest, including Tool.inputSchema becoming Tool.input_schema.

The 500-character threshold on the short-content check is a starting point worth adjusting for your sites.

Most of the rest of the file is response handling. The API reports the page's own status inside results, so a 404 page arrives as HTTP 200 with status_code: 404, and the tool checks both codes. On a 400, the errors array carries a message naming the offending field, such as headless or proxy_pool, and the tool returns it. The allowlist is checked twice, on the supplied URL and on results[0].url, because the API follows redirects and one open redirect would defeat a single check.

Connecting it to your client

In the directory holding server.py, check that the server works before you register it:

DECODO_TOKEN=your_base64_token .venv/bin/python -c \
  "import asyncio; from server import scrape_page; \
   print(asyncio.run(scrape_page(url='https://quotes.toscrape.com'))[:200])"

That tests the token and the dependencies without an MCP client involved. Username invalid means the token isn't base64; Incorrect username or password means it is. Then register the server, using the .venv interpreter, not the system one:

claude mcp add web-scraper -e DECODO_TOKEN=your_base64_token \
  -- /full/path/to/project/.venv/bin/python /full/path/to/project/server.py

That command is for Claude Code, and registers the server to the directory you run it in. For Cursor, Windsurf, or Claude Desktop, the equivalent entry goes in that client's MCP config file:

{
  "mcpServers": {
    "web-scraper": {
      "command": "/full/path/to/project/.venv/bin/python",
      "args": ["/full/path/to/project/server.py"],
      "env": { "DECODO_TOKEN": "your_base64_token" }
    }
  }
}

Ask for a page in plain language, and the agent chooses the arguments itself:

The agent read render_js from the tool description and set it unprompted. That page is worth seeing, because every quote arrives through JavaScript:

Without rendering, the tool returns 174 characters of navigation links, and not one quote. The hint it appends tells the model to retry with render_js=true rather than summarize what little arrived.

The parameter reference lists the rest of what universal accepts, and is where to look before widening the signature.

Bulk work uses different endpoints. POST /v3/task/batch queues many URLs, and GET /v3/task/{id}/results answers 204 until ready. Pass the URLs under url rather than query. Our 3 queued URLs finished within 10 seconds.

We also publish a ready-made server covering search, eCommerce, and social targets, plus code examples in several languages.

Running it in production

Reading a near-empty scrape as an API problem. A missing headless parameter and a genuinely broken page both return HTTP 200, so the status code won't distinguish them. When a page returns too little, try rendering first, then the premium pool, which you can reach by adding a proxy_pool parameter. In our runs, rendering was enough, and the premium pool handles sites that need more. Only successful responses are billed, so a failed attempt is free. Rendered requests are priced separately from standard ones – see the plans page.
Trusting the outer status code. On 1 of 5 passes, Etsy answered HTTP 403 and returned a full results page anyway, with 24 listing links and 33 prices. Rejecting on status alone discards it, so the tool reports the status and returns the content.
Returning raw HTML. A 652,316-character page like those Zillow results arrives as roughly 240K tokens of mostly markup and hides the listings you wanted. markdown: true is the fix.
Treating scraped text as trusted input. A page your agent fetched is attacker-controlled text arriving in a model's context. Indirect prompt injection through tool results is a documented attack on MCP-connected systems. Never let scraped content reach a tool that writes or spends, and have a person approve anything irreversible.
Letting the model choose URLs with no limits. ALLOWED_HOSTS starts empty so the examples run anywhere. Set it to a comma-separated list like example.com,docs.example.com before the server runs unattended. Add a rate limit too, and remember that scraping a site is a decision about its terms, not only a technical one.

Final thoughts

MCP makes the tool portable, and the Web Scraping API handles the proxy and browser work, so neither part does the other's job. Keep the server small, make render_js a parameter, and let the API return Markdown so your context window fills with content instead of markup. Report ranges rather than single runs when you benchmark a site, because the same URL that gave us job listings in the morning needed rendering by evening. A second tool that calls our fast search endpoint lets the same agent search before it scrapes. Get a token from the Decodo's web scraping solutions free plan, run scrape_page on a URL your current tool can't handle, and compare what you get.

Stop Paying Full Price For Every LLM Call

Decodo — Fri, 25 Sep 2026 07:56:11 +0000

An LLM bill has 2 parts: rate and volume. You can adjust both. But a default request barely changes either part. You pay the list price on repeated context, on work that nobody waits for, on easy questions that a cheap model could answer, and on calls that return nothing. The State of FinOps 2026 reports that 98% of teams manage AI costs, up from 31% 2 years ago, and lists AI cost visibility as a main challenge. Managing costs doesn't explain them. This article shows which parts of that bill are optional, and how to stop paying them.

TL;DR

On GPT-5.6, prompt caching lowers the rate on prefix tokens that you reuse, and batching halves both rates. Caching is automatic above a minimum prefix length. Below it, you get neither the discount nor an error. Batching is opt-in, so move async work there.
A 2-model cascade saves money until 9 calls in 10 escalate, so send the cheap model first if the rate gap is 10x and you can check the answer instead of trusting the model.
A failed fetch is re-billed on every later turn. Over 3 attempts, raw HTML costs 136x as much as extracted text. Across 7 protected targets, a plain request returned 0 usable pages, and a better IP alone returned 3, so blocking depends on more than the IP. Decodo's Web Scraping API returned 6, so move retries server-side.

Where you're paying full price

Per token, output costs more than input at gpt-5.6-terra's list rates. By volume, input costs more in total, because an agent resends its context every turn. Reasoning tokens are billed as output, so they raise what you pay per answer. Not every question needs them.

You pay for some calls and get nothing. A failed tool call, a malformed response, and a blocked scrape are all billed for the tokens that they used. Your invoice lists tokens, not results, so cost per request doesn't show you the waste. Sum the token counts by task instead, using the usage field that your API responses already return.

The discounts already on the rate card

Prompt caching discounts the prefix, the front of your prompt, when you reuse it unchanged. Writes cost more than the normal input rate, and a single read already covers it. A prefix that you never reuse costs you the write and returns nothing, so cache only what repeats.

On GPT-5.6, caching is automatic above a minimum prefix length, and below it the discount silently doesn't apply, so measure your shared prefix before you trust it. Because a cache only matches an unchanged prefix, put stable content first and changing content last. If you cache whole responses, expire them when the source data changes.

On the same card, batching halves the rate on input and output, with no extra cost for writes and no minimum length. Batch jobs fit work that nobody waits for, not chat or anything that needs low latency. With the flex option you pay the same discounted rate without the async round trip, and you trade latency and availability for it. You may not keep your prefix discount when you move a workload into batch, so check cached_tokens on your current traffic before you switch.

Paying less for easy questions

Routing is cheap to build. A status code and a look at the body catch the clear blocks, and sorting the head of what remains is easy work. On the same card, input on gpt-5.6-luna is 10x cheaper than on gpt-5.6-terra, so send those pages to the cheap model first and escalate the answers that fail a check:

# pip install litellm
from litellm import completion, model_cost

CHEAP, STRONG = "gpt-5.6-luna", "gpt-5.6-terra"
LABELS = {"content", "block", "captcha", "empty"}
GATE = (f"Label the page as one of: {', '.join(sorted(LABELS))}. "
        "Reply with the label only.")

def classify(head):
    """Cheap model first. Escalate when its answer is not a valid label."""
    def ask(m):
        res = completion(model=m, messages=[
            {"role": "system", "content": GATE},
            {"role": "user", "content": head}])
        return res.choices[0].message.content.strip().lower()
    if (out := ask(CHEAP)) in LABELS:
        return out, CHEAP
    if (out := ask(STRONG)) in LABELS:
        return out, STRONG
    raise ValueError(f"neither model returned a label: {out!r}")

# What that costs, priced from LiteLLM's own cost map. No API key needed.
# The call is small, far under any context limit. Uncached too: the shared
# prefix is the GATE string, well under the cache minimum.
IN_TOK, OUT_TOK, N = 1200, 150, 1000   # page head, label, calls
price = lambda m, n: n * (IN_TOK * model_cost[m]["input_cost_per_token"]
                        + OUT_TOK * model_cost[m]["output_cost_per_token"])

flat = price(STRONG, N)
for rate in (0.1, 0.3, 0.9):
    total = price(CHEAP, N) + price(STRONG, int(N * rate))
    print(f"escalate {rate:>4.0%} -> ${total:>5.2f} per 1K vs ${flat:.2f} flat")

The code compares the cascade with the strong model alone:

escalate  10% -> $ 0.84 per 1K vs $4.20 flat
escalate  30% -> $ 1.68 per 1K vs $4.20 flat
escalate  90% -> $ 4.20 per 1K vs $4.20 flat

If you escalate 3 calls in 10, you save 60%, and break-even is at 90%. A 5x gap breaks even at 80%, because break-even is 1 minus 1 divided by the rate gap, when input and output have the same gap. The code charges the same amount for a failed cheap call and for a successful call, so reasoning on the cheap model lowers the break-even point. Each model in a cascade keeps its own cache, so the saving is smaller than the rate difference.

Check the label, not the confidence that the model reports. Your numbers may differ because LiteLLM refreshes its cost map on import. A valid label isn't always a correct label, so sample against the strong model before you trust the savings.

What a blocked fetch returns

Agents that read protected sites get blocked, and you pay full price for every blocked call. A better IP is the first thing to try, so we tested one. In September 2026, we sent the same httpx GET to each of 7 targets that we picked for their anti-bot protection, first from an ordinary connection, then from a US exit node on Decodo's residential proxies. We also sent the same 7 targets through Decodo's web scraping solutions, which render JavaScript.

Target	Plain fetch	+ residential IP	Web Scraping API
`zillow.com/homes/for_sale/`	403	200, real listings	200
`amazon.com/s?k=laptop`	503	200, real prices	200
`walmart.com/search?q=laptop`	200, bot-check	200, real prices	200
`glassdoor.com/Reviews/index.htm`	403, 147,539 tokens	403	200
`indeed.com/q-software-engineer-jobs.html`	403	403	200
`crunchbase.com/organization/anthropic`	403	403	200
`g2.com/products/decodo/reviews`	403	403	613, refused
All 7	0 usable	3 usable	6 usable

Through a residential IP, the first 3 rows returned real listings and prices. Amazon returned a page in 1 of 3 runs with no delay, and in 5 of 5 runs when we spaced them. Space the retries if you run them yourself. The remaining 4 targets refused the same address on 12 of 12 retries that we spaced 6 seconds apart, so blocking depends on more than the IP.

3 of the remaining 4 targets returned pages because the API wasn't blocked, not because it rendered them, since rendering can't produce content from a refusal.

Walmart returned a failure that looked like a success. On a plain fetch, it returned HTTP 200 with a bot-check body. A tool that branches on status_code reports success and passes that page to the model. The agent gets no signal to retry, so check the body for the data that you want, not just the status code.

Each kind of failure costs you something different. A small 403 costs one turn. A blocked page that you keep retrying costs every later turn. A 200 with no content means you never get the answer.

The retry tax compounds

A retry looks like one extra call. But a retry costs much more, because each turn resends the conversation so far.

That Glassdoor block page is 147,539 tokens of raw HTML. We measured it with o200k_base, like every token count here. We fetched it once, but it's billed on all 3 turns that follow:

891,670 input tokens, no answer.

With 3 attempts, you send 6x as many tokens as with a single attempt, and you pay 11x as much, because turns 3 and 4 are large enough to be billed at a higher rate:

# Agent-loop cost for one web-reading step. Every turn resends the whole
# conversation, so a failed tool result is re-billed on every later turn.
# Arithmetic over measured payloads, not a billing export.
# Rates come from LiteLLM's cost map, uncached, which is the ceiling.
# Input over the tier prices wholly at the long rate, a ceiling. Output is flat.
# That map refreshes on import, so your figures may differ from these.
# Token counts throughout measured with o200k_base.
# pip install litellm
from litellm import model_cost

CARD = model_cost["gpt-5.6-terra"]
IN_SHORT = CARD["input_cost_per_token"] * 1e6                  # $/1M tokens
IN_LONG  = CARD["input_cost_per_token_above_272k_tokens"] * 1e6
OUT_RATE = CARD["output_cost_per_token"] * 1e6
TIER = 272_000        # named in the rate-card key above, not priced as a value
SYSTEM, TASK = 1_500, 16              # assumptions: your prompt, your question
TOOLS_PLAIN, TOOLS_DECODO = 48, 301   # measured: 1 fetch tool vs TOOLSETS=web
CALL, ANSWER = 30, 60

def per_1k(base, payload, attempts):
    ctx, cost, out_tok = base, 0.0, 0
    for _ in range(attempts):
        # each turn pays its own rate
        cost += ctx * (IN_SHORT if ctx <= TIER else IN_LONG)
        out_tok += CALL
        ctx += CALL + payload     # the failure stays in context
    cost += ctx * (IN_SHORT if ctx <= TIER else IN_LONG)
    out_tok += ANSWER
    return (cost + out_tok * OUT_RATE) / 1e6 * 1000

plain = SYSTEM + TASK + TOOLS_PLAIN
for label, payload, n in [("raw HTML", 147_539, 1), ("raw HTML", 147_539, 3),
                          ("text-extracted", 785, 3)]:
    usd = per_1k(plain, payload, n)
    print(f"blocked x{n}, {label:<14} ${usd:>8.2f} per 1K, no answer")

decodo = SYSTEM + TASK + TOOLS_DECODO
served = per_1k(decodo, 951, 1)   # markdown, tokenLimit=2000
print(f"served  x1, markdown       ${served:>8.2f} per 1K, one answer")

The same arithmetic compares the failing runs with the successful one:

blocked x1, raw HTML       $  302.47 per 1K, no answer
blocked x3, raw HTML       $ 3267.09 per 1K, no answer
blocked x3, text-extracted $   24.09 per 1K, no answer
served  x1, markdown       $   10.31 per 1K, one answer

You pay full price for everything the agent keeps. The 136x between the 3-attempt rows comes from the content that stayed in context, not from the block, and extracting the text first closes that gap. Input tokens grow with the square of the attempt count, so if you double your turn limit, you pay roughly 4 times as much.

Caching reduces both 3-attempt numbers by roughly half, but it doesn't close the gap. Caching helps the extracted run slightly more, because a context that grows fast has less to reuse. Compaction and tool-result clearing limit the growth, but they act later, so a page that's large on arrival is billed in full on the turn it arrives. Neither one refunds the first fetch, and neither one produces the answer.

Stopping the retry tax with Decodo's MCP server

3 blocked attempts on raw HTML cost 317x a single served fetch, and they return nothing. Extracting the text first cuts the cost to $24.09, which is still more than a served fetch, and it still returns nothing. So move the blocks and retries out of the agent loop.

Decodo's MCP server gives your agent Decodo's web scraping tools, and it retries outside your context. Your context then contains one response instead of every attempt. A successful fetch costs $10.31 per 1K in gpt-5.6-terra tokens alone, and you also pay Decodo's request pricing.

6 of the 7 returned usable pages through the API, measured by calling it directly. The last target returned a structured refusal in 67 tokens, and a failed call is a stable signal, so branch on the failure rather than on the refusal text. Your own targets will have their own success rate, so measure yours before you budget for it.

Definitions and results both make the context larger, and this server has a setting for each:

TOOLSETS. Name the toolsets that your agent uses, from web, search, ecommerce, social_media, and ai. On 1.2.4, web alone costs 301 tokens of definitions, 93% less than the 4,537 for the full set.
tokenLimit. This setting truncates the response after the server scrapes the page and converts it to markdown, so the limit applies to extracted text rather than raw HTML. A page under the limit arrives whole, so test your worst URL and set the limit to the largest response that you're willing to pay for. The model passes this value per call, so state it in your system prompt.

In your agent, setup takes one config block. The MCP server is a client for the Decodo's scraping solutions and takes a credential from the free account, either base64 of user:pass or an API key:

{
  "mcpServers": {
    "decodo": {
      "command": "npx",
      "args": ["-y", "@decodo/mcp-server@1.2.4"],
      "env": {
        "SCRAPER_API_TOKEN": "<basic_auth_token>",
        "TOOLSETS": "web",
        "MAX_RETRIES": "2"
      }
    }
  }
}

That config registers the web toolset alone, here in Claude Desktop:

MAX_RETRIES moves the retry tax. The server makes those attempts, so the failed attempts never reach your context, and 2 is the default on 1.2.4. jsRender gives the headless rendering that the benchmark used. Approve both tools once, or allow them for a batch run.

Final thoughts

The call that buys nothing costs you the most, so fix how you fetch data first. The other habits save less, so measure by task, cache, batch, then route. None of them costs much, because caching on this card is automatic above the minimum length, batching is opt-in, and routing takes about an afternoon.

Moving the retries out of your context stops the retry tax, because prompt tuning alone won't fix a loop that re-bills a block page every turn. With tiktoken, count the tokens that your agent's worst URL returns and pass that number to per_1k as the payload, then create a free account, point your existing agent at the server, and count again.

Selenium keeps getting blocked? Here's what Cloudflare actually sees

Decodo — Wed, 23 Sep 2026 08:43:58 +0000

Cloudflare's bot protection checks your headers, TLS handshake, and HTTP/2 connection. Some of those checks refuse a request before any script runs. The others cannot refuse on their own, and a request they don't settle gets a JavaScript challenge page that inspects the browser. That's why the usual advice fails. You patch navigator.webdriver, add delays, and install a stealth plugin. The challenge page still comes back because that advice targets the check inside it, not the one that refused you. This guide covers a 2-command diagnostic, the 5 checks, and what cleared the block.

TL;DR

Only 2 of Cloudflare's 5 checks refuse a request on their own: a User-Agent blocklist and an HTTP/2 fingerprint that must match the browser in your headers. Tested separately on the 2 sites measured, each refused 12 out of 12 curl requests.
Selenium drives real Chrome, so the TLS and HTTP/2 checks shouldn't catch it. It leaked HeadlessChrome in the User-Agent. Replacing that header with a normal Chrome string cleared the block in 28 runs out of 28.
Results change from run to run because of the raw HTTP client, not the browser. In a round-robin against both sites, one Python client flipped while curl and Selenium each passed 12 out of 12.
A scraping API can give you many exit addresses instead of 1, and a rate limit you can raise, with no browser processes to run. The API tested here returned a different address for each of 6 requests.

Reproduce the block

The baseline is headless Selenium against a Cloudflare-protected page. I measured against 2 such pages, allrecipes.com/recipes/ and gamespot.com, and both allowed these paths in robots.txt when I ran the measurements. Check it again before you point anything at them. The starting script patches nothing:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless")

driver = webdriver.Chrome(options=options)
driver.get("https://www.allrecipes.com/recipes/")

print(driver.title)
print(driver.execute_script("return navigator.webdriver"))
print(driver.execute_script(
    "return Object.getOwnPropertyNames(window).filter(k => k.startsWith('cdc_')).length"
))

driver.quit()

Every measurement here is mine. The browser tests used Chrome 151 and Selenium 4.38. I ran the client comparisons from 1 residential IP, so the results are comparable across clients.

The first script prints the interstitial as the title, as the advice predicts:

Just a moment...
True
7

Those are the 2 signals that the stealth guides tell you to fix: navigator.webdriver, and the 7 globals ChromeDriver added to window. If you paste those same 2 expressions into a browser you opened yourself, you get false and 0. The true and the 7 are ChromeDriver artifacts.

Patching them should get you through. But the block has little to do with either property, and you can show that without a browser. These 2 requests are identical except for one string:

TARGET="https://www.allrecipes.com/recipes/"
CHROME="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"
HEADLESS="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/151.0.0.0 Safari/537.36"
curl -s -o /dev/null -w "%{http_code}\n" --http1.1 -A "$CHROME"   "$TARGET"
curl -s -o /dev/null -w "%{http_code}\n" --http1.1 -A "$HEADLESS" "$TARGET"

Here’s the result:

200
403

Over 6 runs per site, the Chrome string passed 12 out of 12, and the headless one passed 0. No JavaScript ran, no browser existed, and nothing inspected a canvas or a WebGL renderer. The deciding signal was one substring in one header.
The second pair changes the protocol instead of the string:

curl -s -o /dev/null -w "%{http_code}\n" --http1.1 -A "$CHROME" "$TARGET"
curl -s -o /dev/null -w "%{http_code}\n" --http2   -A "$CHROME" "$TARGET"

Here’s the result:

200
403

Also, 6 runs per site: HTTP/1.1 passed 12 out of 12, and HTTP/2 passed 0. Same binary, same User-Agent, same TLS library. Switching to HTTP/2 changes both how the connection opens and what the handshake advertises in ALPN. The 403 exposes a second check below the header, but it doesn't prove that HTTP/2 alone is the cause.

To run this against your own site, start by checking that it's behind Cloudflare at all, since everything here is scoped to Cloudflare:

TARGET="https://your-own-failing-url.example/path"
curl -sI "$TARGET" | grep -i -E '^(server|cf-ray):'

A Cloudflare site returns Server: cloudflare and usually a CF-RAY header. If the Server header says nginx or anything else, something other than Cloudflare is stopping you, and none of this applies.

Then run the User-Agent pair again, with TARGET changed to your own URL, because the answer isn't the same everywhere. I also checked 18 Cloudflare sites, and 6 returned 200 to the headless string, so your target may not behave like the 2 measured here. Re-checking those URLs within the same hour produced different answers, so treat any single sweep as a snapshot. What each result means:

A 200, then a 403. It's the User-Agent. Changing that one string clears the block.
A 403 both times. The User-Agent isn't the cause. Send the same 2 commands from a mobile hotspot, several times each, since one different result there doesn't prove much on its own. If they pass from the hotspot and not from your own connection, your address is the variable that changed.
A 200 both times. Your headers alone aren't being refused. But curl and your browser differ in almost every other way. Run the first script against your own URL, and see whether Selenium is refused where curl got through.
Different answers on repeat runs. That's usually the client, not the time of day. Only one Python client flipped run to run while curl and Selenium, both sending a Chrome User-Agent, passed every time in the same round-robin.
Any other status code. These 2 sites returned only 200 or 403, so anything else is outside what these runs cover.

What Cloudflare is actually looking at

Cloudflare checks 5 things, and the one that caught you decides what's worth fixing. It doesn't publish its logic or its weighting, so this breakdown is what I observed on 2 sites. Only 2 of the 5 refuse on their own. The other 3 contribute, vary, or are reached only if you're challenged:

The User-Agent blocklist. The string headless anywhere in the User-Agent, matched case-insensitively, got a 403 with a cf-mitigated: challenge header. The 403 came back on 12 out of 12 curl requests across both sites. The same was true of a User-Agent containing curl or Selenium, and of an empty one. It behaved like a blocklist, not a heuristic check. In a single round of 6 runs each with Python's urllib, a User-Agent of Mozilla/5.0 zzzz passed at exactly the same rate as a real Chrome string: 4 of 6 on allrecipes and 2 of 6 on gamespot.
The HTTP/2 fingerprint. A client claiming to be Chrome must open the connection the way Chrome opens it. The SETTINGS frame that curl sends first doesn't match Chrome's:

Setting or frame	Chrome	curl
HEADER_TABLE_SIZE	65536	–
ENABLE_PUSH	0	0
INITIAL_WINDOW_SIZE	6291456	10485760
MAX_HEADER_LIST_SIZE	262144	–
MAX_CONCURRENT_STREAMS	–	100
WINDOW_UPDATE frame	15663105	–

A dash means the client doesn't send that one. I captured Chrome's column there from a Chromium 148 engine. The curl column came from version 8.7.1. Forging those values flipped the result from 403 to 200 on both sites, with the library, the User-Agent, and the address unchanged.

The TLS fingerprint. The handshake differs by client, and TLS fingerprinting reduces that difference to a short string called JA4. An echo service such as tls.peet.ws/api/all reports the one your client is sending, for as long as that service stays up. The clients that differ in TLS library also vary in header count, header order, and HTTP implementation. Even httpx and urllib, both on OpenSSL, behaved differently from each other. Because those differences appear together, these runs can't measure the handshake on its own.
Everything else. Identical requests 2 seconds apart returned different results from that same urllib client, so the remaining signals, which these runs don't identify, don't resolve the same way twice. IP reputation is the usual explanation, but isolating it would need the same client from a different address, and these runs don't include that.
The JavaScript challenge. If you're challenged, the browser gets inspected on the page you get back. In one instrumented run, the challenge script probed for ChromeDriver's injected globals by exact name, 13 times across 3 of the 7. It called Function.prototype.toString 28 times, which is the call that exposes an overridden native function, though I didn't feed the script a patched build to confirm that's what it checks for. It bound 7 behavioral listeners, and it bound a WebRTC listener, which is how a proxied request can leak the address behind it.

The browser fingerprinting is real, but it lives in that challenge page, which is why patching for it doesn't help: the header check decides whether you reach that page. Selenium drives real Chrome, so its handshake should already match Chrome's. I measured Chrome's fingerprint but never captured Selenium's own, so treat that as reasoning rather than measurement, backed by how consistently Selenium passes once the header is fixed. That leaves the User-Agent exposed.

What changes the outcome

If the User-Agent is the exposed part, a matrix should show it. The matrix drives allrecipes through Selenium: 5 configurations, 3 runs each, the last run with the order reversed as a control. All 3 agreed every time:

What you change	UA says Headless	navigator.webdriver	Result
Nothing	yes	true	Just a moment…
Remove the automation flag	yes	false	Just a moment…
Spoof the User-Agent	no	true	page loads
Spoof the User-Agent and remove the flag	no	false	page loads
Drop headless, no other change	no	true	page loads

The second row is the stealth guides' first fix:

options.add_argument("--disable-blink-features=AutomationControlled")
options.add_experimental_option("excludeSwitches", ["enable-automation"])

With the flag gone, the page was still blocked. Cloudflare decided to challenge based on the request headers, so that property never triggered it. The challenge page does read the automation properties, and clearing them still doesn't get you past the header.

The third row is a one-line User-Agent change, the fix that gets dismissed as naive, and it loaded the page with every automation property still in place. That line is:

options.add_argument(f"--user-agent={UA}")

where UA is the value your own Chrome reports for navigator.userAgent. Take it from a browser you opened, so the version matches the Chrome you're running.

The last row is the same fix by another route: headless mode added HeadlessChrome to the string, so turning headless off removed the substring. In full, the script adds the User-Agent and drops the 2 property checks:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

UA = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36")

options = Options()
options.add_argument("--headless")
options.add_argument(f"--user-agent={UA}")

driver = webdriver.Chrome(options=options)
driver.get("https://www.allrecipes.com/recipes/")

print(driver.title)

driver.quit()

The interstitial is gone:

Recipes

The result held: I ran that script 6 times in a row against each target, and 12 out of 12 returned the real page. The browser captured 2 runs itself:

Across every run reported here, headless Selenium with a corrected User-Agent passed 28 times out of 28. On these 2 targets, from 1 address, the naive fix is the whole fix.

What the stealth libraries change

A library built to avoid detection was stopped by the same string. Under nodriver, navigator.webdriver read false, and none of ChromeDriver's cdc_ globals were present. In headless mode against these 2 targets, it was blocked on both because it still announced HeadlessChrome in the User-Agent.

The library, the automation flags, and the headless mode were identical. The only difference was one browser argument overriding the User-Agent. Adding that argument changed the result from blocked to served. nodriver handled the automation properties, but left the User-Agent alone.

The tools that cleared the block avoided the string entirely. CloakBrowser is a Chromium fork patched at the browser source rather than through injected JavaScript, and it didn't announce HeadlessChrome. Camoufox on a Mac reported a Windows machine with an NVIDIA GeForce GTX 980, at a screen size that the laptop didn't have.

Neither is hiding a flag. Both present an entire machine that doesn't exist, consistent from the header down to the GPU. To answer whether Cloudflare cross-checks those layers, you would need a client that matches on some and not others.

SeleniumBase's UC Mode takes a third route: its own patched driver binary instead of vanilla ChromeDriver. In headless mode, it passed both targets 3 times each, with no cdc_ globals and no manual User-Agent change. Whether it also spoofs machine-level signals like Camoufox's wasn't tested here.

What dropping the browser costs

If you move to requests or httpx, you take on 2 failure modes Selenium doesn't have. With a correct Chrome User-Agent, httpx was blocked 12 times out of 12: 3 runs per site over HTTP/1.1, and 3 more over HTTP/2. Something below the header is refusing it, though my runs don't separate its TLS stack from its headers. Neither client runs JavaScript, so anything the page builds client-side is missing from the response.

The run-to-run flakiness also originates below the header, though it's commonly attributed to the browser. I tested 5 client configurations round-robin against both targets, 6 rounds each, so every client met the same conditions at the same moments:

Client	User-Agent	allrecipes	gamespot
curl, HTTP/1.1	Chrome	6/6	6/6
curl, HTTP/1.1	HeadlessChrome	0/6	0/6
curl, HTTP/2	Chrome	0/6	0/6
Python urllib	Chrome	5/6	2/6
Selenium headless	Chrome	6/6	6/6

Exactly one client flips. That rules out the browser and the time of day. A raw client with only 3 headers and a Python TLS stack varies run to run, and no User-Agent changes that.

When you need more than a working script

On these targets, a real browser plus one corrected header loads the page. That combination stops being enough at scale.

Browser processes bring crash loops and restarts. Every request leaves from your address. A local browser gives you 1 concurrent session.

What the API replaces

Decodo's Web Scraping API takes a URL and returns the fetched page, so one POST replaces the browser:

import os
import requests

response = requests.post(
    "https://scraper-api.decodo.com/v2/scrape",
    json={
        "url": "https://www.allrecipes.com/recipes/",
        "proxy_pool": "premium",
        "headless": "html",
    },
    headers={"Authorization": f"Basic {os.environ['DECODO_TOKEN']}"},
    timeout=120,
)
response.raise_for_status()

result = response.json()["results"][0]
print(result["status_code"], len(result["content"]))

DECODO_TOKEN is the Basic authentication token from the dashboard, on the Web Scraping API page under the Playground tab. It's already base64 encoded, so it goes in as-is.

proxy_pool is premium unless you override it, and the snippet sets it explicitly. With headless: _"_html", the API renders the page before returning it.

The free plan covers 667 requests at the premium-with-rendering rate this snippet uses. That plan provides $1 of credits, valid for a year, and doesn't require a credit card. The rate limit runs from 10 requests per second on the entry paid plan up to 200 on the highest published tier.

I ran the snippet 6 times against each target. An allrecipes run printed:

200 531079

That's 12 out of 12 across both Cloudflare-protected targets, and every one was a real page. The character count moves with the page: allrecipes ranged from 531,000 to 598,000 characters across the 6 runs, and gamespot ranged from 462,000 to 475,000. Yours will differ, as will every other count here, so check the status code and confirm there's a page behind it.

I re-ran the snippet, and the answer changed. An earlier run without headless: _"_html" returned 402 from allrecipes. The response body came from the site rather than Cloudflare, and asked visitors with access problems to email support. When I re-ran it later, 6 times per target, it returned 200 every time without the parameter.

The parameter still adds content: gamespot came back at 301,000 characters without it, against 462,000 to 475,000 with it. But it no longer decides whether the request succeeded. That's the same run-to-run instability these targets show, so check what comes back. The parameters reference lists the rest.

What the free tools already do

curl_cffi, primp, and rnet forge Chrome's fingerprint. curl_cffi's JA4 was byte-identical to real Chrome's. Over 6 runs per target, with nothing else changed, requests with impersonation off were blocked 12 times out of 12, and requests with it on passed 11 times out of 12.

primp returned a 469,000-character Allrecipes page with the correct title, 158 internal links, and no interstitial.

If you need a real browser, Camoufox and CloakBrowser both returned the page from each target in headless mode.

rnet has been renamed upstream and is now maintained as wreq, which is the package to install. I used rnet here, before the rename.

Forging a fingerprint improves the odds, but 11 out of 12 is not a guarantee.

What none of them do

None of those free tools rotates your address. I sent 6 requests through the API and they came back from 6 different addresses, spread across 4 /8 ranges. From a local browser, the same 6 requests come from 1 address every time, which is the constraint at volume.

Common pitfalls

An API doesn't help if the diagnostic was wrong. This diagnostic goes wrong in 3 ways:

Mistaking a working patch for understanding. Treat a 1-line fix as evidence of which signal decided the outcome, not of which signals are checked.
Deciding from a single run. Repeat it, and reverse the run order, before believing the result.
Blaming the time of day when the client is the cause. Run the clients round-robin and see which client the flakiness follows.

Final thoughts

A substring in a header stopped these Selenium sessions, and the fix was 1 line. The rest of the detection stack is real but lives in the challenge page. On these targets, the header decided whether you saw that page, so the complicated patches aim at a check that never refused you. The diagnostic comes first for that reason: 2 curl requests, identical except for the User-Agent, will separate a header-level block from everything else in under a minute. The header stops being the interesting problem at volume, when exit addresses matter more than fingerprints. Run that header pair against your own failing URL before patching anything else.

Stop Sending Raw HTML to LLMs

Decodo — Wed, 16 Sep 2026 12:31:42 +0000

A scraper gives you a page as one HTML string: markup, inline styles, scripts, and somewhere inside it the text you wanted. Many pipelines send that string directly to an LLM. That works, so the cost rarely gets measured. But on the 10 pages I measured, the HTML used 3 to 24 times more tokens than the same pages as Markdown. A product page arrived as 267,361 tokens, of which 3,393 were visible text. This guide shows where those tokens go, what they cost, and how to stop sending them.

TL;DR

Visible text was 0.3% to 18.5% of the tokens I fetched, and markup was the rest.
The cost was tokens, latency, and rate limits, not wrong answers. All 3 models I tested answered correctly from the raw HTML, and took up to 5.7 times longer to start.
Setting "markdown": true on Decodo's Web Scraping API returned the same pages with 3 to 24 times fewer tokens, keeping headings, links, and tables.
The same API returns the HTML when you need JSON-LD values like price and rating.

*Every number here is from my own runs, September 2026.

Where HTML tokens go

I fetched each page once with requests.get and a Chrome User-Agent, then removed one category at a time and re-counted with tiktoken's o200k_base. Each category is measured after the ones above it, so an SVG inside nav counts as SVG, and rows sum up to the fetched total:

6 of 10 pages. Markdown keeps structure, so it’s more than that share.

The Hydration JSON is the largest cost on the Next.js docs page, most of it in 65 self.next_f.push calls repeating the content as escaped strings. The Hacker News front page has almost no scripts or styles and is still 90% markup, because it's built from layout tables. Docs, news, and storefronts would each need a different rule.

What HTML waste costs

Tokens. At $2.00 per 1M input tokens, a rate several current models charge, those 10 pages cost $3.04 as HTML and $0.25 as Markdown. Per 1K of the heaviest pages, that's $1,157 instead of $49, plus scrapes at current rates. A different tokenizer changes the totals: across OpenAI, Anthropic, and Google the same HTML was up to 50% larger, while the pooled ratio stayed between 11× and 14×. So, trust the ratio and re-count the dollars for your model.

Time to first token. I asked one question per page on both versions, with a unique first line per call to prevent caching. On gpt-5.6-terra, the raw version took 2.0 to 2.8 times longer to start on the 4 pages above 130K tokens, and 1.0 to 1.4 times longer on the 4 below 75K. The other 2 models were slower, up to 5.7 times.

Rate limits. A 57,266-token page was refused on a gpt-4.1 account plan: Request too large … Limit 30000, Requested 57266. It had 622 tokens of text.

All 3 models across 2 vendors answered every question correctly from the HTML and Markdown, 60 answers each way. With no page attached they answered 1 or 2 of them, so they were reading rather than recalling. I put 4 raw pages into 396K-token prompts and got the right answer in all 3 runs.

What Markdown removes and what it keeps

Markdown conversion removes the markup around the content, including style blocks, SVG paths, class and data- attributes, and the scripts. It keeps headings, lists, tables, and link URLs. On the Hacker News front page, it returned Markdown tables with the story titles, URLs, scores, and comment counts in 3,644 tokens instead of 11,817.

Markdown keeps prose, and JSON-LD is in a script block. When I asked for a product's price, rating, and review count, the model got all 3 from the raw HTML and only the price from the Markdown, because the rest are in JSON-LD. On 3 of the 5 pages with it, ratings, stock status, and publication dates were absent. Markdown still kept 99.4% of that page’s words, because the rating and review count are 2 of 16,844.

Read the application/ld+json blocks with an HTML parser, since quoting varies, and send the model the prose.

Markdown directly from Decodo's API

Decodo's Web Scraping API takes a markdown flag that runs the conversion server-side, so the parsing happens outside your pipeline. The free plan starts immediately after signup.

DECODO_TOKEN is the Basic authentication token in the Playground tab, which builds the same request:

Create a virtual environment, install the packages, and export the token:

python3 -m venv .venv && source .venv/bin/activate
pip install requests tiktoken
export DECODO_TOKEN="paste-your-basic-auth-token"

Once you’ve set up your environment and .env file, create a script file with the following code:

import os
import requests
import tiktoken

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = os.environ["DECODO_TOKEN"]
# o200k_base is OpenAI's. Counts vary by vendor: use your model's encoding
enc = tiktoken.get_encoding("o200k_base")


def fetch(url, markdown):
    # Some targets reject markdown on the standard pool. Retry with "premium".
    payload = {
        "url": url,
        "proxy_pool": "standard",
        "markdown": markdown,
    }
    try:
        response = requests.post(
            API_URL,
            json=payload,
            headers={"Authorization": f"Basic {TOKEN}"},
            timeout=120,
        )
    except requests.RequestException as e:
        raise SystemExit(f"Request failed before a reply: {e}")
    if response.status_code >= 400:
        raise SystemExit(f"HTTP {response.status_code}: {response.text[:200]}")
    try:
        data = response.json()
    except ValueError:
        raise SystemExit(f"Reply was not JSON: {response.text[:200]}")
    if not data.get("results"):
        raise SystemExit(f"Scrape failed: {data.get('message')}")
    result = data["results"][0]
    # results[0] has the target's own status and the URL it reached, so a
    # site's 404 page or a redirect appears even under HTTP 200.
    code = result.get("status_code", 200)
    if code != 200:
        raise SystemExit(f"Target returned {code}, not the page")
    if result.get("url", url) != url:
        print(f"note: redirected to {result['url']}")
    content = result.get("content")
    if not content or not content.strip():
        raise SystemExit("Empty body: a shell or a challenge, not the page")
    return content


url = "https://en.wikipedia.org/wiki/Web_scraping"
# 2 calls here only to compare sizes. In a pipeline, make one call per page:
# markdown for the prose, or HTML when you also need the page's JSON-LD.
html = fetch(url, markdown=False)
md = fetch(url, markdown=True)
# disallowed_special=() keeps pages containing <|endoftext|> from raising
print(f"HTML:     {len(enc.encode(html, disallowed_special=())):>7,} tokens")
print(f"Markdown: {len(enc.encode(md, disallowed_special=())):>7,} tokens")

Running it prints both counts:

HTML:      72,201 tokens
Markdown:  16,553 tokens

Reduction across my 10 pages was 3× to 24×, median 7×. A ratio alone means little, since deleting content improves it. A local extractor reached 856× on a storefront listing by keeping 3.6% of its words, whereas this Markdown kept 99.4% at 24×.
Fetching is the harder half. From one home connection, a plain fetch reached the page on 4 of the 20 commercial sites I tested. It missed a job board at 59× and a property listing at 61×. Free html2text kept more of the words than the API's median 98.4% on pages a plain fetch reaches. The parameters reference lists the fields.

Decodo's MCP server offers this conversion as an agent tool.

Before you trust the output

A 200 that isn't the page. A plain GET to a home-improvement retailer returned HTTP 200 with an "Access Denied" body of 67 visible-text tokens, whereas the API on the premium pool returned the homepage, 3 runs of 3. A department store served a challenge script both ways, 152,000 tokens of it and no visible text. No single limit separated them – a real docs page had 622 visible-text tokens, fewer than a 909-token "browser not supported" page. Check the target's own status code in results[0], then compare each URL with its last good fetch.

Rendering worth confirming. "headless": "html" on a client-rendered page returned the rendered content in 12 of 14 runs. The other 2 returned the unrendered page, 62 tokens, at HTTP 200. The rendered page had 6 times more text.

Extractor coverage. A readability-style extractor kept a median 13% of these pages' words, and 0% on Hacker News. Another extractor kept 99.7% there, and the whole-page conversion kept 69.6%, its lowest. Test yours on a listings page.

Final thoughts

Raw HTML in a prompt is mostly markup. On the pages I measured, visible text was a median 2.3% of what got sent. That waste costs you a larger bill, a slower start, and a rate limit you didn't plan for, not a wrong answer on the 3 models I tested, so check accuracy on yours. Converting server-side removes the cleanup code from your pipeline, and the HTML response has JSON-LD for pages where you need exact fields. When a page needs rendering or a different pool, check the target's own status and the returned content, because both failures return HTTP 200. Run that snippet on a URL your pipeline already fetches, and compare the 2 numbers before you change anything.

How To Build an AI Agent That Browses the Web?

Decodo — Tue, 25 Aug 2026 13:17:19 +0000

Your model provider probably includes a web search tool. It's one line of code, and for a stable fact, it works. It goes wrong on a page that changes or blocks bots, and it rarely warns you. This guide builds the other kind of agent: one tool that fetches a page through Decodo's Web Scraping API, and a short loop that runs the tool.

TL;DR

Use a fetch tool when the answer changes faster than an index can follow, and leave stable facts to search.
Define one tool, fetch_page(url), that calls Decodo's Web Scraping API with markdown: true on the premium pool, and returns truncated text instead of HTML.
Send the question and the tool definition together, then check whether the response contains a function call rather than an answer.
Run fetch_page when the model asks, append the result to the conversation, and repeat until the model answers or the turns run out.

Do you need a fetch tool?

One question for each outcome, run both ways:

Question	Built-in web_search result	Live fetch result	Result
Price of Sharp Objects on books.toscrape.com	£47.82	£47.82	Both correct
Latest Python version on python.org	3.14.6, 4 runs out of 4	3.14.7	Search stale
Latest Node.js LTS on nodejs.org	24.18.0 or 24.18.1 across 3 runs	24.19.0	Search stale
Ranked restaurants on a page that 403s a plain request	1 real, 2 hallucinated, missed the top entry	the correct top 6	Search hallucinated

For a stable fact, a one-line search tool answers correctly, and a fetch loop is wasted work. In the rest, search answered incorrectly without warning, twice from a stale index and once from invented entries.

A plain requests.get() handles the first 3 rows at low volume. The API adds the proxy rotation that the last row needs.

What you'll need

Python 3.11 or newer. The libraries still run on 3.10, but 3.10 loses support in October 2026.
A Decodo account with Web Scraping API credentials. The free plan is enough to follow along.
An OpenAI API key from the platform dashboard. The script sets the model in one constant, so you can use another OpenAI model with function calling.

Install both libraries:

pip install requests openai

Activate a plan, and the Playground tab shows your Basic authentication token:

The script reads DECODO_TOKEN directly, and OpenAI() reads OPENAI_API_KEY automatically, so set both first:

export DECODO_TOKEN="your_decodo_token"
export OPENAI_API_KEY="your_openai_key"

Run these in the same terminal session you'll start the script from. They don't carry over once you close that session.

Step 1: Define the fetch tool

The agent gets one tool. The tool takes a URL, fetches the page through the API, and returns text the model can read.

Three request-body fields matter here. target: universal is the generic template for any URL. markdown: true converts the HTML before it reaches you, and the parameters reference recommends Markdown for LLM input. proxy_pool has to be premium for the conversion.

Here's the tool:

import os
import requests

API_URL = "https://scraper-api.decodo.com/v2/scrape"
DECODO_TOKEN = os.environ["DECODO_TOKEN"]
MAX_CHARS = 8000

def fetch_page(url):
    response = requests.post(
        API_URL,
        json={
            "target": "universal",
            "url": url,
            "proxy_pool": "premium",
            "markdown": True,
        },
        headers={"Authorization": f"Basic {DECODO_TOKEN}"},
        timeout=120,
    )
    response.raise_for_status()

    data = response.json()
    if "results" not in data:
        return f"Fetch failed: {data.get('message')}"

    return data["results"][0]["content"][:MAX_CHARS]

Markdown alone didn't shrink python.org's downloads page enough. In one fetch, the HTML measured 49,413 tokens on OpenAI's o200k_base tokenizer. The Markdown measured 22,542, roughly half. MAX_CHARS cuts that remainder by another 89%, and the first 8,000 characters measure 2,464 tokens.

The results check handles one specific failure. When the API accepts the request but can't scrape the target, it still returns HTTP 200, with a message and no results key, so raise_for_status() passes. Indexing into results then raises KeyError, which nothing catches, ending the run.

Returning the message as a string lets the model read it and try a different URL. A bad DECODO_TOKEN still raises, because the model can't fix it.

Step 2. Set up the agent loop

The model sees only the tools you put in the request. You describe the tool once, in the format the Responses API expects, then send it with every request:

import json
from openai import OpenAI

client = OpenAI()
MODEL = "gpt-5.4-mini"
MAX_TURNS = 5

TOOLS = [
    {
        "type": "function",
        "name": "fetch_page",
        "description": (
            "Fetch a web page and return its text as Markdown. "
            "Use this whenever the answer depends on what a page says now."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "url": {"type": "string", "description": "The full URL to fetch."}
            },
            "required": ["url"],
            "additionalProperties": False,
        },
    }
]

The tool's description tells the model when to call it. The parameters field is a JSON schema for the arguments, and the model fills them from the conversation. The name field comes back as call.name.

Nothing in TOOLS runs fetch_page. The Responses API returns the name and the arguments, and your code runs the tool:

def run(question):
    messages = [{"role": "user", "content": question}]

    for _ in range(MAX_TURNS):
        response = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            tools=TOOLS
        )
        message = response.choices[0].message
        messages.append(message)

        if not message.tool_calls:
            return message.content

        for call in message.tool_calls:
            args = json.loads(call.function.arguments)
            print(f"[tool call] {call.function.name}({args})")

            if call.function.name == "fetch_page":
                result = fetch_page(args["url"])
            else:
                result = f"No tool named {call.function.name}."

            print(f"[tool result] {len(result)} characters of Markdown")
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "name": call.function.name,
                "content": result,
            })

    return f"Stopped at the {MAX_TURNS}-turn limit without a final answer."

MAX_TURNS stops the loop after 5 turns. Using the tool costs at least 2 turns, one to ask for the page and one to answer. Count them before running a long list of questions.

Each result carries call.call_id, matching the result to its own call. call.arguments is a JSON string, not a dict.

The else branch matters because a model can ask for a tool you declared and never implemented. With search_web declared and no branch for it, the model called the tool, read the fallback string, then answered that it couldn't use the tool. Raising would have ended the run.

Step 3. Run a real query against a live page

Pick a question whose answer changes faster than anything can index it. Omit the URL. The Hacker News front page reorders through the day, so training data and search indexes lag behind it. The entry point takes the question from the command line:

import sys

if __name__ == "__main__":
    default_question = "What is the top story on Hacker News right now?"
    question = " ".join(sys.argv[1:]) or default_question
    print(f"[question] {question}")
    print(f"[answer] {run(question)}")

With no argument, 1 run printed:

[question] What is the top story on Hacker News right now?
[tool call] fetch_page({'url': 'https://news.ycombinator.com/'})
[tool result] 8000 characters of Markdown
[answer] The current top story on Hacker News is **“My server is a phone now”** with **169 points** and **71 comments**.

The question named no URL, so the model picked where to look. The title came from this line of the Markdown:

[My server is a phone now](https://seg6.space/posts/phone-server/) ([seg6.space](from?site=seg6.space))

When asked the same question with the built-in search tool, the model named 2 different stories across 3 runs, and neither was on the page. Twice, the model said it couldn't see the live front page, then answered.

The full script

Save the script as agent.py, then run python agent.py:

import json
import os
import sys
import requests
from openai import OpenAI

API_URL = "https://scraper-api.decodo.com/v2/scrape"
DECODO_TOKEN = os.environ["DECODO_TOKEN"]
MAX_CHARS = 8000
MAX_TURNS = 5

client = OpenAI()
MODEL = "gpt-5.4-mini"

def fetch_page(url):
    response = requests.post(
        API_URL,
        json={
            "target": "universal",
            "url": url,
            "proxy_pool": "premium",
            "markdown": True,
        },
        headers={"Authorization": f"Basic {DECODO_TOKEN}"},
        timeout=120,
    )
    response.raise_for_status()

    data = response.json()
    if "results" not in data:
        return f"Fetch failed: {data.get('message')}"

    return data["results"][0]["content"][:MAX_CHARS]

TOOLS = [
    {
        "type": "function",
        "name": "fetch_page",
        "description": (
            "Fetch a web page and return its text as Markdown. "
            "Use this whenever the answer depends on what a page says now."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "url": {"type": "string", "description": "The full URL to fetch."}
            },
            "required": ["url"],
            "additionalProperties": False,
        },
    }
]

def run(question):
    messages = [{"role": "user", "content": question}]

    for _ in range(MAX_TURNS):
        response = client.responses.create(model=MODEL, input=messages, tools=TOOLS)
        messages += response.output
        calls = [item for item in response.output if item.type == "function_call"]

        if not calls:
            return response.output_text

        for call in calls:
            args = json.loads(call.arguments)
            print(f"[tool call] {call.name}({args})")

            if call.name == "fetch_page":
                result = fetch_page(args["url"])
            else:
                result = f"No tool named {call.name}."

            print(f"[tool result] {len(result)} characters of Markdown")
            messages.append(
                {
                    "type": "function_call_output",
                    "call_id": call.call_id,
                    "output": result,
                }
            )

    return f"Stopped at the {MAX_TURNS}-turn limit without a final answer."

if __name__ == "__main__":
    default_question = "What is the top story on Hacker News right now?"
    question = " ".join(sys.argv[1:]) or default_question
    print(f"[question] {question}")
    print(f"[answer] {run(question)}")

Before running the script against any site, check the target's robots.txt and terms of service. urllib.robotparser reads robots.txt. The terms of service are a separate check. Copyright and privacy law can apply.

Common pitfalls

Sending raw HTML. The conversation carries every earlier tool result, so one large page is re-sent on every later turn. Misspelling markdown sends raw HTML, because the API validates values but not key names.
Truncating above the answer. MAX_CHARS cuts blindly at 8,000 characters, and one fetch kept 23 of the 30 front-page stories. Raise the cap when the fact you want sits lower.
Assuming the page is server-rendered. JavaScript rendering is important; that's why headless stays off by default. Without it, a client-side rendered page comes back nearly empty, and the call still succeeds. quotes.toscrape.com/js/ returned 172 characters until headless was set to html. The [tool result] line shows the count.
Trusting the URL the model returns. That widens the attack surface for indirect prompt injection. The fetched page enters the model's context, so its text can influence the next fetch and the answer you get. An allowlist in the fetch_page branch limits where the agent goes, not what it concludes.
Leaving the loop unbounded. The loop exits only on a response with no function call, and a model that keeps calling never sends one. With MAX_TURNS set to 1, the run returns Stopped at the 1-turn limit without a final answer.

These all return success, except the unbounded loop. Watch the character count, not the status code.

Where to take it from here

Add a search tool for stable facts. Decodo's Fast Search API is one option, with its own token.

A second tool is another entry in TOOLS and another branch next to fetch_page. The loop already runs tools as many times as the model asks, one at a time, within the MAX_TURNS limit.

To skip writing the loop, use Decodo's MCP server, which exposes a fetch tool your client can load directly.

Conclusion

Search wins on stable facts. A live fetch wins on pages that change or block bots. Neither reliably tells you when it's wrong. The loop is the easy part. The rest is knowing when to trust the answer.

Build Your First Web Scraper With the Decodo API in 15 Minutes

Decodo — Fri, 21 Aug 2026 10:49:19 +0000

The Decodo Web Scraping API fetches a page and returns the HTML. You send one POST request with the target URL, and the API handles proxy rotation, unblocking, and retries. In about 15 minutes, you'll have a Python script that calls the API and prints the titles and prices from a live page. No cost included, as the free plan covers it.

TL;DR

Copy the Basic authentication token from the Playground tab under Web Scraping API in the Decodo dashboard.
Send a JSON body with your target URL to https://scraper-api.decodo.com/v2/scrape using HTTP Basic auth, and set proxy_pool yourself – standard is cheaper and enough for static pages, while premium is for protected sites.
Read the page from results[0]["content"], which is a string of HTML by default, not a parsed object.
A failed scrape still returns HTTP 200, with "status": "failed" and no results key, so check for results instead of the HTTP status code.

What you'll need

Python 3.8 or newer
A Decodo account with Web Scraping API credentials
The Requests and Beautiful Soup libraries

Install both:

pip install requests beautifulsoup4

*The prices and measurements here are our own, taken in August 2026. The outputs are from real runs, so you can compare them with yours.

Step 1. Get your API credentials

Sign up and pick the free Web Scraping API plan, which gives you $1 of credit. The standard pool costs $0.50 per 1K, so $1 covers 2,000 requests, far more than this build needs.

The same $1 buys 2K standard requests or 667 with premium and JavaScript.

The dashboard tracks what's left of that credit. Open Web Scraping API in the sidebar. Its Pricing tab has the current rates and a Start for free button. Once you start, head to the Playground and find your Basic authentication token. The gear icon beside the token field opens Authentication settings, where you also find your username and password. You only need the token, so simply copy it by clicking on it.

Copy the token into TOKEN.

The token isn't a separate API key. It's the base64 encoding of your username and password, the value that HTTP Basic authentication sends anyway. If you have that pair, auth=(USERNAME, PASSWORD) in Requests builds the same header.

A placeholder constant is fine for a first run. Move the token into an environment variable before you share or commit the script.

Step 2. Make your first request

POST to https://scraper-api.decodo.com/v2/scrape. The body needs a URL, plus 2 more fields worth setting. target picks the template, and proxy_pool picks the IP pool. Use universal for a plain page fetch.

Here is the request on its own:

import requests

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = "YOUR_DECODO_TOKEN"

payload = {
    "target": "universal",
    "url": "https://books.toscrape.com/",
    "proxy_pool": "standard"
}

response = requests.post(
    API_URL,
    json=payload,
    headers={"Authorization": f"Basic {TOKEN}"},
    timeout=60
)
response.raise_for_status()

result = response.json()["results"][0]
print(result["status_code"])
print(result["content"][:120])

If you don't set proxy_pool, the API routes the request through the premium pool. On the free plan, premium costs 2x the standard rate, which cuts that credit from 2K requests to 1K. Standard is enough for a static sandbox page, and premium is for pages with bot protection. The parameters reference documents the rest of the fields.

The Playground pre-selects Premium. That's the pool you get when you don't set proxy_pool in the payload.

A successful call returns a results list. Each entry contains content and status_code, plus other fields. So a single call gives you two status codes. response.status_code comes from the API, and result["status_code"] comes from the target site.

The request makes two hops, and each one has its own status code. A 200 from the API can carry a 404 from the target.

The target's status code comes first:

200
<!DOCTYPE html>
<!--[if lt IE 7]>      <html lang="en-us" class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]

A failed scrape doesn't return a results list. The API still returns HTTP 200, but the body contains "status": "failed", an error code such as 613, and a human-readable message. So raise_for_status() passes, and Python raises a KeyError when you index into results. The full script guards against it.

Step 3. Parse the response and pull out data

content is a string of HTML by default, not a dictionary. If you call .get() on it, Python raises an AttributeError. Pass the string to Beautiful Soup, and you can search it with CSS selectors:

from bs4 import BeautifulSoup

soup = BeautifulSoup(result["content"], "html.parser")

print("Page title:", soup.title.get_text(strip=True))
print()

for book in soup.select("article.product_pod")[:5]:
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").get_text(strip=True)
    print(f"{price}  {title}")

The site truncates the link text on longer titles, and the markup shows "A Light in the …" rather than the full name. So the code takes the title from the title attribute instead. These selectors match books.toscrape.com as of August 2026.

That prints:

Page title: All products | Books to Scrape - Sandbox

£51.77  A Light in the Attic
£53.74  Tipping the Velvet
£50.10  Soumission
£47.82  Sharp Objects
£54.23  Sapiens: A Brief History of Humankind

The full script

Save this as scraper.py and paste your token into TOKEN:

import requests
from bs4 import BeautifulSoup

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = "YOUR_DECODO_TOKEN"

payload = {
    "target": "universal",
    "url": "https://books.toscrape.com/",
    "proxy_pool": "standard"
}

response = requests.post(
    API_URL,
    json=payload,
    headers={"Authorization": f"Basic {TOKEN}"},
    timeout=60
)
response.raise_for_status()

data = response.json()
if "results" not in data:
    raise SystemExit(f"Scrape failed: {data.get('message')}")

result = data["results"][0]
if result["status_code"] != 200:
    raise SystemExit(f"Target returned {result['status_code']}")

soup = BeautifulSoup(result["content"], "html.parser")

print("Page title:", soup.title.get_text(strip=True))
print()

for book in soup.select("article.product_pod")[:5]:
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").get_text(strip=True)
    print(f"{price}  {title}")

That status check is deliberately strict. Decodo counts 4xx as a successful retrieval when content comes back, so loosen it if you scrape pages that return 404 or 403 with a usable body.

Move from the sandbox to a commercial site

The sandbox page accepts all requests, so it doesn't show what the API is capable of. A direct request to a site like Tripadvisor listing page will get you a 403. Through the API, standard doesn't return the page, but premium does, so the payload below uses it.

Check what a live site allows before you point a scraper at it:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://www.tripadvisor.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://www.tripadvisor.com/Restaurants-g187147-Paris.html"))

That returns True for this path. robots.txt isn't the terms of service. Read those too before you scrape a commercial site.

The full script already handles the request. Three edits point it at Tripadvisor. First, add import re alongside the other imports at the top. Then change the payload:

payload = {
    "target": "universal",
    "url": "https://www.tripadvisor.com/Restaurants-g187147-Paris.html",
    "proxy_pool": "premium"
}

Next, replace everything from the soup = BeautifulSoup(...) line to the end of the file:

soup = BeautifulSoup(result["content"], "html.parser")
names = [a.get_text(strip=True) for a in soup.select("a[href*='/Restaurant_Review']")]
ranked = [n for n in names if re.match(r"^\d+\.\s", n)]

for name in ranked[:5]:
    print(name)

The requests.post call and both raise SystemExit guards in between stay exactly as they are.

One run prints:

1. Au Bourguignon Du Marais
2. Bistro L'Olivier
3. Le Poulbot
4. L'Oiseau Blanc Restaurant
5. Chez Pippo

The class names on the Tripadvisor page are non-semantic build hashes rather than named CSS classes. One of them is .biGQs, and a hash like that changes on the next release. The href pattern is more stable, and the numeric prefix separates ranked entries from the review snippets that link to the same page. Our notes on picking selectors that survive a redesign cover the wider trade-offs.

Premium doesn't return this page every time, so re-run before you start debugging. Rankings and markup both change, so treat the output as a sample rather than a fixture. This selector matches Tripadvisor.

Common pitfalls

A bad token stops the script at raise_for_status() with 401 Client Error: Unauthorized, which doesn't say why. Add print(response.text) above that line to see the reason:

401 Incorrect username or password – the token decodes correctly but doesn't match an account. A partial copy causes this.
401 Username invalid. – the token contains a character outside the base64 alphabet, such as a curly quote from a formatted document.
400 Authorization header is required – no credential reaches the API at all.

The API checks the token before the body, so fix auth errors first.

The rest look different:

ValueError: Invalid header value – the token includes a newline. Requests raises this before sending, so there's no response to inspect.
ModuleNotFoundError: No module named 'bs4' – install beautifulsoup4 but import bs4.
An empty list from select() – the selector doesn't match the markup. Check the class names on the live page.
KeyError: 'results' – the scrape itself fails, and the reason is in message.
429 Client Error: Too Many Requests – too much concurrency. The body reads Requests for this page type are temporarily limited.

How to extend the script

Reuse a single requests.Session() when you loop over a list of URLs. It cuts the time for 5 sequential calls from 11.2 to 7.3 seconds. On the free plan, the documented limit is 10 requests per second. When the list gets long, move to the async /v2/task/batch endpoint.

For a target that renders client-side, add "headless": "html" so the API runs JavaScript first. On the free plan, that takes standard from $0.50 to $0.75 per 1K. If the output goes to a model, set "markdown": true to get Markdown instead of HTML. On the sandbox page, that setting cut 51K characters to 10K. It needs the premium pool though, at $1.00 per 1K.

Some targets have a dedicated template rather than the generic one. If you pass an Amazon URL to universal, the API returns The provided 'url' is not supported for the selected target. Scraping Templates lists which sites have one.

There's enough credit left to point this at something you need. If something here doesn't work for your target, say so in the comments.

Conclusion

The request never changes: one POST, one token, one URL. The target decides everything else. The sandbox took the standard pool and a class-name selector. Tripadvisor needed premium, an href pattern, and a re-run when the page didn't come back.

Two habits carry over to whatever you scrape next. Check for results rather than the HTTP status, because a failed scrape still returns 200. And prefer selectors that survive a redesign, because the class names on a commercial site are build hashes.