DEV Community: Decodo

Stop Sending Raw HTML to LLMs

Decodo — Wed, 16 Sep 2026 12:31:42 +0000

A scraper gives you a page as one HTML string: markup, inline styles, scripts, and somewhere inside it the text you wanted. Many pipelines send that string directly to an LLM. That works, so the cost rarely gets measured. But on the 10 pages I measured, the HTML used 3 to 24 times more tokens than the same pages as Markdown. A product page arrived as 267,361 tokens, of which 3,393 were visible text. This guide shows where those tokens go, what they cost, and how to stop sending them.

TL;DR

Visible text was 0.3% to 18.5% of the tokens I fetched, and markup was the rest.
The cost was tokens, latency, and rate limits, not wrong answers. All 3 models I tested answered correctly from the raw HTML, and took up to 5.7 times longer to start.
Setting "markdown": true on Decodo's Web Scraping API returned the same pages with 3 to 24 times fewer tokens, keeping headings, links, and tables.
The same API returns the HTML when you need JSON-LD values like price and rating.

*Every number here is from my own runs, September 2026.

Where HTML tokens go

I fetched each page once with requests.get and a Chrome User-Agent, then removed one category at a time and re-counted with tiktoken's o200k_base. Each category is measured after the ones above it, so an SVG inside nav counts as SVG, and rows sum up to the fetched total:

6 of 10 pages. Markdown keeps structure, so it’s more than that share.

The Hydration JSON is the largest cost on the Next.js docs page, most of it in 65 self.next_f.push calls repeating the content as escaped strings. The Hacker News front page has almost no scripts or styles and is still 90% markup, because it's built from layout tables. Docs, news, and storefronts would each need a different rule.

What HTML waste costs

Tokens. At $2.00 per 1M input tokens, a rate several current models charge, those 10 pages cost $3.04 as HTML and $0.25 as Markdown. Per 1K of the heaviest pages, that's $1,157 instead of $49, plus scrapes at current rates. A different tokenizer changes the totals: across OpenAI, Anthropic, and Google the same HTML was up to 50% larger, while the pooled ratio stayed between 11× and 14×. So, trust the ratio and re-count the dollars for your model.

Time to first token. I asked one question per page on both versions, with a unique first line per call to prevent caching. On gpt-5.6-terra, the raw version took 2.0 to 2.8 times longer to start on the 4 pages above 130K tokens, and 1.0 to 1.4 times longer on the 4 below 75K. The other 2 models were slower, up to 5.7 times.

Rate limits. A 57,266-token page was refused on a gpt-4.1 account plan: Request too large … Limit 30000, Requested 57266. It had 622 tokens of text.

All 3 models across 2 vendors answered every question correctly from the HTML and Markdown, 60 answers each way. With no page attached they answered 1 or 2 of them, so they were reading rather than recalling. I put 4 raw pages into 396K-token prompts and got the right answer in all 3 runs.

What Markdown removes and what it keeps

Markdown conversion removes the markup around the content, including style blocks, SVG paths, class and data- attributes, and the scripts. It keeps headings, lists, tables, and link URLs. On the Hacker News front page, it returned Markdown tables with the story titles, URLs, scores, and comment counts in 3,644 tokens instead of 11,817.

Markdown keeps prose, and JSON-LD is in a script block. When I asked for a product's price, rating, and review count, the model got all 3 from the raw HTML and only the price from the Markdown, because the rest are in JSON-LD. On 3 of the 5 pages with it, ratings, stock status, and publication dates were absent. Markdown still kept 99.4% of that page’s words, because the rating and review count are 2 of 16,844.

Read the application/ld+json blocks with an HTML parser, since quoting varies, and send the model the prose.

Markdown directly from Decodo's API

Decodo's Web Scraping API takes a markdown flag that runs the conversion server-side, so the parsing happens outside your pipeline. The free plan starts immediately after signup.

DECODO_TOKEN is the Basic authentication token in the Playground tab, which builds the same request:

Create a virtual environment, install the packages, and export the token:

python3 -m venv .venv && source .venv/bin/activate
pip install requests tiktoken
export DECODO_TOKEN="paste-your-basic-auth-token"

Once you’ve set up your environment and .env file, create a script file with the following code:

import os
import requests
import tiktoken

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = os.environ["DECODO_TOKEN"]
# o200k_base is OpenAI's. Counts vary by vendor: use your model's encoding
enc = tiktoken.get_encoding("o200k_base")


def fetch(url, markdown):
    # Some targets reject markdown on the standard pool. Retry with "premium".
    payload = {
        "url": url,
        "proxy_pool": "standard",
        "markdown": markdown,
    }
    try:
        response = requests.post(
            API_URL,
            json=payload,
            headers={"Authorization": f"Basic {TOKEN}"},
            timeout=120,
        )
    except requests.RequestException as e:
        raise SystemExit(f"Request failed before a reply: {e}")
    if response.status_code >= 400:
        raise SystemExit(f"HTTP {response.status_code}: {response.text[:200]}")
    try:
        data = response.json()
    except ValueError:
        raise SystemExit(f"Reply was not JSON: {response.text[:200]}")
    if not data.get("results"):
        raise SystemExit(f"Scrape failed: {data.get('message')}")
    result = data["results"][0]
    # results[0] has the target's own status and the URL it reached, so a
    # site's 404 page or a redirect appears even under HTTP 200.
    code = result.get("status_code", 200)
    if code != 200:
        raise SystemExit(f"Target returned {code}, not the page")
    if result.get("url", url) != url:
        print(f"note: redirected to {result['url']}")
    content = result.get("content")
    if not content or not content.strip():
        raise SystemExit("Empty body: a shell or a challenge, not the page")
    return content


url = "https://en.wikipedia.org/wiki/Web_scraping"
# 2 calls here only to compare sizes. In a pipeline, make one call per page:
# markdown for the prose, or HTML when you also need the page's JSON-LD.
html = fetch(url, markdown=False)
md = fetch(url, markdown=True)
# disallowed_special=() keeps pages containing <|endoftext|> from raising
print(f"HTML:     {len(enc.encode(html, disallowed_special=())):>7,} tokens")
print(f"Markdown: {len(enc.encode(md, disallowed_special=())):>7,} tokens")

Running it prints both counts:

HTML:      72,201 tokens
Markdown:  16,553 tokens

Reduction across my 10 pages was 3× to 24×, median 7×. A ratio alone means little, since deleting content improves it. A local extractor reached 856× on a storefront listing by keeping 3.6% of its words, whereas this Markdown kept 99.4% at 24×.
Fetching is the harder half. From one home connection, a plain fetch reached the page on 4 of the 20 commercial sites I tested. It missed a job board at 59× and a property listing at 61×. Free html2text kept more of the words than the API's median 98.4% on pages a plain fetch reaches. The parameters reference lists the fields.

Decodo's MCP server offers this conversion as an agent tool.

Before you trust the output

A 200 that isn't the page. A plain GET to a home-improvement retailer returned HTTP 200 with an "Access Denied" body of 67 visible-text tokens, whereas the API on the premium pool returned the homepage, 3 runs of 3. A department store served a challenge script both ways, 152,000 tokens of it and no visible text. No single limit separated them – a real docs page had 622 visible-text tokens, fewer than a 909-token "browser not supported" page. Check the target's own status code in results[0], then compare each URL with its last good fetch.

Rendering worth confirming. "headless": "html" on a client-rendered page returned the rendered content in 12 of 14 runs. The other 2 returned the unrendered page, 62 tokens, at HTTP 200. The rendered page had 6 times more text.

Extractor coverage. A readability-style extractor kept a median 13% of these pages' words, and 0% on Hacker News. Another extractor kept 99.7% there, and the whole-page conversion kept 69.6%, its lowest. Test yours on a listings page.

Final thoughts

Raw HTML in a prompt is mostly markup. On the pages I measured, visible text was a median 2.3% of what got sent. That waste costs you a larger bill, a slower start, and a rate limit you didn't plan for, not a wrong answer on the 3 models I tested, so check accuracy on yours. Converting server-side removes the cleanup code from your pipeline, and the HTML response has JSON-LD for pages where you need exact fields. When a page needs rendering or a different pool, check the target's own status and the returned content, because both failures return HTTP 200. Run that snippet on a URL your pipeline already fetches, and compare the 2 numbers before you change anything.

How To Build an AI Agent That Browses the Web?

Decodo — Tue, 25 Aug 2026 13:17:19 +0000

Your model provider probably includes a web search tool. It's one line of code, and for a stable fact, it works. It goes wrong on a page that changes or blocks bots, and it rarely warns you. This guide builds the other kind of agent: one tool that fetches a page through Decodo's Web Scraping API, and a short loop that runs the tool.

TL;DR

Use a fetch tool when the answer changes faster than an index can follow, and leave stable facts to search.
Define one tool, fetch_page(url), that calls Decodo's Web Scraping API with markdown: true on the premium pool, and returns truncated text instead of HTML.
Send the question and the tool definition together, then check whether the response contains a function call rather than an answer.
Run fetch_page when the model asks, append the result to the conversation, and repeat until the model answers or the turns run out.

Do you need a fetch tool?

One question for each outcome, run both ways:

Question	Built-in web_search result	Live fetch result	Result
Price of Sharp Objects on books.toscrape.com	£47.82	£47.82	Both correct
Latest Python version on python.org	3.14.6, 4 runs out of 4	3.14.7	Search stale
Latest Node.js LTS on nodejs.org	24.18.0 or 24.18.1 across 3 runs	24.19.0	Search stale
Ranked restaurants on a page that 403s a plain request	1 real, 2 hallucinated, missed the top entry	the correct top 6	Search hallucinated

For a stable fact, a one-line search tool answers correctly, and a fetch loop is wasted work. In the rest, search answered incorrectly without warning, twice from a stale index and once from invented entries.

A plain requests.get() handles the first 3 rows at low volume. The API adds the proxy rotation that the last row needs.

What you'll need

Python 3.11 or newer. The libraries still run on 3.10, but 3.10 loses support in October 2026.
A Decodo account with Web Scraping API credentials. The free plan is enough to follow along.
An OpenAI API key from the platform dashboard. The script sets the model in one constant, so you can use another OpenAI model with function calling.

Install both libraries:

pip install requests openai

Activate a plan, and the Playground tab shows your Basic authentication token:

The script reads DECODO_TOKEN directly, and OpenAI() reads OPENAI_API_KEY automatically, so set both first:

export DECODO_TOKEN="your_decodo_token"
export OPENAI_API_KEY="your_openai_key"

Run these in the same terminal session you'll start the script from. They don't carry over once you close that session.

Step 1: Define the fetch tool

The agent gets one tool. The tool takes a URL, fetches the page through the API, and returns text the model can read.

Three request-body fields matter here. target: universal is the generic template for any URL. markdown: true converts the HTML before it reaches you, and the parameters reference recommends Markdown for LLM input. proxy_pool has to be premium for the conversion.

Here's the tool:

import os
import requests

API_URL = "https://scraper-api.decodo.com/v2/scrape"
DECODO_TOKEN = os.environ["DECODO_TOKEN"]
MAX_CHARS = 8000

def fetch_page(url):
    response = requests.post(
        API_URL,
        json={
            "target": "universal",
            "url": url,
            "proxy_pool": "premium",
            "markdown": True,
        },
        headers={"Authorization": f"Basic {DECODO_TOKEN}"},
        timeout=120,
    )
    response.raise_for_status()

    data = response.json()
    if "results" not in data:
        return f"Fetch failed: {data.get('message')}"

    return data["results"][0]["content"][:MAX_CHARS]

Markdown alone didn't shrink python.org's downloads page enough. In one fetch, the HTML measured 49,413 tokens on OpenAI's o200k_base tokenizer. The Markdown measured 22,542, roughly half. MAX_CHARS cuts that remainder by another 89%, and the first 8,000 characters measure 2,464 tokens.

The results check handles one specific failure. When the API accepts the request but can't scrape the target, it still returns HTTP 200, with a message and no results key, so raise_for_status() passes. Indexing into results then raises KeyError, which nothing catches, ending the run.

Returning the message as a string lets the model read it and try a different URL. A bad DECODO_TOKEN still raises, because the model can't fix it.

Step 2. Set up the agent loop

The model sees only the tools you put in the request. You describe the tool once, in the format the Responses API expects, then send it with every request:

import json
from openai import OpenAI

client = OpenAI()
MODEL = "gpt-5.4-mini"
MAX_TURNS = 5

TOOLS = [
    {
        "type": "function",
        "name": "fetch_page",
        "description": (
            "Fetch a web page and return its text as Markdown. "
            "Use this whenever the answer depends on what a page says now."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "url": {"type": "string", "description": "The full URL to fetch."}
            },
            "required": ["url"],
            "additionalProperties": False,
        },
    }
]

The tool's description tells the model when to call it. The parameters field is a JSON schema for the arguments, and the model fills them from the conversation. The name field comes back as call.name.

Nothing in TOOLS runs fetch_page. The Responses API returns the name and the arguments, and your code runs the tool:

def run(question):
    messages = [{"role": "user", "content": question}]

    for _ in range(MAX_TURNS):
        response = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            tools=TOOLS
        )
        message = response.choices[0].message
        messages.append(message)

        if not message.tool_calls:
            return message.content

        for call in message.tool_calls:
            args = json.loads(call.function.arguments)
            print(f"[tool call] {call.function.name}({args})")

            if call.function.name == "fetch_page":
                result = fetch_page(args["url"])
            else:
                result = f"No tool named {call.function.name}."

            print(f"[tool result] {len(result)} characters of Markdown")
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "name": call.function.name,
                "content": result,
            })

    return f"Stopped at the {MAX_TURNS}-turn limit without a final answer."

MAX_TURNS stops the loop after 5 turns. Using the tool costs at least 2 turns, one to ask for the page and one to answer. Count them before running a long list of questions.

Each result carries call.call_id, matching the result to its own call. call.arguments is a JSON string, not a dict.

The else branch matters because a model can ask for a tool you declared and never implemented. With search_web declared and no branch for it, the model called the tool, read the fallback string, then answered that it couldn't use the tool. Raising would have ended the run.

Step 3. Run a real query against a live page

Pick a question whose answer changes faster than anything can index it. Omit the URL. The Hacker News front page reorders through the day, so training data and search indexes lag behind it. The entry point takes the question from the command line:

import sys

if __name__ == "__main__":
    default_question = "What is the top story on Hacker News right now?"
    question = " ".join(sys.argv[1:]) or default_question
    print(f"[question] {question}")
    print(f"[answer] {run(question)}")

With no argument, 1 run printed:

[question] What is the top story on Hacker News right now?
[tool call] fetch_page({'url': 'https://news.ycombinator.com/'})
[tool result] 8000 characters of Markdown
[answer] The current top story on Hacker News is **“My server is a phone now”** with **169 points** and **71 comments**.

The question named no URL, so the model picked where to look. The title came from this line of the Markdown:

[My server is a phone now](https://seg6.space/posts/phone-server/) ([seg6.space](from?site=seg6.space))

When asked the same question with the built-in search tool, the model named 2 different stories across 3 runs, and neither was on the page. Twice, the model said it couldn't see the live front page, then answered.

The full script

Save the script as agent.py, then run python agent.py:

import json
import os
import sys
import requests
from openai import OpenAI

API_URL = "https://scraper-api.decodo.com/v2/scrape"
DECODO_TOKEN = os.environ["DECODO_TOKEN"]
MAX_CHARS = 8000
MAX_TURNS = 5

client = OpenAI()
MODEL = "gpt-5.4-mini"

def fetch_page(url):
    response = requests.post(
        API_URL,
        json={
            "target": "universal",
            "url": url,
            "proxy_pool": "premium",
            "markdown": True,
        },
        headers={"Authorization": f"Basic {DECODO_TOKEN}"},
        timeout=120,
    )
    response.raise_for_status()

    data = response.json()
    if "results" not in data:
        return f"Fetch failed: {data.get('message')}"

    return data["results"][0]["content"][:MAX_CHARS]

TOOLS = [
    {
        "type": "function",
        "name": "fetch_page",
        "description": (
            "Fetch a web page and return its text as Markdown. "
            "Use this whenever the answer depends on what a page says now."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "url": {"type": "string", "description": "The full URL to fetch."}
            },
            "required": ["url"],
            "additionalProperties": False,
        },
    }
]

def run(question):
    messages = [{"role": "user", "content": question}]

    for _ in range(MAX_TURNS):
        response = client.responses.create(model=MODEL, input=messages, tools=TOOLS)
        messages += response.output
        calls = [item for item in response.output if item.type == "function_call"]

        if not calls:
            return response.output_text

        for call in calls:
            args = json.loads(call.arguments)
            print(f"[tool call] {call.name}({args})")

            if call.name == "fetch_page":
                result = fetch_page(args["url"])
            else:
                result = f"No tool named {call.name}."

            print(f"[tool result] {len(result)} characters of Markdown")
            messages.append(
                {
                    "type": "function_call_output",
                    "call_id": call.call_id,
                    "output": result,
                }
            )

    return f"Stopped at the {MAX_TURNS}-turn limit without a final answer."

if __name__ == "__main__":
    default_question = "What is the top story on Hacker News right now?"
    question = " ".join(sys.argv[1:]) or default_question
    print(f"[question] {question}")
    print(f"[answer] {run(question)}")

Before running the script against any site, check the target's robots.txt and terms of service. urllib.robotparser reads robots.txt. The terms of service are a separate check. Copyright and privacy law can apply.

Common pitfalls

Sending raw HTML. The conversation carries every earlier tool result, so one large page is re-sent on every later turn. Misspelling markdown sends raw HTML, because the API validates values but not key names.
Truncating above the answer. MAX_CHARS cuts blindly at 8,000 characters, and one fetch kept 23 of the 30 front-page stories. Raise the cap when the fact you want sits lower.
Assuming the page is server-rendered. JavaScript rendering is important; that's why headless stays off by default. Without it, a client-side rendered page comes back nearly empty, and the call still succeeds. quotes.toscrape.com/js/ returned 172 characters until headless was set to html. The [tool result] line shows the count.
Trusting the URL the model returns. That widens the attack surface for indirect prompt injection. The fetched page enters the model's context, so its text can influence the next fetch and the answer you get. An allowlist in the fetch_page branch limits where the agent goes, not what it concludes.
Leaving the loop unbounded. The loop exits only on a response with no function call, and a model that keeps calling never sends one. With MAX_TURNS set to 1, the run returns Stopped at the 1-turn limit without a final answer.

These all return success, except the unbounded loop. Watch the character count, not the status code.

Where to take it from here

Add a search tool for stable facts. Decodo's Fast Search API is one option, with its own token.

A second tool is another entry in TOOLS and another branch next to fetch_page. The loop already runs tools as many times as the model asks, one at a time, within the MAX_TURNS limit.

To skip writing the loop, use Decodo's MCP server, which exposes a fetch tool your client can load directly.

Conclusion

Search wins on stable facts. A live fetch wins on pages that change or block bots. Neither reliably tells you when it's wrong. The loop is the easy part. The rest is knowing when to trust the answer.

Build Your First Web Scraper With the Decodo API in 15 Minutes

Decodo — Fri, 21 Aug 2026 10:49:19 +0000

The Decodo Web Scraping API fetches a page and returns the HTML. You send one POST request with the target URL, and the API handles proxy rotation, unblocking, and retries. In about 15 minutes, you'll have a Python script that calls the API and prints the titles and prices from a live page. No cost included, as the free plan covers it.

TL;DR

Copy the Basic authentication token from the Playground tab under Web Scraping API in the Decodo dashboard.
Send a JSON body with your target URL to https://scraper-api.decodo.com/v2/scrape using HTTP Basic auth, and set proxy_pool yourself – standard is cheaper and enough for static pages, while premium is for protected sites.
Read the page from results[0]["content"], which is a string of HTML by default, not a parsed object.
A failed scrape still returns HTTP 200, with "status": "failed" and no results key, so check for results instead of the HTTP status code.

What you'll need

Python 3.8 or newer
A Decodo account with Web Scraping API credentials
The Requests and Beautiful Soup libraries

Install both:

pip install requests beautifulsoup4

*The prices and measurements here are our own, taken in August 2026. The outputs are from real runs, so you can compare them with yours.

Step 1. Get your API credentials

Sign up and pick the free Web Scraping API plan, which gives you $1 of credit. The standard pool costs $0.50 per 1K, so $1 covers 2,000 requests, far more than this build needs.

The same $1 buys 2K standard requests or 667 with premium and JavaScript.

The dashboard tracks what's left of that credit. Open Web Scraping API in the sidebar. Its Pricing tab has the current rates and a Start for free button. Once you start, head to the Playground and find your Basic authentication token. The gear icon beside the token field opens Authentication settings, where you also find your username and password. You only need the token, so simply copy it by clicking on it.

Copy the token into TOKEN.

The token isn't a separate API key. It's the base64 encoding of your username and password, the value that HTTP Basic authentication sends anyway. If you have that pair, auth=(USERNAME, PASSWORD) in Requests builds the same header.

A placeholder constant is fine for a first run. Move the token into an environment variable before you share or commit the script.

Step 2. Make your first request

POST to https://scraper-api.decodo.com/v2/scrape. The body needs a URL, plus 2 more fields worth setting. target picks the template, and proxy_pool picks the IP pool. Use universal for a plain page fetch.

Here is the request on its own:

import requests

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = "YOUR_DECODO_TOKEN"

payload = {
    "target": "universal",
    "url": "https://books.toscrape.com/",
    "proxy_pool": "standard"
}

response = requests.post(
    API_URL,
    json=payload,
    headers={"Authorization": f"Basic {TOKEN}"},
    timeout=60
)
response.raise_for_status()

result = response.json()["results"][0]
print(result["status_code"])
print(result["content"][:120])

If you don't set proxy_pool, the API routes the request through the premium pool. On the free plan, premium costs 2x the standard rate, which cuts that credit from 2K requests to 1K. Standard is enough for a static sandbox page, and premium is for pages with bot protection. The parameters reference documents the rest of the fields.

The Playground pre-selects Premium. That's the pool you get when you don't set proxy_pool in the payload.

A successful call returns a results list. Each entry contains content and status_code, plus other fields. So a single call gives you two status codes. response.status_code comes from the API, and result["status_code"] comes from the target site.

The request makes two hops, and each one has its own status code. A 200 from the API can carry a 404 from the target.

The target's status code comes first:

200
<!DOCTYPE html>
<!--[if lt IE 7]>      <html lang="en-us" class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]

A failed scrape doesn't return a results list. The API still returns HTTP 200, but the body contains "status": "failed", an error code such as 613, and a human-readable message. So raise_for_status() passes, and Python raises a KeyError when you index into results. The full script guards against it.

Step 3. Parse the response and pull out data

content is a string of HTML by default, not a dictionary. If you call .get() on it, Python raises an AttributeError. Pass the string to Beautiful Soup, and you can search it with CSS selectors:

from bs4 import BeautifulSoup

soup = BeautifulSoup(result["content"], "html.parser")

print("Page title:", soup.title.get_text(strip=True))
print()

for book in soup.select("article.product_pod")[:5]:
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").get_text(strip=True)
    print(f"{price}  {title}")

The site truncates the link text on longer titles, and the markup shows "A Light in the …" rather than the full name. So the code takes the title from the title attribute instead. These selectors match books.toscrape.com as of August 2026.

That prints:

Page title: All products | Books to Scrape - Sandbox

£51.77  A Light in the Attic
£53.74  Tipping the Velvet
£50.10  Soumission
£47.82  Sharp Objects
£54.23  Sapiens: A Brief History of Humankind

The full script

Save this as scraper.py and paste your token into TOKEN:

import requests
from bs4 import BeautifulSoup

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = "YOUR_DECODO_TOKEN"

payload = {
    "target": "universal",
    "url": "https://books.toscrape.com/",
    "proxy_pool": "standard"
}

response = requests.post(
    API_URL,
    json=payload,
    headers={"Authorization": f"Basic {TOKEN}"},
    timeout=60
)
response.raise_for_status()

data = response.json()
if "results" not in data:
    raise SystemExit(f"Scrape failed: {data.get('message')}")

result = data["results"][0]
if result["status_code"] != 200:
    raise SystemExit(f"Target returned {result['status_code']}")

soup = BeautifulSoup(result["content"], "html.parser")

print("Page title:", soup.title.get_text(strip=True))
print()

for book in soup.select("article.product_pod")[:5]:
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").get_text(strip=True)
    print(f"{price}  {title}")

That status check is deliberately strict. Decodo counts 4xx as a successful retrieval when content comes back, so loosen it if you scrape pages that return 404 or 403 with a usable body.

Move from the sandbox to a commercial site

The sandbox page accepts all requests, so it doesn't show what the API is capable of. A direct request to a site like Tripadvisor listing page will get you a 403. Through the API, standard doesn't return the page, but premium does, so the payload below uses it.

Check what a live site allows before you point a scraper at it:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://www.tripadvisor.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://www.tripadvisor.com/Restaurants-g187147-Paris.html"))

That returns True for this path. robots.txt isn't the terms of service. Read those too before you scrape a commercial site.

The full script already handles the request. Three edits point it at Tripadvisor. First, add import re alongside the other imports at the top. Then change the payload:

payload = {
    "target": "universal",
    "url": "https://www.tripadvisor.com/Restaurants-g187147-Paris.html",
    "proxy_pool": "premium"
}

Next, replace everything from the soup = BeautifulSoup(...) line to the end of the file:

soup = BeautifulSoup(result["content"], "html.parser")
names = [a.get_text(strip=True) for a in soup.select("a[href*='/Restaurant_Review']")]
ranked = [n for n in names if re.match(r"^\d+\.\s", n)]

for name in ranked[:5]:
    print(name)

The requests.post call and both raise SystemExit guards in between stay exactly as they are.

One run prints:

1. Au Bourguignon Du Marais
2. Bistro L'Olivier
3. Le Poulbot
4. L'Oiseau Blanc Restaurant
5. Chez Pippo

The class names on the Tripadvisor page are non-semantic build hashes rather than named CSS classes. One of them is .biGQs, and a hash like that changes on the next release. The href pattern is more stable, and the numeric prefix separates ranked entries from the review snippets that link to the same page. Our notes on picking selectors that survive a redesign cover the wider trade-offs.

Premium doesn't return this page every time, so re-run before you start debugging. Rankings and markup both change, so treat the output as a sample rather than a fixture. This selector matches Tripadvisor.

Common pitfalls

A bad token stops the script at raise_for_status() with 401 Client Error: Unauthorized, which doesn't say why. Add print(response.text) above that line to see the reason:

401 Incorrect username or password – the token decodes correctly but doesn't match an account. A partial copy causes this.
401 Username invalid. – the token contains a character outside the base64 alphabet, such as a curly quote from a formatted document.
400 Authorization header is required – no credential reaches the API at all.

The API checks the token before the body, so fix auth errors first.

The rest look different:

ValueError: Invalid header value – the token includes a newline. Requests raises this before sending, so there's no response to inspect.
ModuleNotFoundError: No module named 'bs4' – install beautifulsoup4 but import bs4.
An empty list from select() – the selector doesn't match the markup. Check the class names on the live page.
KeyError: 'results' – the scrape itself fails, and the reason is in message.
429 Client Error: Too Many Requests – too much concurrency. The body reads Requests for this page type are temporarily limited.

How to extend the script

Reuse a single requests.Session() when you loop over a list of URLs. It cuts the time for 5 sequential calls from 11.2 to 7.3 seconds. On the free plan, the documented limit is 10 requests per second. When the list gets long, move to the async /v2/task/batch endpoint.

For a target that renders client-side, add "headless": "html" so the API runs JavaScript first. On the free plan, that takes standard from $0.50 to $0.75 per 1K. If the output goes to a model, set "markdown": true to get Markdown instead of HTML. On the sandbox page, that setting cut 51K characters to 10K. It needs the premium pool though, at $1.00 per 1K.

Some targets have a dedicated template rather than the generic one. If you pass an Amazon URL to universal, the API returns The provided 'url' is not supported for the selected target. Scraping Templates lists which sites have one.

There's enough credit left to point this at something you need. If something here doesn't work for your target, say so in the comments.

Conclusion

The request never changes: one POST, one token, one URL. The target decides everything else. The sandbox took the standard pool and a class-name selector. Tripadvisor needed premium, an href pattern, and a re-run when the page didn't come back.

Two habits carry over to whatever you scrape next. Check for results rather than the HTTP status, because a failed scrape still returns 200. And prefer selectors that survive a redesign, because the class names on a commercial site are build hashes.