Decodo for Decodo

Posted on Aug 21 Edited on Aug 25

Build Your First Web Scraper With the Decodo API in 15 Minutes

#webscraping #api #tutorial

The Decodo Web Scraping API fetches a page and returns the HTML. You send one POST request with the target URL, and the API handles proxy rotation, unblocking, and retries. In about 15 minutes, you'll have a Python script that calls the API and prints the titles and prices from a live page. No cost included, as the free plan covers it.

TL;DR

Copy the Basic authentication token from the Playground tab under Web Scraping API in the Decodo dashboard.
Send a JSON body with your target URL to https://scraper-api.decodo.com/v2/scrape using HTTP Basic auth, and set proxy_pool yourself – standard is cheaper and enough for static pages, while premium is for protected sites.
Read the page from results[0]["content"], which is a string of HTML by default, not a parsed object.
A failed scrape still returns HTTP 200, with "status": "failed" and no results key, so check for results instead of the HTTP status code.

What you'll need

Python 3.8 or newer
A Decodo account with Web Scraping API credentials
The Requests and Beautiful Soup libraries

Install both:

pip install requests beautifulsoup4

*The prices and measurements here are our own, taken in August 2026. The outputs are from real runs, so you can compare them with yours.

Step 1. Get your API credentials

Sign up and pick the free Web Scraping API plan, which gives you $1 of credit. The standard pool costs $0.50 per 1K, so $1 covers 2,000 requests, far more than this build needs.

The same $1 buys 2K standard requests or 667 with premium and JavaScript.

The dashboard tracks what's left of that credit. Open Web Scraping API in the sidebar. Its Pricing tab has the current rates and a Start for free button. Once you start, head to the Playground and find your Basic authentication token. The gear icon beside the token field opens Authentication settings, where you also find your username and password. You only need the token, so simply copy it by clicking on it.

Copy the token into TOKEN.

The token isn't a separate API key. It's the base64 encoding of your username and password, the value that HTTP Basic authentication sends anyway. If you have that pair, auth=(USERNAME, PASSWORD) in Requests builds the same header.

A placeholder constant is fine for a first run. Move the token into an environment variable before you share or commit the script.

Step 2. Make your first request

POST to https://scraper-api.decodo.com/v2/scrape. The body needs a URL, plus 2 more fields worth setting. target picks the template, and proxy_pool picks the IP pool. Use universal for a plain page fetch.

Here is the request on its own:

import requests

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = "YOUR_DECODO_TOKEN"

payload = {
    "target": "universal",
    "url": "https://books.toscrape.com/",
    "proxy_pool": "standard"
}

response = requests.post(
    API_URL,
    json=payload,
    headers={"Authorization": f"Basic {TOKEN}"},
    timeout=60
)
response.raise_for_status()

result = response.json()["results"][0]
print(result["status_code"])
print(result["content"][:120])

If you don't set proxy_pool, the API routes the request through the premium pool. On the free plan, premium costs 2x the standard rate, which cuts that credit from 2K requests to 1K. Standard is enough for a static sandbox page, and premium is for pages with bot protection. The parameters reference documents the rest of the fields.

The Playground pre-selects Premium. That's the pool you get when you don't set proxy_pool in the payload.

A successful call returns a results list. Each entry contains content and status_code, plus other fields. So a single call gives you two status codes. response.status_code comes from the API, and result["status_code"] comes from the target site.

The request makes two hops, and each one has its own status code. A 200 from the API can carry a 404 from the target.

The target's status code comes first:

200
<!DOCTYPE html>
<!--[if lt IE 7]>      <html lang="en-us" class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]

A failed scrape doesn't return a results list. The API still returns HTTP 200, but the body contains "status": "failed", an error code such as 613, and a human-readable message. So raise_for_status() passes, and Python raises a KeyError when you index into results. The full script guards against it.

Step 3. Parse the response and pull out data

content is a string of HTML by default, not a dictionary. If you call .get() on it, Python raises an AttributeError. Pass the string to Beautiful Soup, and you can search it with CSS selectors:

from bs4 import BeautifulSoup

soup = BeautifulSoup(result["content"], "html.parser")

print("Page title:", soup.title.get_text(strip=True))
print()

for book in soup.select("article.product_pod")[:5]:
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").get_text(strip=True)
    print(f"{price}  {title}")

The site truncates the link text on longer titles, and the markup shows "A Light in the …" rather than the full name. So the code takes the title from the title attribute instead. These selectors match books.toscrape.com as of August 2026.

That prints:

Page title: All products | Books to Scrape - Sandbox

£51.77  A Light in the Attic
£53.74  Tipping the Velvet
£50.10  Soumission
£47.82  Sharp Objects
£54.23  Sapiens: A Brief History of Humankind

The full script

Save this as scraper.py and paste your token into TOKEN:

import requests
from bs4 import BeautifulSoup

API_URL = "https://scraper-api.decodo.com/v2/scrape"
TOKEN = "YOUR_DECODO_TOKEN"

payload = {
    "target": "universal",
    "url": "https://books.toscrape.com/",
    "proxy_pool": "standard"
}

response = requests.post(
    API_URL,
    json=payload,
    headers={"Authorization": f"Basic {TOKEN}"},
    timeout=60
)
response.raise_for_status()

data = response.json()
if "results" not in data:
    raise SystemExit(f"Scrape failed: {data.get('message')}")

result = data["results"][0]
if result["status_code"] != 200:
    raise SystemExit(f"Target returned {result['status_code']}")

soup = BeautifulSoup(result["content"], "html.parser")

print("Page title:", soup.title.get_text(strip=True))
print()

for book in soup.select("article.product_pod")[:5]:
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").get_text(strip=True)
    print(f"{price}  {title}")

That status check is deliberately strict. Decodo counts 4xx as a successful retrieval when content comes back, so loosen it if you scrape pages that return 404 or 403 with a usable body.

Move from the sandbox to a commercial site

The sandbox page accepts all requests, so it doesn't show what the API is capable of. A direct request to a site like Tripadvisor listing page will get you a 403. Through the API, standard doesn't return the page, but premium does, so the payload below uses it.

Check what a live site allows before you point a scraper at it:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://www.tripadvisor.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://www.tripadvisor.com/Restaurants-g187147-Paris.html"))

That returns True for this path. robots.txt isn't the terms of service. Read those too before you scrape a commercial site.

The full script already handles the request. Three edits point it at Tripadvisor. First, add import re alongside the other imports at the top. Then change the payload:

payload = {
    "target": "universal",
    "url": "https://www.tripadvisor.com/Restaurants-g187147-Paris.html",
    "proxy_pool": "premium"
}

Next, replace everything from the soup = BeautifulSoup(...) line to the end of the file:

soup = BeautifulSoup(result["content"], "html.parser")
names = [a.get_text(strip=True) for a in soup.select("a[href*='/Restaurant_Review']")]
ranked = [n for n in names if re.match(r"^\d+\.\s", n)]

for name in ranked[:5]:
    print(name)

The requests.post call and both raise SystemExit guards in between stay exactly as they are.

One run prints:

1. Au Bourguignon Du Marais
2. Bistro L'Olivier
3. Le Poulbot
4. L'Oiseau Blanc Restaurant
5. Chez Pippo

The class names on the Tripadvisor page are non-semantic build hashes rather than named CSS classes. One of them is .biGQs, and a hash like that changes on the next release. The href pattern is more stable, and the numeric prefix separates ranked entries from the review snippets that link to the same page. Our notes on picking selectors that survive a redesign cover the wider trade-offs.

Premium doesn't return this page every time, so re-run before you start debugging. Rankings and markup both change, so treat the output as a sample rather than a fixture. This selector matches Tripadvisor.

Common pitfalls

A bad token stops the script at raise_for_status() with 401 Client Error: Unauthorized, which doesn't say why. Add print(response.text) above that line to see the reason:

401 Incorrect username or password – the token decodes correctly but doesn't match an account. A partial copy causes this.
401 Username invalid. – the token contains a character outside the base64 alphabet, such as a curly quote from a formatted document.
400 Authorization header is required – no credential reaches the API at all.

The API checks the token before the body, so fix auth errors first.

The rest look different:

ValueError: Invalid header value – the token includes a newline. Requests raises this before sending, so there's no response to inspect.
ModuleNotFoundError: No module named 'bs4' – install beautifulsoup4 but import bs4.
An empty list from select() – the selector doesn't match the markup. Check the class names on the live page.
KeyError: 'results' – the scrape itself fails, and the reason is in message.
429 Client Error: Too Many Requests – too much concurrency. The body reads Requests for this page type are temporarily limited.

How to extend the script

Reuse a single requests.Session() when you loop over a list of URLs. It cuts the time for 5 sequential calls from 11.2 to 7.3 seconds. On the free plan, the documented limit is 10 requests per second. When the list gets long, move to the async /v2/task/batch endpoint.

For a target that renders client-side, add "headless": "html" so the API runs JavaScript first. On the free plan, that takes standard from $0.50 to $0.75 per 1K. If the output goes to a model, set "markdown": true to get Markdown instead of HTML. On the sandbox page, that setting cut 51K characters to 10K. It needs the premium pool though, at $1.00 per 1K.

Some targets have a dedicated template rather than the generic one. If you pass an Amazon URL to universal, the API returns The provided 'url' is not supported for the selected target. Scraping Templates lists which sites have one.

There's enough credit left to point this at something you need. If something here doesn't work for your target, say so in the comments.

Conclusion

The request never changes: one POST, one token, one URL. The target decides everything else. The sandbox took the standard pool and a class-name selector. Tripadvisor needed premium, an href pattern, and a re-run when the page didn't come back.

Two habits carry over to whatever you scrape next. Check for results rather than the HTTP status, because a failed scrape still returns 200. And prefer selectors that survive a redesign, because the class names on a commercial site are build hashes.

DEV Community