How to Extract Website Data with Python and Scrapeless

TL;DR:

• Inspect the original HTML before choosing a browser.
• Select fields inside a record container to keep values aligned.
• Use Scrapeless Web Unlocker for managed acquisition and Agent Browser when the task requires interaction.

A website data extractor needs two things to be correct: the page it reads and the record it creates. A selector can be perfectly valid while operating on a challenge page. A successful network response can contain only the empty shell of a JavaScript application.

This tutorial builds the extraction layer first, then explains how to replace the acquisition layer when the target requires rendering. Keeping that boundary explicit lets you reuse field normalization, validation, and export code across different fetching methods.

Choose a Data Extraction Method

The fastest useful inspection is to compare the initial response with the browser's rendered page. If the fields are already in HTML, an HTTP client and parser are enough. If an authorized, documented endpoint returns those fields as JSON, use that endpoint and its schema. If scripts or interaction produce the data, use a rendering or browser path.

Method	Appropriate starting condition	Application responsibility
HTTP and HTML parsing	Required records exist in the response	Select, normalize, validate
Structured source API	Your source and operation are supported	Map the documented schema
Rendered HTML	JavaScript creates the records	Select from the returned DOM
Controlled browser	Clicking or session state is necessary	Drive actions and extraction

Do not assume that a source-specific API supports another part of the same website. A product-details operation and an autocomplete operation are different contracts. The Scrapeless product-selection guide describes the platform's acquisition surfaces.

Inspect HTML and Available Data Sources

Inspect one record and identify a stable container before extracting individual fields. On an educational quotes page, the container is .quote, with .text and .author beneath it. For a product page, use its actual public markup rather than copying these selectors.

BeautifulSoup parses HTML and XML. It does not run JavaScript. Requests performs the HTTP request and exposes the response for the parser.

Check page identity as well as element count. A page title, a known content container, and a plausible final URL together provide a better acquisition check than status_code == 200 alone. Decide whether an empty result is valid for that target or should be treated as a failed collection.

Extract a Static Page with Python

Install the dependencies, save the following program as extract_quotes.py, and run it in a directory where you want the output files created.

python -m pip install requests beautifulsoup4
python extract_quotes.py

The default target is a public scraping practice site. The optional HTML_FILE input lets you exercise the same parser with a saved capture. A saved fixture is a parser test, not evidence that a remote service fetched a live page.

import csv
import json
import os
from pathlib import Path
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

target_url = os.environ.get("TARGET_URL", "https://quotes.toscrape.com/")
capture_file = os.environ.get("HTML_FILE")
if capture_file:
    html = Path(capture_file).read_text(encoding="utf-8")
    source_url = target_url
else:
    response = requests.get(target_url, timeout=(5, 20))
    response.raise_for_status()
    html, source_url = response.text, response.url

soup = BeautifulSoup(html, "html.parser")
cards = soup.select(".quote")
if not cards:
    raise ValueError("Expected quote records were not present")

records = []
for card in cards:
    text = card.select_one(".text")
    author = card.select_one(".author")
    if text is None or author is None:
        raise ValueError("A record is missing a required field")
    record = {
        "text": text.get_text(" ", strip=True),
        "author": author.get_text(" ", strip=True),
        "source_url": source_url,
    }
    if not record["text"] or not record["author"]:
        raise ValueError("A required field is empty")
    records.append(record)

Path("quotes.json").write_text(
    json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8"
)
with open("quotes.csv", "w", encoding="utf-8", newline="") as handle:
    writer = csv.DictWriter(handle, fieldnames=["text", "author", "source_url"])
    writer.writeheader()
    writer.writerows(records)

next_link = soup.select_one("li.next a[href]")
next_url = urljoin(source_url, next_link["href"]) if next_link else None
print({"records": len(records), "next_url": next_url})

The record loop is deliberately scoped. Collecting all text nodes and all author nodes independently can produce arrays with different lengths. Selecting both fields within each card keeps the relationship intact and exposes malformed records immediately.

The program reports the next page but does not follow it automatically. To extend it, keep a set of visited URLs and a maximum page count. Resolve relative links against the page that supplied them, and limit traversal to the host and paths you intended to collect.

Use Rendered HTML or Browser Interaction

Use rendered HTML when the original response lacks the required records but the page can load them without a custom interaction flow. Scrapeless Web Unlocker can request an HTML representation after JavaScript execution.

Prerequisite: the next request requires your own Scrapeless API key. The endpoint and payload are documented; no authenticated result is implied.

import os
from pathlib import Path
import requests

response = requests.post(
    "https://api.scrapeless.com/api/v2/unlocker/request",
    headers={"x-api-token": os.environ["SCRAPELESS_API_KEY"]},
    json={
        "actor": "unlocker.webunlocker",
        "proxy": {"country": "ANY"},
        "input": {
            "url": "https://quotes.toscrape.com/js/",
            "jsRender": {
                "enabled": True,
                "response": {"type": "html"},
            },
        },
    },
    timeout=(10, 90),
)
response.raise_for_status()
payload = response.json()
if payload.get("code") != 200 or not isinstance(payload.get("data"), str):
    raise ValueError("Expected rendered HTML")
Path("rendered.html").write_text(payload["data"], encoding="utf-8")

Then run the parser with HTML_FILE=rendered.html and TARGET_URL=https://quotes.toscrape.com/js/ in its environment. Store the requested target URL with the capture, and preserve the final page URL separately if the service supplies it. The acquisition path changes; the field extraction rules stay with your application.

Use Agent Browser with Playwright when the task requires controlled actions. Wait for a meaningful record selector, perform the authorized interaction, and extract after the page reaches the intended state. A screenshot that looks complete is useful for inspection, but your code still needs to validate fields.

Normalize Fields and Preserve Source URLs

Normalize according to a declared schema. Keep identifiers as text, separate currency from numeric price, and preserve units. A blank rating is different from a rating of zero. An out-of-stock product is different from a product that failed to load.

Record the source URL and collection time outside the extracted field values. Preserve the original text when conversion might lose information. For example, keep a displayed price string alongside the normalized amount when the page includes ranges or a locale-specific separator.

Never turn a failed fetch into an empty successful dataset just to keep the export step running. Diagnostic records belong in a separate output with enough context to investigate the target and selected path.

Export CSV or JSON and Validate Records

CSV works well for flat records. JSON preserves nested structures and explicit nulls. Python's CSV module handles quoting and embedded commas so you do not need to join values by hand.

Validate the exported artifact too. Read it back, confirm that required columns exist, and compare its row count with the accepted records. For nested JSON, check the schema and type of each field before handing the file to another service.

Conclusion

Reliable extraction begins with the representation that actually contains the data. Keep fetching, parsing, validation, and export separate. That makes it easier to change an acquisition service without changing the meaning of your dataset.

FAQ

Why does BeautifulSoup return an empty list?

The selector may be wrong, the response may be the wrong page, or JavaScript may create the records after the initial HTML response. Inspect the capture before changing code.

Should every target use a browser?

No. Start with the least complex authorized path that contains the required fields.

Can this example collect every page automatically?

It reports the next page only. Add bounded traversal, host checks, and a visited-URL set before extending it.

What should change for another website?

Replace the target, record container, field selectors, page-identity check, and output schema. Keep the acquisition and validation boundaries.

Only collect data you are authorized to access, and review applicable terms and privacy requirements. This article provides technical information, not legal advice.