DEV Community: John Rooney

A data scientist's guide to stress-free product scraping

John Rooney — Thu, 08 Oct 2026 14:00:00 +0000

It’s one of the fastest-growing job categories in business - data scientist positions are expected to swell 34% by 2034.

Retail and e-commerce are embracing data science like no other industry because, in a competitive environment, price analytics has become a needle-mover.

As a data scientist, your job is to find patterns, build models, and generate insights. To do that, you first need to reliably acquire web data. Competitor pricing, product specifications, consumer reviews - you name it, data scientists need it.

The reality for data-gatherers, however, is that you spend 80% of your time scratching your head at 403 Forbidden responses, managing proxy pools, and angrily rewriting broken selectors when a target website adds a new field and changes up its CSS.

But the days of simply firing off a GET request and peacefully parsing DOM elements are now largely over. The modern web is hostile to automated scripts, and traditional scraping playbooks are fundamentally broken.

Let’s look at why the classic approach now fails, the hidden costs of modern workarounds, and how you can get back to actual data science using automated extraction.

Beautiful Soup vs. the modern web

Let me show you the traditional approach to web scraping. It's the first thing you learn in a web data tutorial: you grab the requests library to fetch the page and BeautifulSoup to parse it.

Here is what that looks like:

python

import requests
from bs4 import BeautifulSoup

url = "https://ecommerce-site.com/target-product"
response = requests.get(url)

soup = BeautifulSoup(response.content, "html.parser")
title = soup.select_one("h1").text

print(f"Status Code: {response.status_code}")
print(f"Title: {title}")

You expect to get a 200 OK response and a neatly formatted product name. Instead, you get a 403 Forbidden and a title that says "Just a moment..."

Welcome to the brick wall. Your script didn't fail because your code was wrong; it failed because the server looked at your fingerprint. You didn’t present a browser fingerprint, and because you didn’t render the page with JavaScript you were instantly categorised as a bot.

Down the rabbit hole

Proxies and infrastructure: You realise your IP might be being blocked, so you look into residential proxies. You quickly discover you have to build complex rotation logic, handle retries, and manage bans manually.
Headless browsers: Because the site relies heavily on JavaScript to render the actual product price, requests aren't enough. So, you spin up Selenium or Playwright. Suddenly, your resource consumption skyrockets, and your scraping speed plummets to a crawl.
HTML parsing: Even if you successfully manage the bans, you still have to write custom parsers for hundreds of different e-commerce layouts. And the moment a site updates its UI, your scraper breaks again.
LLMs aren’t as helpful as you’d think: You decide to feed the raw HTML into an LLM to extract the data. It works to a degree, but there are context size problems, you can’t verify the data quality, and when you calculate the token costs for 100,000 product pages you realise you’ll likely need a second mortgage.

Why bandwidth pricing is a trap (and why proxies aren't enough)

Let’s be honest, web scraping isn’t free anymore and to get data at scale, proxies are mandatory. But there is a massive misconception that simply buying a proxy pool will solve all your problems.

Traditional proxy providers charge by the gigabyte (ie, for bandwidth). This sounds fine until you realise what you are actually paying for:

Failed requests: When a site throws a CAPTCHA or blocks your IP, you still pay for the bandwidth of that failed response.
Retries: If it takes your scraper 10 attempts to successfully fetch a specific product page, you are paying for all nine failures.

Useless data: You are paying to download heavy CSS, tracking scripts, and images you don't even need for your dataset.

One call does it all

This is where Zyte API saves you. It is crucial to understand that Zyte API is way more than just a proxy manager. While it does manage massive pools of residential and datacenter proxies under the hood, it is an all-in-one extraction engine. It handles the browser rendering, manages the sessions, manages the anti-bot challenges, and takes all the pain away from accessing publicly available web data.

Because Zyte controls the whole pipeline, we offer a different pricing model: You only pay for successful requests. If Zyte has to rotate through five different IP addresses, spin up a headless browser, and retry a request multiple times to succeed, you don't pay for the retries. You only pay when the API returns a clean 200 OK with your data.

This predictability allows you to accurately forecast your costs (eg, 100,000 product pages equals a fixed cost), eliminating the unpredictable bandwidth tax of modern scraping. There’s still some nuance in which websites you want the data from, but we consistently come top in impartial benchmarks and tests.

Zyte API and AI Extraction

As a data scientist, you don't want to be a proxy manager, and you certainly don't want to be coding hundreds of HTML parsers. You just want structured data to do your job.

Zyte API acts as a single endpoint that abstracts away all the infrastructure headaches. But the real magic for e-commerce data is its AI Extraction feature.

By simply passing 'product': True in your payload, Zyte uses behind-the-scenes AI and ML models to read the page data, identify key e-commerce information into our extensive pre-defined schema, and return a clean JSON object automatically.

And if you need more, we also have a custom attributes feature for you to use.

In all this, you don’t need to specify CSS selectors, you don’t have to budget any spend on LLM tokens, and you don’t have to perform any browser management.

All of this costs you time. Time that could be spent working on your data analysis and providing value to your clients or employer. As e-commerce is fast-moving, up-to-date data is crucial, and a simple website change that would break your pipeline is handled by our AI Extraction tool, making sure you always get a consistent schema.

What does the modern, stress-free approach look like? It’s just one API call:

python

from base64 import b64decode
import requests

api_response = requests.post(
    "https://api.zyte.com/v1/extract",
    auth=("YOUR_API_KEY", ""),
    json={
        "url": "https://books.toscrape.com/catalogue/worlds-elsewhere-journeys-around-shakespeares-globe_972/index.html",
        "httpResponseBody": True,
        "product": True,
        "productOptions": {"extractFrom":"httpResponseBody","ai":True},
        "device": "desktop",
        "followRedirect": True,
    },
)

http_response_body: bytes = b64decode(api_response.json()["httpResponseBody"])
product = api_response.json()["product"]

Instead of wrestling with Beautiful Soup to find the right div, you get back instant e-commerce data in a dependable JSON schema.

Conclusion: Focus on data, not the extraction process

The landscape of web scraping has shifted. Building homegrown infrastructure to try to manage anti-bot systems is a losing game of whack-a-mole that drains your time, budget, and sanity. It’s a specialist skill that we have in droves, so let us help.

By offloading this to our all-in-one solution, Zyte API, you bypass the infrastructure bottlenecks and unpredictable bandwidth taxes entirely.

You stop building brittle scrapers and get back to what you were actually hired to do: building models and analysing data.

FAQ

Why is product scraping often considered stressful for data scientists?

Product scraping can be stressful because e-commerce websites are notoriously dynamic. Frequent layout changes, aggressive bans, and the sheer volume of data can lead to broken spiders and inconsistent datasets, requiring constant manual maintenance.

How does AI help in making web scraping stress-free?

AI-powered tools (like Zyte API's AI extraction) eliminate the need for manual selector maintenance. Instead of writing and fixing CSS or XPath selectors every time a website changes its design, AI can automatically identify and extract product names, prices, and descriptions by "understanding" the page structure, significantly reducing developer workload.

AI extraction

Should data science teams build their scraping tools in-house or outsource them?

While building in-house gives you total control, it often leads to high maintenance costs as you scale. For many teams, using a managed API is less stressful, as it allows data scientists to focus on analysing the data rather than the technical complexities of unblocking websites and maintaining infrastructure.

Is there a free trial for Zyte API

Yes, we offer a no credit card free trial with generous credit.

Originally published on the Zyte blog.

How to parse HTML tables into structured data (CSV/Excel)

John Rooney — Wed, 07 Oct 2026 14:00:00 +0000

You found the data you need. It's sitting right there in a neat HTML table. The problem is, it's stuck in a webpage — and you need it in a spreadsheet.

Most developers assume this is a five-minute job. Then they open the page source, see a tangle of <table>, <thead>, <tbody>, <tr>, and <td> tags — some with colspan attributes, some with nested tables inside them — and suddenly the five minutes becomes an afternoon.

In this guide, you'll learn three things: how HTML tables are actually structured (so the parsing makes sense), how to extract clean tabular data using Python, and how to export it to CSV or Excel without losing formatting or running into encoding nightmares.

Why HTML tables are harder to parse than they look

Here is a question worth answering before you write a single line of code: why do so many developers get tripped up by HTML table parsing when the data looks so simple on screen?

The answer is that what you see in the browser and what's in the markup are often very different things. Browsers do a lot of silent repair work — closing unclosed tags, inferring missing elements, normalising malformed attributes. Your parser does not.

A table that renders beautifully in Chrome might have no <tbody> element at all (browsers add it implicitly), or it might have colspan and rowspan attributes that merge cells visually but leave gaps in the underlying data structure.

There are three common structural issues you'll encounter:

Merged cells. When a cell spans multiple columns or rows using colspan or rowspan, the raw HTML contains fewer cells than the rendered table suggests. A naive row-by-row parser will produce rows of unequal length, and your resulting CSV will be misaligned.

Nested tables. Some websites use tables for layout purposes — there may be a table inside a table cell. You need to make sure you're targeting the right table and not accidentally extracting layout scaffolding alongside your data.

Missing or inconsistent headers. Not every HTML table uses a <thead> element. Some use the first <tr> of the <tbody> as a header row. Some have no headers at all. Your code needs to handle all three.

Understanding these issues upfront means you can choose the right tool for the job, rather than debugging cryptic misalignment errors halfway through.

Setting up your environment

Before writing any parsing code, install the two libraries you'll need:

pip install requests beautifulsoup4 pandas openpyxl lxml

A quick note on what each does:

requests fetches the raw HTML from a URL
beautifulsoup4 parses the HTML into a navigable tree
pandas gives you a DataFrame, which handles table manipulation and export
openpyxl is the engine pandas uses to write .xlsx files
lxml is a fast HTML parser that Beautiful Soup can use under the hood

If you're working with a page that requires JavaScript to render its tables, requests alone won't be enough — we'll cover that case later.

The simplest approach: pandas read_html

For straightforward tables on static pages, pandas has a built-in function that does most of the work for you.

import pandas as pd

url = "https://example.com/data-table"
tables = pd.read_html(url)

# read_html returns a list of all tables found on the page
print(f"Found {len(tables)} table(s)")

# Inspect the first one
print(tables[0].head())

pd.read_html() returns a list of DataFrames — one for each <table> element it finds on the page. If the page has three tables, you'll get a list of three DataFrames.

To export the one you want to CSV:

tables[0].to_csv("output.csv", index=False)

To export to Excel:

tables[0].to_excel("output.xlsx", index=False)

This approach works well when the table is simple and the page is static. When it doesn't work — because the table is dynamically rendered, you need more control over which table to target, or the structure is malformed — you'll need to go a level deeper.

Targeting a specific table with Beautiful Soup

pd.read_html() is convenient but not always precise. If the page has multiple tables and you need a specific one, Beautiful Soup lets you target it by its attributes.

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = "https://example.com/data-table"
response = requests.get(url)
soup = BeautifulSoup(response.text, "lxml")

# Target a table by its ID
table = soup.find("table", {"id": "results-table"})

# Or by a class name
table = soup.find("table", {"class": "data-grid"})

# Parse it with pandas
df = pd.read_html(str(table))[0]
df.to_csv("output.csv", index=False)

By converting the specific <table> element back to a string and passing it to pd.read_html(), you get the precision of Beautiful Soup with the convenience of pandas.

Parsing tables manually when the structure is complex

Sometimes you need full control — for example, when a table has colspan or rowspan attributes that pandas doesn't handle well by default, or when you want to map each cell to a specific header.

Here is a function that handles merged cells correctly by tracking which cells have already been "filled" by a spanning cell:

import requests
from bs4 import BeautifulSoup
import csv

def parse_table_with_spans(table):
    rows = []
    # Track cells occupied by rowspan/colspan
    occupied = {}

    for row_idx, tr in enumerate(table.find_all("tr")):
        row = []
        col_idx = 0

        for cell in tr.find_all(["td", "th"]):
            # Skip columns already occupied by a spanning cell
            while occupied.get((row_idx, col_idx)):
                row.append(occupied[(row_idx, col_idx)])
                col_idx += 1

            cell_text = cell.get_text(strip=True)
            colspan = int(cell.get("colspan", 1))
            rowspan = int(cell.get("rowspan", 1))

            # Fill occupied tracking dict
            for r in range(rowspan):
                for c in range(colspan):
                    if r > 0 or c > 0:
                        occupied[(row_idx + r, col_idx + c)] = cell_text

            row.append(cell_text)
            col_idx += colspan

        rows.append(row)

    return rows

url = "https://example.com/complex-table"
response = requests.get(url)
soup = BeautifulSoup(response.text, "lxml")
table = soup.find("table")

rows = parse_table_with_spans(table)

with open("output.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerows(rows)

This approach fills in the gaps left by spanning cells, so every row in your output has the same number of columns.

Handling pages that require JavaScript

If the table you need is rendered dynamically — meaning it doesn't appear in the raw HTML response but only after JavaScript executes — then requests will return an empty or incomplete page.

You have two options here.

The first is to check whether the data comes from an API. Open your browser's developer tools, go to the Network tab, filter by XHR or Fetch requests, and reload the page. If the table data is loaded via an API call, you may be able to hit that endpoint directly with requests and get the data as JSON — which is even easier to work with than HTML.

The second option is to use a browser automation tool like Playwright or Selenium to render the page before parsing it:

from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
import pandas as pd

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/dynamic-table")
    page.wait_for_selector("table")  # Wait until the table is in the DOM
    html = page.content()
    browser.close()

soup = BeautifulSoup(html, "lxml")
table = soup.find("table")
df = pd.read_html(str(table))[0]
df.to_csv("output.csv", index=False)

For sites with robust anti-bot protection, Playwright alone may not be enough. Zyte API handles browser rendering and unblocking as part of the same request, removing the need to manage a browser instance yourself.

Exporting cleanly to CSV and Excel

Once you have your DataFrame, a few export settings are worth knowing.

For CSV:

df.to_csv(
    "output.csv",
    index=False,        # Don't write row numbers
    encoding="utf-8-sig"  # utf-8-sig adds a BOM so Excel opens it correctly
)

The utf-8-sig encoding is worth noting: plain utf-8 CSVs often display garbled characters when opened in Excel on Windows because Excel expects a byte order mark (BOM) to know the file is UTF-8. Using utf-8-sig adds that marker automatically.

For Excel:

df.to_excel(
    "output.xlsx",
    index=False,
    sheet_name="Data"   # Name the sheet something meaningful
)

If you're writing multiple tables to the same Excel file, use an ExcelWriter object:

with pd.ExcelWriter("output.xlsx") as writer:
    tables[0].to_excel(writer, sheet_name="Table 1", index=False)
    tables[1].to_excel(writer, sheet_name="Table 2", index=False)

Handling common problems

Duplicate column names. If a table has two columns with the same header, pandas will automatically append suffixes (.1, .2) to disambiguate. You can rename columns afterwards with df.columns = ["col_a", "col_b", ...].

Mixed data types in columns. HTML tables store everything as strings. A column of numbers will be read as object type in pandas. Use pd.to_numeric(df["column"], errors="coerce") to convert, which will turn any non-numeric values into NaN rather than raising an error.

Empty rows or header rows in the body. Some tables repeat header rows partway through (common in long tables that span multiple printed pages). You can drop rows that duplicate your header with:

df = df[df["Column Name"] != "Column Name"]

Rate limiting and bot detection. If you're scraping multiple pages of tables, sending too many requests too quickly will often get you blocked. Add delays between requests with time.sleep(), rotate user agents, or use Zyte API's proxy capabilities to handle this transparently.

Putting it all together

Here is a complete script that fetches a page, finds the right table, handles encoding, and exports to both CSV and Excel:

import requests
from bs4 import BeautifulSoup
import pandas as pd

def scrape_table(url, table_id=None, table_class=None):
    headers = {"User-Agent": "Mozilla/5.0"}
    response = requests.get(url, headers=headers)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "lxml")

    if table_id:
        table = soup.find("table", {"id": table_id})
    elif table_class:
        table = soup.find("table", {"class": table_class})
    else:
        table = soup.find("table")

    if not table:
        raise ValueError("No matching table found on page")

    df = pd.read_html(str(table))[0]
    return df

df = scrape_table(
    url="https://example.com/data",
    table_id="results-table"
)

df.to_csv("output.csv", index=False, encoding="utf-8-sig")
df.to_excel("output.xlsx", index=False, sheet_name="Data")

print(f"Exported {len(df)} rows and {len(df.columns)} columns")

What to do when the table is too complex to parse reliably

Some tables are genuinely difficult: deeply nested structures, cells that contain embedded HTML, tables that are actually images, or pages where the same data appears across dozens of paginated tables.

At that point, the question becomes whether the cost of maintaining a custom parser is worth it. Zyte API's automatic extraction can extract structured data from pages without you having to write or maintain selectors — which is particularly useful when table structures change between scraping runs or across different pages of the same site.

For most standard use cases, though, the combination of Beautiful Soup and pandas described here will get you where you need to go.

Next steps

Parsing a table is often just the first step. Once you have clean structured data, you'll likely want to store it somewhere durable, schedule the scrape to run regularly, or combine it with data from multiple pages.

If the website you're scraping uses JavaScript-heavy rendering or has anti-bot measures in place, take a look at our guides on web scraping dynamic websites with Zyte API and managing bans in web scraping. If you want to run this at scale, our guide to large-scale web scraping with Python covers the infrastructure decisions worth making before you scale up.

Originally published on the Zyte blog.

Inside Scrapy's RemoteControl extension

John Rooney — Tue, 06 Oct 2026 14:00:00 +0000

Scrapy 2.19 added an extension called RemoteControl, and it's on by default, which means every crawl you start on the asyncio reactor is already running a small HTTP server that will execute Python inside the crawl for anything holding the right token. Scrapy MCP is the client it was built for, and the announcement post covers that side, but the extension doesn't need MCP at all. This post is about the piece underneath: how it starts, how it's secured, what the code you send can reach, and what you can actually do with it on a running crawl, from reading queues and testing selectors to fixing throughput without a restart. It ends with the traps, methods that look like reads but change state, and what changes once the crawl runs through scrapy-zyte-api. Every number below came from a real crawl, and every snippet works the same whether an agent sends it through Scrapy MCP or you send it with curl.

1. How RemoteControl works

RemoteControl is listed in Scrapy's default extensions and controlled by the REMOTE_CONTROL_ENABLED setting, which defaults to True. It needs asyncio support, so on a crawl without it the extension switches itself off with a warning rather than failing the crawl.

When the engine starts, the extension starts an aiohttp server bound to 127.0.0.1 on a random free port, generates a random bearer token, and only then writes a job file describing the crawl. You can see it happen in the crawl's own log:

uv run scrapy crawl books -o books.jsonl

[scrapy.extensions.remote_control] INFO: Remote control HTTP server listening on port 33093 (job 276910-b678ab03dbe24747a1fc0df7d3404417)

On Linux the job file lands in ~/.local/state/scrapy/job_files/, named after the process ID plus a random suffix, and the REMOTE_CONTROL_JOBS_DIR setting moves it somewhere else. The directory is created with 700 permissions and the file is written atomically with 600, because the token inside it is the only thing standing between a local process and code execution in your crawl:

{"version": 1, "pid": 276910, "port": 33093, "token": "f_lW...pkkXI", "spider": "books",
 "project": "mcp_demo", "scrapy_version": "2.19.0", "start_time": 1790316836.763313}

When the engine stops, the extension deletes the job file first and then shuts the server down. That only happens on a clean shutdown, though. A crawl that's killed or crashes leaves its file behind, and I found 23 of them on my machine from earlier runs, every one pointing at a process that no longer existed. Anything reading these files, Scrapy MCP's list_jobs() included, has to check the process is still alive before trusting one.

Two endpoints

The server has two routes, and both check the Authorization: Bearer header on every request, comparing it with hmac.compare_digest. Without the token you get nowhere:

curl -s http://127.0.0.1:33093/status

{"error": "unauthorized"}   (HTTP 401)

With it, GET /status describes the crawl:

curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:33093/status

{"pid": 276910, "spider": "books", "project": "mcp_demo", "scrapy_version": "2.19.0", "start_time": 1790316836.761879}

POST /execute takes a JSON body with code and an optional timeout_sec, runs the code inside the crawl, and returns whatever it printed:

curl -s -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"code": "print(crawler.stats.get_value(\"response_received_count\"))"}' \
  http://127.0.0.1:33093/execute

{"status": "ok", "output": "5\n", "traceback": null, "elapsed_sec": 0.0}

status is one of ok, error (the code raised, and traceback has the details), compile_error (it didn't parse), or timeout.

What your code can see

The code is compiled with top-level await allowed and run with three names already defined: crawler, the live Crawler instance; stash, a dictionary that persists between calls; and print, swapped for a version that writes into the response instead of the crawl's stdout. It runs on the crawl's own event loop, inside the same process, so there's no sandbox and no copy of the state. A print() reads the actual Crawler that's driving the actual requests, and a line that mutates something mutates the real crawl.

stash is what lets separate calls build on each other. Store a reading in one call, and a later call can pick it up:

stash["t0"] = crawler.stats.get_value("response_received_count")

now = crawler.stats.get_value("response_received_count")
print("then:", stash["t0"], "now:", now)

then: 5 now: 9

It's cleared when the crawl stops.

Limits, and the one that bites

Each call gets a timeout, 30 seconds by default and capped at 600 by REMOTE_CONTROL_TIMEOUT_DEFAULT and REMOTE_CONTROL_TIMEOUT_MAX, and output is truncated past 64 KB, tracebacks past 16 KB. An await asyncio.sleep(5) with timeout_sec set to 1 comes back as {"status": "timeout", ..., "elapsed_sec": 1.001}, exactly as you'd hope.

The catch is that the timeout can only cancel code at an await. Synchronous code runs to the end, and while it runs it holds the event loop that the whole crawl shares. The same test with time.sleep(4) instead:

{"status": "ok", "output": "", "traceback": null, "elapsed_sec": 4.0}

It ignored the one-second timeout, and a /status request I sent a second into it took three seconds to answer, because nothing else on that loop could run, the crawl included. Keep snippets short, prefer await asyncio.sleep() to time.sleep(), and don't loop over something large synchronously on a crawl you care about.

Where Scrapy MCP fits

Scrapy MCP is a thin layer over all of this: list_jobs() reads the job files, status(job_id) calls /status, execute(job_id, code) calls /execute, and inspection_reference() gives the agent a guide to Scrapy's internals so it knows what to ask for. Everything from here on is code sent to /execute, so it works the same from either.

2. Inspecting progress, queues, and active components

Once attached, the scheduler's queue classes and the downloader's per-domain slots are live objects, not documentation:

s = crawler.engine.scheduler
print("pqclass:", s.pqclass.__name__)
print("mqclass:", s.mqclass.__name__)
print("dqclass:", s.dqclass.__name__ if s.dqclass else None)
print("pending:", len(s))

d = crawler.engine.downloader
for key, slot in d.slots.items():
    print(key, "delay=", slot.delay, "concurrency=", slot.concurrency,
          "transferring=", len(slot.transferring), "queue=", len(slot.queue))

pqclass: DownloaderAwarePriorityQueue
mqclass: ScrapyRequestQueue
dqclass: ScrapyRequestQueue
pending: 0
books.toscrape.com delay= 3.0 concurrency= 1 transferring= 0 queue= 1

DownloaderAwarePriorityQueue is the current default. Reading its actual source: it picks the slot with the fewest active downloads, and when several slots are tied on that count, it round-robins among just those tied slots rather than always picking the same one. "Round-robin" alone undersells it, the active-download count is what decides most of the time; the round-robin only breaks ties. pending: 0 here just means the scheduler had already handed everything to the downloader, not that the crawl is idle.

The same live-object approach works for enabled middleware. Settings tell you what's configured; the actual middleware.middlewares tuple tells you what's installed, as real instances, in the order they run:

names = [type(m).__name__ for m in crawler.engine.downloader.middleware.middlewares]
print("downloader middlewares:", names)

downloader middlewares: ['OffsiteMiddleware', 'RobotsTxtMiddleware', 'HttpAuthMiddleware',
'DownloadTimeoutMiddleware', 'DefaultHeadersMiddleware', 'UserAgentMiddleware',
'RetryMiddleware', 'MetaRefreshMiddleware', 'HttpCompressionMiddleware',
'RedirectMiddleware', 'CookiesMiddleware', 'HttpProxyMiddleware', 'DownloaderStats']

If you already know which class you're after, crawler.get_downloader_middleware(Cls) (and the matching get_spider_middleware, get_item_pipeline, get_extension, get_addon) hands back that one instance directly, or None if it isn't enabled:

from scrapy.downloadermiddlewares.robotstxt import RobotsTxtMiddleware
mw = crawler.get_downloader_middleware(RobotsTxtMiddleware)
print("robots middleware instance:", mw)

robots middleware instance: <scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware object at 0x7f420566c980>

3. Testing a selector against a live response

The demo project's stalled spider fetches pages fine but never yields an item, and it holds its last response on self.last_response, which Scrapy doesn't do for you, so that a selector can be tested against real, already-fetched HTML, with no extra request.

print("item_scraped_count:", crawler.stats.get_value("item_scraped_count"))
print("response_received_count:", crawler.stats.get_value("response_received_count"))

sel = crawler.spider.last_response
print("used selector, matches:", len(sel.css("article.product-pod")))
print("candidate selector, matches:", len(sel.css("article.product_pod")))

item_scraped_count: None
response_received_count: 9
used selector, matches: 0
candidate selector, matches: 20

Nine responses in, zero items, and the crawl still looks healthy from the outside since pagination doesn't depend on the broken selector. The candidate selector matching 20 elements against the same live response is the actual fix, confirmed before touching the spider's source at all: article.product-pod should be article.product_pod.

Your own spider won't have last_response unless you add it, which is one line at the top of parse(): self.last_response = response. Without it, the same trackref module section 5 uses can still hand you a response that's in memory:

from scrapy.utils.trackref import get_oldest
resp = get_oldest("HtmlResponse")
print("response:", resp)
print("candidate selector, matches:", len(resp.css("article.product_pod")))

response: <200 https://books.toscrape.com/catalogue/page-1.html>
candidate selector, matches: 20

It returns the oldest response still referenced rather than the latest, and None if nothing is holding one, so check the URL it prints before trusting the result.

4. Diagnosing and fixing throughput without restarting

The books spider sets DOWNLOAD_DELAY = 3.0 and CONCURRENT_REQUESTS_PER_DOMAIN = 1 against a static site with no rate limit of its own. Measuring real throughput means reading a stat twice with a sleep in between, not trusting a setting:

import asyncio
before = crawler.stats.get_value("response_received_count", 0)
await asyncio.sleep(10)
after = crawler.stats.get_value("response_received_count", 0)
print("before:", before, "after:", after, "pages/min:", (after - before) * 6)

before: 7 after: 10 pages/min: 18

That's one call with an await inside it, which fits comfortably in the default 30-second timeout and doesn't hold up the crawl while it waits. For a longer window, keep the first reading in stash and take the second in a later call, as in section 1.

Eighteen pages a minute. Rather than stop the crawl, edit custom_settings, and restart it, patch the live slot directly. Slots are keyed by domain, so swap in one of the keys the section 2 snippet printed for your own crawl:

d = crawler.engine.downloader
slot = d.slots["books.toscrape.com"]
print("before patch, delay=", slot.delay)
slot.delay = 0.25
print("after patch, delay=", slot.delay)

before patch, delay= 3.0
after patch, delay= 0.25

Same measurement again:

before: 23 after: 51 pages/min: 168

Eighteen to a hundred sixty-eight, on a crawl that never stopped running. It finished entirely before I got to the next thing I wanted to try, which is its own kind of answer: sometimes the fix really is one attribute.

One thing worth being clear about: this is a live experiment, not a lasting fix. slot is recreated the next time the crawl restarts, so slot.delay = 0.25 only holds for this run. If the new number is actually correct, put it in the spider's custom_settings or the project's settings.py afterward. What you did live was confirm the number before committing to it, not replace the need to commit to it.

5. Checking what's still allocated

Scrapy tracks live instances of Request, Response, Selector, Item, and Spider through scrapy.utils.trackref. On a long-running crawl where memory keeps climbing, this tells you what's actually still referenced, instead of guessing:

from scrapy.utils.trackref import print_live_refs
import io, contextlib
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
    print_live_refs()
print(buf.getvalue())

Live References

BooksSpider                         1   oldest: 10s ago
HtmlResponse                        5   oldest: 9s ago
Request                             6   oldest: 10s ago
Selector                            5   oldest: 9s ago

RemoteControl only swaps out print inside the snippet you send, and print_live_refs() calls the real one, so without the redirect its output goes to the crawl's own stdout instead of back to you. A count that keeps growing across snapshots, especially Response, usually means something is holding a reference too long: most often a response pinned in request.meta, cb_kwargs, or a callback closure that never lets go.

6. Mutation traps, and the edge of what's safe to patch

Everything above was either a plain attribute read or, in section 4, changing one number on a live object and watching the effect. A few methods look exactly the same from the outside but change the crawl's state the moment you call them, purely to "check" something:

df.request_seen(req) records that fingerprint. Call it just to check whether a URL was already seen, and the crawl now skips that URL for real.
scheduler.next_request() doesn't peek at the next request, it dequeues it, handing it to whoever called the method instead of the engine. Use len(scheduler) for a count.
stats.set_value() / stats.inc_value(), engine.pause() / unpause(), and popping anything off mqs, dqs, or a slot's own queue are all mutations dressed as inspection.

Treat any method whose name suggests it advances, records, or pops something as off-limits unless mutating is actually the goal, and prefer reading an attribute over calling a method when both are available.

Section 4 changed a number on a live object; going further, into replacing a callback's actual code while the crawl runs, is a sharper and much less forgiving version of the same idea. Two things worth knowing before attempting it, verified against Scrapy 2.19.0, kept short because neither was reproduced live here:

Where a queued request's callback comes from depends on which queue it's in. A request sitting in the scheduler's in-memory queue holds the literal bound method it was created with, so reassigning that method on the spider afterward doesn't change requests already queued in memory. A request in a disk queue (JOBDIR set) is serialized by name and only resolved with getattr(spider, name) when it's dequeued, so it does pick up a reassigned method. Same "queued" request, two different behaviors, depending on which queue it landed in.
Swapping a callback's compiled code can defeat Scrapy's own source check. By default (WARN_ON_GENERATOR_RETURN_VALUE, on unless you turn it off), Scrapy calls inspect.getsource() on a generator callback once, caching the result so it isn't repeated on every call. If you replace that callback's __code__ with one compiled from a string sent to /execute, its filename is <execute>, the name RemoteControl compiles every snippet under, which is absent from linecache, so inspect.getsource() raises OSError for it. Scrapy's check only catches IndentationError, and because the error happens before the result gets cached, that cache never kicks in either, the lookup, and the failure, repeat on every call to that callback. The exception itself is loud: it surfaces as a logged spider_exceptions/OSError each time. What's lost quietly is whatever items or requests that call to the callback would have yielded.

Neither of these is a reason to avoid /execute. They're a reason to keep patching scoped to changing a value on an object you can already see, the way section 4 did, and to treat replacing a function's actual code as a different, much riskier category of change.

7. When the crawl runs through scrapy-zyte-api

Everything above assumes a fairly plain Scrapy crawl, fetching directly. Point the same inspection at a crawl running scrapy-zyte-api, and a couple of specifics change. Detectable from the live state: ZYTE_API_KEY in settings, ScrapyZyteAPIHTTPDownloadHandler and ScrapyZyteAPIHTTPSDownloadHandler registered for http and https in downloader.handlers if you use the scrapy-zyte-api add-on, or ScrapyZyteAPIDownloadHandler for both if you configured the settings by hand, and ScrapyZyteAPIDownloaderMiddleware in the enabled middleware list from section 2.

CONCURRENT_REQUESTS still applies. It's passed straight through as the connection pool size for the underlying Zyte API client, so the setting you already know still governs overall concurrency. What's genuinely different is the delay: Zyte API requests run on their own zyte-api@<domain> slot, and whether that slot's delay gets zeroed out depends on ZYTE_API_PRESERVE_DELAY, which defaults to the opposite of whatever AUTOTHROTTLE_ENABLED is set to. With AutoThrottle off, the delay is preserved and behaves like an ordinary slot. With AutoThrottle on, the delay is zeroed so AutoThrottle's own dynamic adjustment can drive it instead of a static DOWNLOAD_DELAY.

Retries and throttling also happen a layer lower than usual, inside the python-zyte-api client, before a Scrapy response exists at all, so read scrapy-zyte-api/* stats rather than assuming retry/count tells the whole story. Two different families of those stats answer two different questions: error_ratio and 429 count every attempt, including ones that were retried and then succeeded, so they describe how much friction the crawl is hitting. fatal_errors and success_ratio describe the final outcome after retries. A request throttled twice and then fine on the third try shows up in both: a real signal of load in the first family, and a completed, successful response in the second. Read both; they're not answering the same question.

Takeaway

RemoteControl is a small piece of code, two endpoints, a token and a job file, but it turns every running crawl into something you can ask questions of. Reading a live crawl through it is safe and genuinely useful: the scheduler's real queue classes, the downloader's actual slot state, a selector tested against a response already in memory, a stat sampled twice. Changing one number on a live object, the way section 4 did, is a live experiment you still have to commit to a setting afterward if it's right. Replacing a callback's actual code is a different, sharper category of change, with failure modes that depend on which queue a request happens to be sitting in. Know which one you're doing, and keep synchronous snippets short, because while they run the crawl doesn't.

Further reading: the RemoteControl source, Using Scrapy with coding agents, the Remote Control extension docs, scrapy/scrapy-mcp-official, and the scrapy-zyte-api docs.

Originally published on the Zyte blog.

Running Playwright at scale: connecting to the Zyte CDP browser

John Rooney — Mon, 05 Oct 2026 14:00:00 +0000

We've just released the Zyte CDP browser, so this post covers what it is, what it's good for, and, just as important, when you shouldn't use it. If you've got existing Playwright or Puppeteer scripts and you're tired of babysitting browser infrastructure, this is for you.

What is CDP?

CDP is the Chrome DevTools Protocol, a way of connecting to and controlling a running browser, and it's neither new nor exotic. Chrome DevTools itself uses this protocol every time you inspect network requests or run JavaScript in the console on an open page.

As web scrapers, we rarely touch CDP directly - instead we use Playwright or Puppeteer as a wrapper around the underlying CDP commands. I'll stick to Playwright here, and through it we can launch browsers, connect to them, run actions on pages, and everything else the job needs.

The problem with local browsers

The standard workflow looks like this: your script downloads a Chrome binary, launches it, and connects to it. This works fine on your laptop, but it falls apart the moment you need scale.

Browsers are resource-heavy, they eat memory, and running them in any real volume is an infrastructure project in its own right. You end up managing containers, memory limits, zombie processes, and crash recovery instead of writing scrapers, and if your scrapers already run in Docker or Kubernetes, the browser pods are the biggest and least reliable things in the cluster.

The fix is to run the browsers somewhere else, connecting to a remote browser on separate infrastructure while keeping your scripts exactly as they are. That's what the Zyte CDP browser is, browsers running on our infrastructure with your Playwright scripts driving them over CDP.

You also get our web scraping technology bundled in, so ban solving and fingerprinting come with the browser and you're not solving those problems yourself either.

An important distinction before you start

If all you need is rendered HTML and you don't have something already build, don't use this, use browserHtml with Zyte API instead. It's simpler and it's the right tool for that job.

The CDP browser makes sense when you have a Playwright script that performs many different actions, needs fine control over the browser, or runs through a flow where state has to carry from one step to the next. Zyte API actions can already handle some of this, but they're much more limited, and direct CDP access removes those limits.

Why we built it

This didn't come out of a product roadmap meeting, it came out of a run of customer conversations where the same few problems kept coming up.

Code nobody wanted to rewrite. One team scraping real estate listings had built their whole dynamic pipeline on Playwright and Puppeteer, and moving to Zyte API meant translating every one of those scripts into API calls or the Actions model. They looked at the size of that job and didn't want to do it, which is fair enough. With a CDP endpoint the scripts stay as they are, the connection line changes and nothing else does.

A second provider. A job-market data startup was already running on another vendor's hosted browser over CDP, and it was failing outright on certain drag-and-drop challenges. They didn't want to switch, they wanted a fallback for when the first option broke, and since the interface is CDP either way, swapping providers is a config change. They also wanted the browser to work out which proxy tier a site needs on its own, datacenter by default and residential only when the site forces it, which is how Zyte API already handles bans and the CDP browser inherits it.

Their sessions never needed to live longer than five minutes, by the way, they just wanted to connect, extract, and disconnect. Not every CDP use case is a long-running one.

Smart Proxy Manager users. SPM supported browser automation, and Zyte API's proxy mode didn't cover everything those users relied on, so with SPM being retired in 2026, people running Puppeteer or Playwright through it needed somewhere to go that didn't mean a rewrite. The CDP browser is that.

Fingerprinting. Websites look at hardware footprint, mouse movement, and keyboard timing, and a stock headless script advertises itself as a bot and gets banned on the first request. Running a customised browser on our side, with our fingerprint and IP handling behind it, takes that maintenance off your plate. I'm not going to claim it gets you everything, but it does mean the browser isn't the weakest link in your stack anymore.

Connecting Playwright to the Zyte browser

Here's the simplest possible example, where you connect with a CDP connection string and authenticate with your Zyte API key:

import asyncio
from base64 import b64encode
from playwright.async_api import async_playwright

ZYTE_API_KEY = "your-api-key"

CDP_URL = "https://browser.zyte.com"
AUTH = b64encode(f"{ZYTE_API_KEY}:".encode()).decode()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(
            CDP_URL,
            headers={"Authorization": f"Basic {AUTH}"},
        )
        page = await browser.new_page()
        await page.goto("https://example.com")
        print(await page.title())
        await browser.close()

asyncio.run(main())

That's it. connect_over_cdp is a standard Playwright method, and if you check its docstring it describes connecting via the Chrome DevTools Protocol, which is exactly what's happening here. Puppeteer's puppeteer.connect takes the same kind of websocket endpoint if that's your tool.

If you already have a Playwright script, this connection block is the only part you change to move from a local or other remote browser onto our infrastructure, and everything after the connection stays the same.

Interactions are where it gets useful

Rendering a page over CDP works, but the value really shows up when your script interacts with pages, because page.click, page.fill, waiting for selectors, anything Playwright exposes, all of it runs against the remote browser:

page = await browser.new_page()
await page.goto("https://example.com/search")

await page.fill("#search-input", "web scraping")
await page.click("#search-button")
await page.wait_for_selector(".results")

data = await page.locator(".results .item").all_text_contents()

The script connects to our infrastructure, runs the whole sequence, and returns the data, and the more complicated your actions, the more this pays off compared to expressing everything through API action definitions.

Blocking images to cut data transfer

One thing I do a lot is block images, since they're usually dead weight in a scraping job and blocking them saves a surprising amount of data transfer and loading time:

await page.route(
    "**/*.{png,jpg,jpeg,gif,webp,svg}",
    lambda route: route.abort(),
)

It's one line, and worth doing by default unless you need the images.

Screenshots

Playwright's page.screenshot runs against the remote browser like everything else:

screenshot = await page.screenshot(full_page=True)
with open("page.png", "wb") as f:
    f.write(screenshot)

Raw CDP commands

Playwright covers most needs, but a direct CDP session gives you access to browser internals that Playwright doesn't wrap, such as overriding the browser's timezone:

cdp = await page.context.new_cdp_session(page)
await cdp.send("Emulation.setTimezoneOverride", {
    "timezoneId": "America/New_York",
})

You can change a lot about the browser this way, and this level of control is the reason to have a direct CDP connection at all, since a request/response API can't give it to you.

Using it with Scrapy

If you're working in Scrapy, the scrapy-playwright extension normally downloads and launches a local browser, which is exactly the pattern I warned about at the start, so point it at our infrastructure with the CDP URL instead:

# settings.py
PLAYWRIGHT_CDP_URL = "https://browser.zyte.com" 

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

If you have existing Playwright scripts doing complicated actions and you want off local browsers, in a lot of cases it really is this straightforward, you add the CDP URL and the work moves to a remote browser.

Giving AI agents browser access

Claude wrote all the example code in this post's companion video, and that's the point here, because if you give an agent the CDP connection string and an API key, it can write its own Playwright scripts for a task and run them against a remote browser.

That means the machine your agent runs on can be tiny, since it doesn't need to run browsers itself when we handle that off-site. The agent writes code, the browser lives on our infrastructure, and you skip the whole problem of provisioning browser-capable compute for agent workloads.

When to use it, when not to

Use the CDP browser when:

You have existing Playwright scripts with many different actions
You need fine control over the browser, including raw CDP commands
Your flow spans several steps and the site has to remember state between them, such as a solved challenge, a login, or a filled form
You're moving off Smart Proxy Manager or another hosted browser and don't want to rewrite anything
You want agents to drive real browsers without hosting them

Skip it when you just need rendered HTML, because Zyte API with browserHtml is the better option there.

Links to pricing, access, and the community are below, and if you build something with this, come tell us what you used it for.

CDP Homepage

CDP Docs

Join our community

Originally published on the Zyte blog.

One Scrapy Spider, Three Browser Setups: Playwright, Patchright, and Zyte

John Rooney — Sun, 04 Oct 2026 14:00:00 +0000

I wanted to see how much difference the browser makes when the spider stays the same. So I ran one Scrapy spider against a test catalogue I'd built to challenge automated browsers, using stock Playwright, Patchright, and a remote browser on Zyte.

Stock Playwright reached a challenge page. Patchright and Zyte returned products. Here's how I configured each setup, and what this small test does, and doesn't, tell us.

The shared setup

Every scrapy-playwright setup starts the same way, in settings.py:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

That wires Playwright into Scrapy's download machinery, but it doesn't send every request through a browser. You opt in per-request:

async def start(self):
    yield scrapy.Request(
        "https://auto.hylnd7.com/",
        meta={"playwright": True, "playwright_include_page": True},
        errback=self.failed,
    )

playwright_include_page hands you the live Playwright Page object in the callback. That matters more than it looks. The response Scrapy gives you is a snapshot taken once, and it won't update as the page keeps running JavaScript. If you need to wait on something, wait on the page, not the response.

I built the test catalogue myself, specifically to challenge automated browsers. It's not a stand-in for a typical e-commerce site, and the results below are what happened against one page I deliberately hardened, not a general ranking of these browsers.

I also ran every stage headful (PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": False}, under xvfb-run on a headless server), since headless mode is one of the easiest things for a site to detect. The spider and selectors below stay identical across all three stages; only the browser setup changes.

Stage one: Playwright's local browser (the default)

Nothing extra to configure. PLAYWRIGHT_BROWSER_TYPE = "chromium" and Playwright launches a Chromium binary on the machine running Scrapy:

PLAYWRIGHT_BROWSER_TYPE = "chromium"
PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": False}

In the callback, we wait for the content we actually need, then hand the rendered HTML back to ordinary Scrapy selectors:

async def parse(self, response):
    page = response.meta["playwright_page"]
    try:
        await page.wait_for_selector(".product-card", timeout=20000)
    except PlaywrightTimeoutError:
        self.logger.warning("No product cards appeared; saving the browser result")
    html = await page.content()
    rendered = scrapy.http.HtmlResponse(url=page.url, body=html.encode(), encoding="utf-8")
    for card in rendered.css(".product-card")[:5]:
        ...
    await page.close()

The upside is that there's nothing to set up. No account, no extra service, no network hop to another provider. You install a browser binary and go, which is exactly why it's still the default.

The downside is that even running headful, a stock Chromium driven by Playwright still has a fingerprint, and this test catalogue checks for it. Here's what came back:

{
  "outcome": "no_products",
  "count": 0,
  "title": "Security Check",
  "url": "https://auto.hylnd7.com/"
}

Zero products. The browser reached the site, Playwright didn't error, and Scrapy happily returned a 200, but the page it rendered was a challenge screen, not the catalogue. This is the trap with browser scraping. An empty result and a blocked result look identical unless you actually inspect what loaded. That's why the spider saves a screenshot on every run: trust the pixels, not the exit code. Running headful wasn't enough on its own here.

Stage two: swap the browser with `PLAYWRIGHT_BROWSER_PROVIDER`

This is the new part. scrapy-playwright added a pluggable provider interface, and its docs now ship example classes for two popular open-source stealth browsers: Patchright (a patched Chromium that removes Playwright's automation tells) and Camoufox (a hardened Firefox build with the same goal). I only ran Patchright for this test; Camoufox is worth trying against your own targets with the same hook.

Add a small provider class to your project:

# browser_demo/providers.py
from contextlib import AsyncExitStack

class PatchrightProvider:
    """Local Chromium provider, following scrapy-playwright's documented example."""

    def __init__(self, config):
        self.config = config
        self.stack = AsyncExitStack()

    async def start(self):
        from patchright.async_api import async_playwright
        driver = await self.stack.enter_async_context(async_playwright())
        self.browser_type = driver.chromium

    async def launch_browser(self):
        return await self.browser_type.launch(**self.config.launch_options)

    async def close(self):
        await self.stack.aclose()

And one settings line switches to it:

PLAYWRIGHT_BROWSER_PROVIDER = "browser_demo.providers.PatchrightProvider"

The spider, the selectors, and the wait_for_selector call don't change. You can change browsers without changing your parsing code. Same headful setting as stage one, same spider. Here's what came back:

{
  "outcome": "success",
  "count": 5,
  "title": "Home - Auto Parts Demo Store",
  "url": "https://auto.hylnd7.com/"
}

Five products. Patchright got through this test. I still had to run the browser, install its dependencies, and supply the network connection myself.

Stage three: connect to a browser on Zyte over CDP

CDP is the Chrome DevTools Protocol, the same protocol Playwright already speaks to control a browser. The difference here is where that browser is running. Instead of launching one, you connect to one that's already alive on Zyte's infrastructure, and you don't need a custom PLAYWRIGHT_BROWSER_PROVIDER class to do it. The default provider that scrapy-playwright ships with already knows how to connect over CDP, so it's two settings:

from w3lib.http import basic_auth_header

PLAYWRIGHT_LAUNCH_OPTIONS = {}   # nothing to launch, the browser already exists
PLAYWRIGHT_CDP_URL = "https://browser.zyte.com/?ttl=60"
PLAYWRIGHT_CDP_KWARGS = {
    "headers": {"Authorization": basic_auth_header(key, "").decode("ascii")}
}

Authentication is Basic auth with your Zyte API key as the username and an empty password. The ttl query parameter bounds how long the remote session stays reserved, so set it to whatever your crawl actually needs. When PLAYWRIGHT_CDP_URL is set, the default provider's launch_browser() calls Playwright's own browser_type.connect_over_cdp() instead of launching a local binary, without adding a hosted service beyond Zyte itself.

Zyte returned five products too:

{
  "outcome": "success",
  "count": 5,
  "title": "Home - Auto Parts Demo Store",
  "url": "https://auto.hylnd7.com/"
}

Real catalogue title, no challenge page, the same result as headful Patchright. Zyte's infrastructure handles the browser fingerprint, session behavior, and proxy routing on its side.

One detail worth keeping if you go this route: closing the page normally only disconnects Playwright's client from the remote browser, it doesn't end the session. To actually close it, send the CDP command directly:

async def cleanup(self, page):
    if self.settings.get("STAGE") == "zyte":
        session = await page.context.browser.new_browser_cdp_session()
        await session.send("Browser.close")
    else:
        await page.close()

Also turn off automatic restarts, so a deliberate close doesn't just spin up another remote browser behind your back:

PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER = False

The browser fingerprint, session handling, and proxy management become Zyte's problem instead of mine, and no browser binary to install or launch option to tune. The catch is that it's a paid, external dependency with an account and business verification requirement, and this comparison changed more than one variable at once: the browser and the network location both moved. So it isn't proof that CDP alone is what got past the check.

A cost that doesn't show up in these numbers: running several contexts at once

Everything above ran one page at a time. I also tried scraping the whole catalogue instead of just the featured five, which meant opening several browser contexts concurrently to speed it up. One context at a time was fine; a handful running together locked up hard enough that I had to kill the process, no errors or timeouts, just stuck.

This isn't the first time I've run into this. Running a real browser, headful, with several contexts going at once on a server, tends to surface exactly this kind of resource contention, whether that's rendering, memory, or something lower in the stack, and it rarely shows up until you actually push concurrency. It's the part of "just run more browsers" that's easy to underestimate.

This is specifically a local-browser problem, not a Zyte CDP one. Zyte's browser infrastructure is built to take as many concurrent connections as you throw at it; the contention above comes from cramming multiple contexts onto one machine, which is exactly the part of the job CDP hands off to Zyte instead.

Takeaway

On this test catalogue, stock Playwright reached a challenge page, while Patchright and Zyte returned five products each. The spider and selectors stayed the same across all three runs.

The provider hook makes it easier to try another local browser, Patchright or Camoufox, without touching your extraction code. Connecting over CDP lets you move the browser to Zyte, with the cost and account requirements that brings. These results come from one target I built myself, but the useful part is how little extraction code had to change: none.

The concurrency hang above is the part I'd weigh most heavily if I were deciding between these options for anything beyond a single-page test. Running your own browser is easy to demo and harder to operate at load.

If you want the fuller case for the Zyte CDP browser, including raw CDP commands and giving agents browser access, I covered that in Running Playwright at scale: connecting to the Zyte CDP browser.

Originally published on the Zyte blog.

I built scraping agents for 30 days - here’s what I learned

John Rooney — Sat, 03 Oct 2026 14:00:00 +0000

The autonomous web data pipeline is dawning. Going deep on the frontier's state of the art surprised me, humbled me, and changed how I work.

The industry is abuzz with talk about autonomous web data-gathering agents. But what's the reality?

For the last 30 days, I did one thing almost exclusively: I built scraping systems with AI agents, from the ground up, across real targets, with real deadlines. Not prototypes designed to impress in a demo, not isolated experiments running against a toy website, but production-grade pipelines that needed to ship and keep running.

I went in with genuine curiosity and came out with something more useful than a hot take: a messy, hard-won picture of where agents genuinely changed my work, where they burned my time, and what I had to unlearn about how I thought development with AI would feel.

The biggest surprise: Agents really can build pipelines

I did not go into this expecting agents to handle the full build cycle. My assumption, going in, was that they would be useful for the boring middle parts: generating boilerplate, suggesting selectors, maybe scaffolding a spider I would then finish myself.

What I found instead was that, with the right tools and domain-specific context in place, an agent could take a target URL and produce a working, tested pipeline without me writing the bulk of the code. Not just a rough scaffold that got me 60% of the way there - a working pipeline, with selectors, page objects, item definitions, and a passing test suite.

The first time this happened cleanly I actually went back and **I checked the output, because I did not quite believe it. The spider ran, the items validated, and the tests passed.

I had been expecting to spend the entire afternoon on it, but it took 45 minutes, and most of that was me reviewing the output rather than the agent producing it. That shift, from writing to reviewing, is the one I keep coming back to when I think about what actually changed over these 30 days.

The caveat I discovered quickly, though, is that the words "right tools" are doing enormous work in that sentence.

The first few sessions where I pointed a general-purpose agent at a blank project and a URL were humbling, with hallucinated selectors, invented conventions, and code that looked plausible - until it completely fell apart on real pages.

It was only once I built out the proper scaffolding around the agent, a structured project to conform to, Zyte-specific skills and tooling, and well-defined context about how we build at Zyte, that the results became something I could trust. If you want a concrete look at that scaffolding, Web Scraping Copilot, the VS Code extension I was leaning on during this month, has since shipped as 1.0, built around exactly this idea of structure around the agent rather than a bare prompt.

Zyte's own research into what makes agents succeed or fail at web scraping tracks closely with what I experienced: specialized tooling and context engineering are what separate useful agents from expensive chaos.

The biggest time sink: Tool design

I did not anticipate that the biggest investment of the 30 days would be in building tools rather than in prompting or orchestration, but that is exactly what happened.

I ended up constructing a set of CLI wrappers around my most-used services - primarily Zyte API and related infrastructure - because **the raw responses were not shaped for an agent to reason with efficiently,* and I kept watching agents get confused by the volume and structure of what came back.*

Each tool I built was designed so that the data coming back to the agent was precisely what it needed for the task and nothing else: no raw HTML dumps, no verbose JSON payloads with 40 fields when three were relevant, no response structures that required the agent to do significant parsing just to understand what it had received.

The difference this made was not subtle. Before those tools, I was watching agents latch onto the wrong signal and carry that error confidently through the rest of their work. After, the reasoning was noticeably cleaner and the output was dramatically more reliable.

What I came to understand is that the bottleneck was never the model, it was the quality of the interface between the agent and its tools, and I had been underinvesting in that interface completely. The work of building good tools felt like plumbing at the time, but, looking back, it was the most valuable engineering I did across the entire 30 days.

# One of my CLI wrappers — the agent gets exactly what it needs, nothing else
zyte-fetch --url "https://example.com/products" --extract product --fields "name,price,sku"
# Returns clean JSON with exactly three fields, not 300

The mistake I made on day four that I am still thinking about

Around day four, an agent built a working spider in under 10 minutes and I shipped it without setting up the review loop I had been meaning to establish.

The spider ran cleanly, the logs looked fine, and I moved on to the next thing. Three hours later, I had 400 ingested records where the price field was either missing or formatted as a raw string with the currency symbol still attached.

It was all because I had not defined what a valid item actually looked like. The agent had made assumptions that were reasonable given what it had to work with, but wrong.

That experience reframed something for me. I had been thinking of good project structure as a professional habit, something worth doing but not load-bearing in the way that, say, a working selector is. What I discovered is that, when an agent is generating and modifying code on your behalf, your project structure becomes the contract the agent conforms tol; gaps in that contract become gaps in the output. The agent will not fill them with good judgment; it will fill them with plausible guesses.

After day four, I started treating typed item definitions, schema validation, fixture-based tests, and clear module boundaries as prerequisites rather than nice-to-haves, and the reliability of the agent's output improved noticeably as a result. Not because the agent got smarter, but because it had less room to go wrong. I later built an actual automated version of this instinct, a Scrapy pipeline that asks a cheap AI model whether each scraped field still looks real and stops the crawl when too many don't, in Stopping a crawl when the data stops looking real. That project exists because of exactly this day-four mistake.

# What my page objects ended up looking like after I tightened the schema contracts

from web_poet import WebPage, Returns, field
from scrapy_poet import handle_urls
from zyte_common_items import Product


@handle_urls("example.co.uk")
class ProductPage(WebPage, Returns[Product]):

    @field
    def name(self) -> str:
        return self.css("h1::text").get()

    @field
    def price(self) -> str:
        return self.css(".price::text").get()

    @field
    def sku(self) -> str:
        return self.css("[data-sku]::attr(data-sku)").get()

The speed is real, and I had to learn not to trust it blindly

By week two I was consistently going from a target domain to a deployed, running spider in under an hour on familiar site patterns, which is a compression of my previous workflow that I still find slightly disorienting to think about. The velocity is genuinely there, and I do not want to understate it.

What I had to learn, more slowly than I would like to admit, is that the speed applies uniformly regardless of output quality. An agent does not slow down when the extraction logic it has produced is brittle, does not flag that it is making assumptions about a selector pattern it has only seen once, and does not hesitate when the approach it is taking will break the moment the target site makes a minor update. It ships at full speed no matter what, which means the pressure to slow down and check has to come entirely from you.

The adjustment I eventually made was to stop treating agent sessions like batch processes I could kick off and return to, and to start treating them more like working alongside a very fast colleague who needed regular check-ins. Short cycles, deliberate review points, and a habit of actually reading the output before deploying it. That felt slower in the moment. Looking back at the pipelines I shipped with that approach versus the ones I did not, it was not.

Why I stopped thinking of human-in-the-loop as a limitation

My initial instinct was to treat the moments where I had to step in and verify something as signs that the agent was not good enough yet, as temporary friction that would disappear as the tooling matured. I do not think that anymore.

Over the 30 days I set up explicit checkpoints in my agent workflows where the agent would surface what it had done, what it was about to do, and where it had uncertainty, and I would review before it proceeded. Setting those up felt like admitting defeat at first. What I actually found was that they caught a meaningful number of decisions I would have made differently, and that the pipelines built with those checkpoints were more reliable and easier to debug than the ones I had let run freely. The checkpoints were not a workaround for agent limitations. They were part of the system design, and removing them would have made the system worse.

I built lightweight observability into every pipeline I shipped as well: item counts per run, field completeness rates, schema validation pass rates, and error ratios. Without those signals I found I genuinely could not tell the difference between a healthy run and one that was quietly degrading, and that gap between "it ran" and "it worked" is exactly where the expensive surprises live. The Web Scraping Copilot is built around the same idea, keeping AI-assisted development fast while keeping the developer in a position to actually know what is happening.

The moment that genuinely impressed me, and the one that humbled me

The most impressive thing I saw over the 30 days was **an agent diagnosing and repairing a broken spider after a target site updated its markup, without me touching a line of code.

The site had changed its product card structure, the extraction started failing, the agent detected the failures, inspected the new HTML, identified where the selectors had drifted, updated the page object, ran the tests against a new fixture, and confirmed the fix. The whole loop took a few minutes. I watched it happen in the logs.

For that class of problem, layout drift and minor structural changes, I came away genuinely convinced that agents are already good enough to handle maintenance work I used to dread.

The ability to detect and repair that kind of breakage automatically is one of the clearest wins I found across the whole 30 days. What I didn't have yet at the time was a clean way to give the agent that live access without wiring it up myself: I ended up building most of that diagnostic loop from scratch. Scrapy MCP now gives an agent exactly that kind of live connection to a running crawl out of the box, discovering jobs, reading engine state, and adjusting behaviour while it runs rather than only after a failure. I've written about what it actually lets you touch in more detail since.

The humbling counterpart came when I hit a target with unusual banning behavior. The agent could see that requests were failing but had no reliable way to diagnose why or adapt its approach in a way that actually worked.

That required me to dig into how Zyte API's unblocking was interacting with the site's specific defenses, and it required the kind of accumulated domain knowledge that does not live in a model. The Zyte post on why AI agents struggle with web scraping maps this ceiling out clearly if you want to understand the mechanics of why access problems are so much harder for agents than structural ones. My experience matched what they describe almost exactly.

What I actually do differently now

After 30 days, the honest version of what changed is not a framework for when to use agents. It is a set of instincts I did not have before.

I now reach for an agent immediately when I am starting something new on an unfamiliar domain, because the exploration work that used to eat half my day happens in minutes.
But I do not reach for one when I am working on a target with unusual access patterns, because I have learned that the agent will spin confidently and expensively without making real progress.
I invest time in tool design before I invest time in prompting, because the tool design pays dividends across every session that follows while the prompting only helps the one in front of me.
And I treat every agent session as collaborative rather than delegative, staying in the loop rather than handing off and walking away.

The 30 days did not resolve the question of whether agents are the future of scraping development, because I do not think that question has a clean answer yet. What they gave me is a much clearer sense of where they already earn their place in the workflow, which is more useful than a verdict.

For a sense of where this is all heading at a systems level, Zyte's piece on the dawn of the autonomous data pipeline is worth reading - the modular, agent-driven architectures they describe feel like a natural extension of what I was building toward by the end of the month, even if I was not thinking about it in those terms at the time.

Want to try Zyte API for free?

start here

Originally published on the Zyte blog.

Supercharging web scraping with Claude skills

John Rooney — Fri, 02 Oct 2026 14:00:00 +0000

As web scraping becomes more complex, the tools we use to navigate that complexity need to become smarter.

I recently took some time to explore how Claude skills can be integrated into a web scraping workflow. My goal was to see if I could improve the process and make data extraction significantly easier for myself.

What I found was that by building a specialized set of instructions, I could transform Claude from a simple coding assistant into a powerful scraping partner, especially for getting data quickly and providing a minimum viable product of the web scraping code.

What are Claude skills?

At its core, a Claude skill is a list of instructions in a folder that the AI can call upon whenever it receives a message from you.

Whenever you send a prompt, Claude checks whether any available skill is appropriate to follow. If so, Claude pulls these instructions into its context.

In itself, the skill is nothing more than a Markdown file. However, that file can also contain code excerpts and the skill folder can contain whole code files.

none

# My Skill

A short description of what this skill does.

## When to use
Use this skill when the user asks you to [trigger condition].

## Instructions
1. Run `script.py` with the provided input
2. Return the output to the user

This allows Claude to call on specific actions like running Python code, execute external scripts, or process data in a highly structured way that is repeatable and consistent.

Once it is called upon, the power of a skill lies in its ability to remain within the conversation context. This means you can query the output further, ask for modifications, or work with the extracted data in real-time without leaving the chat interface. It turns Claude into a specialized environment tailored for your specific technical needs.

Comparing skills to MCP servers

It is important to understand the difference between a skill and a Model Context Protocol (MCP) server. While both expand what the AI can do, they serve different purposes.

An MCP server is generally more like a piece of external tooling designed for a specific, broad task - like connecting to a database or a file system.

Skills, on the other hand, are more lightweight and flexible. They often consist of a single script or a focused set of instructions. Because they stay within the context of the AI, they are easier to iterate on. If you need a script to behave slightly differently for a specific website, you can adjust the skill or the prompt instantly. It provides a more agile way to handle the varying nature of web data.

Skills are also more specific to you and can be easily created and deleted, whereas an MCP server is designed to provide a specific purpose across many platforms and is often provided by companies for their products as integrations.

Top four scraping skills

Here are my favorite skills that I have been trying.

1. Fetcher

Automating HTML acquisition

The first step in any scraping project is getting the raw data. Traditionally, this is a manual, repetitive process. Given how many websites block requests, including directly from your large language model (LLM), you’d need to find another way.

To solve this I built this skill, it uses a Python script integrated with Zyte API to fetch the HTML of any URL you provide.

Why this makes life easier

Less manual work. You no longer have to manually copy and paste code. You give Claude a link, and it handles the retrieval, right inside the chat.
Avoiding blocks. By using a professional API within the skill, you reduce the risk of your request being blocked and being banned.
Speed. You can move from identifying a target to analyzing its code in seconds. This allows for a much faster exploration phase when you are starting a new project.

2. AI Parser

Cleaning data with AI extraction

Raw HTML is often messy. It is filled with navigation menus, footer links, tracking scripts, and styling blocks that have nothing to do with the data you actually want. This extra noise is more than just a distraction - it consumes valuable tokens/context and can make it harder for an AI to focus on the relevant content.

My second skill utilizes Zyte’s AI-powered automatic extraction to return only the main page content. This uses a machine learning model to strip away everything except the main header, body text, and footer. This greatly aids induction into your LLM’s context and provides more accurate results and less hallucination on answers.

How cleaning data saves time

Token efficiency. By removing the junk, you save a massive amount of space in Claude's context window. This allows you to process much longer articles or even multiple pages at once without hitting limits.
Better accuracy. LLMs perform better when they are given high-quality, relevant data. By removing the script tags and images, you ensure the AI focuses only on the information that matters.
Reduced processing time. You don't have to wait for the AI to read through thousands of lines of irrelevant CSS. The "purified" text is ready for immediate analysis.

3. SelectorGen

Generating resilient selectors

Once you have the HTML, you need to write selectors to tell your script exactly where the data lives. Writing these by hand is tedious and can be extremely brittle. If you pick a selector that is too specific, it may break if the site makes a tiny layout change, or even cause you to rewrite all your selectors.

I built a selector generator skill based on the Parsel library. Parsel is the HTML parser behind Scrapy, and it is incredibly efficient at navigating HTML structures.

Why this is powerful

Multiple backups. This skill is programmed to find multiple selectors for the same data point. If your primary selector fails, you have backup options ready to go.
Consistency. It outputs code in a repeatable, standardized style. This makes it very easy to manage your codebase because every scraper you build follows the same logic.
Ease of use. You can copy a snippet of the page's HTML source from your browser and ask the skill to write the selectors for you. It removes the guesswork and the need to manually test dozens of different CSS paths.

4. Extruct HTML

Extracting hidden data

Sometimes, the data you want isn't in the visible HTML at all. Many modern websites store their information in structured formats like JSON-LD or schema metadata hidden inside script tags. This data is often much cleaner and more reliable than the visible text.

I wrote a skill using a library called Extruct to target this specific metadata. It ignores the HTML tags and goes straight for the structured data objects.

Why this is useful

Structural stability. Websites change their visual layout all the time, but they rarely change their JSON-LD structure because that is what Google uses for search rankings. This makes your scrapers much more durable.
Zero parsing logic. Instead of writing complex rules to find a price or a product name, the skill simply hands you a clean JSON object.
Precision. It eliminates the risk of accidentally scraping "related products" or "sponsored content" because those items are rarely included in the primary schema metadata.

Building a workflow by chaining skills

The real strength of these skills is that they do not have to work in isolation. You can build a repository of these tools within Claude and chain them together to create a full automated pipeline.

The chaining can work in two ways depending on how much you want to automate. The simplest approach is manual, where you prompt Claude through each step in sequence, reviewing the output before moving on. This gives you full control and is useful when you're exploring an unfamiliar site.

For a more automated flow, you can embed logic directly into the skill instructions themselves, for example, telling the hidden data skill to automatically pass its output to the selector generator if no JSON-LD is found. This creates a more hands-off pipeline that handles common decision points without you needing to intervene.

In practice, most workflows are a mix of both. The early exploration phase tends to be manual, while the more repetitive extraction steps can be automated once you know what to expect from a site.

I prefer the manual approach, so a typical workflow might look like this:

First, you get the HTML.
Next, you ask if there is any hidden JSON data.
If there is, you are done. If not, you pass that HTML to generate a set of resilient selectors.
Finally, you can have Claude write a full Python script incorporating those selectors and the original fetching logic, giving you a working Python script you can use to carry on testing, or expand on to production code.

This chained approach makes the entire process repeatable. You aren't just getting a one-off answer; you are building a system that can be applied to almost any website with minimal adjustments.

Final thoughts

Web scraping will always require some level of manual oversight, but using Claude skills allows you to focus on the high-level strategy rather than the low-level grind. By automating the fetching, cleaning, and selector generation, you can be effective in your project much quicker.

These four skills have made my workflow significantly more efficient. They allow me to build scrapers that are more resilient and easier to maintain. Hopefully, you can find some benefit in these methods as you build out your own data extraction tools.

Originally published on the Zyte blog.

Zyte MCP: Powerful web data gathering for your agent

John Rooney — Thu, 01 Oct 2026 17:10:18 +0000

AI agents are increasingly where engineers plan, write and debug. But their work can often be thwarted at the first protected web page. The agent times out, mistakes a consent screen for the content or falls back on stale training data.

Even when it gets the page, a web data team still has to jump between code, documentation, dashboards and logs to run and diagnose the crawl.

Today, we’re introducing Zyte MCP Server: a hosted Model Context Protocol server that makes Zyte API and Scrapy Cloud available to MCP-compatible AI and coding agents.

We are exposing all of Zyte, inside your coding agent.

What Zyte MCP makes possible

Model Context Protocol, or MCP, gives AI applications a standard way to discover and invoke external tools. Zyte MCP uses that interface to bring reliable web access, structured extraction and crawl operations into environments such as Claude Code, Cursor, Codex, GitHub Copilot and VS Code.

Zyte MCP is not a new extraction engine. Under the hood, Zyte API still retrieves and extracts the data, while Scrapy Cloud runs spiders. MCP changes where you use those capabilities: inside the agent where the engineering work is already happening. Your agent will be able to:

Fetch protected and JavaScript-heavy pages through Zyte’s access infrastructure.
Return clean Markdown, with browser rendering, screenshots and geolocation available where needed.
Search the web and consume structured results.
Extract structured fields instead of processing a whole page.
Start, stop and monitor Scrapy Cloud jobs from existing deployed spider versions.
Inspect job states, statistics, logs and item samples to diagnose problems.
Create and manage recurring schedules.
Ask about usage, current spend and success rate by domain in natural language.

There is no separate MCP fee. Usage is metered through Zyte API and Scrapy Cloud as it is today.

Why it matters

Agents get the page they actually need

Built-in browsing works for ordinary pages; it is less dependable when a site uses anti-bot protection or requires browser rendering. Zyte MCP gives the agent a specialist web-access tool, so an answer can be grounded in the current page rather than a refusal, consent notice or model memory.

The connection’s real power comes from the underlying Zyte API, independently rated the industry’s #1 solution for unblocking and speed. With 320,000 access strategies and adaptive unblocking under the hood, Zyte API has the highest success rate in the game, now available through fluid, conversational interaction via MCP.

The work after the fetch stays in the agent

A fetch-only tool helps with prototypes. Production web data also needs jobs to be run, monitored and diagnosed.

Zyte MCP lets an agent inspect Scrapy Cloud state, logs, statistics and item samples in the same conversation where an engineer reviews the code. That shortens the path from “this crawl looks wrong” to an evidence-based diagnosis.

Clean content uses fewer tokens

Raw HTML contains navigation, scripts, styling and repeated page furniture that add cost without helping the model reason.

Zyte’s MCP can return clean Markdown instead of messy HTML.

In an internal benchmark of 35 HTML cleanup and HTML-to-Markdown tools plus three hosted services across 299 real pages, Zyte API’s shipped Markdown implementation retained 95% of the main content while removing 85% of unwanted page chrome and reducing HTML token volume by about 97%.

Five ways to use it

With Zyte MCP Server enabled in your agent as a connector, you can use Zyte fluidly and conversationally, with prompts like these:

1. Ground research in the live web

“Search for the latest official sources about [topic]. Fetch the five best pages, including any ordinary browsing cannot read, and cite a URL for every claim.”

The agent can search for sources, retrieve clean content and synthesise an answer from the pages themselves.

2. Extract current product data

“Extract the name, price, currency, availability and SKU from this product URL. Return JSON and flag any field that is missing from the page.”

The agent receives structured fields rather than parsing noisy HTML in its context window.

3. Diagnose a failed crawl

“Find the latest incomplete run for this spider. Compare it with the previous successful job, inspect the relevant errors and show three affected item samples.”

The agent gathers the evidence from Scrapy Cloud and helps distinguish an access problem from a spider or coverage problem.

4. Turn a working spider into a recurring job

“Start this deployed spider with these arguments. If the item sample is correct, schedule it for 06:00 every weekday.”

The agent carries out the routine operational steps through structured tools, subject to the user’s approval controls — developers stay in control throughout.

5. Understand where your spend is going

“Where did most of my spend go in the last billing period? Break it down by product and by domain, and highlight any anomalies.”

The agent can pull usage and billing signals into a quick summary, so you can spot cost drivers and unexpected spikes without manually digging through dashboards.

Compose it with the rest of your stack

MCP-enabled tools can work together around one outcome. For example:

Pair GitHub and Zyte MCP to review a spider change, start the deployed version and inspect the result.
Zyte already has plugins for Claude Code, Codex, Visual Studio Code and GitHub, plus a range of agent-neutral agent skills. Use your own tools to build spiders, fetch the most defended pages and run your web scraping operations without leaving your coding agent.
Send a daily crawl-health summary to Slack or another messaging tool, including only jobs that need attention.
Fetch difficult-to-access sources and save a cited research brief to a knowledge base.
Search, fetch and extract web data before passing an approved sample to an analytics or storage destination.

How to connect

https://www.youtube.com/watch?v=wbaVOWvCUgc

Find our MCP documentation in the Zyte API docs.

Zyte MCP is hosted, so there is no server package to maintain.

Add this remote Streamable HTTP endpoint to a compatible MCP client:

https://mcp.zyte.com/v1/mcp

A typical configuration is:

{
  "mcpServers": {
    "zyte": {
      "type": "http",
      "url": "https://mcp.zyte.com/v1/mcp"
    }
  }
}

For Claude Code, the expected command is:

claude mcp add --transport http zyte https://mcp.zyte.com/v1/mcp

On first connection, complete the Zyte sign-in flow. OAuth gives the client a short-lived, scoped token instead of placing a permanent Zyte API key in the agent’s configuration.

Then ask for an outcome rather than a tool name:

“Fetch this page and return the main content as Markdown.”

“Extract the current product price and availability from this URL.”

“List my recent failed Scrapy Cloud jobs and explain the most common error.”

What’s next

Check out the docs for full details of capabilities, setup instructions, billing and usage.

More tools and capabilities for our MCP server are in the pipeline.

And we have ambitions for agentic web data beyond even MCP.

Fluid data gathering is already here. Connect your client to mcp.zyte.com/v1/mcp and ask your agent to fetch a page it could not read before.

Stopping a crawl when the data stops looking real

John Rooney — Tue, 22 Sep 2026 16:46:13 +0000

TL;DR

I built scrapy-jev, a Scrapy pipeline that asks a fast, cheap AI model whether each scraped field still looks like a real product name, price, SKU, or category, and stops the crawl when too many items fail the check. It works, and it caught the kind of breakage that normally goes unnoticed until someone opens the exported CSV. It also has a narrow job: it catches values that look wrong, not selectors that point at the wrong thing but happen to still return something plausible.

The problem I was trying to solve

Scrapers don't announce when they break. A site renames a CSS class, or restructures a product card, and the extractor starts returning the wrong text, or nothing at all, while every other signal in the crawl looks healthy. No exception fires. Nothing appears in the log. You find out when someone downstream opens the export and the price column is full of category names.

What I wanted was something that could ask, mid-crawl, "does this still look like a real value for this field?" and stop before that damage spread across thousands of pages, rather than finding out afterward from a customer complaint or a broken pipeline.

What I built it with

I wrote the client, pipeline, and add-on with DeepSeek V4 Pro doing most of the coding, working from a spec of the endpoint and the pipeline behavior I wanted.

The core idea only works if judging a field is cheap and fast enough to run inline, on every crawl, without turning into its own cost center. A general-purpose LLM generating a paragraph of reasoning per field would have been too slow and too expensive to run at any real scale.

I used Jev, a "System One" model from TypeSafe AI, reached through OpenRouter's alpha endpoint. Instead of generating text, Jev returns typed, calibrated decisions. You send it a state and one or more typed questions, and it returns structured answers with probabilities attached, in roughly 70 to 500 ms.

I wrote more about what Jev actually is, and where a model like it fits into web scraping, in an earlier post.

For each field, I sent a two-level score question, "does this look like a plausible, correctly-scraped value for this field, or not," which meant the returned score was directly interpretable as P(plausible). One request carried one question per field, evaluated against the whole item at once, so a five-field item cost one call, not five.

On top of that I built three pieces:

JevClient, a thin wrapper that reads item fields through itemadapter, so it works against Scrapy Item, plain dicts, attrs, and dataclasses, and returns a score per field.
QualityGatePipeline, an item pipeline that scores a sample of items, requires every field on an item to clear a threshold to count as a pass (not a majority, since one bad field should fail the item), and stops the spider if the sample's pass rate falls below a configured threshold.
Addon, so installing the package doesn't require hand-editing ITEM_PIPELINES. It self-disables if no fields are configured, so adding the dependency does nothing until you turn it on.

Two design choices were deliberate rather than obvious. First, judging a fixed-size sample rather than every item keeps the cost of the check independent of crawl size. A 1,000-page crawl and a 100,000-page crawl spend the same amount on verification. Second, a missing field counts as a failure rather than being skipped, because a selector silently returning None is the most common way scrapers actually break, and skipping it would have defeated the entire point.

I checked the actual dollar cost against real calls logged during testing rather than guessing. A three-field scoring request, one field was missing and never sent, ran to 570 input tokens and cost $0.000024, which works out to $0.042 per million input tokens, output is free. Scaled to a full four-field item across a 1-million-item crawl, that's roughly 700 million input tokens, somewhere around $30. Scoring a fixed sample of 20 items per crawl instead, regardless of whether the crawl has 1,000 items or 1,000,000, costs a fraction of a cent.

At Jev's actual pricing, the dollar difference between sampling and scoring everything isn't the main argument, $30 per million items is trivial either way. The real argument for sampling is that scoring every item means every item's pipeline waits on a round trip to an external API, and the crawl now depends on roughly a million of those calls all succeeding rather than twenty.

Trying it yourself

Installing it doesn't touch ITEM_PIPELINES by hand:

uv add scrapy-jev

Turning it on is three lines in settings.py: register the add-on, and tell it which fields to judge.

ADDONS = {
    "scrapy_jev.Addon": 350,
}

JEV_FIELDS = ["name", "price", "sku", "category"]

JEV_API_KEY, or the OPENROUTER_API_KEY environment variable, has to be set too. If neither is present, or JEV_FIELDS is empty, the add-on raises NotConfigured and gets out of the way rather than doing nothing silently.

The rest is optional and has defaults tuned for a small verification sample rather than exhaustive checking:

Setting	Default	Meaning
`JEV_FIELDS`	`[]`	Fields to judge on each item.
`JEV_SAMPLE_SIZE`	`20`	Items to judge before deciding.
`JEV_PASS_RATE_THRESHOLD`	`0.7`	Minimum share (0–1) of items that must pass.
`JEV_FIELD_THRESHOLD`	`0.5`	Per-field plausibility below which a field fails.
`JEV_MODEL`	`~typesafe/jev-latest`	Model id for the decisions endpoint.
`JEV_BASE_URL`	`https://openrouter.ai/api/alpha/decisions`	Endpoint.
`JEV_PIPELINE_PRIORITY`	`400`	`ITEM_PIPELINES` priority used by the add-on.
`JEV_API_KEY` / `OPENROUTER_API_KEY`	—	API key (setting or environment variable).
`JEV_STOP_REASON`	`"Extraction quality fell below threshold (Jev plausibility check)"`	Closing reason logged when the gate stops the spider.

The bug that mattered more than the feature

The most useful thing I learned building this had nothing to do with AI. My first version raised CloseSpider from inside the pipeline to stop the crawl, and the crawl didn't stop. Scrapy's item-processing path wraps process_item in a bare except Exception, logs "Error processing item", drops the item, and keeps going. CloseSpider is an Exception subclass, so it gets swallowed like any other error. The gate fired every time, and the crawl just kept going anyway, for another thousand pages, retries and all.

The fix was to stop asking politely and instead tell the engine directly:

from scrapy.utils.defer import deferred_from_coro
deferred_from_coro(crawler.engine.close_spider_async(reason=reason))

That's not a Jev problem or an AI problem. It's the kind of thing you only find by actually watching a crawl you expected to stop keep running, and it's a reminder that the flashy part of a project is rarely where the real debugging happens.

Whether it actually worked

I tested it against a demo target, a fictional 1,000-product coffee shop with markup I'd deliberately built to drift, crawled through scrapy-poet page objects.

Breaking the sku selector so it returned the product title instead: Jev scored sku around 0.08 to 0.35 while the other fields stayed above 0.9, and the gate stopped the crawl.
Breaking the name selector so it returned nothing: every item in the sample failed, and the crawl stopped with a clear reason logged.
Renaming the price CSS class: the gate didn't catch it, and correctly so. The demo's price extraction was a regex against a currency-prefixed string, not a class-based selector, so the rename didn't actually break anything. The check didn't cry wolf.

Here's what one of those failures actually looked like in the logs, an item where name came back None while the other three fields scored a clean 1.0:

2026-09-21 16:08:28 [scrapy_jev.client] INFO: Jev response: {'model': 'typesafe/jev-1.13-20260917', 'answers': {'price': {'type': 'score', 'score': 1, 'legend': {'0': 'Does not look like a real value for this field', '1': 'Looks like a plausible, correctly-scraped value for this field'}, 'probabilities': {'0': 0, '1': 1}, 'confidence': 0.99}, 'sku': {'type': 'score', 'score': 1, 'legend': {'0': 'Does not look like a real value for this field', '1': 'Looks like a plausible, correctly-scraped value for this field'}, 'probabilities': {'0': 0, '1': 1}, 'confidence': 0.99}, 'category': {'type': 'score', 'score': 1, 'legend': {'0': 'Does not look like a real value for this field', '1': 'Looks like a plausible, correctly-scraped value for this field'}, 'probabilities': {'0': 0, '1': 1}, 'confidence': 0.99}}, 'usage': {'input_tokens': 570, 'output_tokens': 43, 'cost': 2.394e-05}, 'id': 'gen-dec-1790003308-TkGiCpI2ZIe3KtT8fFSn', 'provider': 'TypeSafe'}
2026-09-21 16:08:28 [scrapy_jev.pipeline] WARNING: scrapy_jev: item failed quality check: name=None

name isn't in the answers at all, because a missing value never gets sent to Jev in the first place. JevClient only asks about fields that have something to judge, and the pipeline treats the absence itself as the failure. price, sku, and category came back at confidence 0.99, which is exactly what should happen when three fields are fine and only one selector is broken.

That last result mattered more than the first two. A quality gate that fires on cosmetic changes gets disabled by the second team it annoys. This one only fired when the extracted value was actually wrong.

Where it doesn't help

This is not a general defense against scraper drift, and I don't want to oversell it as one.

It only catches breakage that produces an implausible-looking value. A selector that starts pointing at the wrong product on the page, but returns something that's still shaped like a real name and a real price, will sail through untouched. Jev has no way to know the name belongs to the wrong item. It's a plausibility check, not a correctness check.

It's sample-based, which is the right tradeoff for cost but means a crawl can run through a chunk of bad items before or after the sampled window and never trigger the gate, depending on where the breakage falls relative to the sample.

It also adds a hard dependency on an external API mid-crawl, OpenRouter's alpha endpoint in this case, not even TypeSafe's own. That's one more thing that can be slow, rate-limited, or unavailable at exactly the moment you need the crawl to keep running smoothly. I didn't build in a fallback behavior for that; right now a Jev outage just means the gate isn't checking anything, silently, which is close to the same failure mode the whole project set out to fix.

The thresholds are guesses too: sample size, pass rate, per-field cutoff, all tuned against one demo site. I have no evidence yet that a 0.7 pass rate and a 0.5 per-field threshold are the right defaults for a real production catalog with more natural variance in its data.

Was it worth it

As a package: yes, with caveats. scrapy-jev is published, on PyPI, with trusted publishing so there's no stored token to leak, and it does the one thing it claims to do. It caught two different kinds of real selector breakage in testing and correctly ignored a change that didn't matter. That's a genuine, narrow win, and the packaging overhead, src layout, CI, changelog, was small next to the core pipeline work.

As a general answer to "how do I know when my scraper is broken": no, not by itself. It's a plausibility net under specific fields, not a correctness check, and it depends on an external model call succeeding at the exact moment you need it to. It's worth having if you already have a low-cost way to bolt it onto a pipeline, but it isn't something to build a monitoring strategy around by itself. The main thing I got out of it wasn't even the AI check. It was being forced to actually watch a crawl fail to stop, and finding a bug in my assumptions about Scrapy's exception handling that I'd have shipped otherwise.

Takeaway

The interesting result wasn't that an AI model can score whether a scraped value looks real. It can, cheaply and fast enough to run inline. What matters more is that the check is only as good as its blind spots, and the most valuable thing the project produced was finding a case where the failure mode I was defending against, a crawl silently continuing after something breaks, was hiding in my own stop-the-spider code, not in the site I was scraping.

Give an AI Agent a Browser: From Instructions to Actions

John Rooney — Fri, 11 Sep 2026 14:10:00 +0000

Most browser automation examples start with selectors. Find the button, identify the input, write the script, and hope the page does not change before the next run.

That approach works, but it becomes awkward when a site requires real browser interaction. Some information only appears after a click. Some pages run browser checks. And if the agent is running on a remote server, even launching and maintaining a local browser can become a project of its own.

In this walkthrough, I connected an agent to a Zyte CDP browser through Playwright CLI and gave it instructions in plain language. The interesting part was not simply connecting to a remote browser. It was seeing what the agent could do once it had access to a real, interactive browser session.

Video timestamps are included throughout this post.

follow along

The setup: a short instruction file and Playwright CLI [00:18]

I started with a small Markdown file that explains how the agent should use the Zyte CDP browser with Playwright CLI. It tells the agent to make sure Playwright CLI is installed and meets the required version, then gives it the basic connection instructions.

This creates a simple bridge between the agent and the browser. Once the browser is connected, the agent can use the operations available through Playwright CLI: opening pages, finding fields, clicking buttons, entering text, and reading the resulting page.

The difference is that I do not have to write a separate Playwright script for every interaction. I can describe the task and let the agent work out how to perform it on the page.

A deliberately interactive demo site [00:39]

The demo website is intentionally simple. It contains categories, a search box, product controls, and several buttons. It also includes a browser check, which appears briefly when the page loads.

The site has a useful interaction flow:

Search for products.
Select products.
Click a compare button.
Read the comparison table that appears.
Open an individual product page.
Click a button to check availability.

Some of this information might be available through Ajax requests or an underlying API. In a real project, it can be worth investigating those options. But there are also cases where the most reliable path is to use the website as a user would and click through the interface.

That is where a browser-connected agent becomes useful.

Replacing selector work with instructions [01:13]

The traditional Playwright workflow requires you to inspect the page and choose selectors. You need to work out which element is the search field, which button triggers comparison, and which part of the page contains the result.

With the agent, the instruction can be much closer to the actual task:

Go to the website, search for "brake", and return the products you find.

The agent can inspect the page, locate the relevant field, enter the search term, and read the results. It is still using selectors under the hood, but you do not have to discover and maintain those selectors yourself for each request.

This is especially useful when the task changes from run to run. The instruction can describe the outcome rather than encoding every click in a fixed script.

Connecting the agent securely [01:28]

For the demonstration, I used OpenCode with DeepSeek Flash through OpenRouter. This task does not require a large amount of reasoning. The agent mainly needs to follow the browser instructions and carry out a sequence of actions, so a relatively inexpensive model is enough.

The connection also needs authentication. The API key is read from an environment variable and used to create a short-lived configuration file in a temporary location. That file is passed to Playwright CLI when it connects to the browser.

The important practice here is to keep the key out of the instruction file and out of the agent's visible working material. The connection configuration should be created securely and cleaned up according to the needs of the environment.

After that, the agent can use the browser through Playwright CLI.

Searching without writing selectors [03:02]

The first task was simple: open the site and search for "brake".

The agent found the search bar, entered the term, and returned the matching products. During the process, Playwright CLI reported the elements it found and the actions it took. The model was verbose, but the important result was that no custom selector script was needed.

The same approach works when the page is more complicated than the demo. You can describe the field or action in terms a user would understand, and the agent can inspect the current page before deciding what to interact with.

Using the site's compare feature [04:07]

Next, I asked the agent to take the top three products, use the website's compare feature, and return the comparison table.

This matters because the table does not appear until the products have been selected and the compare action has been triggered. It is not simply a block of data sitting in the initial HTML.

The agent had to find the product controls, select the relevant items, click the compare button, and read the table that appeared below the results. Again, the instruction described the goal rather than the selectors:

For the top three products, use the website's compare feature and return the table it shows.

The example is deliberately basic, and the same information might have been available elsewhere on this particular site. The point is to demonstrate the interaction pattern. Once the agent has a browser, it can perform actions that depend on the page's state and then collect the result.

Checking availability on each product page [05:13]

I then asked the agent to visit each product page and retrieve the availability information.

The product pages include a "check availability" button. The agent can follow the product information it already collected, open each page, find that button, click it, and return the availability result.

This is the kind of task that can become a collection of small scripts when written manually. The agent can handle the sequence from a single higher-level instruction, provided the task is clear and the browser session remains available.

Why use a remote browser? [05:48]

If Playwright can launch a browser locally, why connect to a remote one?

The two main reasons are access and maintenance.

Access to sites that expect a real browser

Some sites do not respond well to a basic automation setup. They may perform browser checks or apply other forms of traffic protection. In the demo, the site was configured so that a standard Playwright launch would not get access, while the remote browser could connect successfully.

There are open-source stealth browser options, and some of them work well. But they need to be maintained. Browser behavior changes, sites change their checks, and the tools need to keep up.

Less local infrastructure to maintain

Running a browser locally can also create deployment work. An agent may be running on a server without a display, which means you may need a virtual display such as Xvfb. Then there is the browser installation, version management, operating system configuration, and the rest of the environment around it.

None of these problems is impossible to solve. They are just extra moving parts, and every extra moving part is another thing that can fail in an agentic workflow.

A remote browser moves much of that browser infrastructure outside the agent's environment. The agent only needs to connect, perform the work, collect the result, and disconnect.

Session limits and closing the connection [02:28, 07:34]

The browser session in the demonstration lasted five minutes. That limit is important for two reasons: the agent needs enough time to finish its work, and the connection should be closed as soon as the work is complete.

When I spent too long talking through the demonstration, the session expired. The agent reconnected and continued with the task, but this is a useful reminder to design the workflow around short, purposeful sessions.

In a production-style flow, the agent might receive one instruction, connect to the Zyte CDP browser, complete the interaction, return the data, and close the browser. Closing the connection ends the session and avoids paying for unnecessary additional browser time.

This cleanup step should be part of the instructions or application logic, not something left to chance.

The larger idea

The real benefit here is not that an agent can click a few buttons on a demo site. It is that browser automation can be expressed in terms of the work you want done:

Search for a product.
Compare the first three results.
Visit each product page.
Check availability.
Return the results.

The agent still needs a browser automation tool, authentication, sensible session limits, and clear instructions. But you do not have to begin by turning every task into a collection of selectors and hard-coded steps.

That makes a remote CDP browser a useful option for agentic web interaction, especially when a site requires clicks, state changes, browser checks, or other behavior that a simple scraping request cannot reproduce.

The instructional Markdown file and the browser CDP resources used in the demonstration are linked below the video. If you are experimenting with browser-connected agents, this is a straightforward pattern to try: give the agent a small set of browser instructions, let it act on the page, and make sure it closes the connection when the job is done.

Zyte CDP browser documentation:
https://docs.zyte.com/zyte-api/usage/cdp.html

Zyte headless browser:
https://www.zyte.com/zyte-api/headless-browser/

Instruction Markdown file:
https://raw.githubusercontent.com/zytelabs/zyte-cdp-examples/refs/heads/main/agent-onboard.md

40% of pages are empty over plain HTTP

John Rooney — Tue, 08 Sep 2026 15:25:00 +0000

40.6% of popular landing pages need JavaScript to show you anything useful. That is from State of Web Access, an audit Zyte (where I work) ran on 11,100 landing pages. The test was blunt. Fetch the page over plain HTTP, fetch it again in a headless browser, and if rendering grew the meaningful HTML by more than half, count it as JavaScript-dependent.

On the pages that failed the test, rendering added 140,923 bytes on average. Three quarters of the final HTML did not exist until a browser ran the scripts. Restaurants and airlines top the list at 66%, travel at 65%.

The report calls this architecture rather than defence, and I agree. Single page apps won. The HTML that comes over the wire is a shell and the content arrives in a second round trip.

What rendering costs you

The instinct on seeing an empty response is to reach for Playwright. Here is what that does to the bill, using the audit's own tier data. Zyte's tiers map to the infrastructure needed to fetch a page reliably, and the report scored every site both ways:

Access method	Simple	Easy	Moderate	Complex	Advanced	Mean tier
Plain HTTP	85.2%	9.4%	2.5%	2.0%	0.9%	1.24
Headless browser	58.4%	31.3%	8.0%	1.8%	0.5%	1.55

The gap is not sites defending themselves. A browser carries a heavier fingerprint and a heavier compute cost even on a site with no barriers at all. Switching to rendering moves a quarter of all sites up a tier for no reason other than the method.

So before rendering, I look in three places. The data is very often already in the response.

First, measure

This is the audit's test, rewritten to compare visible text instead of HTML bytes, which is stricter. Run it once per website before you decide anything.

import asyncio
import sys

import httpx
from playwright.async_api import async_playwright
from selectolax.parser import HTMLParser

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"


def visible_text_len(html: str) -> int:
    tree = HTMLParser(html)
    for node in tree.css("script, style, noscript, template"):
        node.decompose()
    return len(tree.body.text(separator=" ", strip=True)) if tree.body else 0


async def js_dependence(url: str) -> dict:
    plain = httpx.get(url, headers={"User-Agent": UA}, follow_redirects=True, timeout=20).text
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page(user_agent=UA)
        await page.goto(url, wait_until="networkidle")
        rendered = await page.content()
        await browser.close()
    a, b = visible_text_len(plain), visible_text_len(rendered)
    return {"plain_chars": a, "rendered_chars": b, "js_dependent": b > a * 1.5}


if __name__ == "__main__":
    print(asyncio.run(js_dependence(sys.argv[1])))

If js_dependent is false, stop here. You have a plain HTTP site and a browser will only make it slower and more expensive.

Second, harvest the state the framework left behind

SPAs hydrate from data the server already embedded. Next.js puts it in a script tag with the id __NEXT_DATA__. Nuxt uses a global. Apollo, Redux and a dozen others assign to window.__SOMETHING__. E-commerce and news sites also carry JSON-LD for search engines, which is often cleaner than the visible page.

A Scrapy spider that pulls all of it from the plain response, with no browser involved:

import json
import re

import scrapy

STATE_SELECTORS = {
    "next": "script#__NEXT_DATA__::text",
    "jsonld": 'script[type="application/ld+json"]::text',
}

STATE_GLOBALS = [
    r"window\.__INITIAL_STATE__\s*=\s*",
    r"window\.__PRELOADED_STATE__\s*=\s*",
    r"window\.__APOLLO_STATE__\s*=\s*",
]


class EmbeddedStateSpider(scrapy.Spider):
    name = "embedded_state"

    def parse(self, response):
        for source, sel in STATE_SELECTORS.items():
            for raw in response.css(sel).getall():
                try:
                    yield {"source": source, "url": response.url, "data": json.loads(raw)}
                except json.JSONDecodeError:
                    self.logger.debug("unparseable %s block on %s", source, response.url)

        for script in response.css("script::text").getall():
            for pattern in STATE_GLOBALS:
                m = re.search(pattern, script)
                if m:
                    obj = self._leading_json(script[m.end():])
                    if obj is not None:
                        yield {"source": "global", "url": response.url, "data": obj}

    @staticmethod
    def _leading_json(s: str):
        # Parses the JSON object at the start of s and ignores whatever follows
        # (a semicolon, more script). Avoids trying to find the closing brace by hand.
        try:
            obj, _ = json.JSONDecoder().raw_decode(s)
            return obj
        except json.JSONDecodeError:
            return None

The raw_decode trick is the useful part. You never have to find the end of the object yourself. Nuxt is the awkward one, since its global is usually a function call rather than a literal, and you either evaluate it or fall through to the next step.

Third, replay the request the page makes

Open DevTools, Network tab, filter to Fetch/XHR, reload. The JSON endpoint the page calls is right there, and it usually returns cleaner data than the HTML ever will. Right click, copy as cURL, and check what it needs. If it works with a static header set, you have a plain HTTP scraper for a JavaScript site. If it needs a token that a script computes on the client, that is your signal to render, and now you know why.

When rendering is the right call

Three cases. The API is signed or needs a challenge token from a script. The content genuinely does not exist until a user interacts. Or the TLS probe and the 403 classifier from the earlier posts in this series put the site at Moderate or above, at which point you need a browser-shaped session anyway and rendering comes with it.

Everything else is a plain HTTP job wearing a React costume.

The full JavaScript breakdown by industry and site size is on the report page, and the tier tables are under scraping cost. The tier model is Zyte's own pricing model applied to Zyte's own data, so trust the ordering between industries more than the absolute percentages.

Disclosure: I work at Zyte. The audit is ours. The code and the opinions are mine.

One TLS handshake predicts the whole anti-bot stack

John Rooney — Mon, 07 Sep 2026 15:19:39 +0000

When I size up a new website I used to start with headers. Copy them out of DevTools, match the order, add the cookies, see what happens. Now I start with one TLS handshake, because it tells me more about the site than an afternoon of header work.

The reason is a set of numbers from State of Web Access, the audit Zyte (my employer) ran on 11,100 popular landing pages. 13.8% of them screen the TLS handshake. Look at what else those sites run compared to everyone else:

Barrier	TLS-fingerprinting sites	All other sites
Rate limiting	78.1%	13.2%
Antibot	45.0%	14.2%
CAPTCHA	37.3%	20.2%
JavaScript required	27.0%	42.8%

A TLS check is the strongest single predictor in the whole dataset. If the handshake is being inspected, you are almost certainly going to hit rate limiting too, and you have a coin-flip chance of a dedicated bot manager on top. If it is not being inspected, the site probably stops at a CDN firewall. One request and you know which world you are in.

The JavaScript row runs the other way, and I find it the most interesting. TLS-checking sites need a browser less often. They are server-rendered, performance-minded operations that enforce access at the connection, not in the page.

Why nobody notices this layer

TLS fingerprinting fails before HTTP starts. Your client sends a ClientHello announcing which TLS versions, cipher suites and extensions it supports, in a particular order. Python on OpenSSL, Go's crypto/tls and Chrome's BoringSSL all produce different ones. A site can see "Chrome user agent, OpenSSL handshake" and know the client is lying before it reads a single header.

JA3 hashed those fields into a fingerprint in 2017. Chrome started shuffling its extension order in 2023, which broke JA3, and JA4 replaced it by sorting the fields first. The detail does not matter much for scraping. What matters is that the check is silent. Some sites return a bare 403 with an empty body. Some just close the connection, and your logs say "connection reset by peer", which looks like a network blip. Nothing in the failure says TLS.

Vendor attribution was only possible for 45% of the TLS-checking sites in the audit. The other 55% show the behaviour with no named signature. Where it could be attributed, a single bot management vendor was 88.8% of it. So this is mostly one vendor's check, bundled into a product a lot of sites bought for other reasons.

The probe

This is the method the report used, cut down. Two requests, identical headers, from the same IP. The only thing that differs is the handshake. aiohttp presents a standard Python TLS signature. curl_cffi presents Chrome's. If the first fails and the second passes, the site is looking at TLS.

import asyncio
import sys

import aiohttp
from curl_cffi.requests import AsyncSession

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}


async def python_tls(url: str) -> tuple[int, int]:
    try:
        async with aiohttp.ClientSession(headers=HEADERS) as s:
            async with s.get(url, timeout=aiohttp.ClientTimeout(total=20)) as r:
                return r.status, len(await r.read())
    except aiohttp.ClientError:
        return 0, 0          # reset or refused: count it as a block


async def chrome_tls(url: str) -> tuple[int, int]:
    async with AsyncSession(impersonate="chrome", headers=HEADERS) as s:
        r = await s.get(url, timeout=20)
        return r.status_code, len(r.content)


async def probe(url: str) -> dict:
    py, ch = await asyncio.gather(python_tls(url), chrome_tls(url))
    filtered = py[0] != ch[0] and ch[0] < 400
    return {"python_tls": py, "chrome_tls": ch, "tls_filtered": filtered}


if __name__ == "__main__":
    print(asyncio.run(probe(sys.argv[1])))

Install with pip install aiohttp curl_cffi. The impersonate part follows whatever Chrome version curl_cffi ships, so it drifts less than a hand-maintained profile would.

Because both requests leave the same machine, IP reputation is held constant. This is a controlled experiment with one variable, which is more than most scraping diagnostics manage.

How I read the result

If tls_filtered is true, I plan for the stack the table above predicts. Browser-shaped TLS from request one, session handling, per-domain concurrency of one or two, content validation on every response. I do not spend a day on header tweaks because headers were never the problem.

If it is false and I am still blocked, I have ruled out one layer for the cost of one request. The block is IP class, headers, cookies or behaviour, and I look there instead.

If it is false and both requests pass, I have a Simple-tier site and I should stop reaching for Playwright. The audit's tier data puts 85% of landing pages at plain HTTP. Rendering when you do not need to is the most common way I see scraping budgets disappear.

One caveat the report makes and I will repeat. Everything here is landing pages, scanned once, from datacentre IPs. Search and product pages are usually defended harder than the front door. Treat the percentages as a floor.

The industry split is on the TLS page. Jewellery and luxury leads at 34%. Reference sites sit at 3%, because blocking crawlers would kill the thing they exist for.

Disclosure: I work at Zyte. The dataset is ours. The probe is a cut-down version of what the audit ran, and the opinions are mine.

DEV Community: John Rooney

A data scientist's guide to stress-free product scraping

Beautiful Soup vs. the modern web

Down the rabbit hole

Why bandwidth pricing is a trap (and why proxies aren't enough)

One call does it all

Zyte API and AI Extraction

Conclusion: Focus on data, not the extraction process

FAQ

Why is product scraping often considered stressful for data scientists?

How does AI help in making web scraping stress-free?

Should data science teams build their scraping tools in-house or outsource them?

Is there a free trial for Zyte API

How to parse HTML tables into structured data (CSV/Excel)

Why HTML tables are harder to parse than they look

Setting up your environment

The simplest approach: pandas read_html

Targeting a specific table with Beautiful Soup

Parsing tables manually when the structure is complex

Handling pages that require JavaScript

Exporting cleanly to CSV and Excel

Handling common problems

Putting it all together

What to do when the table is too complex to parse reliably

Next steps

Inside Scrapy's RemoteControl extension

1. How RemoteControl works

Two endpoints

What your code can see

Limits, and the one that bites

Where Scrapy MCP fits

2. Inspecting progress, queues, and active components

3. Testing a selector against a live response

4. Diagnosing and fixing throughput without restarting

5. Checking what's still allocated

6. Mutation traps, and the edge of what's safe to patch

7. When the crawl runs through scrapy-zyte-api

Takeaway

Running Playwright at scale: connecting to the Zyte CDP browser

What is CDP?

The problem with local browsers

An important distinction before you start

Why we built it

Connecting Playwright to the Zyte browser

Interactions are where it gets useful

Blocking images to cut data transfer

Screenshots

Raw CDP commands

Using it with Scrapy

Giving AI agents browser access

When to use it, when not to

One Scrapy Spider, Three Browser Setups: Playwright, Patchright, and Zyte

The shared setup

Stage one: Playwright's local browser (the default)

Stage two: swap the browser with PLAYWRIGHT_BROWSER_PROVIDER

Stage three: connect to a browser on Zyte over CDP

A cost that doesn't show up in these numbers: running several contexts at once

Takeaway

I built scraping agents for 30 days - here’s what I learned

The biggest surprise: Agents really can build pipelines

The biggest time sink: Tool design

The mistake I made on day four that I am still thinking about

The speed is real, and I had to learn not to trust it blindly

Why I stopped thinking of human-in-the-loop as a limitation

The moment that genuinely impressed me, and the one that humbled me

What I actually do differently now

Want to try Zyte API for free?

Supercharging web scraping with Claude skills

What are Claude skills?

Comparing skills to MCP servers

Top four scraping skills

1. Fetcher

Automating HTML acquisition

2. AI Parser

Cleaning data with AI extraction

3*.* SelectorGen

Generating resilient selectors

4. Extruct HTML

Extracting hidden data

Building a workflow by chaining skills

Stage two: swap the browser with `PLAYWRIGHT_BROWSER_PROVIDER`

3. SelectorGen