DEV Community: Extract by Zyte

What the Scrapy Maintainer Thinks About AI-Generated Scrapers

Neha Setia — Sat, 11 Apr 2026 15:13:41 +0000

I sat down with Adrian Chaves, one of the lead Scrapy maintainers, who also works at Zyte, to ask him the questions I've been chewing on since Zyte launched Web Scraping Copilot: what happens when an LLM writes your spider(the web scraping code)? What gets easier? What doesn't change?

His answers surprised me. A few highlights:

On vibe coding: Adrian has thoughts about developers treating scraper generation as a black box, and why Scrapy's design philosophy matters more, not less, when an LLM is writing the code.
The bottleneck isn't what you think. He argues the hard part of scraping in 2026 isn't writing code. It's reading pages. And that's the part LLMs still struggle with.
What "good design meeting the future halfway" means. Why frameworks like Scrapy that were built for humans are turning out to be the best frameworks for AI agents too.
Where LLMs actually help. The concrete places where AI makes a scraper developer's life better, and where it just adds complexity.

Full conversation is on the Zyte blog, linked below. If you're building scrapers, thinking about adding AI to your extraction pipeline, or just curious what someone who's been maintaining one of the most widely used scraping frameworks for years thinks about all of this, it's worth a read.

Read the full interview on zyte.com

Happy to discuss in the comments.
What are you using AI for in your scraping workflow right now, and where have you hit walls?

Tags: web scraping • scrapy • ai • opus • anti-bot • Claude AI • sonnet • open source

Stop using Python `requests` for web scraping: there are better & modern libraries instead

Ayan Pahwa — Thu, 09 Apr 2026 11:09:31 +0000

While the 'Requests' library remains the default choice for many Python developers due to its reliability and extensive documentation, the Python HTTP landscape has evolved considerably.

Modern alternatives now offer significant advantages, including built-in asynchronous support, HTTP/2 compatibility, enhanced performance, and up-to-date TLS handling.

This article introduces and compares three such contemporary clients: HTTPX, curl_cffi, and rnet, detailing their unique features and practical applications.

The problem with Requests for web scraping

It's important to clarify Requests' limitations before proceeding; for simple API interactions with well-behaved endpoints, it still remains the de facto standard.

However, a major drawback of the Requests library when it comes to web scraping is its predictable HTTP client fingerprint. This fingerprint, a unique combination of TLS version, cipher suites, HTTP headers, and connection characteristics, is sent with every request, and is well-known and cataloged by anti-bot systems.

Consequently, if you're interacting with any endpoint, including APIs or services protected by anti-ban vendors, your request can be blocked purely based on how the requests library identifies itself. This happens even before your credentials or payload are scrutinized, highlighting a significant limitation when targeting systems that perform client-side validation.

In addition to issues like fingerprinting, a major limitation of the requests library is its lack of native asynchronous support. This absence of async capability is particularly problematic when handling workloads that involve numerous HTTP requests. Without it, the calls execute sequentially, and the program's thread remains blocked for the entire duration of each individual request.

For straightforward scenarios, the standard requests API call remains perfectly functional, as demonstrated in a quick example.

import requests

response = requests.get(
    "https://jsonplaceholder.typicode.com/posts/1",
    timeout=10,
)
response.raise_for_status()
data = response.json()
print(data["title"])

Clean and simple. For a one-off call to a standard REST API, this is fine. The gaps start showing when you need concurrency, HTTP/2, or when the target endpoint does any kind of client validation.

Install the Alternatives

pip install httpx       or  uv add https
pip install curl-cffi       or  uv add curl-cffi
pip install rnet        or  uv add rnet &&
                    uv add asyncio

1. HTTPX

HTTPX is the most direct upgrade from Requests as the API is nearly identical. If you know Requests, you already know most of HTTPX. What it adds is first-class async support, HTTP/2, and a more modern internal architecture.

Where it differs from Requests is the explicit use of a Client context manager (strongly recommended over module-level function calls) and the AsyncClient for async usage. This gives you connection pooling and proper resource cleanup by default.

HTTPX is the right starting point if you're looking for a migration that requires minimal code changes.

Example: Sync

import httpx

with httpx.Client(timeout=10.0) as client:
    response = client.get("https://jsonplaceholder.typicode.com/posts/1")
    response.raise_for_status()
    data = response.json()

print(data["title"])

Example: Async (calling the Zyte API)

Async is where HTTPX really earns its keep. Here it's used to fire multiple requests to the Zyte API concurrently, each request blocks on the server side until extraction is complete, but your event loop stays free to send others in parallel:

import os
import asyncio
import httpx

API_KEY = os.environ["ZYTE_API_KEY"]
ENDPOINT = "https://api.zyte.com/v1/extract"

urls = [
    "https://example.com",
    "https://httpbin.org",
]

async def fetch(client: httpx.AsyncClient, url: str) -> dict:
    response = await client.post(
        ENDPOINT,
        json={"url": url, "browserHtml": True},
        auth=(API_KEY, ""),
    )
    response.raise_for_status()
    return response.json()

async def main():
    async with httpx.AsyncClient(timeout=60.0) as client:
        results = await asyncio.gather(*[fetch(client, url) for url in urls])
    for result in results:
        print(result["url"], "—", len(result["browserHtml"]), "chars")

asyncio.run(main())

Notes:

raise_for_status() raises httpx.HTTPStatusError on 4xx/5xx responses.
HTTP/2 support requires pip install httpx[http2] and passing http2=True to the client.
The 60-second timeout accounts for the Zyte API's server-side blocking behavior — it holds the connection open until extraction completes.

2. curl_cffi

curl_cffi wraps libcurl with Python bindings and adds something HTTPX doesn't have: TLS fingerprint impersonation. It can show the TLS handshake of Chrome, Firefox, Safari, and other browsers. For API calls hitting endpoints protected by anti-ban or similar systems, this can be the difference between getting a response and getting a 403.

The interface closely mirrors Requests, with the addition of the impersonate parameter. It supports both sync and async usage. For most API calls where fingerprinting isn't a concern, curl_cffi behaves just like Requests, the impersonate parameter is opt-in.

Example: Sync

from curl_cffi import requests

response = requests.get(
    "https://jsonplaceholder.typicode.com/posts/1",
    impersonate="chrome",
    timeout=10,
)
response.raise_for_status()
data = response.json()
print(data["title"])

Example: Async (calling the Zyte API)

import os
import asyncio
from curl_cffi.requests import AsyncSession

API_KEY = os.environ["ZYTE_API_KEY"]
ENDPOINT = "https://api.zyte.com/v1/extract"

payload = {
    "url": "https://example.com",
    "browserHtml": True,
}

async def call_zyte_api():
    async with AsyncSession(impersonate="chrome") as session:
        response = await session.post(
            ENDPOINT,
            json=payload,
            auth=(API_KEY, ""),
            timeout=60,
        )
        response.raise_for_status()
        data = response.json()
        print(data["url"], "—", len(data["browserHtml"]), "chars")

asyncio.run(call_zyte_api())

Notes:

impersonate="chrome" sends Chrome's TLS fingerprint on every request made through this session.
Other supported values include "firefox", "safari", "chrome110", and more — check the curl-cffi docs for the full list.
The sync interface (from curl_cffi import requests) is nearly identical to the requests module, making it the easiest drop-in if you only need sync.

3. rnet

rnet is the newest of the three. Like a lot of modern Python, it's built on Rust, making it async-first and performance-oriented. Like curl_cffi, it supports TLS impersonation, but its primary differentiator is throughput. It is designed for high-concurrency workloads where you're firing many requests simultaneously.

The API surface is different from Requests, so it's not a drop-in replacement. But the patterns are clean and modern, and for async-heavy workloads it's worth the minor adjustment.

Example: Sample library code

import asyncio
from rnet import Impersonate, Client


async def main():
    # Build a client
    client = Client(impersonate=Impersonate.Firefox139)

    # Use the API you're already familiar with
    resp = await client.get("https://tls.peet.ws/api/all")

    # Print the response
    print(await resp.text())


if __name__ == "__main__":
    asyncio.run(main())

Notes:

rnet is async-first; sync support is limited.
Response body methods like .json() and .text() are awaitable.
The Rust core makes it particularly well-suited for high-throughput concurrent workloads.

Comparison Table

Feature	Requests	HTTPX	curl_cffi	rnet
Sync Support	✅ Yes	✅ Yes	✅ Yes	⚠️ Limited
Async support	❌ No	✅ Yes	✅ Yes	✅ Yes (primary)
HTTP/2	❌ No	✅ With extra dependencies	✅ Via libcurl	✅ Built-in
Performance	Baseline	Good	Good–High	High
TLS changes	❌ No	❌ No	✅ Yes	✅ Yes

When to use which

Use Requests for simple, one-off scripts, internal tooling, or any situation where you're hitting a cooperative API endpoint and don't need concurrency. Nothing wrong with it in that context.

Use HTTPX when you need async, want the closest migration path from Requests, or need HTTP/2. It's the safest default upgrade for most projects.

Use curl_cffi when TLS fingerprint control matters, whether that's because you're hitting an anti-ban wall or an API with strict client validation, or any service that checks how a client identifies itself at the TLS layer.

Use rnet when raw async performance is the priority. Its Rust foundation makes it the strongest choice for high-concurrency workloads where you're firing many requests simultaneously and need low overhead.

The optimal choice is determined by several factors: your concurrency requirements, the target endpoint's sensitivity to client identification, and the desired similarity between the new code and your existing requests implementation.

Writing production-ready Scrapy spiders with opencode

John Rooney — Wed, 08 Apr 2026 09:19:39 +0000

AI-enabled code editors can now conjure scraping code on command. But anyone who has used a generic coding agent to build a spider knows what comes next: a plausible-looking file that falls apart the moment it hits a real website. The selectors are fragile, the error handling is missing, and the structure ignores everything Scrapy actually expects from production code.

The problem is not the AI. It's the prompts, the context, and knowing where to let the agent drive and where to stay in control. This article walks through using opencode to build Scrapy spiders that are actually deployable, covering setup, the prompts that work, and the pitfalls that will burn you if you are not careful.

Why opencode works well for scraping projects

Most AI coding agents are designed around general-purpose software projects. opencode is different in one important way: it is terminal-native, model-agnostic, and designed to operate inside your actual working directory. It reads your project, understands your file structure, and writes code into the files that already exist rather than pasting snippets into a chat window.

For Scrapy projects specifically, this matters. A spider is not a standalone script. It depends on items, settings, middlewares, pipelines, and page objects. An agent that can see all of those files at once produces far better output than one operating on a blank context.

opencode also supports custom commands stored as Markdown files. That means you can encode your own Scrapy conventions as reusable prompts and call them every time you start a new spider, without retyping the same context.

Getting set up

Install opencode with the one-liner:

curl -fsSL https://opencode.ai/install | bash

On macOS and Linux, the Homebrew tap gives you the fastest updates:

brew install anomalyco/tap/opencode

On Windows, use WSL for the best experience. The choco install opencode path works but the terminal experience is noticeably smoother inside a Linux environment.

Once installed, connect your model provider. The /connect command in the terminal user interface walks you through it. If you want to avoid managing API keys from multiple providers, opencode Zen gives you a curated set of pre-tested models through a single subscription at opencode.ai/auth.

For scraping work, choose a model with a large context window. Spider files, page objects, items, and a sample HTML fixture can easily fill 20,000 tokens before you have written a single prompt. Models with at least 64k context are the practical minimum.

Initialize your Scrapy project first

Before you open opencode, scaffold your Scrapy project as you normally would:

scrapy startproject myproject
cd myproject

Then initialize opencode inside the project root:

opencode init

This creates an AGENTS.md file. Commit it. opencode reads this file on every session to understand how your project is structured. Fill it with the conventions your project follows: which item classes exist, which middlewares are active, whether you are using scrapy-poet page objects, and which version of Zyte API or other HTTP backends you are using. The more context AGENTS.md carries, the less you repeat yourself in prompts.

A minimal AGENTS.md for a Scrapy project looks like this:

# Project conventions

- Python 3.12, Scrapy 2.12
- All spiders use scrapy-poet page objects (never parse in the spider class itself)
- Item classes are defined in items.py using dataclasses
- Zyte API is configured via scrapy-zyte-api; ZYTE_API_KEY is in .env
- Settings live in settings.py; never hardcode values in spider files
- All spiders output to JSON Lines via FEEDS setting
- Test fixtures live in tests/fixtures/ as .html files

The prompts that actually work

Generic prompts produce generic code. The prompts below are tested patterns that produce Scrapy-idiomatic output.

Starting a new spider

The most common mistake is asking opencode to "write a spider for X." That produces a working script, not a Scrapy spider. Be specific about structure:

Create a Scrapy spider for https://books.toscrape.com that:
- Uses a scrapy-poet page object called BookListPage for list pages and BookDetailPage for detail pages
- Extracts: title, price, availability, star rating, and product URL
- Handles pagination by following the "next" link
- Stores results in a BookItem dataclass in items.py
- Does not put any CSS selector logic inside the spider class itself

Start with the page objects in pages.py, then write the spider in spiders/books.py.

The explicit constraint against putting selectors in the spider class is important. Without it, the agent will inline everything, which defeats scrapy-poet's purpose and makes the code harder to test.

Asking for resilient selectors

Generated selectors are often too specific. They target a class that is only present on one layout variant, or chain through five levels of nesting that will break on the next site deploy.

Prompt the agent to justify its selector choices:

Write the CSS selectors for BookDetailPage. For each field, explain why you chose
that selector over alternatives. Prefer attribute-based selectors (like [itemprop] or
[data-*]) over class names where both options exist.

This produces more defensive selectors and, more importantly, gives you enough reasoning to judge whether to accept them.

Adding error handling

The agent will skip error handling unless you ask for it explicitly:

Add error handling to BookDetailPage:
- If price is missing, log a warning and return None (do not raise an exception)
- If star rating cannot be parsed, default to 0
- Add a try/except around the availability field and log the raw text if parsing fails

Never assume the agent will add graceful degradation on its own. It optimizes for the happy path.

Writing tests

opencode is genuinely useful for generating pytest fixtures and test scaffolding. Give it a concrete fixture to work from:

Write a pytest test for BookDetailPage using the HTML fixture at
tests/fixtures/book_detail.html. Test that:
- title is extracted as a non-empty string
- price is a float greater than zero
- availability is one of: "In stock", "Out of stock"
- star_rating is an integer between 0 and 5

Use pytest parametrize if testing multiple fixture variants.

Pitfalls to watch for

The agent assumes the HTML is static

By default, any spider the agent generates will use response.css() or response.xpath() on raw HTML. If your target site renders content with JavaScript, those selectors return nothing. Before you run any generated spider, check whether the target page is JavaScript-rendered by viewing source in your browser. If the data you need is absent from the raw HTML, prompt the agent to use Zyte API's headless browser or a Playwright download handler instead of a plain HTTP request.

Selectors written against one page break on others

The agent writes selectors against whatever HTML you give it. If you paste a single product page, it will produce selectors that work on that product page. Run the spider against 10 or 20 URLs from the same site before treating the selectors as reliable.

Ask the agent to help you validate coverage:

Here are three different product page HTML snippets from the same site (pasted below).
Identify any selectors in BookDetailPage that would fail on snippet 2 or snippet 3,
and suggest more robust alternatives.

Context window exhaustion mid-session

Long sessions that involve large HTML files, multiple spider files, and back-and-forth debugging will eventually exhaust the model's context. When this happens, the agent starts contradicting earlier decisions or forgetting your project conventions.

The fix is to keep sessions short and focused. One session per spider, or one session per refactor task. Use your AGENTS.md to carry conventions across sessions rather than re-explaining them in chat.

Generated settings override your existing configuration

When the agent writes setup instructions, it often suggests adding settings directly to settings.py. If you already have a settings file, this can clobber existing values or introduce conflicts. Review every settings change the agent proposes before accepting it.

The agent does not know about anti-bot measures

opencode has no knowledge of whether a site actively blocks scrapers. It will happily generate a spider that will be blocked immediately in production. Anti-bot handling, rate limiting, and request fingerprinting are your responsibility to layer in. Zyte API handles the blocking and fingerprinting side; you still need to configure the integration yourself rather than expecting the agent to know it is necessary.

Useful custom commands for scraping

opencode custom commands let you encode reusable prompts as Markdown files in ~/.config/opencode/commands/. Here are three worth setting up for any Scrapy workflow.

`user:new-spider`

# New scrapy-poet spider

Create a new Scrapy spider for the URL provided by the user.
- Use scrapy-poet page objects (list page + detail page if applicable)
- Put all selector logic in page objects, nothing in the spider class
- Use item dataclasses from items.py (create new ones if needed)
- Include pagination handling
- Add logging for missing fields (warning level)
- Write page objects first, then the spider

Ask the user for the target URL before starting.

`user:harden-selectors`

# Harden selectors

Review the page objects in the current file. For each CSS or XPath selector:
1. Identify whether it targets a class, ID, tag, or attribute
2. If it targets a class name, suggest an attribute-based or structural alternative
3. Flag any selectors that chain more than three levels deep as fragile

Output a revised version of the file with improved selectors and inline comments
explaining each change.

`user:gen-tests`

# Generate pytest tests

Given a page object file and an HTML fixture provided by the user:
1. Write a pytest test file that covers all extracted fields
2. Test that required fields are non-null and the correct type
3. Test that optional fields handle absence gracefully (None, not exception)
4. Use parametrize if multiple fixture variants are present

Ask for the fixture file path before starting.

Where opencode fits in the workflow

Think of opencode as a fast first-draft tool, not an autonomous spider factory. The right workflow is:

Scaffold the project and write AGENTS.md manually
Use opencode to generate page objects and the spider skeleton
Review every selector by hand before trusting it
Run the spider against a sample of real URLs and inspect the output
Use opencode to patch failures and write tests
Handle anti-bot, rate limiting, and deployment yourself

The agent saves the most time on the repetitive structural work: boilerplate item classes, pagination logic, field extraction scaffolding, and test stubs. The judgment calls around which selectors are robust, whether a site is JavaScript-rendered, and how to handle blocking remain entirely in your hands.

That division of labor is what makes this approach work at production scale rather than just for prototypes.

Try it yourself

Install opencode, initialize it in an existing Scrapy project, and start with the user:new-spider custom command above. Pick a publicly accessible, static site like books.toscrape.com to test the workflow before applying it to a site with more complexity.

For the JavaScript-rendered sites and anything with active anti-bot measures, pair opencode's code generation with Zyte API to handle the access layer. You can sign up for a free trial and have a working integration running in minutes. The Zyte documentation covers the scrapy-zyte-api configuration in detail.

Small models, big ideas: what Google Gemma and MoE mean for developers

Ayan Pahwa — Tue, 07 Apr 2026 12:50:03 +0000

We at zyte-devrel try to stay plugged into what is happening in the AI and developer tooling space, not just because it is interesting, but because a lot of it starts having real implications for how we build and think about web data pipelines. Lately, one development that has had us genuinely curious is Google's new Gemma 4 model family, and specifically the direction it points toward with Mixture of Experts (MoE) architecture.

This is not a deep tutorial. It is more of a "hey, here is what we have been poking at" - the kind of update we would share in a Slack channel or over coffee. If you wanna participate in such discussions, our discord is always a welcoming platform.

What is Gemma 4?

Gemma has been dubbed as stripped down versions of Google Gemini. The new Gemma 4 is Google's latest family of open-weight language models, released last week. The lineup covers four sizes:

2B: ultra-efficient, built for mobile and edge devices
4B: enhanced multimodal capabilities, still edge-deployable
26B: sparse model using Mixture of Experts architecture (more on this below)
31B: dense model for more demanding tasks

All four variants support multimodal input (text and images), over 140 languages, a 128K-256K token context window, and agentic workflows with tool use and JSON output. The 2B and 4B models are specifically designed to run fully offline on modern edge devices like smartphones, with no internet dependency at all.

According to Google's Gemma 4 model page, the family ranks third among open-weighted models on the LM Arena leaderboard and uses 2.5 times fewer tokens than comparable models for equivalent tasks.

The Gemma 4 26B MoE, specially caught my attention because unlike other variants it's based on MoE architecture and it does make a difference :

What is MoE, and why does it matter?

Mixture of Experts (MoE) is one of those ideas that sounds complex but is actually pretty intuitive once you hear the analogy.

In a traditional dense neural network, every parameter in the model activates for every input. It is like calling your entire company into a meeting every time someone has a question. It works, but it is expensive.

MoE works differently. Instead of one large model doing everything, you have a set of smaller "expert" sub-networks, each specialized in different patterns, plus a router that looks at each incoming token and decides which one or two experts to activate. Most of the model sits idle at any given moment.

The result: you get the quality of a much larger model at a fraction of the inference cost.

The Gemma 4 26B model is a great illustration of this. It has 26 billion total parameters, but during inference it only activates around 3.8 billion of them. You get near-26B quality at roughly 3.8B compute cost. That is the MoE advantage in one number.

Other models that take the same approach:

Mixtral 8x7B: eight experts, two active per token; it outperforms Llama 2 70B on most benchmarks at far lower inference cost
Kimi: Moonshot AI's model, also MoE-based, has been making similar waves in the open-model space

For a deep dive on how MoE works under the hood, the Hugging Face guide to mixture of experts is well worth the read.

Since the models are free, if you have the right machine you can host them lcoally using Ollama or call them using API services like OpenRouter.

My prefered way of using a new mode is through Claude, but I believe Gemma4 has a different tool calling structure so it is not compatible yet, but you can use it with LMStudio, or skil all that because you can now

Run Gemma 4 offline on an iPhone

Here is the part worth sharing, because it genuinely surprised us.

Using the Google Edge AI Gallery app from the App Store, you can load a Gemma 4 model and run it with airplane mode on. No API calls, no cloud round-trips, no data leaving the device. Just the model running locally on your phone.

The experience is not going to replace a foundational frontier model for complex reasoning. But that is not the point. For quick classification, summarization, or just experimenting with local inference, the 2B and 4B variants are remarkably capable, and there are zero API costs with no data leaving your device. And since it is multi-modal you can practically point your phone camera to a paper recipt and ask it to save the details in a spreadhseet.

If you have not tried running a local large language model (LLM) yet, this is probably the lowest-friction entry point on hardware you already own.

Why should developers building data pipelines care?

Here is where it connects back to what a lot of us are building.

When LLMs run on-device or at the edge, the calculus around data pipelines shifts in a few useful directions:

Tokens are getting expensive and when a model as good as Gemma 4 or Qwen-3.5 is free and open-weighted it's a welcome development. Everyone's complaining about running out of their claude usage quota last couple of weeks or getting huge bills, thanks to giving Opus API Keys to OpenClaw. These things can be significantly addressed using Open Models.

No API round-trips: on-device inference eliminates latency from cloud API calls. For classification tasks running inside a scraping pipeline, this is a meaningful difference.
Data privacy: running extraction locally means scraped content never leaves your infrastructure. For regulated industries or sensitive datasets, that is a significant advantage.
Cost at scale: if you are doing high-volume classification — is this a product page? is this content in the target language? — running a small local model beats paying per-token at scale.
Edge preprocessing: a small LLM can filter and classify pages before they ever reach a more expensive cloud model for deeper analysis, and I am personally looking forward to run them on SBCs like a Raspberry Pi.
Open Weights: people often confuse open-weights models with open-source models, while the lines may be blurry and even I don't fully understand the difference, one thing I know for sure is that Gemma 4 is available under the Apache 2.0 license, which allows building and selling products on top of it and open-weights allows you to fine-tune it for your use-case or application.

Here's me playing it with it on my iPhone 16, completely offline:

Just checking in

We do not have grand proclamations here. This is a space that is moving fast, and we are learning alongside everyone else.

If you have been experimenting with local LLMs in your scraping or data extraction workflows, we would genuinely love to hear about it. Drop a comment below, or find us on the Zyte discord and read more interesting blogs on Zyte Blog.

If you want to try this yourself, here are three good starting points:

Google Edge Gallery: available on the App Store and Playstore, runs Gemma 4 locally on iOS
Gemma models on Hugging Face: for running on desktop or server
Google's Gemma 4 model page: full family overview, benchmarks, and architecture details

Build Scrapy spiders in 23.54 seconds with this free Claude skill

John Rooney — Mon, 30 Mar 2026 17:50:39 +0000

I built a Claude skill that generates Scrapy spiders in under 30 seconds — ready to run, ready to extract good data. In this post I'll walk through what I built, the design decisions behind it, and where I think it can go next.

What it does

The skill takes a single input: a category or product listing URL. From there, Claude generates a complete, runnable Scrapy spider as a single Python script. No project setup, no configuration files, no boilerplate to write. Just a script you can run immediately.

Here's what that looks like in practice. I opened Claude Code in an empty folder with dependencies installed, activated the skill, and said: "Create a spider for this site" — and pasted a URL.

Within seconds, the script was generated. I ran it, watched the products roll in, piped the output through jq, and had clean structured product data. Start to finish: under a minute.

Why a single-file script, not a full Scrapy project?

Scrapy is usually a full project — multiple files, lots of moving parts, a proper setup process. Running it from a script instead is generally discouraged for production work, but for this use case it's actually the right call.

The goal here is what I'd call pump-and-dump scraping: give Claude a URL, get a spider, run it for a couple of days, move on. It's not designed to scrape millions of products every day for years. For that kind of scale you need proper infrastructure, robust monitoring, and serious logging. This isn't that — and that's intentional.

What you do get, even in the single-file approach, is almost everything Scrapy offers: middleware, automatic retries, and concurrency handling. You'd have to build all of that yourself with a plain requests script. Scrapy gives it to you for free, even when running from a script.

The key design decision: AI extraction

The other major call I made was to lean entirely on Zyte API's AI extraction rather than generating CSS or XPath selectors.

Specifically, the skill uses two extraction types chained together: productNavigation on the category or listing page, which returns product URLs and the next page link, and product on each product URL, which returns structured product data including name, price, availability, brand, SKU (stock keeping unit), description, images, and more.

This means the spider doesn't need to know anything about the structure of the site it's crawling. There are no selectors to generate, no schema to define, no user confirmation step. The AI on Zyte's end handles all of that. It does cost slightly more than a raw HTTP request, but given how little time it takes to go from URL to working spider, the trade-off makes sense.

I've hardcoded httpResponseBody as the extraction source — it's faster and more cost-efficient than browser rendering. If a site is JavaScript-heavy and you're not getting the data you need, you can switch to browserHtml with a one-line change. The spider logs a warning to remind you of this.

The use case is deliberately narrow

This skill is designed for e-commerce sites, and only e-commerce sites. That's not a limitation I stumbled into — it's a feature.

Because the scope is narrow, the spider structure is simple and predictable: category pages with pagination, product links, and detail pages. Zyte API's productNavigation and product extraction types handle this reliably. Widening the scope to arbitrary crawling would require a lot more of Scrapy's machinery and would quickly exceed what makes sense for a lightweight script like this.

What it doesn't do: deep subcategory crawling, link discovery, or full-site crawls. If a page renders all its products without pagination, that works fine — the next page just returns nothing.

Logging and output

I replaced Scrapy's default logging with Rich logging, which gives cleaner terminal output. Scrapy's logs are verbose in ways that aren't useful when you're running a short-lived script — I wanted something concise enough that if something went wrong, it would be obvious at a glance.

Output goes to a .jsonl file named after the spider, alongside a plain .log file. Both are derived from the spider name, which is itself derived from the domain. Run example_com.py, get example_com.jsonl and example_com.log.

Where this goes next

The immediate next step I have in mind is selector-based extraction as an alternative path — useful for sites where the AI extraction isn't quite right, or where you want more control over exactly what gets pulled.

The longer-term vision is running this fully agentically. URLs get submitted somewhere — a queue, a database table, a form — an agent picks them up, builds the spider, and maybe runs a quick validation. The spider then goes into a pool to be run on a schedule, and data lands in a database rather than a flat file. Give Claude access to a virtual private server (VPS) via terminal and most of this is achievable without much extra infrastructure. The skill is already the hard part.

Download the skill

The skill is free to download and use. It's a single .skill file you can install directly into Claude Code. You'll need:

pip install scrapy scrapy-zyte-api rich
export ZYTE_API_KEY=your_key_here

Scrapy 2.13 or above is required for AsyncCrawlerProcess.

The link to the repo and the skill download are in the video description, and here. If you've built something similar, or have thoughts on the design decisions — especially around the extraction approach or the logging setup — I'd love to hear it in the comments. GitHub links to your own scrapers are very welcome too.

If you're interested in more agentic scraping patterns, I also built a Claude skill that helps spiders recover from excessive bans — you can watch that video here.

I Built a Self-Healing Web Scraper to Auto-Solve 403s

John Rooney — Mon, 16 Mar 2026 11:09:31 +0000

Web scraping has a recurring enemy: the 403. Sites add bot detection, anti-scraping tools update their challenges, and scrapers that worked fine last week start silently failing. The usual fix is manual — check the logs, diagnose the cause, update the config, redeploy. I wanted to see if an agent could handle that loop instead.

So I built a self-healing scraper. After each crawl, a Claude-powered agent reads the failure logs, probes the broken domains with escalating fetch strategies, and rewrites the config automatically. By the next run, it's already fixed itself.

How it works

The project has two parts: a scraper and a self-healing agent.

The scraper

main.py is a straightforward Python scraper driven entirely by a config.json file. Each domain entry tells the scraper which URLs to fetch and how to fetch them:

{
  "id": "books",
  "zyte": true,
  "browser_html": false,
  "urls": ["https://www.bookstocsrape.co.uk/products/..."]
}

There are three fetch modes:

Direct — a plain requests.get(). Fast, free, works for sites that don't block bots.
Zyte API (httpResponseBody) — routes the request through Zyte's residential proxy network. Good for sites that block datacenter IPs.
Zyte API (browserHtml) — spins up a real browser via Zyte, executes JavaScript, and returns the fully-rendered DOM. Required for sites using JS-based bot challenges.

Every request is logged to scraper.log in the same format:

2026-03-14 09:12:01 url=https://... domain_id=scan status=200

If a request throws any exception, it's recorded as a 403. That keeps the log clean and gives the agent a consistent signal to act on.

The self-healing agent

agent.py is a Claude-powered agent that runs after each crawl. It uses the Claude Agent SDK and has access to three tools: Read, Bash, and Edit — enough to operate completely autonomously.

The agent works through a staged process:

Read the log — finds every domain that returned a 403
Cross-reference the config — skips domains already configured to use Zyte
Stage 1 probe — uses the zyte-api CLI to fetch one URL per failing domain with httpResponseBody, then inspects the page <title>
Challenge detection — if the title contains phrases like "Just a moment", "Checking your browser", or "Verifying you are human", the page is flagged as a bot challenge
Stage 2 probe — challenge pages are re-probed using browserHtml, which runs a real browser to bypass JS-based detection
Config update — the agent edits config.json directly, setting zyte: true and/or browserHtml: true for domains that now work

The next crawl automatically uses the right fetch strategy. No manual intervention needed.

Design decisions

Config-driven, not code-driven

Everything lives in config.json. Adding a new domain is a one-liner, and the scraper doesn't need to know anything about individual sites — it just reads the config and follows instructions. The agent writes to the same file, so the loop closes itself naturally.

Graduated fetch strategy

Not every site needs an expensive browser render. By escalating from direct to httpResponseBody to [browserHtml](https://www.zyte.com/zyte-api/headless-browser/) only when necessary, I keep costs manageable. Browser renders are slower and consume more API credits — reserving them for sites that actually need them makes a meaningful difference at scale.

Letting the agent handle the heuristics

The challenge detection logic — matching titles against known bot-detection phrases — is exactly the kind of fuzzy heuristic that's tedious to maintain as code but natural for a language model to reason about. Claude also handles edge cases gracefully: if the zyte-api CLI isn't installed, if the log is empty, if a domain is already correctly configured. A rule-based script would need explicit handling for every one of those scenarios.

The limitations

It's worth being honest about where this approach falls short.

It's reactive, not proactive. The agent only runs after a failed crawl. If a site starts blocking mid-run, those URLs fail silently until the next cycle.

Title-based detection is fragile. Most bot-challenge pages say "Just a moment…" — but a legitimate site could theoretically use that phrase. A false positive would cause the scraper to wastefully use browser rendering where it isn't needed.

One URL per domain. The agent probes only the first failing URL for each domain. Different URL patterns on the same domain can have different bot-detection behaviour, which this doesn't account for.

No rollback. Once the config is updated, there's no way to detect if a Zyte setting later stops working and revert it automatically.

Cost opacity. The scraper logs HTTP status codes, not Zyte API credit consumption. There's no visibility into what each domain actually costs to fetch.

Where I'd take it next

Smarter challenge detection. Rather than keyword-matching on the title, the agent could read the full page HTML and make a more nuanced call — is this a product page, a login wall, or a soft block with a CAPTCHA? Each requires a different response.

Proactive monitoring. A lightweight probe running daily against each configured domain, independent of the main crawl, would let the agent update the config before a full scrape run hits a known-bad configuration.

Per-URL config. Right now zyte and browser_html are set at the domain level. Some sites serve static product pages on one path and JS-rendered category pages on another — granular per-URL settings would handle that cleanly.

Structured data extraction. Right now parse_page only pulls the page title. The natural next step is structured product extraction — price, availability, name, images — either via CSS selectors in the config or Zyte's product extraction type, which uses ML models to parse product data from any page.

Multi-agent parallelism. The self-healing loop is currently a single agent. As the config grows, a coordinator could spawn one subagent per failing domain, each running its own probe pipeline concurrently. The Claude Agent SDK supports subagents natively, so this would be a relatively small change.

The core idea is simple: a scraper that observes its own failures and reconfigures itself. What I found interesting about building it wasn't the scraping itself — it was seeing how little scaffolding the agent actually needed. Three tools, a clear task, and it handles the diagnostic work that would otherwise fall to me.

How I get Claude to build HTML parsing code the way I want it

John Rooney — Wed, 11 Mar 2026 07:13:06 +0000

Getting HTML off a page is only the first step. Once you have it, the real work begins: pulling out the data that actually matters — product names, prices, ratings, specifications — in a clean, structured format you can actually do something with.

That's what the parser skill is for. If you haven't read the introduction to skills in our fetcher post, it's worth a quick look first. But the short version is this: a skill is a SKILL.md file that gives Claude precise, reusable instructions for using a specific tool. The parser skill is one of three that together form a complete web scraping pipeline.

zytelabs / claude-webscraping-skills

claude-webscraping-skills

A collection of claude skills and other tools to assist your web-scraping needs.

video explanations:

https://youtu.be/HH0Q9OfKLu0 https://youtu.be/P2HhnFRXm-I

Other claude tools for web scraping

zyte-fetch-page-content-mcp-server

A Model Context Protocol (MCP) server that runs locally using docker desktop mcp-toolkit and help you extracts clean, LLM-friendly content from any webpage using the Zyte API. Perfect for AI assistants that need to read and understand web content. by Ayan Pahwa

Improve Claude Code WebFetch with Zyte API

When Claude encounters a WebFetch failure, it reads the CLAUDE.md instructions and makes a curl request to the Zyte API endpoint. The API returns base64-encoded HTML, which Claude decodes and processes just like it would with a normal WebFetch response. by Joshua Odmark

Claude skills vs MCP vs Web Scraping CoPilot

View on GitHub

What is a skill?

A skill is a small markdown file that tells Claude how to use a specific script or tool — what it does, when to use it, and step-by-step how to run it. Claude reads the file and follows the instructions as part of a broader workflow, with no manual intervention required.

Skills are composable by design. The fetcher skill hands raw HTML to the parser skill, which hands structured JSON to the compare skill. Each one does one job well, and they're built to work together.

The parser skill's front matter sets out its purpose immediately:

---
name: parser
description: "Extracts structured product data from raw HTML. Tries JSON-LD via"
Extruct first, falls back to CSS selectors via Parsel.
---

Two methods, one fallback. That single description line captures the entire logic of the skill.

What the parser skill does

The parser skill takes raw HTML as input and returns a structured JSON object. It uses two extraction methods in sequence, trying the more reliable one first and falling back to the more flexible one if needed.

The primary method uses Extruct to find JSON-LD data embedded in the page. JSON-LD is a structured data format that many modern sites include in their HTML specifically to make their content machine-readable — it's used for search engine optimisation and data portability. When it's present, Extruct can read it cleanly and reliably, with no need to write or maintain selectors.

If no usable JSON-LD is found, the skill falls back to Parsel, which uses CSS selectors to locate data heuristically across the page. This is more flexible but inherently tied to the page's visual structure, which can change.

When to use it

## When to use
Use this skill when you have raw HTML and need to extract structured data from
it — product details, prices, specs, ratings, or any page content.

In practice, that means the parser skill is almost always the second step in a pipeline — running immediately after the fetcher skill has retrieved your HTML. It works with any page type, and handles the most common product fields out of the box.

How it works

## Instructions
1. Save the HTML to a temporary file `page.html`
2. Run `parser.py` against it:
   python parser.py page.html

3. The script outputs a JSON object. Check the `method` field:
   - "extruct" — clean structured data was found, use it directly
   - "parsel" — fell back to CSS selectors, review fields for completeness
4. If key fields are missing from the Parsel output, ask the user which fields
   they need and re-run with --fields:
   python parser.py page.html --fields "price,rating,brand"

5. Return the parsed JSON to the conversation for use in the Compare skill.

The method field in the output is particularly useful. It tells you immediately how the data was extracted and how much trust to place in it. An "extruct" result is clean and stable. A "parsel" result is worth reviewing, especially if you're working with an unusual page layout.

The --fields flag is a practical escape hatch. Rather than requiring you to dig into the script when key data is missing, it lets you specify exactly what you need and re-run — a much more efficient loop.

Why prefer Extruct?

The notes section of the skill file makes this explicit:

## Notes
- Always prefer the Extruct path — it is more stable and requires no maintenance
- Parsel selectors are generated heuristically and may need adjustment for
  unusual page layouts
- Run once per page; pass all outputs together into the Compare skill

Parsel selectors break when sites redesign. JSON-LD, by contrast, is structured data the site publishes independently of its visual layout. A site can completely overhaul its design and its JSON-LD will often remain untouched. That stability is worth prioritising wherever possible.

What comes next

Once you've run the parser skill across all your target pages, you have a set of structured JSON objects ready to compare. That's where the compare skill picks up — generating tables, summaries, and side-by-side analysis from the extracted data.

Do you need a skill?

The parser skill works well when the data you need maps cleanly onto fields that Extruct or Parsel can find — product names, prices, ratings, and similar structured attributes that sites commonly expose through JSON-LD or consistent HTML patterns. For that category of task, the skill is fast to apply and requires no custom code.

See our post about Skills vs MCP vs Web Scraping Copilot (our VS Code Extension):

zyte.com

But not every extraction problem fits that mould. If you're working with pages that don't include JSON-LD and have highly irregular layouts, Parsel's heuristic selectors may return incomplete or inconsistent results, and you'll spend time debugging field by field. In those cases, a purpose-built extraction script using Parsel or BeautifulSoup directly — with selectors you've written and tested against the specific target — will be more reliable.

For larger-scale or more complex extraction work, Zyte API's automatic extraction capabilities go further still. Rather than relying on selectors at all, automatic extraction uses AI to identify and return structured data from a page without requiring you to specify fields or maintain selector logic. If you're extracting data from many different site structures, or you need extraction to keep working through site redesigns without manual intervention, that's a more robust foundation than a skill-based approach. The parser skill is best understood as a practical middle ground: fast to use, good enough for a wide range of common cases, and easy to slot into a pipeline — but not a replacement for extraction tooling built for scale or resilience.

I gave Claude access to a web scraping API

John Rooney — Wed, 11 Mar 2026 07:10:43 +0000

If you've worked with Claude for any length of time, you've probably noticed it can do a lot more than answer questions. With the right setup, it can take actions — running scripts, processing files, working through multi-step workflows autonomously. Skills are what make that possible.

What is a skill?

A skill is a small, self-contained instruction set that tells Claude how to use a specific tool or script to accomplish a well-defined task. Technically, it's a markdown file — a SKILL.md — that describes what a tool does, when to reach for it, and exactly how to run it. Claude reads that file and follows the instructions as part of a larger workflow.

Skills are designed to be composable. Each one does one thing well, and they're built to hand off to each other. The fetcher skill retrieves HTML. The parser skill extracts data from it. The compare skill turns multiple parsed outputs into a structured comparison. Together, they form a complete scraping pipeline — and Claude orchestrates the whole thing.

See our skills here: https://github.com/zytelabs/claude-webscraping-skills

The skill format looks like this:

---
name: fetcher
description: "Fetches raw HTML from a URL using httpx, with automatic fallback to Zyte API if blocked."
---

That front matter is what Claude uses to match the right skill to the right task. The description is deliberately precise: it tells Claude not just what the skill does, but how it does it, so Claude can reason about whether it's the right tool for the job.

What the fetcher skill does

The fetcher skill's job is exactly what it sounds like: given a URL, fetch the raw HTML and return it. It uses httpx as its primary HTTP client — a modern, performant Python library well suited to scraping workloads.

What makes it more than a simple wrapper is the fallback logic. A significant number of sites actively block automated requests. Without a fallback, a blocked request just fails, and you're left manually diagnosing why. The fetcher skill handles this automatically. If a request comes back with a BLOCKED status, it retries via Zyte API, which provides built-in unblocking. Most of the time, you get your HTML without ever needing to intervene.

When to use it

The skill's SKILL.md is explicit about this:

## When to use
Use this skill when the user provides one or more URLs and asks you to fetch,
retrieve, scrape, or get the HTML or page content.

In practice, that means any time you're starting a scraping or data extraction task and you have a URL to work from. It's the entry point for the pipeline.

How it works

The instructions in the skill file are straightforward:

## Instructions
1. Run `fetcher.py` with the URL as an argument:
   python fetcher.py <url>

2. If the script returns a successful HTML response, return the HTML to the
   conversation for use in the next step.
3. If the script returns a `BLOCKED` status, re-run with the `--zyte` flag:
   python fetcher.py <url> --zyte

4. Inform the user if a URL could not be fetched after both attempts.

The two-step process keeps things efficient. httpx is fast and lightweight, so it handles the majority of requests without needing to route through Zyte API. The fallback only kicks in when it's needed. If both attempts fail, Claude surfaces that to you clearly rather than silently moving on.

For multiple URLs, the script runs once per URL — there's no batching — so Claude loops through a list sequentially.

Transparency about failure

One detail worth highlighting is the final instruction: inform the user if a URL could not be fetched after both attempts. This might seem obvious, but it reflects a design principle worth being explicit about. A skill that silently drops failed URLs would produce incomplete data downstream, and you might not notice until you're looking at a comparison table with missing rows. Surfacing failures immediately keeps the pipeline honest.

What comes next

The fetcher skill's output is raw HTML — exactly what the parser skill expects as its input. The two are designed to be used in sequence. Once you have the HTML, the parser skill takes over, extracting structured data through JSON-LD or CSS selectors depending on what the page contains.

That handoff is documented in the skill's notes:

## Notes
- For multiple URLs, run the script once per URL
- Pass the raw HTML output into the Parser skill for extraction

The pipeline continues from there.

Do you need a skill?

Skills are a good fit when you have a well-defined, repeatable task that benefits from consistent behaviour across many runs. Fetching HTML from a URL is a clear example: the inputs and outputs are predictable, the fallback logic is always the same, and packaging that into a skill means Claude applies it reliably without you having to re-explain the process each time.

Read our break down of Skills vs MCP vs Web Scraping Copilot here - our VS Code extension

That said, skills aren't always the right tool. If you only need to fetch a handful of pages once, asking Claude to write a quick httpx script directly may be faster and more flexible. Similarly, if your target sites have unusual behaviour — rate limiting, JavaScript rendering, login walls, or multi-step navigation — a bespoke Scrapy spider built with Zyte API gives you far more control than a general-purpose fetch wrapper. Scrapy's middleware architecture, item pipelines, and scheduling make it better suited to large-scale or complex crawls where you need precise control over every aspect of the request cycle.

The fetcher skill sits in the middle: more structured than an ad hoc script, less complex than a full Scrapy project. It's the right choice when you want Claude to handle straightforward retrieval as part of a larger automated workflow, without the overhead of setting up and maintaining a dedicated spider.

I built a Claude Code skill that screenshots any website (and it handles anti-bot sites too)

Ayan Pahwa — Fri, 06 Mar 2026 14:40:32 +0000

TLDR;

Automate screenshot capture for any URL with JavaScript rendering and anti-ban protection — straight from your AI assistant.

Taking a screenshot of a webpage sounds trivial, until you need to do it at scale. Modern websites throw every obstacle imaginable in your way: JavaScript-rendered content that only appears after a React bundle loads, bot-detection systems that serve blank pages to automated headless browsers, geo-blocked content, and CAPTCHAs that appear the moment traffic patterns look non-human. For a handful of URLs you can get away with Puppeteer or Playwright. For hundreds or thousands? You need infrastructure built for the job.

The Zyte API was designed specifically for this problem. It handles JavaScript rendering, anti-bot fingerprinting, rotating proxies, and headless browser management so you don't have to and what better way to do it straight from your LLM supplying the URLs? Hence I created this zyte-screenshots Claude Skill, which you can use to trigger the entire workflow- API call, base64 decode, PNG save on your filesystem, all just by chatting with Claude.

In this tutorial, we'll walk through exactly how the skill works, how to set it up, and how to use it to capture production-quality screenshots of any URL.

Why Use the Zyte API for Screenshots?

Before diving into the skill itself, it's worth understanding what makes the Zyte API uniquely suited to screenshot capture at scale.

1. Full JavaScript Rendering

Single-page applications built with React, Vue, Angular, or Next.js don't serve their content in the raw HTML response, they render it client-side after the page loads. Tools that capture the raw HTTP response will get a blank shell. Zyte's screenshot endpoint fires a real headless browser, waits for the DOM to fully settle, then captures the final rendered state.

2. Anti-Bot and Anti-Ban Protection

Enterprise-grade sites use fingerprinting libraries to detect automation. They check TLS fingerprints, browser headers, canvas rendering patterns, mouse movement entropy, and dozens of other signals. Zyte's infrastructure is battle-tested to pass these checks so your screenshots won't return a "Access Denied" page.

3. Scale Without Infrastructure

Managing a fleet of headless browser instances, proxy rotation, retries, and residential IP pools is a serious engineering investment. Zyte abstracts all of this into a single API call.

4. One API, Any URL

Whether the target is a static HTML page, a JS-heavy SPA, a behind-login dashboard (with session cookies), or a geo-restricted site, the same API call structure works. The skill you're about to install uses this endpoint.

What Is the zyte-screenshots Claude Skill?

Claude Skills are reusable instruction packages that extend Claude's capabilities with domain-specific workflows. The zyte-screenshots skill teaches Claude how to:

Accept a URL from the user in natural language
Read the ZYTE_API_KEY environment variable
Construct and execute the correct curl command against https://api.zyte.com/v1/extract
Pipe the JSON response through jq and base64 --decode to produce a PNG file
Derive a clean filename from the URL (e.g. https://quotes.toscrape.com becomes quotes.toscrape.png)
Report the exact file path and describe what's visible in the screenshot in one sentence

In practice, this means you can open Claude, say "screenshot https://example.com", and have a pixel-perfect PNG on your filesystem in seconds, no browser, no script, no Puppeteer config.

Prerequisites

Before installing the skill, make sure you have the following:

Tools

curl: Pre-installed on macOS and most Linux distributions. On Windows, use WSL or Git Bash.
jq: A lightweight JSON processor. Install via brew install jq (macOS) or sudo apt install jq (Ubuntu/Debian).
base64: Standard on all Unix-like systems.
Claude desktop app with Skills support enabled.

A Zyte API Key

Sign up at zyte.com and navigate to your API credentials. The free tier includes enough credits to get started with testing. Copy your API key, you'll set it as an environment variable.

💡 Pro tip: Set your ZYTE_API_KEY in your shell profile (~/.zshrc or ~/.bashrc) so it's always available: export ZYTE_API_KEY="your_key_here" or pass it along your prompt

Installing the Skill

Step 1: Download the Skill from GitHub

The skill is open source and available at github.com/apscrapes/claude-zyte-screenshots. Download the latest release ZIP from the repository's Releases page, or clone it directly:

git clone https://github.com/apscrapes/claude-zyte-screenshots.git

Step 2: Import into Claude

Open the Claude desktop app or go to Claude.ai
Navigate to Settings → Skills
Click Import Skill
Select the SKILL.md file from the downloaded ZIP
Claude will confirm the skill has been installed

Step 3: Set Your API Key

In your terminal, export your Zyte API key or pass it along your prompt:

export ZYTE_API_KEY="your_zyte_api_key_here"

To make this permanent across terminal sessions, add the export line to your ~/.zshrc or ~/.bashrc file.

Using the Skill: Step-by-Step

Basic Screenshot

Once the skill is installed and your API key is set, using it is as simple as asking Claude:

Screenshot https://quotes.toscrape.com using zyte-screenshots skill

Claude will:

Verify ZYTE_API_KEY is available in the environment
Derive the output filename: quotes.toscrape.png
Execute the following command:

curl -s https://api.zyte.com/v1/extract \
  -u "$ZYTE_API_KEY": \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://quotes.toscrape.com",
    "screenshot": true
  }' \
| jq -r '.screenshot' \
| base64 --decode > quotes.toscrape.png

Verify the file is non-empty
Open and inspect the image
Return the file location and a one-line visual description

Example output from Claude:

✅ Screenshot captured!

📁 Location: /home/you/quotes.toscrape.png

🖼️  What's in it: A clean light-themed page listing inspirational quotes
   with author attributions and tag filters in the sidebar.

Under the Hood: How the API Call Works

Let's break down the exact curl command the skill executes:

curl -s https://api.zyte.com/v1/extract \
  -u "$ZYTE_API_KEY": \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://target-site.com",
    "screenshot": true
  }' \
| jq -r '.screenshot' \
| base64 --decode > output.png

curl -s — Silent mode; suppresses progress output.

-u "$ZYTE_API_KEY": — HTTP Basic Auth. Zyte uses the API key as the username with an empty password.

-H "Content-Type: application/json" — Tells the API to expect a JSON body.

-d '{...}' — The JSON request body. Setting screenshot: true instructs Zyte to return a base64-encoded PNG of the fully rendered page.

| jq -r '.screenshot' — Extracts the raw base64 string from the JSON response.

| base64 --decode — Decodes the base64 string into binary PNG data.

> output.png — Writes the binary data to a PNG file.

The Zyte API handles everything in between — spinning up a headless Chromium instance, loading the page with real browser fingerprints, waiting for JavaScript execution to complete, and rendering the final DOM to a pixel buffer.

This was a fun weekend project I put together, let me know your thoughts on our Discord and feel free to play around with it. I'd also love to know if you create any useful claude skills or mcp server, so say hi on our discord.

Tags: web scraping • Zyte API • screenshots at scale • JavaScript rendering • anti-bot • Claude AI • Claude Skills • automation • headless browser • site APIs

Why your Python request gets 403 Forbidden

John Rooney — Wed, 04 Mar 2026 09:11:29 +0000

If you’ve had your HTTP request blocked despite using correct headers, cookies, and clean IPs, there’s a chance you are running into one of the simplest forms of blocking, and one of the most confusing for beginners.

Chances are, you will recognise the problem. You found the hidden API, and your request works perfectly in Postman... but it fails instantly within your Python code.

It’s called TLS fingerprinting. But the good news is, you can solve it. In fact, when I showed this to some developers at Extract Summit, they couldn’t believe how straightforward it was to fix.

CAPTION: “I copied the request -> matching headers, cookies and IP, but it still failed?”

Your TLS fingerprint

Let’s start with a question. How do the servers and websites know you’ve moved from Postman to making the request in Python? What do they see that you can’t? The key is your TLS fingerprint.

To use an analogy: We’ve effectively written a different name on a sticker and stuck it to our t-shirt, hoping to get past the bouncer at a bar.

Your nametag (headers) says "Chrome."

But your t-shirt logo (TLS handshake) very obviously says "Python."

It’s a dead giveaway. This mismatch is spotted immediately. We need to change our t-shirt to match the nametag.

To understand how they spot the “logo”, we need to look at the initial “Client Hello” packet. There are three key pieces of information exchanged here:

Cipher suites: The encryption methods the client supports.
TLS extensions: Extra features (like specific elliptic curves).
Key exchange algorithms: How they agree on a password.

This is because Python’s requests library uses OpenSSL, while Chrome uses Google's BoringSSL. While they share some underlying logic, their signatures are notably different. And that’s the problem.

OpenSSL vs. BoringSSL

The root cause of this mismatch lies in the underlying libraries.

Python’s requests library relies on OpenSSL, the standard cryptographic library found on almost every Linux server. It is robust, predictable, and remarkably consistent.

Chrome, however, uses BoringSSL, Google’s own fork of OpenSSL. BoringSSL is designed specifically for the chaotic nature of the web and it behaves very differently.

The biggest giveaway between the two is a mechanism called GREASE (Generate Random Extensions And Sustain Extensibility).

[
  "TLS_GREASE (0xFAFA)",
   ....
]

Chrome (BoringSSL) intentionally inserts random, garbage values into the TLS handshake - specifically, in the cipher suites and extensions lists. It does this to "grease the joints" of the internet, ensuring that servers don't crash when they encounter unknown future parameters.

This is one of the key changes

Chrome: Always includes these random GREASE values (e.g., 0x0a0a).
Python (OpenSSL): Never includes them. It only sends valid, known ciphers.

So, when an anti-bot system sees a handshake claiming to be "Chrome 120" but lacking these random GREASE values, it knows instantly that it is dealing with a script. It’s not just that your shirt has the wrong logo; it’s that your shirt is too clean.

JA3 hash

Anti-bot companies take all that handshake data and combine it into a single string called a JA3 fingerprint.

Salesforce invented this years ago to detect malware, but it found its way into our industry as a simple, effective way to fingerprint HTTP requests. Security vendors have built databases of these fingerprints.

It is relatively straightforward to identify and block any request coming from Python’s default library because its JA3 hash is static and well-known.

This code snippet would yield the below JSON response.

def get_ja3_info():
    url = "https://tls.peet.ws/api/clean"
    with requests.Session() as session:
        response = session.get(url)
        response.raise_for_status()
        data = response.json()
        print(json.dumps(data))

Note the lack of akamai_hash

{
  "ja3": 
"771,4866-4867-4865-49196-49200-49195-49199-52393-52392-49188-49192-49187-49191-159-158-107-103-255,0-11-10-16-22-2
3-49-13-43-45-51-21,29-23-30-25-24-256-257-258-259-260,0-1-2",
  "ja3_hash": "a48c0d5f95b1ef98f560f324fd275da1",
  "ja4": "t13d1812h1_85036bcba153_375ca2c5e164",
  "ja4_r": 
"t13d1812h1_0067,006b,009e,009f,00ff,1301,1302,1303,c023,c024,c027,c028,c02b,c02c,c02f,c030,cca8,cca9_000a,000b,000
d,0016,0017,002b,002d,0031,0033_0403,0503,0603,0807,0808,0809,080a,080b,0804,0805,0806,0401,0501,0601,0303,0301,030
2,0402,0502,0602",
  "akamai": "-",
  "akamai_hash": "-",
  "peetprint": 
"772-771|1.1|29-23-30-25-24-256-257-258-259-260|1027-1283-1539-2055-2056-2057-2058-2059-2052-2053-2054-1025-1281-15
37-771-769-770-1026-1282-1538|1||4866-4867-4865-49196-49200-49195-49199-52393-52392-49188-49192-49187-49191-159-158
-107-103-255|0-10-11-13-16-21-22-23-43-45-49-51",
  "peetprint_hash": "76017c4a71b7a055fb2a9a5f70f05112"
}

Putting the above JA3 hash into ja3.zone clearly shows this is a python3 request, using urllib3:

What’s the solution?

As mentioned, simply changing headers and IP addresses won’t make a difference, as these are not part of the TLS handshake. We need to change the ciphers and Extensions to be more like what a browser would send.

The best way to achieve this in Python is to swap requests for a modern, TLS-friendly library like curl_cffi or rnet.

Here is how easy it is to switch to curl_cffi:

from curl_cffi import requests
# note the impersonate argument & import above
def get_ja3_info():
    url = "https://tls.peet.ws/api/clean"
    with requests.Session() as session:
        response = session.get(url, impersonate="chrome")
        response.raise_for_status()
        data = response.json()
        print(json.dumps(data))

"akamai_hash": "52d84b11737d980aef856699f885ca86"

CAPTION: Note - I searched via the akamai_hash here as the fingerprint from the JA3 hash wasn’t in this particular database.

By adding that impersonate parameter, you are effectively putting on the correct t-shirt.

Summary

Make curl_cffi or rnet your default HTTP library in Python. This should be your first port of call before spinning up a full headless browser.

A simple change (which brings benefits like async capabilities) means you don’t fall foul of TLS fingerprinting. curl-cffi even has a requests-like API, meaning it's often a drop-in replacement.

Hybrid scraping: The architecture for the modern web

John Rooney — Wed, 25 Feb 2026 17:05:36 +0000

If you scrape the modern web, you probably know the pain of the JavaScript challenge.

Before you can access any data, the website forces your browser to execute a snippet of JavaScript code. It calculates a result, sends it back to an endpoint for verification, and often captures extensive fingerprinting data in the process.

Once you pass this test, the server assigns you a session cookie. This cookie acts as your "access pass." It tells the website, "This user has passed the challenge," so you don’t have to re-run the JavaScript test on every single page load.

For web scrapers, this mechanism creates a massive inefficiency.

It looks like you are forced to use a headless browser (like Puppeteer or Playwright) for every single request just to handle that initial check. But browsers are heavy, they are slow and they consume massive amounts of RAM and bandwidth.

Running a browser for thousands of requests can quickly become an infrastructure nightmare. You end up paying for CPU cycles just to render a page when all you wanted was the JSON payload.

The solution: Hybrid scraping

The answer to this problem is a technique I’ve started calling hybrid scraping.

This involves using the browser only to open the initial request, grab the cookie, and create a session. Once you have them, you extract that session data and hand it over to a standard, lightweight HTTP client.

This architecture gives you the access of a browser with the speed and efficiency of a script.

Implementing this in Python

To build this in Python, we need two specific packages:

A browser: We will use ZenDriver, a modern wrapper for headless Chrome that handles the "undetected" configuration for us.
HTTP client: We will use rnet, a Rust-based HTTP client for Python.

But why rnet? Well, within the initial TLS handshake where the client/server “hello” is sent, the information traded here can be fingerprinted, taking in things like the TLS version and the ciphers available for encryption. This can be hashed into a fingerprint and profiled.

Python’s requests package, which uses urllib from the standard library, has a very distinctive TLS fingerprint, containing ciphers (amongst other things) that aren’t seen in a browser. This makes it very easy to spot. Both rnet, and other options such as curl-cffi, are able to send a TLS fingerprint similar to that of a browser. This reduces the chances of our request being blocked.

Here is how we assemble the pipeline.

Step 1: Load the page (The handshake)

First, we define our browser logic. Notice that we are not trying to parse HTML here. Our only goal is to visit the site, pass the initial JavaScript challenge, and extract the session cookies.

import zendriver as zd
import asyncio

async def get_cookies():
    """
    Use ZenDriver to launch a browser, navigate to the page, 
    and retrieve the cookies.
    """

    browser = await zd.start()

    # Hit the homepage to trigger the check
    await browser.get("https://auto.hylnd7.com")

    # Wait briefly for the JS challenge to complete
    await asyncio.sleep(1) 

    # Extract the cookies
    requests_style_cookies = await browser.cookies.get_all()
    await browser.stop()

    return requests_style_cookies

What’s happening here:

We launch the browser, visit the site, and wait just one second for the JS challenge to run. Once we have the cookies, we call browser.stop(). This is the most important line: we do not want a browser instance wasting resources when we don’t need it.

Step 2: Use the cookies

Now that we have the "access pass," we can switch to our lightweight HTTP client. We take those cookies and inject them into the rnet client headers.

from rnet import Client, Emulation

async def http_request_rnet(cookies=None):
    """
    Make a fast request using RNet with the borrowed cookies.
    """
    headers = {
        "referer": "https://auto.hylnd7.com/",
    }

    # Format the browser cookies into a simple HTTP header string
    if cookies:
        cookie_list = []
        for cookie in cookies:
            cookie_list.append(f"{cookie.name}={cookie.value}")
        headers["Cookie"] = "; ".join(cookie_list)

    # We use Emulation.Chrome142 to change the TLS Fingerprint.
    # This is site dependent - but worth using
    client = Client(emulation=Emulation.Chrome142, headers=headers)

    response = await client.get("https://auto.hylnd7.com/api/products?page=1&limit=8")
    return response

What’s happening here:

We convert the browser's cookie format into a standard header string. Note the “Emulation.Chrome142” parameter. We are layering two techniques here: hybrid scraping (using real cookies) and TLS fingerprinting (using a modern HTTP client). This double-layer approach covers all our bases.

(Note: Many HTTP clients have a cookie jar that you could also use; for this example, sending the header directly worked perfectly).

Step 3: Run the code

Finally, we tie it together. For this demo, we use a simple argparse flag to show the difference with and without the cookie.

async def main(use_cookies: bool):
    cookies = None

    # The Decision Logic
    if use_cookies:
        cookies = await get_cookies() # Run the heavy browser

    # Always run the fast HTTP client
    resp = await http_request_rnet(cookies)

    status_code = resp.status
    print("Status Code:", status_code)

    if status_code == 200:
        print("Response Body:", await resp.json())
    else:
        print("Request blocked")

Get the complete script

Want to run this yourself? We’ve put the full, copy-pasteable script (including the argument parsers and imports) in the block below.

uv init
uv add zendriver rnet rich
# linux/mac
source .venv/bin/activate\
# windows
.venv\Scripts\activate

import argparse
import asyncio

import zendriver as zd
from rnet import Client, Emulation
from rich import print


async def http_request_rnet(cookies=None):
    """
    Make an HTTP GET request using rnet with the provided cookies. Cookies are sent in the headers. Note for this site we need the referer too.
    Return the Response Object.
    """
    headers = {
        "referer": "https://auto.hylnd7.com/",
    }

    if cookies:
        cookie_list = []
        for cookie in cookies:
            # Adjust based on the actual structure of the cookie object from zendriver
            # If it's a dict: cookie['name'], cookie['value']
            # If it's an object: cookie.name, cookie.value
            cookie_list.append(f"{cookie.name}={cookie.value}")
        headers["Cookie"] = "; ".join(cookie_list)

    client = Client(emulation=Emulation.Chrome142, headers=headers)
    response = await client.get("https://auto.hylnd7.com/api/products?page=1&limit=8")
    return response

async def get_cookies():
    """
    Use zendriver to launch a browser, navigate to a page, and retrieve cookies.
    """
    browser = await zd.start()
    await browser.get("https://auto.hylnd7.com")
    await asyncio.sleep(1)
    requests_style_cookies = await browser.cookies.get_all()
    await browser.stop()
    return requests_style_cookies

async def main(use_cookies: bool):
    cookies = None
    if use_cookies:
        cookies = await get_cookies()

    resp = await http_request_rnet(cookies)
    status_code = resp.status
    print("Status Code:", status_code)

    if status_code == 200:
        print("Response Body:", await resp.json())

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Make HTTP request with optional browser cookies")
    parser.add_argument(
        "--cookies",
        type=lambda x: x.lower() == "true",
        default=False,
        help="Set to 'true' to launch browser and get cookies, 'false' to skip (default: false)"
    )
    args = parser.parse_args()
    asyncio.run(main(args.cookies))

Pros and Cons of Hybrid Scraping

Feature	Pros	Cons
Efficiency	Reduces RAM usage massively compared to pure browser scraping.	Higher complexity: You must manage two libraries (zendriver and rnet) and the glue code.
Speed	HTTP requests complete in milliseconds. Browsers take seconds.	State management: You need logic to handle cookie expiry. If the cookie dies, you must "wake up" the browser.
Access	You get the verification of a real browser without the drag.	Maintenance: You are debugging two points of failure: the browser's ability to solve the challenge, and the client's ability to fetch data.

Final thoughts

For smaller jobs, it might be easier to just use the browser; the benefits won’t necessarily outweigh the extra complexity required.

But for production pipelines, this approach is the standard. It treats the browser as a luxury resource: used only when strictly necessary to unlock the door, so the HTTP client can do the real work. It’s this session and state management that allows you to scrape harder-to-access sites effectively and efficiently.

If building this orchestration layer yourself feels like too much overhead, this is exactly what the Zyte API handles internally. We manage the browser/HTTP switching logic automatically, so you just make a single request and get the data.

Raspberry Pi & E-ink scrapes & displays the price of Gold today

Ayan Pahwa — Tue, 24 Feb 2026 08:39:31 +0000

They say that “data is the new oil”, but there’s another hot commodity that’s setting markets alight - precious metals.

In the last 12 months, the value of gold has surged about 75%, while silver has boomed more than 200%. That’s why I, like a growing number of others, now trade in the metal markets.

These days, it is possible to buy digital versions of precious metals. But I think of myself as a collector - I like to buy real, solid coins or bullions whenever I get a chance.

In the last two years, I have acquired a small collection of gold bullions and silver coins, which have appreciated healthily. But I am not planning to sell and book a profit just yet. In fact, I want to buy more, especially when there’s a dip in the price.

There’s just one problem that hits this hobby - prices of actual physical gold and silver bullions are very different in the retail market from stock exchanges’ spot prices and keeping track of them manually is cumbersome specially with a full time job.

To take advantage of the dips and price arbitrage, I need to automate my decisions. To buy gold old-style, I need a key resource from the modern trading toolset - data.

Turn data into gold

The All-India Gem And Jewellery Domestic Council (GJC), a national trade federation for the promotion and growth of trade in gems and jewellery across, is the go-to site listing latest retail rates for gold and silver.

Alas, it doesn’t offer an API to access that data. But fear not - with web scraping skills and Zyte API, I can extract these prices quickly and regularly.

And I can do it using some of the tech I love to tinker with.

I call it ExtractToInk - a custom project that pulls the latest prices on a two-inch, 250x122 e-ink display powered by a retired Raspberry Pi (total cost under US$50).

This is the story of how I power my quest for rapid riches using cheap old hardware and the world’s best web scraping engine - and how you can, too.

Mining for data

Like many modern sites, GJC’s includes both JavaScript rendering for HTML and protection mechanisms- technologies that can break brittle traditional scraping solutions.

This project connects all the dots:

Web → Extract → Parse → Render → Physical display

Tech stack

Hardware

Raspberry Pi (tested on Pi Zero 2 W), it should run on any Raspberry Pi Board
Pimoroni Inky pHAT (Black, SSD1608)

Software

Python 3
Zyte API: to get rendered HTML
BeautifulSoup: to parse HTML
Pillow and Inky Python libraries: for e-ink display stuff

Now let’s get building.

Step 1: Prepare hardware

Setup your Raspberry Pi. In my case, I am using Raspberry Pi OS booted from the SD card.

Depending on which display you use, it most probably will be connected to the Pi over i2c bus or SPI bus protocol - so, enable your display type by entering:

sudo raspi-config

Now attach your e-ink display and do a quick reboot

You might need to install libraries to use your e-ink display.

Step 2: Fetching rendered HTML with Zyte API

The source site, GJC, renders prices dynamically, using JavaScript - something which can make plain HTTP requests unreliable.

No problem. By accessing the page through Zyte API, we can set browserHTML mode to return the page content as though rendered in an actual browser.

Instead of fighting JavaScript, we let Zyte handle it.

html = requests.post(`  
    `"https://api.zyte.com/v1/extract",`  
    `auth=(ZYTE_API_KEY, ""),`  
    `json={"url": URL, "browserHtml": True},`  
`).json()["browserHtml"]

Note: there is no Selenium here, and no headless browsers. This is much more reliable for production-style scraping

Step 3: Parsing with CSS selectors

Once we have clean HTML, parsing becomes straightforward.

Gold prices

Let’s locate the actual prices in the page mark-up.

for row in soup.select(".gold_rate table tr"):`  
    `label = row.select_one("td strong")`  
    `values = row.select("td strong")`

    `if not label or len(values) < 2:`  
        `continue`

    `text = label.get_text(strip=True)`  
    `priceText = values[1].get_text()`

    `if "Standard Rate Buying" in text:`  
        `goldBuying = re.search(r"\d[\d,]*", priceText).group(0)`

    `if "Standard Rate Selling" in text:`  
        `goldSelling = re.search(r"\d[\d,]*", priceText).group(0)

We’re deliberately using:

CSS selectors (easy to find from your browser’s DevTools).
Minimal regular expressions (only for numeric extraction).
Defensive checks to avoid brittle parsing.

Silver prices

Silver appears outside the main table, so we filter it carefully:

for strong in soup.select("p > strong"):`  
    `text = strong.get_text(" ", strip=True)`

    `if "Standard Rate Selling" in text and not strong.find_parent("table"):`  
        `silver = re.search(r"\d[\d,]*", text).group(0)`  
        `break

Step 4: Rendering for e-ink

For this project, I did not want to pipe data into a web dashboard on a computer monitor.

E-ink is always-on, low power, distraction-free and perfect for “ambient information” like this.

So, it’s a great fit for data like prices, weather, status indicators and system health.

But e-ink displays are not normal screens.

They are typically black-and-white, have high contrast and are slow to refresh.

What’s more, no two e-ink displays are made the same way. Every vendor has different support packages so, whichever you end up using, make sure to read the documentation and change the code accordingly.

In my case, I am using Pimoroni inky PHat. The supplied Python library has great built-in examples to get you quickly up and running. I used the helper function to render texts on the display, ex, the build in draw.text() function comes handy:

Draw silver selling price

    draw.text((x, y), f"Silver : {silverPrice}", fill=(0, 0, 0), font=fontBig)

Taking it furtherSection about the finished product

I built this project to use web data thoughtfully, connecting it to the physical world, and building pipelines that feel calm, reliable, and purposeful. When I am at my work-desk the project actively tells me the current prices so I can buy new coins if I see a price drop.

I can further extend this to place automatic orders on the website and secure me a coin at my desired strike price.

If you want to take this further, you could also:

Run it via cron every 10 minutes : The website I am targeting only refreshes prices twice a day, so my cron job runs every 12 hours, but, if you need faster data, you can scrape a site with more real-time updates.
Add more commodities or currencies.
Turn it into a systemd service to run at start time.
Swap e-ink for another output (PDF, LED, dashboard).

If you’re exploring Zyte API, or looking for real-world scraping examples beyond CSVs and JSON files, this project is a great place to start.

You can get my code in the ExtractToInk GitHub repository now.