DEV Community: Extract by Zyte

Stopping a crawl when the data stops looking real

John Rooney — Tue, 22 Sep 2026 16:46:13 +0000

TL;DR

I built scrapy-jev, a Scrapy pipeline that asks a fast, cheap AI model whether each scraped field still looks like a real product name, price, SKU, or category, and stops the crawl when too many items fail the check. It works, and it caught the kind of breakage that normally goes unnoticed until someone opens the exported CSV. It also has a narrow job: it catches values that look wrong, not selectors that point at the wrong thing but happen to still return something plausible.

The problem I was trying to solve

Scrapers don't announce when they break. A site renames a CSS class, or restructures a product card, and the extractor starts returning the wrong text, or nothing at all, while every other signal in the crawl looks healthy. No exception fires. Nothing appears in the log. You find out when someone downstream opens the export and the price column is full of category names.

What I wanted was something that could ask, mid-crawl, "does this still look like a real value for this field?" and stop before that damage spread across thousands of pages, rather than finding out afterward from a customer complaint or a broken pipeline.

What I built it with

I wrote the client, pipeline, and add-on with DeepSeek V4 Pro doing most of the coding, working from a spec of the endpoint and the pipeline behavior I wanted.

The core idea only works if judging a field is cheap and fast enough to run inline, on every crawl, without turning into its own cost center. A general-purpose LLM generating a paragraph of reasoning per field would have been too slow and too expensive to run at any real scale.

I used Jev, a "System One" model from TypeSafe AI, reached through OpenRouter's alpha endpoint. Instead of generating text, Jev returns typed, calibrated decisions. You send it a state and one or more typed questions, and it returns structured answers with probabilities attached, in roughly 70 to 500 ms.

I wrote more about what Jev actually is, and where a model like it fits into web scraping, in an earlier post.

For each field, I sent a two-level score question, "does this look like a plausible, correctly-scraped value for this field, or not," which meant the returned score was directly interpretable as P(plausible). One request carried one question per field, evaluated against the whole item at once, so a five-field item cost one call, not five.

On top of that I built three pieces:

JevClient, a thin wrapper that reads item fields through itemadapter, so it works against Scrapy Item, plain dicts, attrs, and dataclasses, and returns a score per field.
QualityGatePipeline, an item pipeline that scores a sample of items, requires every field on an item to clear a threshold to count as a pass (not a majority, since one bad field should fail the item), and stops the spider if the sample's pass rate falls below a configured threshold.
Addon, so installing the package doesn't require hand-editing ITEM_PIPELINES. It self-disables if no fields are configured, so adding the dependency does nothing until you turn it on.

Two design choices were deliberate rather than obvious. First, judging a fixed-size sample rather than every item keeps the cost of the check independent of crawl size. A 1,000-page crawl and a 100,000-page crawl spend the same amount on verification. Second, a missing field counts as a failure rather than being skipped, because a selector silently returning None is the most common way scrapers actually break, and skipping it would have defeated the entire point.

I checked the actual dollar cost against real calls logged during testing rather than guessing. A three-field scoring request, one field was missing and never sent, ran to 570 input tokens and cost $0.000024, which works out to $0.042 per million input tokens, output is free. Scaled to a full four-field item across a 1-million-item crawl, that's roughly 700 million input tokens, somewhere around $30. Scoring a fixed sample of 20 items per crawl instead, regardless of whether the crawl has 1,000 items or 1,000,000, costs a fraction of a cent.

At Jev's actual pricing, the dollar difference between sampling and scoring everything isn't the main argument, $30 per million items is trivial either way. The real argument for sampling is that scoring every item means every item's pipeline waits on a round trip to an external API, and the crawl now depends on roughly a million of those calls all succeeding rather than twenty.

Trying it yourself

Installing it doesn't touch ITEM_PIPELINES by hand:

uv add scrapy-jev

Turning it on is three lines in settings.py: register the add-on, and tell it which fields to judge.

ADDONS = {
    "scrapy_jev.Addon": 350,
}

JEV_FIELDS = ["name", "price", "sku", "category"]

JEV_API_KEY, or the OPENROUTER_API_KEY environment variable, has to be set too. If neither is present, or JEV_FIELDS is empty, the add-on raises NotConfigured and gets out of the way rather than doing nothing silently.

The rest is optional and has defaults tuned for a small verification sample rather than exhaustive checking:

Setting	Default	Meaning
`JEV_FIELDS`	`[]`	Fields to judge on each item.
`JEV_SAMPLE_SIZE`	`20`	Items to judge before deciding.
`JEV_PASS_RATE_THRESHOLD`	`0.7`	Minimum share (0–1) of items that must pass.
`JEV_FIELD_THRESHOLD`	`0.5`	Per-field plausibility below which a field fails.
`JEV_MODEL`	`~typesafe/jev-latest`	Model id for the decisions endpoint.
`JEV_BASE_URL`	`https://openrouter.ai/api/alpha/decisions`	Endpoint.
`JEV_PIPELINE_PRIORITY`	`400`	`ITEM_PIPELINES` priority used by the add-on.
`JEV_API_KEY` / `OPENROUTER_API_KEY`	—	API key (setting or environment variable).
`JEV_STOP_REASON`	`"Extraction quality fell below threshold (Jev plausibility check)"`	Closing reason logged when the gate stops the spider.

The bug that mattered more than the feature

The most useful thing I learned building this had nothing to do with AI. My first version raised CloseSpider from inside the pipeline to stop the crawl, and the crawl didn't stop. Scrapy's item-processing path wraps process_item in a bare except Exception, logs "Error processing item", drops the item, and keeps going. CloseSpider is an Exception subclass, so it gets swallowed like any other error. The gate fired every time, and the crawl just kept going anyway, for another thousand pages, retries and all.

The fix was to stop asking politely and instead tell the engine directly:

from scrapy.utils.defer import deferred_from_coro
deferred_from_coro(crawler.engine.close_spider_async(reason=reason))

That's not a Jev problem or an AI problem. It's the kind of thing you only find by actually watching a crawl you expected to stop keep running, and it's a reminder that the flashy part of a project is rarely where the real debugging happens.

Whether it actually worked

I tested it against a demo target, a fictional 1,000-product coffee shop with markup I'd deliberately built to drift, crawled through scrapy-poet page objects.

Breaking the sku selector so it returned the product title instead: Jev scored sku around 0.08 to 0.35 while the other fields stayed above 0.9, and the gate stopped the crawl.
Breaking the name selector so it returned nothing: every item in the sample failed, and the crawl stopped with a clear reason logged.
Renaming the price CSS class: the gate didn't catch it, and correctly so. The demo's price extraction was a regex against a currency-prefixed string, not a class-based selector, so the rename didn't actually break anything. The check didn't cry wolf.

Here's what one of those failures actually looked like in the logs, an item where name came back None while the other three fields scored a clean 1.0:

2026-09-21 16:08:28 [scrapy_jev.client] INFO: Jev response: {'model': 'typesafe/jev-1.13-20260917', 'answers': {'price': {'type': 'score', 'score': 1, 'legend': {'0': 'Does not look like a real value for this field', '1': 'Looks like a plausible, correctly-scraped value for this field'}, 'probabilities': {'0': 0, '1': 1}, 'confidence': 0.99}, 'sku': {'type': 'score', 'score': 1, 'legend': {'0': 'Does not look like a real value for this field', '1': 'Looks like a plausible, correctly-scraped value for this field'}, 'probabilities': {'0': 0, '1': 1}, 'confidence': 0.99}, 'category': {'type': 'score', 'score': 1, 'legend': {'0': 'Does not look like a real value for this field', '1': 'Looks like a plausible, correctly-scraped value for this field'}, 'probabilities': {'0': 0, '1': 1}, 'confidence': 0.99}}, 'usage': {'input_tokens': 570, 'output_tokens': 43, 'cost': 2.394e-05}, 'id': 'gen-dec-1790003308-TkGiCpI2ZIe3KtT8fFSn', 'provider': 'TypeSafe'}
2026-09-21 16:08:28 [scrapy_jev.pipeline] WARNING: scrapy_jev: item failed quality check: name=None

name isn't in the answers at all, because a missing value never gets sent to Jev in the first place. JevClient only asks about fields that have something to judge, and the pipeline treats the absence itself as the failure. price, sku, and category came back at confidence 0.99, which is exactly what should happen when three fields are fine and only one selector is broken.

That last result mattered more than the first two. A quality gate that fires on cosmetic changes gets disabled by the second team it annoys. This one only fired when the extracted value was actually wrong.

Where it doesn't help

This is not a general defense against scraper drift, and I don't want to oversell it as one.

It only catches breakage that produces an implausible-looking value. A selector that starts pointing at the wrong product on the page, but returns something that's still shaped like a real name and a real price, will sail through untouched. Jev has no way to know the name belongs to the wrong item. It's a plausibility check, not a correctness check.

It's sample-based, which is the right tradeoff for cost but means a crawl can run through a chunk of bad items before or after the sampled window and never trigger the gate, depending on where the breakage falls relative to the sample.

It also adds a hard dependency on an external API mid-crawl, OpenRouter's alpha endpoint in this case, not even TypeSafe's own. That's one more thing that can be slow, rate-limited, or unavailable at exactly the moment you need the crawl to keep running smoothly. I didn't build in a fallback behavior for that; right now a Jev outage just means the gate isn't checking anything, silently, which is close to the same failure mode the whole project set out to fix.

The thresholds are guesses too: sample size, pass rate, per-field cutoff, all tuned against one demo site. I have no evidence yet that a 0.7 pass rate and a 0.5 per-field threshold are the right defaults for a real production catalog with more natural variance in its data.

Was it worth it

As a package: yes, with caveats. scrapy-jev is published, on PyPI, with trusted publishing so there's no stored token to leak, and it does the one thing it claims to do. It caught two different kinds of real selector breakage in testing and correctly ignored a change that didn't matter. That's a genuine, narrow win, and the packaging overhead, src layout, CI, changelog, was small next to the core pipeline work.

As a general answer to "how do I know when my scraper is broken": no, not by itself. It's a plausibility net under specific fields, not a correctness check, and it depends on an external model call succeeding at the exact moment you need it to. It's worth having if you already have a low-cost way to bolt it onto a pipeline, but it isn't something to build a monitoring strategy around by itself. The main thing I got out of it wasn't even the AI check. It was being forced to actually watch a crawl fail to stop, and finding a bug in my assumptions about Scrapy's exception handling that I'd have shipped otherwise.

Takeaway

The interesting result wasn't that an AI model can score whether a scraped value looks real. It can, cheaply and fast enough to run inline. What matters more is that the check is only as good as its blind spots, and the most valuable thing the project produced was finding a case where the failure mode I was defending against, a crawl silently continuing after something breaks, was hiding in my own stop-the-spider code, not in the site I was scraping.

Jev, the model that cannot write a word, and where it fits in web scraping (does it?)

Ayan Pahwa — Mon, 21 Sep 2026 15:41:26 +0000

My first reaction to Jev was that I did not get it, and I suspect I was not the only one. You give it something, you ask a question, and it comes back with yes or no and a confidence number. This is where AI started for me. Is this a photo of a dog? Yes, 92% confident. That demo is older than most people's careers, so when the launch thread was filled with people calling it a new category of model, I assumed I was missing the joke. Rather than keep arguing with a comment section, I spent an afternoon putting it into the scraping workflow I actually use.
The part that makes Jev odd is that it cannot write. It does not write badly or write short; it has no ability to produce a string at all. You hand it data and a list of typed questions, and it hands back probabilities and choices which could be decision directions your agents can take, so my first thought was to use it as a complimentary block with LLMs for agentic application use-case to increase accuracy of output or to reduce cost for my agents. More on this later.
Initially it also sounded useless for web scraping, since extraction means producing text. That turned out to be the wrong way around, because there is a job in every pipeline that is not extraction at all: deciding whether the record you just built is any good.

What the hype is about

TypeSafe AI put Jev on Hacker News on September 15, 2026, and the launch thread reached more than 1,900 points and 509 comments. Three days later a project called OpenJev drew 714 points of its own.
The appeal is easy enough to state. Output tokens are free, the answer always comes back in the shape you asked for, and a per-record call is quick enough not to be the bottleneck.

What it is

Jev belongs to a category TypeSafe calls System One models. An ordinary language model is autoregressive: it predicts a token, appends it to the context, predicts the next one, and repeats until it decides to stop, which is why output costs money.
Jev is not trained to generate text.
You give it a state, which is your data, and a map of typed questions, and it evaluates every question against that state in parallel. TypeSafe describes what comes back as typed decisions and probabilities rather than generated text. The answer to your fifth question is not waiting on your first.

Everything you can ask is one of three shapes. A Noul is a yes or no question returning a probability between 0 and 1, and it has no confidence field, which catches people out: the probability is the answer, not a measure of how sure the model is. A Choice picks one option from a set you define, up to 255 of them, and returns the winner, the distribution, and a confidence. A Score places the input on a rubric you write, and its number is probability-weighted across your levels, so it can land between two rungs.

How it works, in the simplest case

A request is one flat JSON object, and the answers come back under keys you chose yourself:

{
  "state": "Help! My payouts have been failing for three days.",
  "model": "jev-latest",
  "questions": {
    "is_urgent": { "type": "noul", "instructions": "Does this convey urgency?" },
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": "Payments, invoicing, refunds",
        "technical": "Bugs, outages, integrations"
      }
    }
  }
}

You get is_urgent back as a float and team as one of your two strings plus confidence. No parsing, no retry loop, no prompt engineering to coax valid JSON out of a model that would rather write a paragraph.

How it differs from a language model, and what it costs you

Output tokens are free. Input runs at $0.042 per million tokens and everything the model produces costs nothing, which inverts the usual incentive: with a generative model you ask the fewest questions you can get away with, and here you ask everything you might want to know. The answer is always a valid instance of the type you requested. Responses land inside two seconds. TypeSafe says the probabilities are calibrated; I did not test that, and one result below makes me want to.
The launch coverage mostly stops there. The next bit matters more. You will always get a well-formed answer, it can be completely wrong, and arriving in a tidy shape makes it easier to trust than it has earned.
Against that, it cannot generate a string, so it can never fill in a field for you. TypeSafe's own model jaggedness page is blunt about the consequence: "For data extraction, it is better to extract possible options using regex or a generative model and let jev-1.13 pick the correct extraction." Your code proposes, the model decides.
A Choice always returns one of the options you gave it, and the confidence does not reliably warn you when none of them fit: in a larger run it labeled a category listing page poetry at a confidence of 1.00.
The documentation tells you to add an other or none of the above options for exactly that reason, and I did not, which you will see the cost of below. Context is capped at 64k tokens per request, with 32k for the state plus your longest question. It is text only, English first, and the documentation is honest that CJK scripts are handled but not equally well.

Where it could fit in web scraping

My filter, after getting this wrong twice: if a regular expression, a status code, or a CSS class can answer the question, do not ask a model. Those signals are structural, and code beats a non-deterministic model on structure every time, for nothing.
What is left is the questions where the answer only exists in the language, and where the alternative is a hand-maintained list of phrases that is never finished. Classifying a product into a category, or deciding whether a description actually describes the thing it is attached to, are not regex problems, and I have a number for that claim further down.
A gate runs after the work, on a record you have already built, and decides whether to trust it. A switch runs before, and decides what the pipeline does next, so it is asking ahead of the expensive thing instead of auditing after it. Switches are the more interesting group if cost is what you care about, and they are not unique to scraping: routing a support ticket to a person or a canned reply, sending an uploaded document to the right parser, and deciding whether a log line is worth waking anyone over are all the same shape.

The scraping switch I keep coming back to is choosing an extraction type, because Zyte API will not let you combine multiple automatic extraction fields in one request, so something upstream perhaps could decide between product, article, job posting, and the rest before you spend the call. I have not tested that one yet, and my own filter argues against it: most sites announce their page type structurally, in the URL, in JSON-LD @type, or in an og:type tag, and code should read those first. A Choice earns a look only for the pages where none of that is present.

One real example

The shape I ended up with is the gate, and it is about as plain as it looks:

I scraped a single book from books.toscrape.com, using nothing but requests and BeautifulSoup:

import re
import requests
from bs4 import BeautifulSoup
def scrape(url):
    html = requests.get(url, timeout=30).text
    soup = BeautifulSoup(html, "html.parser")
    cells = [c.get_text(strip=True) for c in soup.select("table.table-striped td")]
    heading = soup.select_one("h1")
    desc = soup.select_one("#product_description ~ p")
    price = cells[2] if len(cells) > 2 else None
    return {
        "name": heading.get_text(strip=True) if heading else None,
        "author": None,                       # the site does not publish one
        "description": desc.get_text(strip=True) if desc else None,
        "price": re.sub(r"[^\d.]", "", price) if price else None,
        "currency": "GBP" if price and "£" in price else None,
        "category": soup.select("ul.breadcrumb li a")[-1].get_text(strip=True)
                    if len(soup.select("ul.breadcrumb li a")) > 1 else None,
        "availability": cells[5] if len(cells) > 5 else None,
    }

Those if heading else None guards are not decoration. Point these selectors at a page that does not exist and they all come back empty, and without the guards the parse raises before you reach the interesting part.
Then one call to Jev API carrying six questions about that record at once:

QUESTIONS = {
    "is_a_book":      {"type": "noul", "instructions": "Is this record a single real book, rather than an error page or a listing page?"},
    "price_present":  {"type": "noul", "instructions": "Is the `price` field filled in with a real price?"},
    "author_present": {"type": "noul", "instructions": "Is the `author` field filled in with a real author name?"},
    "currency_right": {"type": "noul", "instructions": "Is the `currency` field the right currency for the `price` shown on this listing?"},
    "category_right": {"type": "noul", "instructions": "Does the `category` field match what the title and description are actually about?"},
    "genre": {
        "type": "choice",
        "instructions": "Which genre does this book belong to?",
        "criteria": {g: None for g in ["poetry", "travel", "mystery", "science fiction",
                                       "history", "business", "self help", "art", "music", "humor"]},
    },
}
def ask_jev(state):
    response = requests.post(
        "https://api.typesafe.ai/v1/systemone",
        headers={"Authorization": f"Bearer {TYPESAFE_API_KEY}"},
        json={"state": state, "model": "jev-latest", "questions": QUESTIONS},
        timeout=120,
    )
    response.raise_for_status()
    return response.json()

That is the whole integration: a dictionary in, six typed answers out. Basically a Jev powered data quality checker.

A good record, a bad one, and one that is quietly wrong

Jev responded :

{ "name": "A Light in the Attic", "author": null,
  "description": "It's hard to imagine a world without A Light i...",
  "price": "51.77", "currency": "GBP", "category": "Poetry",
  "availability": "In stock (22 available)" }
  is_a_book        0.85
  price_present    0.71
  author_present   0.01
  currency_right   0.61
  category_right   0.96
  genre            poetry (confidence 1.00)

On a URL that does not exist:

{ "name": "404 Not Found", "author": null, "description": null,
  "price": null, "currency": null, "category": null, "availability": null }
  is_a_book        0.04
  category_right   0.08
  genre            humor (confidence 0.41)

Jev rejects it, and author_present reports 0.01 on both, because this site publishes no author anywhere.
That second record should never have reached the model, though. The response was a 404, raise_for_status() would have ended it a line earlier, and a null check catches it anyway. My own filter says so. I show it because it is the failure people actually ship, not because it needs a model. This next one does.

The bug no check can see

Take the correctly scraped record and swap in the description of a different book.
Think of this as the zip bug, or the pagination bug, the off-by-one in a list comprehension, and I have shipped it more than once. Every field is populated, every type is right, the price is positive, and the status code is 200.

  name        : 'A Light in the Attic'
  description : 'WICKED above her hipbone, GIRL across her heart...'
  null check  : PASSES
  type check  : price float() = 51.77, positive
  category_right   0.04 <-- DATA QUALITY :D
  genre            mystery (confidence 0.69)

category_right falls from 0.96 to 0.04, and the genre follows the planted description rather than the title, which is a tell that the model read the field instead of pattern-matching the name. Across 59 correct records and the same 59 with descriptions shifted by one, that question caught 46 crossings at a 0.5 threshold with zero false alarms, and 52 at 0.7 with one. Most of those crossings land within the same genre, which is the harder case.

Now look again at the good record, because two of those answers are weaker than they should be. price_present came back 0.71 on a record where the price is there, and that is a question a null check answers perfectly and for free. currency_right came back 0.61, and that one is my fault: my parser strips the pound sign before building the record, so the state I sent Jev contained no evidence of any currency at all. It was right to shrug. The jaggedness page tells you to point a question at the relevant state, and I had deleted it. The strong answers, 0.96 and 1.00, are the judgment calls.

What this saves

I sent the identical record and the identical six questions to a cheap generative model, gpt-5.6-luna, routed through OpenRouter so its latency carries a hop that Jev's does not, three runs each:
| | latency | tokens | cost |
|---|---|---|---|
| Jev | 0.92s | 776 in, output free | $0.000033 |
| generative model | 3.8s to 8.9s | 457 in, 117 to 142 out | $0.000232 to $0.000262 |
Roughly 7X the cost, on one record. The latency spread is the more interesting half: Jev sat between 0.92 and 0.97 seconds across every run, while the generative model ranged from under 4 seconds to nearly 9. You can plan around the first number.
One deduction, in fairness: the shape advantage is smaller than it looks, because a generative API can be pushed into a schema with structured outputs. Cost and consistency are the real differences.
At a million records a day with six questions each, that gap is roughly $33 against $232, though TypeSafe's published ceiling of 1,200 requests a minute puts a million a day at about 58% of the limit before you ask for more. The marginal question is nearly free: a seventh check, or a twentieth, costs a few more input tokens for the wording and no extra time, because they are evaluated together.
That last point deserves a number against the alternative it replaces. On a separate run over 59 books, using the site's own category as the answer key and showing the model only the title and description, one Choice got 55 right against 43 for the best keyword list I could write. All four of Jev's misses were arguable rather than wrong: business against self help, twice, and travel against art. That run cost $0.00188 in total.

Your mileage will vary

Everything here ran against jev-1.13.0 on one teaching site, in English, on one afternoon, with three runs of the comparison. Treat it as a first look, not a benchmark.
Before relying on any of it, know this. Jev is not deterministic. Across repeated identical calls the confident answers held steady while the borderline ones drifted by several points, so do not put a threshold anywhere near where the answers wobble. I was ready to present that as my own finding until I read further into TypeSafe's cookbooks and found they had measured the same thing: six of eight Nouls came back with a standard deviation of exactly zero across five repeats while two carried noise, and their conclusion is the useful one. "The noise is a property of the question, not of how you batch."
A well-formed request can also encode the wrong question. I once passed a Choice's options as a list nested under a key called options, expecting an error. None came, because criteria accepts a map whose values may be arrays, so what I had sent was a valid one-option Choice. Jev returned options as the winner at a confidence of 1.0. Nothing was broken and the answer was useless: the type is guaranteed, the meaning is not.

Other ways to use it

None of these are tested. Triaging spider monitor alerts so a human only sees the ambiguous ones, which pairs naturally with Spidermon or a preflight check like scrapy-spidey-sense. Adjudicating deduplication candidates your own code has already shortlisted. Flagging listings whose description contradicts their own attributes.
One caution applies to all of them. The state you send is untrusted text off the internet, and TypeSafe is direct about it: "Content written to adversarially steer the model, whether that is an injected instruction, a deliberately misleading framing, or text that argues for its own classification, can move the answer." That last case should worry a scraper, because a page has every incentive to argue for its own classification. Their cookbook is blunter still: nothing here is a security boundary. The same thinking applies as to any model reading a hostile page.

Closing

Jev may never write you a selector. For now it is a cheap and very fast second opinion on data you already have, and the free output tokens mean you can afford that opinion on every record rather than on a sample.
I would ship it for the narrow job, with thresholds tuned on my own data and every cheap check running in front of it. I would not replace a null check with a model call.
As for the dog photo, it is the same shape of question I was writing classifiers for a decade ago. What changed is that I did not train anything. No labeled set, no feature engineering, and when I wanted an eleventh genre I added a key to a dictionary instead of collecting examples. That, and the price, is what changes which questions are worth asking at all.
Originally published on Zyte.

uv Python cheatsheet: what changed in 0.12 and what still trips you up

Ayan Pahwa — Fri, 18 Sep 2026 18:05:05 +0000

Use the uv cheatsheet we created for Zyte's community : https://github.com/zytelabs/uv-cheatsheet

uv init does not do what it did in April, the very few issues I had with it has been fixed in the recent releases but it's one good package manager that I never felt a need to update and hence couldn't really enjoy the latest and greatest, until this blog.
I keep a working reference in my repository, the kind you paste into a colleague's message when they ask how to start a project, and when I checked it against a current build the two entries I trusted most did not survive. One had been wrong since late July. The other had never been right on either version I tested, and I had been passing it around anyway.

This is normal for a tool moving at uv's pace rather than a scandal. The big ideas hold: uv is still the fast Rust-based package and project manager that replaced an awkward pile of older tools, and it is still the right default. What drifts is the layer underneath, the part you actually type.

Everything below was reproduced against uv 0.12.15, released on Tuesday, September 15, 2026, with uv 0.11.7 installed alongside, so every before and after is real terminal output rather than recollection.

The command that changed under everyone

Since uv 0.12.0, released on Tuesday, July 28, 2026, projects created with uv init define a build system and are packaged by default, so you get a src layout, a [project.scripts] entry, and a [build-system] table:

$ uv init demo
Initialized project `demo` at `/private/tmp/demo`
$ find demo -type f -not -path '*/.git/*' | sort
demo/.gitignore
demo/.python-version
demo/pyproject.toml
demo/README.md
demo/src/demo/__init__.py

[project.scripts]
demo = "demo:main"
[build-system]
requires = ["uv_build>=0.12.15,<0.13.0"]
build-backend = "uv_build"

The older behavior gave you a bare main.py at the top level with no build system, which meant your own code was not installed into its own virtual environment and was importable only by accident of whichever directory you happened to be standing in. That layout still exists behind uv init --no-package, and existing projects are untouched, so nothing breaks.
The entry in my notes said the reverse. It said uv init --package was the flag you needed for a build system and a src layout, and that without it your imports worked by luck. True in April 2026, backwards now, and exactly the sort of small thing a reader copies into a project and then carries for a year.

Why switch?

I asked John, our developer engagement manager, what uv replaced for him, because I wanted the version of this that comes from using the thing rather than reading about it. His answer was pip and venv, which he called fine, just slow and dated next to modern tooling. He had tried Poetry, liked the idea of it, and never got it to stick. What he actually wanted was cargo for Python.
Two things closed that gap. One was speed. The other was the piece pip never had, which is a way to run a tool you have not installed. Node developers have had npx for years and Python had nothing, and uvx is that.

What he does now is unremarkable in the best way. Everything goes through uv, his agents run uv sync, and pyproject.toml turned dependency management and Dockerfiles into something standard rather than something every project reinvents. Scrapy and Scrapy Cloud work exactly as they did before.
His only complaint was having to delete the main.py file that uv init leaves behind.
That complaint is not an issue anymore. It is the same 0.12.0 change from the section above. The packaged default produces src/<name>/__init__.py and no main.py at all: WIN!!

$ uv init mp && find mp -type f -not -path '*/.git/*' | sort

On uv 0.11.7 that leaves you a main.py to delete:

mp/.gitignore
mp/.python-version
mp/main.py
mp/pyproject.toml
mp/README.md

On 0.12.15 it does not:

mp/.gitignore
mp/.python-version
mp/pyproject.toml
mp/README.md
mp/src/mp/__init__.py

His one real gripe with uv is already fixed.

Where uv already lives in the Scrapy world?

Scrapy runs its continuous integration on uv today. The workflow pins astral-sh/setup-uv, drives the test matrix through uvx --with tox-uv tox, and sets UV_PYTHON_PREFERENCE: only-system with a comment explaining why, which is to make uv use the interpreter actions/setup-python already installed instead of downloading one of its own. Steal that last setting. In continuous integration you have usually already paid for a specific interpreter, and letting uv helpfully fetch a second one gives you a build that passes against a Python your users are not running.
Although Scrapy's installation guide doesn't mention uv. It covers pip, it covers conda, it covers virtual environments, and it stops. Documentation usually trails practice so this is nobody's failure, but it is real and fixable, and since Zyte maintains Scrapy it is a gap we can close rather than complain about. If you have ever wondered what a genuinely useful first contribution to a large open-source project looks like, updating an installation page to match what the maintainers already do is a strong candidate.

The two modes that do not mix

uv has two personalities. In project mode, pyproject.toml and uv.lock are the source of truth, .venv is disposable output, and you drive everything with uv add, uv sync, uv lock, and uv run. In pip mode, uv venv and uv pip install reproduce the workflow you already know, and the source of truth is whatever you remember typing. Both are called uv, and nothing in the interface tells you which one you are in.

Everything uv does from the first command onward moves between three files, and almost every confusion is about which one a command writes to.

What each uv command actually writes to.

The folklore, which I had written down myself, says that uv pip install inside a project gets silently undone by the next uv run or uv sync. Tested on both 0.11.7 and 0.12.15, that is half right. uv run prunes nothing:

$ uv pip install six          # six is not a project dependency
$ uv run python -c "pass"
$ uv run --no-sync python -c "import six"
# six is still there

uv sync is what removes it, because uv sync is exact by default and makes the environment match the lockfile:

$ uv sync
Resolved 1 package in 2ms
Uninstalled 1 package in 0.47ms
 - six==1.17.0

There is also an escape hatch almost nobody mentions. uv sync --inexact installs your declared dependencies and leaves anything extra alone, so the package survives the round trip:

$ uv pip install six && uv sync --inexact
# six is still there

Memorize the precise version rather than the vague one: uv sync is exact by default, --inexact opts out, and uv run does not prune. The practical advice is unchanged, since inside a project the durable ways to add a package are uv add or an edit to pyproject.toml followed by a lock, but knowing which command did the deleting is the difference between fixing a problem and performing a ritual.

The hash check that did nothing

If you pin dependencies by hash, you have probably written a requirements.txt beginning with the --require-hashes directive. Before uv 0.12.0, uv read that directive, told you it was unsupported, and installed your packages anyway without checking a single hash. Here is uv 0.11.7 against a file containing --require-hashes and one pinned requirement carrying no hash:

warning: Ignoring unsupported option in `requirements.txt`: `--require-hashes` (hint: pass `--require-hashes` on the command line instead)
Resolved 1 package in 82ms
Installed 1 package in 1ms
 + six==1.17.0

The warning is accurate and the outcome is still wrong. You asked for a guarantee, and what you got was a note explaining the guarantee had been declined, followed by the install proceeding regardless. Scrolling past in a long build log, that line is invisible.
uv 0.12.0 made the directive real. The same file on 0.12.15:

error: In `--require-hashes` mode, all requirements must have a hash, but none were provided for: six==1.17.0

Nothing installs. The build fails, correctly, because a requirement with no hash cannot be hash checked.
The consequence deserves stating plainly. If you put --require-hashes in a requirements file, installed it with uv before July 28, 2026, and did not also pass the flag on the command line, you did not have hash-checked installs, whatever your build log implied. Any pipeline claiming hash pinning is worth revisiting to confirm which uv it ran on, and if the answer is pre-0.12, whether the directive was passed on the command line rather than only living in the file.

Pin uv itself

Most of us pin our dependencies. Far fewer pin the tool doing the pinning, and uv is pre-1.0 software sitting in the most load-bearing step of the build.
On Tuesday, September 15, 2026, uv shipped 0.12.14 and 0.12.15 on the same day, because 0.12.14 carried a regression that rejected valid installation commands, including uv pip install --system inside the official python Docker images. If your build used that command without pinning uv, it broke. The fix arrived within hours, which is a good outcome you would still rather have watched from a distance.

A quieter version of the same lesson turned up while writing this. My machine's package manager was serving uv 0.12.13 while upstream was on 0.12.15. Neither number is wrong, but "latest" and "latest from your package manager" are different claims, and only one of them reproduces for a colleague on another operating system. Pin the version in your images and your continuous integration, then upgrade deliberately, the way you would treat any dependency you cannot easily roll back.

Five traps worth five minutes

None of these are bugs. They are places where a reasonable expectation meets a different design decision, and all five were reproduced on 0.12.15.

The first is that uv run runs a script while uvx runs a tool. uv run script.py executes the file and reads its PEP 723 inline dependency header; uvx is the tool runner and does not, which matters whenever a single-file program is driven by some other command. That trap nearly shipped in my earlier article on running web data workflows in a reactive notebook, where the documented command worked locally and would have failed for every reader who cloned the repository fresh. A clean-room test in an empty directory caught it before publication. To uv's credit, the error now signposts the way out.

$ uvx s.py
error: It looks like you tried to run a Python script at `s.py`, which is not supported by `uvx`
hint: Use `uv run s.py` instead

Second, --locked asserts and --frozen ignores. uv sync --locked fails when uv.lock no longer matches pyproject.toml, which is what you want guarding continuous integration. uv sync --frozen skips the check and installs against the stale lockfile without comment. Two flags that look like synonyms and behave like opposites.

$ uv sync --locked
error: The lockfile at `uv.lock` needs to be updated, but `--locked` was provided.
hint: To update the lockfile, run `uv lock`.

Third, uv python pin does not rebuild your environment. It rewrites .python-version and stops, and the existing .venv keeps whatever interpreter it had until uv sync deletes and recreates it. Note also that requires-python outranks the pin, so pinning 3.12 under requires-python = ">=3.13" fails loudly rather than disagreeing in silence, which is the right call.
Fourth, uv virtual environments ship without pip. This is deliberate and almost always fine, right until a legacy tool shells out to python -m pip and reports no module named pip. uv venv --seed puts it back.
Fifth, uv add writes a lower bound rather than a pin. uv add six puts six>=1.17.0 in pyproject.toml, and the resolved version lives only in uv.lock. Commit the lockfile, for applications and libraries alike, because it is the only artifact recording what you actually tested against.

Locking a crawler

Scrapy is a good place to watch all of this land at once, because a crawler has the properties that make dependency management interesting: a lockfile that has to survive a rebuild, an image rebuilt every time a selector changes, and a continuous integration run where a stale lock should fail loudly rather than pass quietly.
Starting one takes two commands, and the second is the one that matters:

$ uv init crawler && cd crawler
$ uv add scrapy
$ uv run scrapy version
Scrapy 2.19.0

That wrote scrapy>=2.19.0 into pyproject.toml and 47 packages into uv.lock. The 47 is the number worth noticing, because Scrapy pulls in Twisted, lxml, cryptography, and a long tail beneath them, so the distance between "I installed Scrapy" and "I can rebuild this exact environment in six months" is 46 packages you never chose. uv sync --locked is what closes that distance, and in continuous integration it is the difference between a build that fails on a stale lockfile and one that quietly resolves something new.
For the container, ordering does the work. Astral publishes uv as an image, and the pattern in their Docker guide copies the binary in at a pinned version rather than installing it:

FROM python:3.13-slim
COPY --from=ghcr.io/astral-sh/uv:0.12.15 /uv /uvx /bin/
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --locked --no-install-project
COPY . .
RUN uv sync --locked
CMD ["uv", "run", "scrapy", "crawl", "products"]

Two details there earn their place. Dependencies are synced before your source is copied, so editing a spider leaves the layer holding Twisted, lxml, and cryptography untouched instead of rebuilding it. And the uv version is pinned in the COPY line, which is the concrete form of the advice from earlier: on September 15, that pin was the difference between a broken build and an uneventful one. If your crawls drive a browser, the same ordering matters more, because the dependency layer gets considerably heavier once a browser is in it, which John covers in running Playwright at scale.

What I would still change

An honest cheatsheet should say where a tool annoys its own users, and uv's issue tracker is unusually clear about this, since the requests are heavily upvoted and have been open a long time. As of Thursday, September 17, 2026, the most-supported open requests are using uv run as a task runner at 702 up-votes and 254 comments, an upgrade --all option at 536, a dedicated uv upgrade for bumping pyproject.toml at 532, and a uv shell activation command at 416.
Those four cluster around one theme, which is that uv replaced the tools people used for everyday chores without replacing the chores. Upgrading a dependency and writing the new bound back into pyproject.toml remains a two-step dance, and running a project's common commands still needs make, just, or a pile of shell aliases.

It is also worth knowing what to stop repeating. The criticism I still see most often, that Dependabot cannot read uv.lock, is out of date. uv is now a first-class ecosystem in GitHub's supported-ecosystems table, with its own uv value in the configuration rather than being routed through pip. It is not flawless, and dependabot-core carries several open issues about how it updates uv.lock, but "unsupported" is no longer the right word.
The larger open question is governance. On Thursday, March 19, 2026, Astral founder Charlie Marsh announced that the company had "entered into an agreement to join OpenAI as part of the Codex team", writing that "OpenAI will continue supporting our open source tools after the deal closes." I have no inside knowledge and no prediction. At the time of writing uv remains permissively licensed, actively developed, and pre-1.0 with no announced 1.0 date, and that is a fact worth holding alongside the adoption figures, which are substantial: on Thursday, September 17, 2026, pypistats reported 141,324,434 downloads of uv in the preceding 30 days, and the project repository showed 89,920 stars.

The cheatsheet, grouped by intent

Alphabetical command lists are useless when you cannot remember the command, so this is grouped by what you are trying to do. It also lives in a repository at zytelabs/uv-cheatsheet, along with a one-page printable version, so you can correct it when it goes stale, which it will.
Start something. uv init name for a packaged project with a src layout, uv init name --no-package for the old flat script layout, and uv init --lib when you are writing a library.
Change dependencies inside a project. uv add pkg, uv add --dev pkg for tooling that never ships, and uv remove pkg. Never uv pip install here. Use uv lock --upgrade-package pkg to bump one thing and uv lock --upgrade to bump everything.
Reproduce an environment. uv sync for exact, uv sync --inexact to leave extras alone, uv sync --locked in continuous integration so a stale lockfile fails the build, and uv sync --frozen only when you deliberately want the old lock.
Run things. uv run cmd inside a project, uv run script.py for a single file with a PEP 723 header, and uvx tool for a tool you have not installed. Never source .venv/bin/activate, because uv run syncs first and wins over an activated environment anyway, warning that a mismatched VIRTUAL_ENV "will be ignored" unless you pass --active.
Manage interpreters. uv python list, uv python install 3.13, and uv python pin 3.12 followed by uv sync to make it real. Set UV_PYTHON_PREFERENCE=only-system in continuous integration, or only-managed when you want uv's own builds.
Escape hatches. uv venv --seed when something needs pip in the environment, uv tree --invert --package pkg when you need to know who dragged a dependency in, and uv export when a downstream tool insists on a requirements.txt.
Containers. Pin the uv version in the image, use UV_PROJECT_ENVIRONMENT to control where the environment lands, and install dependencies before copying your source so the dependency layer caches. For scraping work, where images get rebuilt constantly as selectors change, that ordering is the difference between rebuilding a spider and rebuilding Twisted.

Where this leaves you

uv is infrastructure now. It sits under a very large number of builds, including the continuous integration of the framework this company maintains, and it is still pre-1.0 and moving fast enough that a six-month-old reference misleads rather than merely lags.
So treat your uv knowledge like a dependency. Give it a version, check it occasionally, and be willing to find that something you were confident about moved underneath you. I found two in my own notes, and I was not looking hard.
Pin the version, put --locked in continuous integration, and see what your build has been getting away with. The cheatsheet above is in zytelabs/uv-cheatsheet if you would rather print it than scroll it, and pull requests are the fastest way to make this article wrong.
Originally published on Zyte.

Give an AI Agent a Browser: From Instructions to Actions

John Rooney — Fri, 11 Sep 2026 14:10:00 +0000

Most browser automation examples start with selectors. Find the button, identify the input, write the script, and hope the page does not change before the next run.

That approach works, but it becomes awkward when a site requires real browser interaction. Some information only appears after a click. Some pages run browser checks. And if the agent is running on a remote server, even launching and maintaining a local browser can become a project of its own.

In this walkthrough, I connected an agent to a Zyte CDP browser through Playwright CLI and gave it instructions in plain language. The interesting part was not simply connecting to a remote browser. It was seeing what the agent could do once it had access to a real, interactive browser session.

Video timestamps are included throughout this post.

follow along

The setup: a short instruction file and Playwright CLI [00:18]

I started with a small Markdown file that explains how the agent should use the Zyte CDP browser with Playwright CLI. It tells the agent to make sure Playwright CLI is installed and meets the required version, then gives it the basic connection instructions.

This creates a simple bridge between the agent and the browser. Once the browser is connected, the agent can use the operations available through Playwright CLI: opening pages, finding fields, clicking buttons, entering text, and reading the resulting page.

The difference is that I do not have to write a separate Playwright script for every interaction. I can describe the task and let the agent work out how to perform it on the page.

A deliberately interactive demo site [00:39]

The demo website is intentionally simple. It contains categories, a search box, product controls, and several buttons. It also includes a browser check, which appears briefly when the page loads.

The site has a useful interaction flow:

Search for products.
Select products.
Click a compare button.
Read the comparison table that appears.
Open an individual product page.
Click a button to check availability.

Some of this information might be available through Ajax requests or an underlying API. In a real project, it can be worth investigating those options. But there are also cases where the most reliable path is to use the website as a user would and click through the interface.

That is where a browser-connected agent becomes useful.

Replacing selector work with instructions [01:13]

The traditional Playwright workflow requires you to inspect the page and choose selectors. You need to work out which element is the search field, which button triggers comparison, and which part of the page contains the result.

With the agent, the instruction can be much closer to the actual task:

Go to the website, search for "brake", and return the products you find.

The agent can inspect the page, locate the relevant field, enter the search term, and read the results. It is still using selectors under the hood, but you do not have to discover and maintain those selectors yourself for each request.

This is especially useful when the task changes from run to run. The instruction can describe the outcome rather than encoding every click in a fixed script.

Connecting the agent securely [01:28]

For the demonstration, I used OpenCode with DeepSeek Flash through OpenRouter. This task does not require a large amount of reasoning. The agent mainly needs to follow the browser instructions and carry out a sequence of actions, so a relatively inexpensive model is enough.

The connection also needs authentication. The API key is read from an environment variable and used to create a short-lived configuration file in a temporary location. That file is passed to Playwright CLI when it connects to the browser.

The important practice here is to keep the key out of the instruction file and out of the agent's visible working material. The connection configuration should be created securely and cleaned up according to the needs of the environment.

After that, the agent can use the browser through Playwright CLI.

Searching without writing selectors [03:02]

The first task was simple: open the site and search for "brake".

The agent found the search bar, entered the term, and returned the matching products. During the process, Playwright CLI reported the elements it found and the actions it took. The model was verbose, but the important result was that no custom selector script was needed.

The same approach works when the page is more complicated than the demo. You can describe the field or action in terms a user would understand, and the agent can inspect the current page before deciding what to interact with.

Using the site's compare feature [04:07]

Next, I asked the agent to take the top three products, use the website's compare feature, and return the comparison table.

This matters because the table does not appear until the products have been selected and the compare action has been triggered. It is not simply a block of data sitting in the initial HTML.

The agent had to find the product controls, select the relevant items, click the compare button, and read the table that appeared below the results. Again, the instruction described the goal rather than the selectors:

For the top three products, use the website's compare feature and return the table it shows.

The example is deliberately basic, and the same information might have been available elsewhere on this particular site. The point is to demonstrate the interaction pattern. Once the agent has a browser, it can perform actions that depend on the page's state and then collect the result.

Checking availability on each product page [05:13]

I then asked the agent to visit each product page and retrieve the availability information.

The product pages include a "check availability" button. The agent can follow the product information it already collected, open each page, find that button, click it, and return the availability result.

This is the kind of task that can become a collection of small scripts when written manually. The agent can handle the sequence from a single higher-level instruction, provided the task is clear and the browser session remains available.

Why use a remote browser? [05:48]

If Playwright can launch a browser locally, why connect to a remote one?

The two main reasons are access and maintenance.

Access to sites that expect a real browser

Some sites do not respond well to a basic automation setup. They may perform browser checks or apply other forms of traffic protection. In the demo, the site was configured so that a standard Playwright launch would not get access, while the remote browser could connect successfully.

There are open-source stealth browser options, and some of them work well. But they need to be maintained. Browser behavior changes, sites change their checks, and the tools need to keep up.

Less local infrastructure to maintain

Running a browser locally can also create deployment work. An agent may be running on a server without a display, which means you may need a virtual display such as Xvfb. Then there is the browser installation, version management, operating system configuration, and the rest of the environment around it.

None of these problems is impossible to solve. They are just extra moving parts, and every extra moving part is another thing that can fail in an agentic workflow.

A remote browser moves much of that browser infrastructure outside the agent's environment. The agent only needs to connect, perform the work, collect the result, and disconnect.

Session limits and closing the connection [02:28, 07:34]

The browser session in the demonstration lasted five minutes. That limit is important for two reasons: the agent needs enough time to finish its work, and the connection should be closed as soon as the work is complete.

When I spent too long talking through the demonstration, the session expired. The agent reconnected and continued with the task, but this is a useful reminder to design the workflow around short, purposeful sessions.

In a production-style flow, the agent might receive one instruction, connect to the Zyte CDP browser, complete the interaction, return the data, and close the browser. Closing the connection ends the session and avoids paying for unnecessary additional browser time.

This cleanup step should be part of the instructions or application logic, not something left to chance.

The larger idea

The real benefit here is not that an agent can click a few buttons on a demo site. It is that browser automation can be expressed in terms of the work you want done:

Search for a product.
Compare the first three results.
Visit each product page.
Check availability.
Return the results.

The agent still needs a browser automation tool, authentication, sensible session limits, and clear instructions. But you do not have to begin by turning every task into a collection of selectors and hard-coded steps.

That makes a remote CDP browser a useful option for agentic web interaction, especially when a site requires clicks, state changes, browser checks, or other behavior that a simple scraping request cannot reproduce.

The instructional Markdown file and the browser CDP resources used in the demonstration are linked below the video. If you are experimenting with browser-connected agents, this is a straightforward pattern to try: give the agent a small set of browser instructions, let it act on the page, and make sure it closes the connection when the job is done.

Zyte CDP browser documentation:
https://docs.zyte.com/zyte-api/usage/cdp.html

Zyte headless browser:
https://www.zyte.com/zyte-api/headless-browser/

Instruction Markdown file:
https://raw.githubusercontent.com/zytelabs/zyte-cdp-examples/refs/heads/main/agent-onboard.md

40% of pages are empty over plain HTTP

John Rooney — Tue, 08 Sep 2026 15:25:00 +0000

40.6% of popular landing pages need JavaScript to show you anything useful. That is from State of Web Access, an audit Zyte (where I work) ran on 11,100 landing pages. The test was blunt. Fetch the page over plain HTTP, fetch it again in a headless browser, and if rendering grew the meaningful HTML by more than half, count it as JavaScript-dependent.

On the pages that failed the test, rendering added 140,923 bytes on average. Three quarters of the final HTML did not exist until a browser ran the scripts. Restaurants and airlines top the list at 66%, travel at 65%.

The report calls this architecture rather than defence, and I agree. Single page apps won. The HTML that comes over the wire is a shell and the content arrives in a second round trip.

What rendering costs you

The instinct on seeing an empty response is to reach for Playwright. Here is what that does to the bill, using the audit's own tier data. Zyte's tiers map to the infrastructure needed to fetch a page reliably, and the report scored every site both ways:

Access method	Simple	Easy	Moderate	Complex	Advanced	Mean tier
Plain HTTP	85.2%	9.4%	2.5%	2.0%	0.9%	1.24
Headless browser	58.4%	31.3%	8.0%	1.8%	0.5%	1.55

The gap is not sites defending themselves. A browser carries a heavier fingerprint and a heavier compute cost even on a site with no barriers at all. Switching to rendering moves a quarter of all sites up a tier for no reason other than the method.

So before rendering, I look in three places. The data is very often already in the response.

First, measure

This is the audit's test, rewritten to compare visible text instead of HTML bytes, which is stricter. Run it once per website before you decide anything.

import asyncio
import sys

import httpx
from playwright.async_api import async_playwright
from selectolax.parser import HTMLParser

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"


def visible_text_len(html: str) -> int:
    tree = HTMLParser(html)
    for node in tree.css("script, style, noscript, template"):
        node.decompose()
    return len(tree.body.text(separator=" ", strip=True)) if tree.body else 0


async def js_dependence(url: str) -> dict:
    plain = httpx.get(url, headers={"User-Agent": UA}, follow_redirects=True, timeout=20).text
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page(user_agent=UA)
        await page.goto(url, wait_until="networkidle")
        rendered = await page.content()
        await browser.close()
    a, b = visible_text_len(plain), visible_text_len(rendered)
    return {"plain_chars": a, "rendered_chars": b, "js_dependent": b > a * 1.5}


if __name__ == "__main__":
    print(asyncio.run(js_dependence(sys.argv[1])))

If js_dependent is false, stop here. You have a plain HTTP site and a browser will only make it slower and more expensive.

Second, harvest the state the framework left behind

SPAs hydrate from data the server already embedded. Next.js puts it in a script tag with the id __NEXT_DATA__. Nuxt uses a global. Apollo, Redux and a dozen others assign to window.__SOMETHING__. E-commerce and news sites also carry JSON-LD for search engines, which is often cleaner than the visible page.

A Scrapy spider that pulls all of it from the plain response, with no browser involved:

import json
import re

import scrapy

STATE_SELECTORS = {
    "next": "script#__NEXT_DATA__::text",
    "jsonld": 'script[type="application/ld+json"]::text',
}

STATE_GLOBALS = [
    r"window\.__INITIAL_STATE__\s*=\s*",
    r"window\.__PRELOADED_STATE__\s*=\s*",
    r"window\.__APOLLO_STATE__\s*=\s*",
]


class EmbeddedStateSpider(scrapy.Spider):
    name = "embedded_state"

    def parse(self, response):
        for source, sel in STATE_SELECTORS.items():
            for raw in response.css(sel).getall():
                try:
                    yield {"source": source, "url": response.url, "data": json.loads(raw)}
                except json.JSONDecodeError:
                    self.logger.debug("unparseable %s block on %s", source, response.url)

        for script in response.css("script::text").getall():
            for pattern in STATE_GLOBALS:
                m = re.search(pattern, script)
                if m:
                    obj = self._leading_json(script[m.end():])
                    if obj is not None:
                        yield {"source": "global", "url": response.url, "data": obj}

    @staticmethod
    def _leading_json(s: str):
        # Parses the JSON object at the start of s and ignores whatever follows
        # (a semicolon, more script). Avoids trying to find the closing brace by hand.
        try:
            obj, _ = json.JSONDecoder().raw_decode(s)
            return obj
        except json.JSONDecodeError:
            return None

The raw_decode trick is the useful part. You never have to find the end of the object yourself. Nuxt is the awkward one, since its global is usually a function call rather than a literal, and you either evaluate it or fall through to the next step.

Third, replay the request the page makes

Open DevTools, Network tab, filter to Fetch/XHR, reload. The JSON endpoint the page calls is right there, and it usually returns cleaner data than the HTML ever will. Right click, copy as cURL, and check what it needs. If it works with a static header set, you have a plain HTTP scraper for a JavaScript site. If it needs a token that a script computes on the client, that is your signal to render, and now you know why.

When rendering is the right call

Three cases. The API is signed or needs a challenge token from a script. The content genuinely does not exist until a user interacts. Or the TLS probe and the 403 classifier from the earlier posts in this series put the site at Moderate or above, at which point you need a browser-shaped session anyway and rendering comes with it.

Everything else is a plain HTTP job wearing a React costume.

The full JavaScript breakdown by industry and site size is on the report page, and the tier tables are under scraping cost. The tier model is Zyte's own pricing model applied to Zyte's own data, so trust the ordering between industries more than the absolute percentages.

Disclosure: I work at Zyte. The audit is ours. The code and the opinions are mine.

Is your ai agent ready for the hostile web? Join Zyte 2nd virtual community meet-up to learn

Ayan Pahwa — Tue, 08 Sep 2026 13:33:07 +0000

An agent that answers a question in a chat window is easy to trust, because a person is reading every word before anything happens. An agent that runs unattended is a different animal entirely: it fetches pages, calls tools, and takes action on a schedule, with nobody in the loop to catch the moment something goes wrong. We already runs agents in production, writing and maintaining spiders, sometimes even without a person watching each run, and that experience surfaces two questions that only matter once you take the human out of the loop: what can the pages your agent reads talk it into doing, and can the thing running your agent be trusted to behave the same way twice.

Join our next virtual community meetup, happening on 24th September 2026 to learn more on this topic. Register here : https://luma.com/wci93kpz

The page your agent reads is now the attack surface

For years, the input a security team worried about was what a user typed into a form. That assumption breaks the moment an agent is left to browse and act on its own, because now the attack surface is every page it fetches, every tool result it parses, every document it's asked to summarize.

A price-monitoring agent that scrapes a competitor's product page every night is doing exactly what it was built to do, and if a single line of fine print on that page is written to manipulate the model reading it, the agent can walk sensitive numbers, such as its own cost basis or floor price, straight back out. Nothing in the logs looks wrong. No rule was broken, and no exploit was used. The agent simply used a tool it was allowed to use on data it was told to read, and that is precisely what makes this class of failure so hard to catch after the fact.

The same shape of problem shows up anywhere an agent treats fetched content as data when the page is treating it as instructions (Prompt Injection). A support agent that reads incoming tickets can be told, inside a ticket, to escalate its own privileges. A research agent that summarizes PDFs can be told, inside a PDF, to email its findings somewhere else first.

None of these need a vulnerability in the traditional sense. They need only an agent that reads text and a model that can't yet tell the difference between "here is information about the page" and "here is a command from the page's author." That distinction used to be free, because a human was doing the reading. Once the agent reads unattended, it has to be built in on purpose.
The fix is not a single filter bolted onto the input. It is a discipline with three parts:

map where untrusted content enters the agent and what it can reach once it is in,
turn each identified threat into an adversarial test that runs on every change to the agent, and
keep watching after that because a new tool, a new model, or a page that changes its content can quietly reopen a hole that was already closed, without a single line of the agent's own code changing.

A discipline like that needs an agent you can rebuild identically

Testing an agent on every change only works if "the agent" is something precise enough to rerun. A definition that lives partly in a notebook, partly in environment variables, and partly in whichever model happened to be configured that week cannot be tested with any confidence, because there is no fixed thing to test against.
That is the argument for treating the coding agent itself as a portable, declarative artifact rather than a one-off script wired to a single provider. Define an agent once, and run that same definition locally or as a background job in the cloud, swapping the harness it runs on or the language model behind it without a rewrite.
The two ideas depend on each other: security testing needs an agent stable enough to test repeatedly, and a reproducible agent definition is what makes that testing possible in the first place. Most teams have neither piece in place yet.

See both in one session

Reading about this is one thing. Watching a real agent fail live, and then watching the fix hold on a second attempt, is what actually changes how you build the next one. That is what Zyte's next Developer Community Meetup is for: a joint session with Humanbound titled "Ship Agents That Survive the Real Web," Thursday, September 24, 2026, 3:00 to 4:00 PM BST, virtual over Zoom.

Demetris Gerogiannis, co-founder and co-CEO of Humanbound, walks through the model, test, and monitor discipline on a real price-monitoring agent, including the moment a failing security test becomes a guardrail exported into a stock LangChain agent in two lines of code.

Konstantin Lopukhin, Zyte's Head of R&D, opens up the design behind Zyte's new open-source library for running coding agents as declarative, portable background jobs.

Every attendee leaves with both repositories, free usage keys, and a one-line command to test their own agent the same day.
Register for the meetup on lu.ma to save your seat.

Originally published on Zyte.

One TLS handshake predicts the whole anti-bot stack

John Rooney — Mon, 07 Sep 2026 15:19:39 +0000

When I size up a new website I used to start with headers. Copy them out of DevTools, match the order, add the cookies, see what happens. Now I start with one TLS handshake, because it tells me more about the site than an afternoon of header work.

The reason is a set of numbers from State of Web Access, the audit Zyte (my employer) ran on 11,100 popular landing pages. 13.8% of them screen the TLS handshake. Look at what else those sites run compared to everyone else:

Barrier	TLS-fingerprinting sites	All other sites
Rate limiting	78.1%	13.2%
Antibot	45.0%	14.2%
CAPTCHA	37.3%	20.2%
JavaScript required	27.0%	42.8%

A TLS check is the strongest single predictor in the whole dataset. If the handshake is being inspected, you are almost certainly going to hit rate limiting too, and you have a coin-flip chance of a dedicated bot manager on top. If it is not being inspected, the site probably stops at a CDN firewall. One request and you know which world you are in.

The JavaScript row runs the other way, and I find it the most interesting. TLS-checking sites need a browser less often. They are server-rendered, performance-minded operations that enforce access at the connection, not in the page.

Why nobody notices this layer

TLS fingerprinting fails before HTTP starts. Your client sends a ClientHello announcing which TLS versions, cipher suites and extensions it supports, in a particular order. Python on OpenSSL, Go's crypto/tls and Chrome's BoringSSL all produce different ones. A site can see "Chrome user agent, OpenSSL handshake" and know the client is lying before it reads a single header.

JA3 hashed those fields into a fingerprint in 2017. Chrome started shuffling its extension order in 2023, which broke JA3, and JA4 replaced it by sorting the fields first. The detail does not matter much for scraping. What matters is that the check is silent. Some sites return a bare 403 with an empty body. Some just close the connection, and your logs say "connection reset by peer", which looks like a network blip. Nothing in the failure says TLS.

Vendor attribution was only possible for 45% of the TLS-checking sites in the audit. The other 55% show the behaviour with no named signature. Where it could be attributed, a single bot management vendor was 88.8% of it. So this is mostly one vendor's check, bundled into a product a lot of sites bought for other reasons.

The probe

This is the method the report used, cut down. Two requests, identical headers, from the same IP. The only thing that differs is the handshake. aiohttp presents a standard Python TLS signature. curl_cffi presents Chrome's. If the first fails and the second passes, the site is looking at TLS.

import asyncio
import sys

import aiohttp
from curl_cffi.requests import AsyncSession

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}


async def python_tls(url: str) -> tuple[int, int]:
    try:
        async with aiohttp.ClientSession(headers=HEADERS) as s:
            async with s.get(url, timeout=aiohttp.ClientTimeout(total=20)) as r:
                return r.status, len(await r.read())
    except aiohttp.ClientError:
        return 0, 0          # reset or refused: count it as a block


async def chrome_tls(url: str) -> tuple[int, int]:
    async with AsyncSession(impersonate="chrome", headers=HEADERS) as s:
        r = await s.get(url, timeout=20)
        return r.status_code, len(r.content)


async def probe(url: str) -> dict:
    py, ch = await asyncio.gather(python_tls(url), chrome_tls(url))
    filtered = py[0] != ch[0] and ch[0] < 400
    return {"python_tls": py, "chrome_tls": ch, "tls_filtered": filtered}


if __name__ == "__main__":
    print(asyncio.run(probe(sys.argv[1])))

Install with pip install aiohttp curl_cffi. The impersonate part follows whatever Chrome version curl_cffi ships, so it drifts less than a hand-maintained profile would.

Because both requests leave the same machine, IP reputation is held constant. This is a controlled experiment with one variable, which is more than most scraping diagnostics manage.

How I read the result

If tls_filtered is true, I plan for the stack the table above predicts. Browser-shaped TLS from request one, session handling, per-domain concurrency of one or two, content validation on every response. I do not spend a day on header tweaks because headers were never the problem.

If it is false and I am still blocked, I have ruled out one layer for the cost of one request. The block is IP class, headers, cookies or behaviour, and I look there instead.

If it is false and both requests pass, I have a Simple-tier site and I should stop reaching for Playwright. The audit's tier data puts 85% of landing pages at plain HTTP. Rendering when you do not need to is the most common way I see scraping budgets disappear.

One caveat the report makes and I will repeat. Everything here is landing pages, scanned once, from datacentre IPs. Search and product pages are usually defended harder than the front door. Treat the percentages as a floor.

The industry split is on the TLS page. Jewellery and luxury leads at 34%. Reference sites sit at 3%, because blocking crawlers would kill the thing they exist for.

Disclosure: I work at Zyte. The dataset is ours. The probe is a cut-down version of what the audit ran, and the opinions are mine.

When one vendor changes its mind, every scraper breaks the same morning

John Rooney — Mon, 07 Sep 2026 15:18:55 +0000

Two incidents look the same on most scraping dashboards.

In the first, one spider starts failing because the site changed a CSS class. In the second, forty spiders start failing in the same hour because a bot management vendor pushed a new model. Both show up as a drop in items and a rise in "blocked". The fix for the first is a code change. The fix for the second is to change identity and wait, and touching the spider code will make it worse.

The second kind is more common than it should be, and State of Web Access, the audit Zyte (where I work) ran on 11,100 landing pages, has the numbers on why.

One vendor decides what a browser is

18.5% of landing pages run a dedicated bot manager, the kind that looks at behaviour, canvas rendering and attribute consistency rather than just IPs. One vendor is 75.9% of those deployments. The next four combined are under a quarter.

The TLS layer is more concentrated still. Of the TLS-fingerprinting sites the audit could attribute, the same vendor is 88.8%.

And at the firewall layer, 61.9% of sites run a single WAF vendor in its default mode. 1.4% run more than one. Roughly three in ten sites run a WAF and nothing else, and that WAF arrived bundled with the CDN. Nobody chose it.

Put those together and for most of the defended web, one company's model of what a real browser looks like is the definition of a real browser. When that model changes, the change lands everywhere at once. Your spiders are not failing independently. They are failing together, for one reason, and the reason is not in your repository.

Why a "blocked" counter cannot see this

Most Scrapy projects I have looked at count blocks in one bucket. Maybe two, split by status code. That hides the two things you need to know when the pager goes off: which layer refused you, and whether it is one domain or all of them.

The audit found the same four layers over and over. TLS, which shows up as a connection reset or an empty 403. WAF, which shows up as a 403 with a block page. Challenge, which shows up as a 200 or 403 carrying a JavaScript challenge. And rendering, where the response is fine but the data is not in it. Each has a different fix. A single counter makes them all look like the same fire.

Middleware that counts by layer and domain

This is a Scrapy downloader middleware. It classifies each response by the wording of the page and increments a stat per layer and per domain. Connection-level failures get their own bucket, because that is where TLS filtering hides.

import re
from urllib.parse import urlparse

from scrapy import Request
from scrapy.http import Response
from twisted.internet.error import ConnectionLost, ConnectionRefusedError, TimeoutError

SIGNATURES = {
    "challenge/js":      [r"checking your browser", r"enable javascript and cookies",
                          r"just a moment", r"verify you are human"],
    "challenge/captcha": [r"captcha", r"are you a robot", r"unusual traffic"],
    "waf/block-page":    [r"access denied", r"request blocked", r"you have been blocked",
                          r"reference\s*(#|number|id)"],
}

CONNECTION_ERRORS = (ConnectionLost, ConnectionRefusedError, TimeoutError, OSError)


def classify(response: Response) -> str | None:
    body = response.text[:50_000] if hasattr(response, "text") else ""
    for label, patterns in SIGNATURES.items():
        if any(re.search(p, body, re.I) for p in patterns):
            return label
    if response.status == 429:
        return "ratelimit/429"
    if response.status == 403:
        return "waf/unattributed"
    if response.status == 200 and len(body) < 2_000:
        return "render/empty"
    return None


class BlockLayerStatsMiddleware:
    def __init__(self, stats):
        self.stats = stats

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.stats)

    def _record(self, layer: str, url: str) -> None:
        domain = urlparse(url).netloc
        self.stats.inc_value(f"blocked/{layer}")
        self.stats.inc_value(f"blocked/{layer}/{domain}")

    def process_response(self, request: Request, response: Response, spider):
        layer = classify(response)
        if layer:
            self._record(layer, response.url)
        return response

    def process_exception(self, request: Request, exception, spider):
        if isinstance(exception, CONNECTION_ERRORS):
            # TLS filtering often looks like a reset, not a status code.
            self._record("tls-or-network/reset", request.url)
        return None

Enable it in settings:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.BlockLayerStatsMiddleware": 543,
}

The stats land in the crawl's stats collector, so they end up wherever you already ship Scrapy stats. If you use Spidermon, that is where the rules below go.

Two rules that tell the incidents apart

The first rule is per spider. If blocked/* for one domain rises past a threshold and other domains are quiet, it is that site. Somebody changed a selector or added a rule. Open the spider.

The second rule is fleet-wide. If the same layer, say challenge/js, rises across three or more domains inside ten minutes, it is not your code. It is a vendor change. Do not deploy anything. Rotate identity, drop concurrency, and give it a few hours before anyone edits a spider. The person who owns the fetch layer gets paged, not the person who owns the parser.

The layer name in the stat key is what makes the second rule possible. blocked/403 across forty domains could be forty separate problems. blocked/challenge/js across forty domains is one problem.

The unattributed bucket is not a bug

You will see a lot of waf/unattributed. The audit could only attribute 45% of TLS-fingerprinting sites to any vendor, and plenty of block pages carry no recognisable wording at all. Keep the bucket. A spike in unattributed 403s across many domains is still a correlated failure, and the replay test from the first post in this series, one request from a different IP class, tells you whether it is reputation or something else.

The vendor shares by industry are on the antibot page and the WAF breakdown is on the WAF page. Adult content, furniture and gambling lead bot management adoption at 30% and above. Those are sectors with checkout flows and inventory, and the bot manager is guarding revenue, not text.

Disclosure: I work at Zyte. The audit is ours. The middleware and the opinions are mine.

Web data in a reactive notebook: an introduction to marimo

Ayan Pahwa — Mon, 07 Sep 2026 14:20:05 +0000

Most of us keep a short list of tools we reach for without thinking about it: something to fetch pages, something to hold the rows, something to draw a chart, and a notebook to keep all three in one place. This article is a proposal to add one more to that list, marimo, together with a working notebook to try it on.
marimo is a reactive Python notebook. Its cells form a dataflow graph built from which cells declare variables and which cells read them, so running a cell reruns everything downstream of it and nothing else, instead of leaving you to remember what you clicked and in what order. Its user interface elements are bound to Python values too, which is where the interesting part of this article ends up. It is also not a small project any more: as of Wednesday, August 19, 2026, its GitHub repository sits at 22,393 stars, and it was downloaded 2,625,051 times from PyPI in the preceding month, which puts it in the same order of magnitude as Scrapy, measured at 3,224,433 downloads a month around the same time.
There is a second reason to write this down. marimo's curated gallery, checked on Tuesday, September 1, 2026, holds 103 notebooks across sixteen categories, and not one of them mentions scraping, crawling, or HTTP. Going by their descriptions they all start from data that already exists: a CSV someone saved, a dataset someone else collected. Collection is treated as the step that happens elsewhere and finishes before the notebook opens. It does not have to be, and a reactive notebook is an unusually good place to put it, because the fetch is normally the slowest and most expensive thing in the file, and a dataflow graph is exactly the thing that knows when not to repeat it.

What makes marimo different

Three properties do the work here, and each shows up later in something concrete.
The execution model is reactive, so a cell that reads a variable reruns whenever the cell that defines it runs. In the default configuration that means you do not get stale output, because marimo reruns whatever depended on the thing you changed. You can turn autorun off when the work is expensive, and marimo then marks the affected cells as stale rather than leaving them looking current.
That rule is the whole notebook, drawn once. Every box below is a cell, and an edge is one cell reading a variable another cell defines — nothing more exotic than that builds the graph the two Zyte API calls, the join, the chart, and the table all sit on:

The file is plain Python. A marimo notebook is a .py file, which means it goes through code review as a diff, runs under python at the command line, and can have its top-level functions imported by other files. There is no JSON envelope wrapped around your code.
The user interface elements are bound to Python values. When you assign mo.ui.slider(...) to a global variable and then reference that variable in another cell, marimo reruns that cell every time the slider moves, with the new value already in place. marimo's documentation states the rule directly: "When a UI element assigned to a global variable is interacted with, marimo automatically runs all cells that reference the variable (but don't define it)."
Everything below lives in one file, notebook.py, including its dependency list, which sits in a PEP 723 header at the top so that uv can resolve it with no install step:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#     "marimo",
#     "zyte-api",
#     "polars",
#     "altair",
#     "duckdb",
#     "sqlglot",
# ]
# ///

Building the scraper in two stages, with no selectors

The notebook scrapes books.toscrape.com, which is a catalogue that exists so that people can practice scraping it. Its own footer says so: "This is a demo website for web scraping purposes. Prices and ratings here were randomly assigned and have no real meaning." Worth knowing before you read a price chart built on those prices. It changes nothing about the mechanics, which are the two stages any catalogue scrape needs: find the product URLs, then fetch each product.
What is worth noticing is that neither stage involves a CSS selector or a line of HTML parsing. Zyte API has two extraction types that map onto the two stages directly, productList for a listing page and product for a product page, and both return structured records. If you have not used this before, the closing section of my earlier article on Scrapy item types covers what automatic extraction hands back and how it maps to a fixed schema.
Stage one asks for the listing:

from zyte_api import ZyteAPI
client = ZyteAPI(api_key=api_key)
queries = [{"url": u, "productList": True} for u in category_urls]
listings = list(client.iter(queries))

Six category pages came back with 105 product URLs between them, along with the category name for each page, which saves deriving it from the breadcrumb trail. The client's iter method sends the requests in parallel, up to 15 concurrent connections by default, and yields each result as it arrives rather than making you wait for the slowest one. Parallel is not instant, though: the cold run further down made all 111 requests in 104.7 seconds. One detail the snippet above glosses over is that iter yields an exception in place of a result when a request fails, so real code needs an isinstance(item, Exception) branch, which the notebook has.
Stage two asks for each product, with the same call shape and product in place of productList. That gives the full record: the name, the price, the currency, the availability, and the stock keeping unit.
Here is the part I did not expect, and it is the reason the second stage earns its cost. On the listing pages, 59 of the 105 book names were truncated, arriving as strings like In a Dark, Dark ..., because the catalogue's own listing markup cuts them short. All 59 came back complete from the product pages. The prices, on the other hand, were identical between the two stages for all 105 records, every single one. So stage two is not buying you better prices, and if prices were all you needed, six requests would have done the job instead of 111. Stage two is buying you names.

What the extraction actually returns

Three details about the returned data will save you a debugging session, and the notebook's tests pin all three so they stay honest.
Every price arrives as a JSON string. The record reads "19.63", not 19.63, so anything numeric needs an explicit cast before it reaches a chart. Related, and more useful than it first looks: the currency is split from its symbol, with currency holding "GBP" and currencyRaw holding "£", which means nothing in your code has to parse £19.63 apart. Availability comes back normalized against schema.org, so it reads "InStock" rather than whatever phrasing the page happened to use.
The third detail is the one to take seriously. Every record carries a metadata.probability value, because automatic extraction is probabilistic rather than guaranteed. Across all 105 product records the probability sat at 0.99 or above, which is reassuring, but I saw the other end of that range by accident: while writing the notebook I guessed at a product URL rather than using one that stage one had discovered, and the guess did not exist. Zyte API still returned a product record for it. The name was 404 Not Found, every other field was null, and the probability was 0.10. That number is the signal, and a pipeline that ignores it will happily store a page of nothing as a product. Filter on it.

Dragging a selection on the chart back into Python

This is the section the article exists for. In marimo you can wrap an Altair chart in mo.ui.altair_chart, and the selection a reader makes with the mouse becomes a dataframe in Python, in a cell that reruns automatically. marimo's documentation states it plainly: "selections you make on the frontend are automatically made available as Pandas dataframes in Python." In practice the frame you get back matches the frame you put in, so feeding it polars gives you polars.

brush = alt.selection_interval(encodings=["x"])
base = (
    alt.Chart(books)
    .mark_circle(size=90, opacity=0.65)
    .encode(
        x=alt.X("price:Q", title="Price (GBP)"),
        y=alt.Y("category:N", title=None),
        color=alt.condition(brush, alt.value("#c026d3"), alt.value("#cbd5e1")),
        tooltip=["name", "category", "price", "availability", "probability"],
    )
    .add_params(brush)
)
prices = mo.ui.altair_chart(base, chart_selection=False, legend_selection=False)

Then, in a different cell, the selected rows are simply available:

mo.ui.table(prices.value, selection=None, page_size=8)

That is the whole mechanism. Drag across a price band on the chart and the table below it shows exactly those books, because the table's cell references prices, and marimo reran it the moment the selection changed. You can get to something similar in Jupyter with ipywidgets and a callback, but it is a noticeably larger amount of machinery for the same result, and it is the machinery that tends to break when someone else opens the notebook.
Worth being precise about what "reran" means here, because it is the part a linear notebook cannot do. The drag only invalidates the two cells that actually read prices. It does not touch books, and it does not touch either of the two Zyte API calls, so dragging the chart back and forth all afternoon spends no additional credit:

Two practical notes. marimo adds a default selection based on the chart's mark type, and when you want to control that behavior yourself its plotting guide tells you to set chart_selection and legend_selection to False and add the selection to the Altair chart directly with .add_params, which is exactly what the code above does. And selections stream to Python as you drag, which is fine at this size and worth debouncing if the downstream work is expensive.

Asking the same dataframe a SQL question

marimo also has SQL cells, which run against your existing dataframes rather than requiring a database, and which return a dataframe so the result flows onward like anything else. Having scraped into polars, I can group the same data without switching mental models:

SELECT
    category,
    count(*)                AS books,
    round(median(price), 2) AS median_price,
    sum(was_truncated::int) AS truncated_names
FROM books
GROUP BY category
ORDER BY median_price DESC

For a certain kind of question, and grouped aggregates are exactly that kind, this is simply the clearer way to write it.

Keeping a metered API from surprising you

Zyte API is billed per successful request, so a notebook that fetches on every keystroke would be an expensive notebook. marimo's guide for expensive notebooks opens by framing the goal as preventing "expensive cells, which may call APIs or take a long time to run, from accidentally running," which is a fair description of the problem.
The notebook uses two mechanisms. The first is a gate, so that opening the file sends no requests at all:

headless = mo.app_meta().mode == "script"
mo.stop(
    not headless and not fetch.value,
    mo.md("Press **Fetch through Zyte API** above. No requests are sent until you do."),
)

The second is a disk cache on the function that does the fetching, using mo.persistent_cache, whose cache key includes the arguments, so re-running with the same URLs and the same requested fields reads from disk instead of calling the API again. The effect is easy to measure: the cold run against an empty cache took 104.7 seconds for 111 requests, and the next run took 1.1 seconds, made no API calls at all, and produced the same 105 rows. On the pricing side, this scrape used two data types, since productList and product are billed separately, and Zyte's pricing page puts automatic extraction at $0.0004 to $0.0016 per data type before volume discounts, with rate-limited and unsuccessful responses free. If you want the account-level version of the same discipline rather than the notebook-level one, Zyte shipped spending controls and usage insights in May 2026.

That mo.app_meta().mode check in the gate is worth a second look, because it is what makes the next section work.

The same file as an app and as a cron job

marimo reports its mode as edit in the notebook, run in an app, and script when the file is executed by Python. The gate above only applies in the first two, where there is a human present to press a button, which means the identical file runs unattended without modification:

uvx marimo edit --sandbox notebook.py   # the notebook
uvx marimo run --sandbox notebook.py    # an app, with the code hidden
uv run notebook.py                      # a plain script, for cron

The middle command is the one I would not have predicted finding useful. It serves the same notebook as a small web application with the code hidden and only the inputs, the chart, and the table showing, which is a reasonable thing to hand to a colleague who wants to look at prices and does not want to look at Python.

What you give up

An honest comparison has to include the costs, and marimo documents its own.
Because the file is Python rather than JSON, your outputs are not stored in it, so a notebook in version control shows the code and not the plots. There is a setting that snapshots to HTML or ipynb alongside the file, and marimo export ipynb for when you need the other format.
IPython magics do not work, so %pip, %%time, and !ls all need replacing, and marimo publishes a table of equivalents for the common ones.
The restriction that takes the longest to absorb is that the same variable cannot be defined in more than one cell, which is what allows marimo to build the graph in the first place. If you are used to redefining df in six consecutive cells as you clean it up, that habit has to go: merge the cells, alias the dataframe, or prefix throwaway variables with an underscore to make them local to a cell.
If you already have a notebook you like, the conversion is one command, and it is a reasonable way to see what your own code looks like under a dataflow model:

marimo convert your_notebook.ipynb -o your_notebook.py

Try it yourself

The notebook, its tests, and the fixture behind them are on GitHub at zytelabs/zytelabs-marimo-web-data, and the whole thing is one file plus a dependency header, so there is nothing to install beyond uv. The 18 tests are deliberately offline and run against a saved response set, which means every number in this article can be re-checked without spending a Zyte credit. Signing up for Zyte API comes with $5 of free credit for the first billing month, and because the notebook caches to disk, going back for a second look at the same data costs nothing.
The gallery gap I opened with is still there: nothing in it goes and gets its own data, and this one is my attempt at the first. It is a thin category to be the only entry in, so if you build something in the same shape, publish it and say so.
And if the reactive idea appeals to you but your interest is in giving tools to an agent rather than to a person, I wrote about giving a coding agent a fetch tool that survives the real web in August 2026. Either way the argument is the same one. The notebook is a perfectly good place to go and get the data, and treating it as somewhere you only inspect data that arrived by other means sells it short.
Originally published on the Zyte blog.

Rendering Javascript pages without giving up Scrapy

John Rooney — Sun, 06 Sep 2026 08:00:00 +0000

A page can contain more JavaScript than HTML and still not need a browser. Often the data is already in the response, in a script tag, or behind a JSON endpoint the application calls itself. Reaching for browser automation before checking those places is an expensive habit.

The modern approach is to treat JavaScript rendering as an acquisition choice. Keep Scrapy for scheduling, retries, item processing, pipelines, logging, and monitoring. Use an ordinary Scrapy request when it is enough, call an authorized data endpoint when that is the real source, and render in a browser only when you genuinely need the browser's DOM or state.

A JavaScript application is not the same thing as a browser-only data source.

That distinction is the difference between a crawler that scales predictably and one that launches a browser for every pagination link.

Find the data source before choosing a renderer

Start by looking at the response Scrapy receives. The Scrapy guide to dynamically loaded content suggests saving a fetched response locally, then using browser developer tools to inspect the requests that actually supply the data.

scrapy fetch --nolog https://example.com/products > response.html

The following decision table is deliberately boring. That is a strength. It prevents a rendering problem from becoming the default architecture for the whole project.

What you find	First choice	Why
Data in the HTML response	Normal Scrapy request and selectors	Lowest operational cost and simplest failure mode
JSON-LD or serialized state in a script element	Normal Scrapy request and JSON parsing	The browser has nothing useful to add
A documented or otherwise authorized JSON endpoint	Scrapy request to that endpoint	You receive the data in its native form
Content appears only after client-side code runs	`scrapy-playwright` for those requests	A rendered DOM is the actual input
A public page needs interaction or browser-bound state	A targeted browser flow and explicit context management	State is part of acquisition, not an incidental header

The second and third rows are easy to miss when looking at a polished application UI. They are often the better solution.

Call the API with Scrapy when the API is the source of truth

Suppose the product page fetches a JSON document from an endpoint your project is permitted to use. Make that request with Scrapy, not with an ad hoc requests loop inside a callback. Scrapy can schedule it, retry it, log it, and feed the resulting items into the same pipeline and Spidermon checks from the earlier parts of this series.

JsonRequest serializes a JSON request body and sets the appropriate content type. The Scrapy request and response documentation also documents response.json(), which decodes the response body for you.

# catalog/spiders/products_api.py
import scrapy
from scrapy.http import JsonRequest

from catalog.items import Product


class ProductsApiSpider(scrapy.Spider):
    name = "products_api"
    api_url = "https://api.example.com/v1/products/search"

    async def start(self):
        yield JsonRequest(
            self.api_url,
            data={"category": "widgets", "page": 1},
            callback=self.parse_products,
        )

    def parse_products(self, response):
        payload = response.json()

        for product in payload["products"]:
            yield Product(
                name=product["name"],
                price=product["price"],
                description=product.get("description", ""),
                url=response.urljoin(product["url"]),
            )

Real APIs have their own pagination, rate limits, and terms. Treat those as part of the spider's contract. Do not treat a network-panel request as permission to use an endpoint.

This route also fits neatly with Part 2. The Product item stays the stable output contract even when the input changes from an HTML Page Object to an API response. That makes it possible to replace an acquisition strategy without rewriting exports, validation, or downstream consumers.

Add Playwright only for the requests that need a browser

When the page needs client-side rendering, scrapy-playwright is the useful integration point. It is a Scrapy download handler backed by Playwright, so rendered requests still go through Scrapy's regular scheduling and item-processing workflow.

Install the Python package and the browser binaries:

pip install scrapy-playwright
playwright install chromium

Then configure the handler and asyncio-compatible Twisted reactor. Recent Scrapy projects already use this reactor by default, but keeping it explicit removes an easy source of confusion when updating an older project.

# catalog/settings.py
DOWNLOAD_HANDLERS = {
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

That setting does not mean every HTTPS request launches Chromium. A request uses Playwright only when it carries meta={"playwright": True}. Keep listing pages, robots-aware discovery, and API calls on Scrapy's normal downloader unless rendering is specifically necessary.

Make browser rendering the project default only when it really is the default

scrapy-playwright does not provide a setting that turns on Playwright for every request. Its download handler deliberately falls back to Scrapy's normal HTTP handler unless a request has the playwright meta flag.

If a narrowly scoped project truly needs browser rendering for every request, set that flag in a downloader middleware. Downloader middlewares run before Scrapy selects the download handler.

# catalog/middlewares.py
class EnablePlaywrightMiddleware:
    def process_request(self, request, spider):
        request.meta.setdefault("playwright", True)

# catalog/settings.py
DOWNLOADER_MIDDLEWARES = {
    "catalog.middlewares.EnablePlaywrightMiddleware": 543,
}

This preserves a request that has explicitly set playwright to False, which makes an opt-out possible for a public API or static asset endpoint. In most mixed crawls, the opposite policy is easier to operate: leave the default as False and mark only the rendered paths. The scrapy-playwright activation documentation explains why registering its handler alone does not enable rendering for every request.

Wait for a meaningful element, not an arbitrary number of seconds

The common first browser spider waits five seconds and hopes. The reliable version waits for an observable condition that says the field you need has appeared.

PageMethod lets the download handler perform an action before it returns the final rendered response. The callback then receives a normal Scrapy response whose selectors see the browser-rendered DOM.

# catalog/spiders/rendered_products.py
import scrapy
from scrapy_playwright.page import PageMethod

from catalog.items import Product


class RenderedProductsSpider(scrapy.Spider):
    name = "rendered_products"
    start_urls = ["https://example.com/widgets"]

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            yield response.follow(
                href,
                callback=self.parse_product,
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_selector", ".product-price"),
                    ],
                },
            )

    def parse_product(self, response):
        yield Product(
            name=response.css("h1::text").get(default="").strip(),
            price=response.css(".product-price::text").get(default="").strip(),
            description=" ".join(
                response.css(".description *::text").getall()
            ).strip(),
            url=response.url,
        )

The example keeps the extraction short to show the boundary. In the project from Part 2, move those selectors into the appropriate scrapy-poet Page Object once the rendering path is understood. Browser acquisition should not become an excuse to put all extraction back in callbacks.

Avoid time.sleep() and fixed browser delays. They are slow when the page is fast and unreliable when the page is slow. A selector, response condition, or explicit page action describes what the spider needs rather than guessing how long the target will take.

Treat browser pages and contexts as limited resources

A browser context holds state such as cookies and local storage. A page consumes memory and a concurrency slot. Neither should be created casually.

Use a named context when public browsing needs a coherent state, such as a chosen locale, and cap the amount of browser work your crawler can do at once:

# catalog/settings.py
PLAYWRIGHT_CONTEXTS = {
    "catalog": {
        "locale": "en-GB",
    },
}

PLAYWRIGHT_MAX_CONTEXTS = 4
PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 2
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 20 * 1000

# catalog/spiders/localized_products.py
import scrapy


class LocalizedProductsSpider(scrapy.Spider):
    name = "localized_products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/widgets",
            callback=self.parse_product,
            meta={
                "playwright": True,
                "playwright_context": "catalog",
            },
        )

    def parse_product(self, response):
        return {"url": response.url}

Only set playwright_include_page when the callback must directly use the Playwright Page, for example for a multi-step interaction that cannot be expressed as PageMethod calls. Otherwise let scrapy-playwright close it after producing the response. Leaving pages open is a quiet way to stall a crawl.

Connect to a managed browser when local Chromium is not the right runtime

Local launch is the simplest starting point. It is not the only option. scrapy-playwright can connect to a browser that another system owns, which is useful when browser lifecycle, isolation, or capacity is managed outside the spider process.

Connection	Setting	Use it when	Important constraint
Chrome DevTools Protocol	`PLAYWRIGHT_CDP_URL`	You have a remote Chromium browser exposing CDP	Chromium only; local launch options are ignored
Playwright WebSocket connection	`PLAYWRIGHT_CONNECT_URL`	You connect to a Playwright server endpoint	Client and server Playwright versions must be compatible

Here is a CDP connection to a remote Chromium instance:

# catalog/settings.py
PLAYWRIGHT_CDP_URL = "http://browser-worker.internal:9222"
PLAYWRIGHT_CDP_KWARGS = {
    "timeout": 20 * 1000,
}

Do not configure PLAYWRIGHT_CDP_URL and PLAYWRIGHT_CONNECT_URL together. They are alternative connection models, and both bypass PLAYWRIGHT_LAUNCH_OPTIONS. The scrapy-playwright connection settings explain the lifecycle and compatibility implications in detail.

The operational question is more important than the setting name: who restarts a disconnected browser, who limits capacity, and how are browser logs correlated with the Scrapy job? Answer those before moving a spider to remote browsers.

Use browser-provider integrations for more options

Current scrapy-playwright releases expose PLAYWRIGHT_BROWSER_PROVIDER, an extension point for a provider that owns browser startup, connection, and teardown. The default provider supports local launch, CDP, WebSocket connections, and persistent contexts. The project also documents examples for Playwright-compatible alternatives such as Patchright and Camoufox.

These are sometimes described as stealth-browser options. That phrase promises more than the setting does. They are browser-provider integrations, not a guarantee that a target will accept a crawl or that its access rules can be ignored.

Use a provider only when you can name the concrete browser behavior or runtime constraint it addresses, and test it as a separate acquisition implementation. Keep the same Items, Page Objects, logs, and monitors around it. Respect the site's terms, permissions, and applicable requirements regardless of which browser implementation is underneath.

Monitor rendered spiders as a separate operating mode

Part 3 used Spidermon to monitor output quality. Browser-backed spiders need those checks, plus a few browser-specific signals:

navigation timeouts and Playwright request failures;
an unexpected rise in error pages or interstitial pages;
a drop in rendered item count compared with the expected scope; and
pages or contexts left open long enough to reduce crawl throughput.

A rendered response can be syntactically valid HTML and still be the wrong page. Validate the resulting item, not just the HTTP status.

Start with an item-count monitor and required-field validation. Then add a periodic monitor when a long-running browser crawl can waste meaningful capacity after a failure. The Spidermon periodic-monitoring documentation covers that pattern.

Browser automation also changes the economics of a spider. It is worth keeping a normal Scrapy path for the pages that do not need rendering, then using monitoring to prove that the smaller browser surface still produces the expected data.

Recent work on using Web Scraping Copilot in VS Code can help with the initial mechanics of a Scrapy project. It does not replace the technical judgment in the table above. The same caution appears in a review of AI-generated Scrapy projects: a spider that runs is not necessarily one with a sound acquisition strategy. For teams that need a managed alternative to operating browser infrastructure, Zyte API's headless browser capability is another option to evaluate against your requirements.

Questions developers ask about Scrapy and JavaScript rendering

Do I need Playwright because a site uses JavaScript?

No. Inspect the Scrapy response and network requests first. If the data is in HTML, serialized state, or an authorized JSON response, use Scrapy without a browser. Use Playwright when the rendered DOM, client-side interaction, or browser state is genuinely required.

Can one spider mix ordinary and rendered requests?

Yes. That is usually the point. Register the download handler once, then set meta={"playwright": True} only on the individual requests that need it. The rest continue through Scrapy's standard downloader.

When should I use CDP instead of `PLAYWRIGHT_CONNECT_URL`?

Use CDP when the remote browser is Chromium and exposes a CDP endpoint. Use the Playwright connection URL for a Playwright server endpoint. Pick one connection model, document who owns browser lifecycle, and test reconnect behavior before relying on it in scheduled crawls.

Should I enable a stealth-oriented browser provider by default?

No. Start with the default provider and add another only for a defined, permitted requirement. A provider increases the number of moving parts, so it should earn its place through a concrete operational need, not a vague expectation that every modern site needs it.

Keep the browser at the edge of the architecture

The series began with Scrapy's core components, moved extraction into Page Objects, and added monitoring around the data contract. Browser rendering belongs at the edge of that architecture: it changes how a selected response is acquired, not how the rest of the project is designed.

For your next JavaScript-heavy target, inspect one product page and one listing page. Write down where each field comes from. Then choose the least expensive permitted acquisition method that gives you that source, and make the resulting Items pass the same checks as every other spider.

Spider monitoring made easy

John Rooney — Sat, 05 Sep 2026 08:00:00 +0000

A Scrapy job can exit cleanly after collecting zero products. It can also export 20,000 products with no prices. Neither failure needs to raise an exception in your spider, which means neither failure is visible if the only definition of success is an exit code of zero.

Spidermon gives a Scrapy project a place to define success in terms that matter to the data: how many items arrived, whether required fields are present, and what should happen when those expectations are not met.

A finished crawl tells you that the process stopped. A healthy crawl tells you that the data is still useful.

Part 1 put logging, pipelines, and feed exports in the right places. Part 2 moved page-specific extraction into typed Page Objects. This article closes the loop: it turns the resulting crawl statistics and item validation into monitors, then connects a failed monitor to an action that somebody can see.

What should a production spider prove?

Start with the failure modes of the specific spider, not a generic monitoring checklist. Our running catalog spider should not pass merely because it issued requests and wrote a file. It should show that it found enough products and that those products still meet the Product schema from Part 2.

Signal	What it catches	What it does not prove
Exit status	Unhandled process-level failure	That the expected pages or fields were collected
`item_scraped_count`	Empty or unexpectedly small output	That individual fields are valid
JSON Schema validation	Missing fields and incompatible values	That a price is commercially plausible
A Spidermon failure action	A failed expectation reaches the right place	That a person has responded

The last column matters. Monitoring is not a promise that data is perfect. It is a way to turn specific, known bad states into visible events before downstream users discover them in a dashboard or a customer-facing report.

Enable Spidermon before writing a monitor

Install Spidermon and jsonschema, which its JSON Schema validation feature uses:

pip install spidermon jsonschema

Enable the extension and tell it which monitor suite should run when the spider closes. Keep the configuration with the rest of your project settings, not hidden in a deployment script.

# catalog/settings.py
SPIDERMON_ENABLED = True

EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}

SPIDERMON_SPIDER_CLOSE_MONITORS = (
    "catalog.monitors.CatalogCloseMonitorSuite",
)

Spidermon runs monitors in suites at defined points in the spider lifecycle. A spider-close suite is the natural first choice because the final Scrapy stats and the item-validation results are both available by then. The Spidermon getting-started guide documents the extension configuration and monitor lifecycle.

Do not begin by alerting on every retry or every 404. Start with the conditions under which you would not ship the resulting dataset.

Define a minimum item count that means something

The first monitor checks an intentionally modest, project-specific threshold. In a real catalog, derive it from the spider's normal output, its configured scope, and the consequences of partial data. A development crawl limited to one category should not be judged by the threshold of a nightly full crawl.

# catalog/monitors.py
from spidermon import Monitor, MonitorSuite, monitors


@monitors.name("Catalog crawl")
class CatalogStatsMonitor(Monitor):
    @monitors.name("Minimum product count")
    def test_minimum_product_count(self):
        item_count = getattr(self.data.stats, "item_scraped_count", 0)
        minimum_count = 100

        self.assertGreaterEqual(
            item_count,
            minimum_count,
            msg=(
                f"Expected at least {minimum_count} products, "
                f"but extracted {item_count}"
            ),
        )


class CatalogCloseMonitorSuite(MonitorSuite):
    monitors = [CatalogStatsMonitor]

Run the spider locally, then deliberately set minimum_count above the current output once to see the failure format in the logs:

scrapy crawl products

That small exercise is worth doing. A monitor that has never failed in development is only assumed to be connected to your runtime.

Scrapy already collects the stats that make monitors possible. The Scrapy stats documentation is useful when you need to inspect the exact counter name produced by a downloader, middleware, pipeline, or your own component.

Validate items at the point where they leave the spider

Part 2 gave our Page Object a typed Product item. That prevents plenty of accidental field mistakes in Python, but the data still comes from the web. Schema validation gives the project an independent check on the item it is about to export.

Place the validation pipeline after the project's own cleanup pipeline. Validating before a pipeline normalizes a price or URL creates false alarms, while validating before a later pipeline changes the item creates false confidence.

# catalog/settings.py
ITEM_PIPELINES = {
    "catalog.pipelines.ProductPipeline": 300,
    "spidermon.contrib.scrapy.pipelines.ItemValidationPipeline": 800,
}

SPIDERMON_VALIDATION_SCHEMAS = ["schemas/product.json"]
SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS = True

Here is a deliberately small schema. It requires all four fields from the Product item and rejects empty name and price values. Your schema should grow with your data contract, not with every possibility you can imagine on the first day.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "name": {"type": "string", "minLength": 1},
    "price": {"type": "string", "minLength": 1},
    "description": {"type": "string"},
    "url": {"type": "string", "format": "uri"}
  },
  "required": ["name", "price", "description", "url"]
}

With SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS enabled, invalid items carry validation information in an _validation field. That is useful while diagnosing a new rule. For a production export, decide whether downstream systems should receive flagged records or whether the pipeline should drop invalid items using SPIDERMON_VALIDATION_DROP_ITEMS_WITH_ERRORS.

The Spidermon item-validation guide recommends making the validation pipeline the last one, and documents the schema, error-field, and item-dropping settings.

Fail the crawl when validation errors become data loss

Adding validation errors to items is observability. A monitor turns those errors into a decision.

# catalog/monitors.py
from spidermon import Monitor, monitors
from spidermon.contrib.monitors.mixins import StatsMonitorMixin


@monitors.name("Catalog data quality")
class ProductValidationMonitor(Monitor, StatsMonitorMixin):
    @monitors.name("No product validation errors")
    def test_no_product_validation_errors(self):
        error_count = getattr(
            self.stats,
            "spidermon/validation/fields/errors",
            0,
        )

        self.assertEqual(
            error_count,
            0,
            msg=f"Found {error_count} product field validation errors",
        )

Add it to the suite from the first monitor:

class CatalogCloseMonitorSuite(MonitorSuite):
    monitors = [
        CatalogStatsMonitor,
        ProductValidationMonitor,
    ]

This rule is intentionally strict. It fits a catalog where a product without a price is not acceptable output. Other use cases may tolerate a small number of missing optional descriptions, or may care about a percentage rather than an absolute count. Spidermon provides validation-monitor helpers for field-level counts and percentages when that is the more honest definition of failure. Do not set a tolerance just to silence a monitor. Explain what the tolerated records mean for the people using the data.

Use periodic monitors to stop bad long-running crawls early

Spider-close monitors are necessary, but they arrive too late for every problem. If a broad crawl has been returning error pages for 45 minutes, waiting for it to complete wastes capacity and leaves an operator with a larger incident to untangle.

Periodic monitors run a normal monitor suite at a configured interval while the spider is still running. They are a good fit for failures that become expensive as they continue: a rising error count, no increase in extracted items, or a job running past its intended duration. Keep final data-quality validation in the spider-close suite, where the complete output is available.

A periodic monitor is a circuit breaker for a crawl, not a replacement for its final quality gate.

For the catalog spider, close the crawl after 20 logged errors. The number is an example, not a threshold to copy. Choose one that reflects the target site's normal failure behavior and the point at which continuing is no longer useful.

# catalog/actions.py
from spidermon.core.actions import Action


class CloseSpiderAction(Action):
    def run_action(self):
        spider = self.data["spider"]
        spider.logger.error("Closing spider after a periodic monitor failure")
        spider.crawler.engine.close_spider(
            spider,
            reason="closed_by_spidermon",
        )

# catalog/monitors.py
from spidermon import Monitor, MonitorSuite, monitors

from catalog.actions import CloseSpiderAction


@monitors.name("Catalog periodic health")
class CatalogPeriodicMonitor(Monitor):
    @monitors.name("Error count remains below the limit")
    def test_error_count(self):
        error_count = self.data.stats.get("log_count/ERROR", 0)
        maximum_errors = 20

        self.assertLessEqual(
            error_count,
            maximum_errors,
            msg=f"Crawl logged {error_count} errors, limit is {maximum_errors}",
        )


class CatalogPeriodicMonitorSuite(MonitorSuite):
    monitors = [CatalogPeriodicMonitor]
    monitors_failed_actions = [CloseSpiderAction]

Schedule the suite every 60 seconds:

# catalog/settings.py
SPIDERMON_PERIODIC_MONITORS = {
    "catalog.monitors.CatalogPeriodicMonitorSuite": 60,
}

Spidermon also provides built-in periodic monitors for execution time and item-count increase. Those are often a better first choice

A failed monitor should trigger an action

Logs are fine when you are running one spider in a terminal. They are not a monitoring strategy for scheduled jobs. Attach a failure action to the monitor suite so that a broken data contract reaches a channel where it can be triaged.

# catalog/monitors.py
from spidermon.contrib.actions.slack.notifiers import (
    SendSlackMessageSpiderFinished,
)


class CatalogCloseMonitorSuite(MonitorSuite):
    monitors = [
        CatalogStatsMonitor,
        ProductValidationMonitor,
    ]
    monitors_failed_actions = [
        SendSlackMessageSpiderFinished,
    ]

Keep credentials out of version control. For example, load the Slack configuration from the environment used to run the spider:

# catalog/settings.py
import os

SPIDERMON_SLACK_SENDER_TOKEN = os.environ["SPIDERMON_SLACK_SENDER_TOKEN"]
SPIDERMON_SLACK_SENDER_NAME = "catalog-monitor"
SPIDERMON_SLACK_RECIPIENTS = ["#data-alerts"]

Spidermon also supports actions for email, Telegram, Discord, Sentry, reports, and custom integrations. Start with one destination that has an owner. Alerting an unmonitored channel is only a more elaborate way to write to a log. The actions documentation lists the built-in and custom-action options.

Choose the action by the failure

Monitor result	First action	What to investigate
Zero or very few products	Alert the spider owner	Scope change, blocked listing page, pagination, or selector failure
A few invalid optional fields	Report and track the trend	A template variant or a cleanup rule
Invalid required fields	Alert and quarantine or drop affected output	A changed product page, response substitution, or broken extractor
A system-wide run failure	Escalate through the job platform	Credentials, deployment, networking, or a shared service

Do not make the first automated reaction "run the same spider again." Retrying can be sensible for a transient timeout. It cannot repair an HTML template that no longer has a .price element.

Keep the monitor suite small enough to trust

An early suite needs only a few checks that protect against known bad outcomes. For the catalog spider, start here:

At least the expected minimum number of products were extracted.
No required product fields failed schema validation.
A failed check sends one actionable notification to the owner.

Add checks when you can state the response to failure. For example, a sudden rise in retry count might warrant an alert only if it has a named owner and a clear threshold. Otherwise, it is a graph waiting to be ignored.

Zyte Web Data for Claude Code and Web Scraping Copilot 1.0 can speed up the work of creating spiders and Page Objects. The review loop is still yours. The same is true of the concerns raised in this practical review of AI-generated Scrapy projects: code that runs needs checks that say whether it produced usable data.

Questions developers ask about Spidermon

Should every validation error fail the crawl?

No. Fail on the errors that make the dataset unfit for its purpose. A missing product URL is usually serious. A missing optional marketing description may deserve a report and a follow-up instead. Make the distinction explicit in the schema and monitor threshold.

Can Spidermon replace tests for Page Objects?

No. Page Object fixture tests catch a regression against a known response before deployment. Spidermon checks the behavior and output of a real crawl. Use both, because they fail at different points in the workflow.

Should invalid items be dropped immediately?

Sometimes. During development, attaching _validation errors to items makes debugging easier. In production, dropping an invalid record may be safer than exporting it, provided the monitor still alerts you to the loss. Do not silently drop records and call the job successful.

Make a failure visible before the next scheduled crawl

Add one item-count monitor, one required-field check, and one failure action to an existing spider. Then break each condition on purpose in a safe environment and follow the signal from the Scrapy stats to the monitor result to the notification.

That is the beginning of operating a spider as a data system. Part 4 will return to acquisition: how to choose between Scrapy, APIs, and browser automation when the target site is more complicated than static HTML.

Building maintainable spiders with scrapy-poet

John Rooney — Fri, 04 Sep 2026 08:00:00 +0000

A Scrapy callback can begin life as six lines of selectors and end up responsible for following links, interpreting a page, normalizing an item, deciding whether data is missing, and fetching a second endpoint when the price is loaded separately. It still runs, so it is easy to call that design successful.

Until the site changes. Then the code that needs changing is also the code that decides how to crawl, and a selector fix becomes a risky edit to the spider's control flow.

scrapy-poet is valuable because it gives extraction logic its own home. It integrates web-poet, a Page Object framework, with Scrapy so that a spider can concentrate on crawling while Page Objects describe how to turn a particular kind of page into a typed item.

This is not an argument that every 20-line spider needs another abstraction. It is an argument that the callback is the wrong long-term home for reusable extraction logic.

The callback starts simple, then quietly owns too much

Here is the product callback from Part 1 of this series, expanded with a description and a JSON-LD fallback for a price. Nothing here is outrageous. That is why this pattern lasts so long.

import json


def parse_product(self, response):
    name = response.css("h1::text").get(default="").strip()
    price = response.css(".price::text").get(default="").strip()

    if not price:
        structured_data = response.css(
            "script[type='application/ld+json']::text"
        ).get()
        if structured_data:
            data = json.loads(structured_data)
            price = data.get("offers", {}).get("price", "")

    yield {
        "name": name,
        "price": price,
        "description": " ".join(response.css(".description *::text").getall()).strip(),
        "url": response.url,
    }

Now imagine a second spider needs the same product interpretation, or that the catalog moves to a new domain with different selectors but the same output schema. Copying the callback is fast. Keeping copies aligned is where the cost arrives.

The boundary we want is straightforward:

The spider owns where to go and which links to follow.
A Page Object owns what one page means and how to extract it.
A pipeline still owns rules applied to every yielded item, such as the price validation from Part 1.

That division keeps an ordinary selector change out of the crawling strategy. It also makes the right unit of reuse obvious: the page, not a large utility function passed a response, a spider, and a few flags.

Put product extraction and its output contract in a Page Object

A web-poet Page Object receives page inputs, such as the downloaded response, and exposes an item. Use an attrs class instead of a dictionary for that item. It makes a misspelled field or a missing required value fail close to the extractor, rather than silently creating a new key in a result file.

Returns[Product] declares the item contract for the Page Object. Combined with @field, it lets web-poet build the Product instance automatically, so there is no hand-written to_item() dictionary to keep in sync with the model.

# catalog/items.py
from attrs import define


@define
class Product:
    name: str
    price: str
    description: str
    url: str

# catalog/pages/products.py
import json

from web_poet import Returns, WebPage, field, handle_urls

from catalog.items import Product


@handle_urls("example.com")
class ProductPage(WebPage, Returns[Product]):
    @field
    def name(self) -> str:
        return self.css("h1::text").get(default="").strip()

    @field
    def price(self) -> str:
        visible_price = self.css(".price::text").get(default="").strip()
        if visible_price:
            return visible_price

        structured_data = self.css(
            "script[type='application/ld+json']::text"
        ).get()
        if not structured_data:
            return ""

        data = json.loads(structured_data)
        return data.get("offers", {}).get("price", "")

    @field
    def description(self) -> str:
        return " ".join(self.css(".description *::text").getall()).strip()

    @field
    def url(self) -> str:
        return self.response.url

The Page Object is not magically more correct than the callback. Its advantage is that the responsibilities and output schema are visible. A reviewer looking at ProductPage can discuss selector behavior and the Product contract without first understanding pagination, request metadata, or retry policy.

Keep the item class deliberately dull. Reuse Product across sites that produce the same schema, and keep site-specific parsing and cleanup in Page Object fields. A typed item is a contract, not a second place to hide extraction logic.

This is also a useful place to be honest about the limitations. A page with invalid JSON-LD will raise JSONDecodeError in the example above. Whether that should fail the item, produce a warning, or fall back to a different source is a data-quality decision. Do not swallow it just to make the crawl look healthier. Part 3 will deal with making those failures observable.

The scrapy-poet basic tutorial shows the same separation with a smaller example. The web-poet items guide explains its recommendation to use item classes, while the Page Object documentation goes deeper into available page inputs.

Enable scrapy-poet as a Scrapy add-on

For Scrapy 2.10 and later, scrapy-poet is enabled through Scrapy's add-on system. Add-ons are designed to package related Scrapy component configuration, rather than asking every project to manually copy middleware and fingerprinter settings. Scrapy's add-on documentation explains how their priorities and overrides work.

pip install scrapy-poet

# catalog/settings.py
ADDONS = {
    "scrapy_poet.Addon": 300,
}

SCRAPY_POET_DISCOVER = ["catalog.pages"]

SCRAPY_POET_DISCOVER tells the integration where to find Page Objects that have URL rules. Even if your first Page Object is imported directly by a spider, set this up now. It makes the project layout predictable once you add site-specific Page Objects. The setup guide documents both settings and the configuration required for older Scrapy versions.

Ask scrapy-poet for the item the spider needs

With a URL rule and Returns[Product] in place, a spider callback can ask for Product directly. scrapy-poet chooses the Page Object that can produce that item for the requested URL, builds it, and injects the completed item into the callback. The callback no longer needs the response for extraction.

# catalog/spiders/products.py
import scrapy

from catalog.items import Product
from scrapy_poet import DummyResponse


class ProductsSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/catalog"]

    def parse(self, response):
        links = response.css(".product-card a::attr(href)").getall()
        yield from response.follow_all(links, callback=self.parse_product)

    def parse_product(self, response: DummyResponse, product: Product):
        yield product

DummyResponse makes the dependency explicit: this callback consumes the item, not Scrapy's response object. If you need the Page Object itself, perhaps because the callback combines it with crawling logic, request the Page Object and await page.to_item() instead.

For the common case where a product page produces one item, callback_for() removes the callback entirely:

import scrapy

from catalog.pages.products import ProductPage
from scrapy_poet import callback_for


class ProductsSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/catalog"]

    parse_product = callback_for(ProductPage)

    def parse(self, response):
        links = response.css(".product-card a::attr(href)").getall()
        yield from response.follow_all(links, callback=self.parse_product)

Store callback_for(ProductPage) on the spider rather than creating it inline in response.follow_all(). The scrapy-poet documentation notes that the inline form does not work with Scrapy disk queues. That is exactly the kind of implementation detail you would rather learn before a crawl needs persistent scheduling.

Use URL rules when one spider needs more than one page implementation

The Page Object above describes one site's product pages. Reuse becomes more interesting when the crawl strategy stays the same while the page markup changes by domain.

web-poet can select a Page Object implementation using URL rules. Annotate a concrete class with handle_urls(), keep the common Product contract in a base class, and let discovery register the rule.

# catalog/pages/example_store.py
from web_poet import handle_urls

from catalog.pages.products import ProductPage


@handle_urls("example-store.com", instead_of=ProductPage)
class ExampleStoreProductPage(ProductPage):
    @field
    def name(self) -> str:
        return self.css("h1.product-title::text").get(default="").strip()

This does not mean every multi-site project should become a generic crawler. Different sites often have genuinely different navigation, login, pagination, or data contracts. Share the crawler only where the strategy is actually the same, and keep site-specific work explicit where it is not.

The payoff is most visible when it is time to replace a selector. You edit the relevant Page Object, test it against its fixture, and leave the rest of the crawl alone. The rules section of the scrapy-poet tutorial covers the fuller pattern, including explicit overrides.

Test an extraction rule without running a whole crawl

Most spider tests either mock so much that they no longer resemble a page, or run a real crawl and wait for the network. Page Objects give you a better middle ground: save the inputs and expected output for a representative page, then test the parser locally.

scrapy-poet supplies savefixture to capture a Page Object's dependencies and its to_item() result. The generated fixture can then be discovered by the pytest plugin that comes with web-poet.

scrapy savefixture catalog.pages.products.ProductPage \
  "https://example.com/products/widget-42"

python -m pytest

The fixture becomes a regression test. When a selector change causes a price or name to move, the test can fail before a scheduled crawl produces a quiet column of empty values. The scrapy-poet testing guide and web-poet test documentation explain the fixture layout, expected output, and how individual fields are tested.

Fixtures are not a substitute for live checks. They tell you whether your extraction code still interprets a known response as expected. They cannot tell you that the site has changed, that a login page arrived instead, or that the listing page stopped linking to products. You need both fixture tests and a small, monitored live crawl.

When scrapy-poet is the wrong extension point

Use scrapy-poet when the problem is page interpretation, reusable extraction, or dependencies needed to interpret a page. Do not use it as a container for every Scrapy customization.

Use an item pipeline for project-wide item validation and persistence.
Use downloader middleware for request and response behavior that applies before the spider sees a page.
Use a Scrapy extension and signals for crawl-wide behavior, such as reporting or metrics.
Keep code in the spider when it is navigation strategy that only that spider owns.

This matters because a framework can make bad boundaries easier to decorate. A Page Object that schedules all requests, sends Slack messages, and writes to a database is still a callback with a nicer name.

If you are using AI-assisted tooling to create Page Objects, keep the review focused on the contract: which URL does this class handle, which inputs does it need, and exactly which item does it promise to produce? Zyte Web Data for Claude Code and Web Scraping Copilot 1.0 both make Page Objects part of the development workflow. The gaps between runnable code and a shippable Scrapy project are also the subject of this recent practical review. None of these tools remove the need to decide where project behavior belongs.

Make one callback smaller this week

Pick the callback that makes you hesitate before editing it. Move only the page-specific extraction into a Page Object, leave crawling and pipelines where they are, and generate one fixture for the page that has caused trouble before.

If that leaves the spider with a clear description of how it moves through the site, you have found a useful boundary. In Part 3, we will use Spidermon to define what a healthy crawl looks like and act when the spider stops meeting that definition.