Stopping a crawl when the data stops looking real

TL;DR

I built scrapy-jev, a Scrapy pipeline that asks a fast, cheap AI model whether each scraped field still looks like a real product name, price, SKU, or category, and stops the crawl when too many items fail the check. It works, and it caught the kind of breakage that normally goes unnoticed until someone opens the exported CSV. It also has a narrow job: it catches values that look wrong, not selectors that point at the wrong thing but happen to still return something plausible.

The problem I was trying to solve

Scrapers don't announce when they break. A site renames a CSS class, or restructures a product card, and the extractor starts returning the wrong text, or nothing at all, while every other signal in the crawl looks healthy. No exception fires. Nothing appears in the log. You find out when someone downstream opens the export and the price column is full of category names.

What I wanted was something that could ask, mid-crawl, "does this still look like a real value for this field?" and stop before that damage spread across thousands of pages, rather than finding out afterward from a customer complaint or a broken pipeline.

What I built it with

I wrote the client, pipeline, and add-on with DeepSeek V4 Pro doing most of the coding, working from a spec of the endpoint and the pipeline behavior I wanted.

The core idea only works if judging a field is cheap and fast enough to run inline, on every crawl, without turning into its own cost center. A general-purpose LLM generating a paragraph of reasoning per field would have been too slow and too expensive to run at any real scale.

I used Jev, a "System One" model from TypeSafe AI, reached through OpenRouter's alpha endpoint. Instead of generating text, Jev returns typed, calibrated decisions. You send it a state and one or more typed questions, and it returns structured answers with probabilities attached, in roughly 70 to 500 ms.

I wrote more about what Jev actually is, and where a model like it fits into web scraping, in an earlier post.

For each field, I sent a two-level score question, "does this look like a plausible, correctly-scraped value for this field, or not," which meant the returned score was directly interpretable as P(plausible). One request carried one question per field, evaluated against the whole item at once, so a five-field item cost one call, not five.

On top of that I built three pieces:

• JevClient, a thin wrapper that reads item fields through itemadapter, so it works against Scrapy Item, plain dicts, attrs, and dataclasses, and returns a score per field.
• QualityGatePipeline, an item pipeline that scores a sample of items, requires every field on an item to clear a threshold to count as a pass (not a majority, since one bad field should fail the item), and stops the spider if the sample's pass rate falls below a configured threshold.
• Addon, so installing the package doesn't require hand-editing ITEM_PIPELINES. It self-disables if no fields are configured, so adding the dependency does nothing until you turn it on.

Two design choices were deliberate rather than obvious. First, judging a fixed-size sample rather than every item keeps the cost of the check independent of crawl size. A 1,000-page crawl and a 100,000-page crawl spend the same amount on verification. Second, a missing field counts as a failure rather than being skipped, because a selector silently returning None is the most common way scrapers actually break, and skipping it would have defeated the entire point.

I checked the actual dollar cost against real calls logged during testing rather than guessing. A three-field scoring request, one field was missing and never sent, ran to 570 input tokens and cost $0.000024, which works out to $0.042 per million input tokens, output is free. Scaled to a full four-field item across a 1-million-item crawl, that's roughly 700 million input tokens, somewhere around $30. Scoring a fixed sample of 20 items per crawl instead, regardless of whether the crawl has 1,000 items or 1,000,000, costs a fraction of a cent.

At Jev's actual pricing, the dollar difference between sampling and scoring everything isn't the main argument, $30 per million items is trivial either way. The real argument for sampling is that scoring every item means every item's pipeline waits on a round trip to an external API, and the crawl now depends on roughly a million of those calls all succeeding rather than twenty.

Trying it yourself

Installing it doesn't touch ITEM_PIPELINES by hand:

uv add scrapy-jev

Turning it on is three lines in settings.py: register the add-on, and tell it which fields to judge.

ADDONS = {
    "scrapy_jev.Addon": 350,
}

JEV_FIELDS = ["name", "price", "sku", "category"]

JEV_API_KEY, or the OPENROUTER_API_KEY environment variable, has to be set too. If neither is present, or JEV_FIELDS is empty, the add-on raises NotConfigured and gets out of the way rather than doing nothing silently.

The rest is optional and has defaults tuned for a small verification sample rather than exhaustive checking:

Setting	Default	Meaning
JEV_FIELDS	[]	Fields to judge on each item.
JEV_SAMPLE_SIZE	20	Items to judge before deciding.
JEV_PASS_RATE_THRESHOLD	0.7	Minimum share (0–1) of items that must pass.
JEV_FIELD_THRESHOLD	0.5	Per-field plausibility below which a field fails.
JEV_MODEL	~typesafe/jev-latest	Model id for the decisions endpoint.
JEV_BASE_URL	https://openrouter.ai/api/alpha/decisions	Endpoint.
JEV_PIPELINE_PRIORITY	400	ITEM_PIPELINES priority used by the add-on.
JEV_API_KEY / OPENROUTER_API_KEY	—	API key (setting or environment variable).
JEV_STOP_REASON	"Extraction quality fell below threshold (Jev plausibility check)"	Closing reason logged when the gate stops the spider.

The bug that mattered more than the feature

The most useful thing I learned building this had nothing to do with AI. My first version raised CloseSpider from inside the pipeline to stop the crawl, and the crawl didn't stop. Scrapy's item-processing path wraps process_item in a bare except Exception, logs "Error processing item", drops the item, and keeps going. CloseSpider is an Exception subclass, so it gets swallowed like any other error. The gate fired every time, and the crawl just kept going anyway, for another thousand pages, retries and all.

The fix was to stop asking politely and instead tell the engine directly:

from scrapy.utils.defer import deferred_from_coro
deferred_from_coro(crawler.engine.close_spider_async(reason=reason))

That's not a Jev problem or an AI problem. It's the kind of thing you only find by actually watching a crawl you expected to stop keep running, and it's a reminder that the flashy part of a project is rarely where the real debugging happens.

Whether it actually worked

I tested it against a demo target, a fictional 1,000-product coffee shop with markup I'd deliberately built to drift, crawled through scrapy-poet page objects.

• Breaking the sku selector so it returned the product title instead: Jev scored sku around 0.08 to 0.35 while the other fields stayed above 0.9, and the gate stopped the crawl.
• Breaking the name selector so it returned nothing: every item in the sample failed, and the crawl stopped with a clear reason logged.
• Renaming the price CSS class: the gate didn't catch it, and correctly so. The demo's price extraction was a regex against a currency-prefixed string, not a class-based selector, so the rename didn't actually break anything. The check didn't cry wolf.

Here's what one of those failures actually looked like in the logs, an item where name came back None while the other three fields scored a clean 1.0:

2026-09-21 16:08:28 [scrapy_jev.client] INFO: Jev response: {'model': 'typesafe/jev-1.13-20260917', 'answers': {'price': {'type': 'score', 'score': 1, 'legend': {'0': 'Does not look like a real value for this field', '1': 'Looks like a plausible, correctly-scraped value for this field'}, 'probabilities': {'0': 0, '1': 1}, 'confidence': 0.99}, 'sku': {'type': 'score', 'score': 1, 'legend': {'0': 'Does not look like a real value for this field', '1': 'Looks like a plausible, correctly-scraped value for this field'}, 'probabilities': {'0': 0, '1': 1}, 'confidence': 0.99}, 'category': {'type': 'score', 'score': 1, 'legend': {'0': 'Does not look like a real value for this field', '1': 'Looks like a plausible, correctly-scraped value for this field'}, 'probabilities': {'0': 0, '1': 1}, 'confidence': 0.99}}, 'usage': {'input_tokens': 570, 'output_tokens': 43, 'cost': 2.394e-05}, 'id': 'gen-dec-1790003308-TkGiCpI2ZIe3KtT8fFSn', 'provider': 'TypeSafe'}
2026-09-21 16:08:28 [scrapy_jev.pipeline] WARNING: scrapy_jev: item failed quality check: name=None

name isn't in the answers at all, because a missing value never gets sent to Jev in the first place. JevClient only asks about fields that have something to judge, and the pipeline treats the absence itself as the failure. price, sku, and category came back at confidence 0.99, which is exactly what should happen when three fields are fine and only one selector is broken.

That last result mattered more than the first two. A quality gate that fires on cosmetic changes gets disabled by the second team it annoys. This one only fired when the extracted value was actually wrong.

Where it doesn't help

This is not a general defense against scraper drift, and I don't want to oversell it as one.

It only catches breakage that produces an implausible-looking value. A selector that starts pointing at the wrong product on the page, but returns something that's still shaped like a real name and a real price, will sail through untouched. Jev has no way to know the name belongs to the wrong item. It's a plausibility check, not a correctness check.

It's sample-based, which is the right tradeoff for cost but means a crawl can run through a chunk of bad items before or after the sampled window and never trigger the gate, depending on where the breakage falls relative to the sample.

It also adds a hard dependency on an external API mid-crawl, OpenRouter's alpha endpoint in this case, not even TypeSafe's own. That's one more thing that can be slow, rate-limited, or unavailable at exactly the moment you need the crawl to keep running smoothly. I didn't build in a fallback behavior for that; right now a Jev outage just means the gate isn't checking anything, silently, which is close to the same failure mode the whole project set out to fix.

The thresholds are guesses too: sample size, pass rate, per-field cutoff, all tuned against one demo site. I have no evidence yet that a 0.7 pass rate and a 0.5 per-field threshold are the right defaults for a real production catalog with more natural variance in its data.

Was it worth it

As a package: yes, with caveats. scrapy-jev is published, on PyPI, with trusted publishing so there's no stored token to leak, and it does the one thing it claims to do. It caught two different kinds of real selector breakage in testing and correctly ignored a change that didn't matter. That's a genuine, narrow win, and the packaging overhead, src layout, CI, changelog, was small next to the core pipeline work.

As a general answer to "how do I know when my scraper is broken": no, not by itself. It's a plausibility net under specific fields, not a correctness check, and it depends on an external model call succeeding at the exact moment you need it to. It's worth having if you already have a low-cost way to bolt it onto a pipeline, but it isn't something to build a monitoring strategy around by itself. The main thing I got out of it wasn't even the AI check. It was being forced to actually watch a crawl fail to stop, and finding a bug in my assumptions about Scrapy's exception handling that I'd have shipped otherwise.

Takeaway

The interesting result wasn't that an AI model can score whether a scraped value looks real. It can, cheaply and fast enough to run inline. What matters more is that the check is only as good as its blind spots, and the most valuable thing the project produced was finding a case where the failure mode I was defending against, a crawl silently continuing after something breaks, was hiding in my own stop-the-spider code, not in the site I was scraping.