5 lessons from 5 interviews on AI agents and web scraping

Neha Setia — Tue, 23 Jun 2026 08:46:14 +0000

Over the last few months, I’ve been interviewing people at Zyte about AI, agents, web scraping, data quality, and what actually changes when LLMs enter developer workflows.

These were not abstract “AI will change everything” conversations. They were grounded in the daily work of people building and maintaining scraping systems, QA workflows, AI-assisted developer tools, and agentic infrastructure.

Five conversations stayed with me:

Adrian Chaves on Web Scraping Copilot and AI-generated scraping code
Julia Medina and Mihaela Popova on how LLMs changed their day-to-day development work
Javier on why AI feels like superpowers only when you already have judgment
Tomasz Lesiak on why AI cannot fix data quality until you define what “good” means
Konstantin Lopukhin on agent skills, scraping, and why the best skill often says the least

Here are the five lessons I’m taking from them.

1. Simpler code is usually better scraping code

In my interview with Adrian Chaves, we talked about what “correct” AI-generated scraping code actually means.

The obvious answer is: it extracts the right data.

But Adrian added a second standard that I keep coming back to: the code also has to be simple.

That matters because web scraping is not the kind of software where you write code once and forget about it. Websites change. Layouts move. Fields disappear. New variants appear. Pagination breaks. A selector that worked last month may quietly fail next month.

AI-generated code can easily become too defensive. It tries to handle every possible future case, and the result is often harder for a human to read, debug, and maintain.

The better standard is not “write code that imagines every future website change.”

It is:

Write the simplest code that gets the right value now, and surround it with tests and breakage detection.

That feels especially important in an AI-assisted workflow. If an agent generates a spider, the human still needs to understand what it produced. Maintainability is not a nice-to-have. It is part of correctness.

2. Developers are moving from coding-first to design-first

In my conversation with Julia Medina and Mihaela Popova, both described a similar shift in how they work with LLMs.

The work is less about typing every line yourself and more about deciding what should exist.

That sounds small. It is not.

When an AI assistant can generate code, the developer’s leverage moves upstream:

What are we trying to build?
What should the finished output look like?
What examples should the agent learn from?
What schema defines success?
What tests will tell us the output is wrong?
What should we not automate yet?

Julia described Web Scraping Copilot as a test-driven loop: give it a schema, create fixtures, generate expected output, write extraction code, run the tests, and iterate when the output is wrong.

Mihaela described LLMs inside production scraping workflows, where the question is not just “can the model extract this?” but where in the pipeline LLM judgment is actually useful.

That is the real shift I heard: developers are not disappearing. They are moving into design, evaluation, review, and system boundaries.

When agents can produce more code, the human job becomes more about defining what good looks like.

3. AI multiplies judgment. It does not replace it.

Javier described his experience with a line I loved:

“I have superpowers now.”

But the interesting part was what those superpowers were made of.

Not magic. Not blind delegation. Not “the AI knows everything now.”

His superpower was years of engineering judgment moving faster.

He uses AI to discuss approaches, document decisions, preserve context, and accelerate tedious parts of the work. But the quality still comes from knowing what is good, what is risky, what deserves skepticism, and when the assistant is simply being too agreeable.

That distinction matters.

A junior developer and a senior developer can use the same AI tool and get very different results. The difference is not only prompt quality. It is taste, context, pattern recognition, and the ability to evaluate the answer.

AI can compress the time between idea and artifact. But it does not remove the need to know whether the artifact is any good.

If anything, it makes that judgment more visible.

4. Data quality begins before AI enters the workflow

My conversation with Tomasz Lesiak was a useful antidote to AI abstraction.

Before asking whether AI can fix data quality, he brought the problem back to three questions:

Did we catch every item?
Did we fill every required field?
Is the information actually correct?

Every QA process, whether manual, automated, or AI-assisted, is trying to answer some version of those questions.

This is easy to forget because AI can make messy output look polished. It can summarize incomplete data. It can normalize wrong values. It can make a broken extraction feel more usable than it is.

But if the crawl missed 40% of the products, a beautiful summary does not help.

If the price belongs to the wrong product, valid JSON does not help.

If a required field is empty even though the website had the information, the workflow failed.

So before AI enters the quality process, the team still needs a schema, field definitions, expected outputs, manual review, and tests. LLMs can help with fields that are hard to check using regex or simple rules, but they cannot define “good data” on behalf of the project.

That definition has to come first.

5. The best agent skill does not over-explain

The fifth conversation, with Konstantin Lopukhin, helped me name something I had been noticing in agentic workflows.

A good skill is not the longest possible instruction manual.

In fact, too much instruction can make an agent worse. If you tell it exactly what path to follow in every case, it may miss the simpler and more reliable answer sitting in front of it.

For web scraping, that matters a lot.

A general-purpose coding agent may try to scrape visible HTML when the page has a clean embedded API. It may use curl and get blocked. It may pick the first visible price without checking whether it belongs to the right product. It may produce a one-time answer when the user actually needs a repeatable spider.

A useful scraping skill gives the agent the domain knowledge it needs:

access is part of the problem
the least fragile data source is usually best
structured output must be checked
tests matter
evaluation stays foundational
repeatability matters more than a one-off answer

But it should not over-script the agent into brittle behavior.

The best skill says enough to prevent obvious mistakes, but not so much that it prevents good judgment.

The pattern across all five

The pattern I keep seeing is this:

AI is not making web scraping less human.

It is moving the human work.

From typing to designing.

From scraping once to building loops.

From code generation to evaluation.

From “can the agent do it?” to “can we trust the system?”

That is the thread I want to keep following.

Because the future of web scraping is not just agents that scrape.

It is developers designing reliable web-data systems where agents, tools, infrastructure, tests, and human judgment all know their place.

What the Scrapy Maintainer Thinks About AI-Generated Scrapers

Neha Setia — Sat, 11 Apr 2026 15:13:41 +0000

I sat down with Adrian Chaves, one of the lead Scrapy maintainers, who also works at Zyte, to ask him the questions I've been chewing on since Zyte launched Web Scraping Copilot: what happens when an LLM writes your spider(the web scraping code)? What gets easier? What doesn't change?

His answers surprised me. A few highlights:

On vibe coding: Adrian has thoughts about developers treating scraper generation as a black box, and why Scrapy's design philosophy matters more, not less, when an LLM is writing the code.
The bottleneck isn't what you think. He argues the hard part of scraping in 2026 isn't writing code. It's reading pages. And that's the part LLMs still struggle with.
What "good design meeting the future halfway" means. Why frameworks like Scrapy that were built for humans are turning out to be the best frameworks for AI agents too.
Where LLMs actually help. The concrete places where AI makes a scraper developer's life better, and where it just adds complexity.

Full conversation is on the Zyte blog, linked below. If you're building scrapers, thinking about adding AI to your extraction pipeline, or just curious what someone who's been maintaining one of the most widely used scraping frameworks for years thinks about all of this, it's worth a read.

Read the full interview on zyte.com

Happy to discuss in the comments.
What are you using AI for in your scraping workflow right now, and where have you hit walls?

Tags: web scraping • scrapy • ai • opus • anti-bot • Claude AI • sonnet • open source

DEV Community: Neha Setia