Over the last few months, I’ve been interviewing people at Zyte about AI, agents, web scraping, data quality, and what actually changes when LLMs enter developer workflows.
These were not abstract “AI will change everything” conversations. They were grounded in the daily work of people building and maintaining scraping systems, QA workflows, AI-assisted developer tools, and agentic infrastructure.
Five conversations stayed with me:
- Adrian Chaves on Web Scraping Copilot and AI-generated scraping code
- Julia Medina and Mihaela Popova on how LLMs changed their day-to-day development work
- Javier on why AI feels like superpowers only when you already have judgment
- Tomasz Lesiak on why AI cannot fix data quality until you define what “good” means
- Konstantin Lopukhin on agent skills, scraping, and why the best skill often says the least
Here are the five lessons I’m taking from them.
1. Simpler code is usually better scraping code
In my interview with Adrian Chaves, we talked about what “correct” AI-generated scraping code actually means.
The obvious answer is: it extracts the right data.
But Adrian added a second standard that I keep coming back to: the code also has to be simple.
That matters because web scraping is not the kind of software where you write code once and forget about it. Websites change. Layouts move. Fields disappear. New variants appear. Pagination breaks. A selector that worked last month may quietly fail next month.
AI-generated code can easily become too defensive. It tries to handle every possible future case, and the result is often harder for a human to read, debug, and maintain.
The better standard is not “write code that imagines every future website change.”
It is:
Write the simplest code that gets the right value now, and surround it with tests and breakage detection.
That feels especially important in an AI-assisted workflow. If an agent generates a spider, the human still needs to understand what it produced. Maintainability is not a nice-to-have. It is part of correctness.
2. Developers are moving from coding-first to design-first
In my conversation with Julia Medina and Mihaela Popova, both described a similar shift in how they work with LLMs.
The work is less about typing every line yourself and more about deciding what should exist.
That sounds small. It is not.
When an AI assistant can generate code, the developer’s leverage moves upstream:
- What are we trying to build?
- What should the finished output look like?
- What examples should the agent learn from?
- What schema defines success?
- What tests will tell us the output is wrong?
- What should we not automate yet?
Julia described Web Scraping Copilot as a test-driven loop: give it a schema, create fixtures, generate expected output, write extraction code, run the tests, and iterate when the output is wrong.
Mihaela described LLMs inside production scraping workflows, where the question is not just “can the model extract this?” but where in the pipeline LLM judgment is actually useful.
That is the real shift I heard: developers are not disappearing. They are moving into design, evaluation, review, and system boundaries.
When agents can produce more code, the human job becomes more about defining what good looks like.
3. AI multiplies judgment. It does not replace it.
Javier described his experience with a line I loved:
“I have superpowers now.”
But the interesting part was what those superpowers were made of.
Not magic. Not blind delegation. Not “the AI knows everything now.”
His superpower was years of engineering judgment moving faster.
He uses AI to discuss approaches, document decisions, preserve context, and accelerate tedious parts of the work. But the quality still comes from knowing what is good, what is risky, what deserves skepticism, and when the assistant is simply being too agreeable.
That distinction matters.
A junior developer and a senior developer can use the same AI tool and get very different results. The difference is not only prompt quality. It is taste, context, pattern recognition, and the ability to evaluate the answer.
AI can compress the time between idea and artifact. But it does not remove the need to know whether the artifact is any good.
If anything, it makes that judgment more visible.
4. Data quality begins before AI enters the workflow
My conversation with Tomasz Lesiak was a useful antidote to AI abstraction.
Before asking whether AI can fix data quality, he brought the problem back to three questions:
- Did we catch every item?
- Did we fill every required field?
- Is the information actually correct?
Every QA process, whether manual, automated, or AI-assisted, is trying to answer some version of those questions.
This is easy to forget because AI can make messy output look polished. It can summarize incomplete data. It can normalize wrong values. It can make a broken extraction feel more usable than it is.
But if the crawl missed 40% of the products, a beautiful summary does not help.
If the price belongs to the wrong product, valid JSON does not help.
If a required field is empty even though the website had the information, the workflow failed.
So before AI enters the quality process, the team still needs a schema, field definitions, expected outputs, manual review, and tests. LLMs can help with fields that are hard to check using regex or simple rules, but they cannot define “good data” on behalf of the project.
That definition has to come first.
5. The best agent skill does not over-explain
The fifth conversation, with Konstantin Lopukhin, helped me name something I had been noticing in agentic workflows.
A good skill is not the longest possible instruction manual.
In fact, too much instruction can make an agent worse. If you tell it exactly what path to follow in every case, it may miss the simpler and more reliable answer sitting in front of it.
For web scraping, that matters a lot.
A general-purpose coding agent may try to scrape visible HTML when the page has a clean embedded API. It may use curl and get blocked. It may pick the first visible price without checking whether it belongs to the right product. It may produce a one-time answer when the user actually needs a repeatable spider.
A useful scraping skill gives the agent the domain knowledge it needs:
- access is part of the problem
- the least fragile data source is usually best
- structured output must be checked
- tests matter
- evaluation stays foundational
- repeatability matters more than a one-off answer
But it should not over-script the agent into brittle behavior.
The best skill says enough to prevent obvious mistakes, but not so much that it prevents good judgment.
The pattern across all five
The pattern I keep seeing is this:
AI is not making web scraping less human.
It is moving the human work.
From typing to designing.
From scraping once to building loops.
From code generation to evaluation.
From “can the agent do it?” to “can we trust the system?”
That is the thread I want to keep following.
Because the future of web scraping is not just agents that scrape.
It is developers designing reliable web-data systems where agents, tools, infrastructure, tests, and human judgment all know their place.
Top comments (0)