Two months ago I needed a marketplace to look alive on day one, so I wrote a scraper. That part took an afternoon, because it always does now. Model-assisted or not, producing the first working version stopped being the hard part of a lot of software projects a while ago. What comes after is where most of them actually die, and it rarely announces itself as a design flaw. It looks like a 403.
The first portal had Cloudflare in front of it. The scraper went through. The second had DataDome: session tokens rotating on every page render, TLS fingerprinting, header ordering, request timing, device identifiers bound to the session. The endpoint I cared about returned "Suspicious activity" the moment a session went stale, which was often.
That is all I am going to say about how any of it was located. The method is not the interesting part. What happened to the code afterward is.
The first enforcer is not a lawyer
For most people writing this kind of code, the first enforcer will not be a lawyer. It will be an anti-bot vendor, and the penalty arrives as a 403 in the first week.
That is the cheapest lesson available on this entire subject. It costs an afternoon and some ego. The penalties that arrive as letters arrive later, cost more, and by then the thing is in production with a client's name on it.
Almost nobody prices the second kind. Almost everybody eventually meets the first.
What a 403 does to your architecture
The workaround was not cleverer code. It was: stop batching, one request every few minutes, refresh cookies out of a live browser by hand.
That is not automation. That is a person with extra steps.
It is worth sitting with, because this is the part that generalizes. The anti-bot layer did not stop me. It changed what I was. It converted a pipeline into a human process with a database attached, and it did that through economics rather than through blocking. Nothing ever returned a hard no. The throughput just fell until the automation stopped being automation.
Then I made an assumption
Having been pushed into that mode, I assumed I had also stepped out from under the legal problem.
The reasoning felt solid. Manual is what a person does. Scraping is what a machine does. Every argument anyone has about this uses the word "scraping", and I was no longer doing the thing the word describes. I was a guy with a browser and a spreadsheet.
I checked, eventually. The assumption turned out to be about three quarters right, which is worse than being wrong, because three quarters right is the kind of thing you act on.
Three moved. One did not.
Terms of service moved. They usually single out automated access specifically, and a human reading and typing does not trip an automation-specific clause merely by being manual. That is real relief, and it is narrower than it sounds: the same documents almost always carry separate clauses on extraction, reuse and commercial copying, and those do not care how you got there.
The database right moved. It targets systematic extraction of a substantial part of a database, and repeated extraction of insubstantial parts where the cumulative effect reconstructs a substantial one. A person hand-picking a few hundred ordinary records usually lowers that exposure dramatically. Also narrower than it sounds: "substantial" is qualitative as well as quantitative, so a small hand-picked set of the most valuable rows is not automatically safe.
The third one moved and nobody writes it down. "Manual market research" and "scraper across six European portals" describe identical rows and read as opposite companies to anyone deciding whether to fund you. Not a legal exposure. Priced anyway.
Then the one that did not move.
Article 4(2) GDPR defines processing as operations performed on personal data "whether or not by automated means". There is a qualifier in Article 2(1) for non-automated processing: it is in scope where the data form part of a filing system, or are intended to form part of one. Article 4(6) defines a filing system as any structured set of personal data accessible according to specific criteria.
A structured spreadsheet can be a filing system. In my case, that was the whole point of the spreadsheet.
And notice there is no moment where that gate swings shut. It does not close at some volume threshold, or the first time you automate a fetch. Intent to file is enough, and you had the intent before you opened the file. Filing the result was the entire reason you were collecting.
So a mode change forced on me by a vendor's bot detection moved three exposures and left the personal-data gate exactly where it had always been.
Why this is an engineering post
Because the failure is a familiar one wearing different clothes.
"Is scraping legal" fails the same way "it works on my machine" fails. It is an unqualified verdict standing in for a claim that needs a predicate. There were three legal or contractual exposures here and one commercial one. Four dimensions, four different consequences, and no reason to expect them to move together.
The habit that transfers: when a constraint forces a mode change, enumerate what the mode change actually touches and check each one separately. Do not let one word carry four different exposures.
You already know this. It is why you do not have a single boolean called isValid.
What it cost
The scraper took an afternoon. The 403 cost me a week. The assumption underneath survived the whole project.
The 403 I had to build something to find. The definition was readable the entire time.
Provisions here are described as gates and thresholds, not as advice. If you are actually building this, the person you need is a lawyer, not a blog post. I wrote a longer version with the criminal gate and the case law in it over on Substack.
Top comments (2)
This is a great example of how the hardest part of a scraper usually isn't getting the first request to work, but understanding what happens when the environment changes. The distinction between technical feasibility, ToS restrictions, database rights, and GDPR processing is especially important. A 403 may look like a technical problem, but sometimes it's really an architecture and risk-management problem.
I've seen similar lessons while working on projects around CodeCan.net—the initial implementation is often the easy part; building something that remains sustainable under real-world constraints is where the engineering actually begins.
That gap gets skipped constantly in a first implementation, and it's easy to skip twice with AI-generated code, since the thing runs against today's target and looks finished, so nobody budgets for the environment changing under it. It's also getting harder to postpone. Anti-bot vendors iterate faster than they used to, so the window between it works and it needs an architecture change keeps shrinking, and the first time you hit that wall is usually the first time anyone thought about it at all.