Web scraping often looks cheap from the outside.
A crawler collects pages.
A dataset grows.
A model gets more examples.
The product team gets more material to train, test, classify, summarize, or evaluate.
But when scraped data contains personal data, the cost is not only infrastructure.
There is a privacy cost.
There is a documentation cost.
There is a cleanup cost.
There is a review cost.
That is the useful signal from the EDPB’s July 2026 draft guidelines on web scraping in the context of generative AI.
The European Data Protection Board says GDPR applies to web scraping when it includes personal data processing operations, including collection, storage, organisation, and retrieval. The Board also points teams toward questions around legal basis, purpose limitation, transparency, accuracy, data minimisation, and special-category data.
For software teams, the practical lesson is simple:
Scraped data is not automatically usable just because it is publicly visible.
Why this matters for AI teams
A lot of AI work depends on data that looks available.
Product reviews. Public profiles. Forum posts. Help pages. Job listings. Marketplace entries. Documentation pages. Social content. Website text. Support examples. Public code comments. Knowledge-base content.
Some of that data may be harmless to collect and use.
Some may include personal data.
Some may include sensitive personal data.
Some may include inaccurate, outdated, or copied information.
Some may be visible publicly but still not appropriate for the AI purpose the team has in mind.
That is where the economics change.
The team may think it is collecting “free training material.” But if that collection creates privacy review, filtering, deletion, transparency, accuracy, and governance work, the dataset is not free.
It has operating cost.
What changed
On 8 July 2026, the EDPB announced draft guidelines on web scraping in the context of generative AI and guidelines on anonymisation. The web-scraping guidance is open for public feedback until 30 October 2026.
The EDPB describes web scraping as large-scale automated data extraction that often happens without people being aware and may create risks for personal data protection.
It also clarifies that GDPR applies when scraping includes personal data processing operations.
That matters because AI teams cannot treat the act of scraping, storing, cleaning, and retrieving personal data as a neutral technical step.
It becomes data processing.
And data processing needs a lawful and documented basis.
The hidden cost in scraped data
The obvious cost is crawling, storage, and processing.
The less obvious cost sits around the dataset.
Teams may need to answer:
- Why was this data collected?
- What lawful basis supports the use?
- Which sources were used?
- When was it collected?
- Was personal data included?
- Was special-category data included?
- Was inaccurate data removed?
- Was the data minimised?
- Can data be deleted or excluded later?
- Can the team explain the purpose?
These questions take time.
If the team answers them late, the cost is higher.
That is why the best time to design the data process is before collection starts.
The 7-check scraping readiness review
1. Purpose before collection
Do not start with “collect everything and decide later.”
Start with the task.
Is the data needed for model training, evaluation, retrieval, classification, moderation, quality testing, benchmarking, or product analytics?
Each purpose may need a different level of review.
If the purpose is vague, the dataset will grow beyond the product need.
Check:
Can the team explain the specific AI task the scraped data supports?
2. Source reliability
The EDPB recommends scraping only from reliable sources, recording the timestamp, and validating data before using it for AI training.
That is an important practical point.
A model trained or evaluated on unreliable data can produce worse outputs, not better ones.
Teams should record:
- source URL,
- collection date,
- collection method,
- source type,
- update frequency,
- known quality issues,
- and whether the data is likely to include personal information.
Check:
Do we know where the data came from and when it was collected?
3. Personal-data filter
Public text can still include personal data.
That may include names, emails, usernames, work history, photos, location references, contact details, comments, complaints, account identifiers, or anything that can identify a person directly or indirectly.
The team should filter before the data enters training, evaluation, or indexing workflows.
This can include:
- removing obvious identifiers,
- detecting contact details,
- excluding profile pages,
- excluding private or semi-private contexts,
- reducing unnecessary fields,
- and retaining only what is needed for the stated purpose.
Check:
What personal data might enter the dataset, and can we reduce it before use?
4. Special-category risk
The EDPB reminds teams that processing special categories of personal data is generally prohibited unless both a GDPR Article 6 lawful basis and an Article 9(2) exception apply.
That matters because scraped data may accidentally include health, political views, religion, trade-union membership, biometric data, sexual orientation, or other sensitive categories.
Even if the team did not intend to collect it, the risk can still appear in large-scale scraping.
Check:
Could the dataset include sensitive personal data, even by accident?
5. Transparency plan
The EDPB says particular attention should be paid to transparency. It also notes that depending on how processing is designed, personal notice may not always be required if it is impossible or would require excessive effort.
This means teams still need a transparency position.
For product teams, that may include:
- privacy notice updates,
- dataset descriptions,
- source categories,
- opt-out or objection channels where applicable,
- internal documentation,
- and support-ready language.
Transparency is not only a legal page.
It is the ability to explain what the product is doing with data.
Check:
Can we explain the data source and purpose without hiding behind vague language?
6. Data minimisation
A large dataset is not automatically a better dataset.
If the task only needs short examples, do not store full pages.
If the model only needs public documentation, do not collect comments.
If evaluation only needs categories, do not keep identifiers.
Data minimisation means collecting and retaining only what is necessary for the purpose.
This can reduce privacy risk, storage cost, review burden, and future cleanup.
Check:
Are we keeping data because it is needed, or because it was easy to collect?
7. Removal and exclusion path
AI datasets need a way to change.
A source may become unreliable. A person may object. A site may update its policy. The team may discover sensitive data. The product purpose may change.
The team should be able to remove or exclude data from future use.
That does not mean every system can perfectly forget every past influence in a model. But teams should still design practical controls:
- dataset versioning,
- source exclusion lists,
- deletion workflows,
- retraining or re-indexing rules,
- audit logs,
- and clear ownership for data removal decisions.
Check:
If this source becomes unsuitable, can we remove it from the active data workflow?
A simple decision model
Before scraping data for an AI workflow, place it into one of four buckets.
Safe to use with basic records
Data that is non-personal, reliable, and directly tied to the AI task.
Example: public technical documentation used for retrieval.
Use only after filtering
Data that may contain personal details but can be reduced safely.
Example: forum text after removing identifiers and irrelevant fields.
Use only with stronger review
Data that may contain sensitive context, user-generated content, profiles, or mixed personal data.
Example: public social posts, support discussions, community profiles.
Do not use for this purpose
Data that is too sensitive, unreliable, unrelated, or impossible to govern.
Example: scraped personal profiles for unrelated AI training.
The goal is not to block useful AI work.
The goal is to stop treating every public page as equally usable.
Founder takeaway
Scraping can make AI work feel faster at the beginning.
It can also create hidden work later.
The important question is not only:
Can we collect this data?
It is:
Can we justify, minimise, validate, explain, and remove it if needed?
That is where the cost of AI data becomes visible.
A small, governed dataset can often be more useful than a large dataset nobody can explain.
Top comments (0)