Is Web Scraping Legal? The Question Our Code Could Not Answer

The crawler worked.

It fetched public pages, respected a small concurrency limit, extracted the required fields, and wrote clean records to storage.

Then someone asked the question the integration test could not answer:

“Are we allowed to do this?”

There is no universal boolean called is_scraping_legal. The answer depends on the source, authorization, access method, data, jurisdiction, contract, and what happens after collection.

Publicly visible does not automatically mean unrestricted, and technically accessible does not automatically mean authorized.

This is the review framework I wish had existed before the first line of crawler code.

Start with the access path

Ask how the data is reached:

official API or export
public page without authentication
page available only after login
paid or subscription content
private endpoint
content behind an access control

These paths do not carry the same risk. Authentication, paywalls, CAPTCHAs, explicit denials, and technical restrictions are not ordinary parser problems.

If access requires defeating a control, stop and obtain legal and security review.

Then classify the data

A list of public product prices is different from personal profiles, health information, financial data, employment records, precise locations, or account activity.

For every field, document:

• why it is needed;
• the legal or contractual basis for collecting it;
• who can access it;
• how long it will be retained;
• whether it will be combined with other datasets;
• and how deletion or correction requests are handled.

Data minimization is an engineering requirement. If a field is not required, do not collect it “just in case.”

Terms and permission belong in the design

Terms of service, licenses, robots directives, API policies, customer contracts, and written permissions may all affect the approved route.

Robots rules are not a complete statement of law, and compliance with them does not answer every legal question. But ignoring an explicit site preference is also a poor default.

The useful artifact is a written collection specification:

source: supplier-catalog
authorized_paths:
  - /public/products/
prohibited_paths:
  - /account/
data_fields:
  - sku
  - public_price
max_concurrency: 2
retention_days: 90
owner: market-data-team
escalation: [e-mail]

Now the crawler has boundaries it can enforce.

Downstream use can change the risk

Collection is only the first step.

The same dataset may be used for internal research, a public search product, automated pricing, direct outreach, model training, or resale. Copyright, privacy, consumer-protection, database, and contract issues can depend on that use.

Do not let “the page was public” become the entire review.

Build compliance into the job state

I want the scheduler to know whether a source is approved:

from dataclasses import dataclass
from datetime import date


@dataclass(frozen=True)
class CollectionPolicy:
    approved: bool
    expires_on: date
    allowed_prefixes: tuple[str, ...]
    max_concurrency: int


def may_schedule(url: str, policy: CollectionPolicy) -> bool:
    return (
        policy.approved
        and date.today() <= policy.expires_on
        and any(url.startswith(prefix) for prefix in policy.allowed_prefixes)
    )

This is not a legal opinion encoded in Python. It is enforcement of a decision made by the responsible people.

What the crawler should do when conditions change

Stop or quarantine the job when it encounters:

• authentication that was not part of the approved route;
• a CAPTCHA or explicit denial;
• a new category of personal data;
• a redirect outside the approved domain;
• changed terms or API policy;
• or output that no longer matches the reviewed schema.

Automatic retries are appropriate for some transient failures. They are not appropriate for unresolved permission changes.

Where managed infrastructure fits

Tools such as Nstdata Crawl can handle retrieval, rendering, bounded discovery, and page outputs for approved public-web workflows. Proxy products can support legitimate localization and testing.

None of those capabilities decide whether a collection project is lawful. Infrastructure executes policy; it does not create it.

The pre-launch checklist

Before production, I would require clear answers to:

1. What exact source and paths are in scope?
2. What permission or legal basis supports access and use?
3. Are authentication or access controls involved?
4. Does the dataset include personal or regulated information?
5. What fields are necessary?
6. How will data be secured, retained, corrected, and deleted?
7. What rate and crawl boundaries apply?
8. Who owns policy changes and incident response?
9. What downstream uses are permitted?
10. What event automatically stops collection?

If several answers are “we will decide later,” the crawler is not ready.

Final takeaway

Web scraping is not categorically legal or illegal. It is a collection method operating inside a specific factual and legal context.

Engineers should not make the legal determination alone, but they should make the approved decision enforceable: narrow scope, explicit paths, minimal fields, bounded traffic, audit logs, expiration dates, and stop conditions.

This article is general technical information, not legal advice. Consult qualified counsel for the relevant jurisdictions and facts.

What policy decision does your current scraper assume without recording anywhere?