Why a scraping API rejects private and metadata URLs

A scraper accepts a URL and makes a server-side network request. That makes URL validation a security boundary, not a convenience check. If an API worker can fetch 127.0.0.1, a private network address, or a cloud metadata service, an untrusted caller may use the worker to inspect infrastructure that was never meant to be public.

MESSORA applies an SSRF guard to /scrape, /crawl, and discovered crawl links. URLs in RFC1918 ranges, loopback, link-local ranges, and cloud metadata targets are rejected. The request fails with 422; it does not create a job or return a partial page.

The API is the final authority

A client-side check is useful for fast feedback, but it cannot replace the server guard. DNS can resolve differently from one network to another, redirects can point to a different host, and a public hostname may resolve to an internal address from the worker's network.

import ipaddress
import requests
from urllib.parse import urlparse


def obvious_private_target(url: str) -> bool:
    parsed = urlparse(url)
    if parsed.scheme not in {"http", "https"} or not parsed.hostname:
        return True

    try:
        address = ipaddress.ip_address(parsed.hostname)
    except ValueError:
        # A hostname still needs server-side validation and DNS-aware policy.
        return False

    return any(
        (
            address.is_private,
            address.is_loopback,
            address.is_link_local,
            address.is_reserved,
        )
    )


for candidate in [
    "http://127.0.0.1:8080/health",
    "http://192.168.1.20/admin",
    "https://public.example.com/docs",
]:
    print(candidate, obvious_private_target(candidate))

This helper deliberately does not claim that a hostname is safe when it is not a literal IP. It only avoids sending obvious bad input. Submit accepted URLs to the API and handle its 422 response as the authoritative decision.

Crawl validation happens twice

A crawl starts with one seed, then follows links discovered in the returned pages. Validating only the seed would leave a gap: a public page could contain a link to an internal address. The API applies the guard to the seed and each discovered link before that link enters the breadth-first frontier.

This is why a crawl can complete with fewer pages than max_pages. A link may be outside the allowed URL pattern, point to another host when follow_subdomains is false, or be rejected by the SSRF policy. The job's results and stopped_reason tell you whether the frontier ended normally; the client should not infer a security failure from page count alone.

Do not turn failures into retries

A 422 for a private target is deterministic. Retrying the same URL only repeats validation and adds noise to your logs. Separate input rejection from transient fetch outcomes:

response = requests.post(
    f"{API}/scrape",
    headers={"X-API-Key": os.environ["MESSORA_API_KEY"]},
    json={"url": url, "formats": ["markdown"]},
    timeout=90,
)

if response.status_code == 422:
    detail = response.json().get("detail", "invalid request")
    raise ValueError(f"URL rejected by API policy: {detail}")

response.raise_for_status()

Do not log full URLs if they may contain credentials or sensitive query strings. Normalize and redact those values before putting the rejection into an operator-facing log. The API key also belongs only in the request header and environment, never in the URL.

Safe URL intake is a system property

If users can submit arbitrary URLs, validate the scheme, enforce a maximum length, and keep redirects under the same network policy. If a crawler accepts a user-supplied regex, bound its length and reject malformed patterns before enqueueing. These controls reduce accidental work, but the worker-side SSRF guard remains necessary because only the worker sees the final network destination.

A 422 is the correct result here: the request was understood, but its target violates the API's input policy. Treating it as a retriable outage obscures the real boundary. Treating it as a successful empty scrape risks silently dropping a security decision. Keep the distinction explicit in the client and in your metrics.