Keeping a crawl inside the documentation tree

A crawl that starts at /docs/ can reach a login page, a changelog, a language switcher, and every marketing link in the site header. The crawler is doing what the HTML tells it to do. The boundary has to be part of the request.

MESSORA's /crawl endpoint exposes three controls for that boundary: url_regex filters discovered absolute URLs, max_pages caps the total work, and follow_subdomains decides whether links on subdomains are eligible. The endpoint runs a breadth-first crawl and returns a job_id immediately.

Start with a small page budget

import os
import requests

API = "https://api.messora.dev"
HEADERS = {"X-API-Key": os.environ["MESSORA_API_KEY"]}

response = requests.post(
    f"{API}/crawl",
    headers=HEADERS,
    json={
        "url": "https://docs.example.com/",
        "max_pages": 10,
        "max_depth": 2,
        "url_regex": r"^https://docs\.example\.com/(guide|reference)(/.*)?$",
        "follow_subdomains": False,
        "only_main_content": True,
        "include_links": True,
    },
    timeout=30,
)
response.raise_for_status()       # 202 Accepted
job_id = response.json()["job_id"]
print(job_id)

max_pages is required and accepts values from 1 to 50. A value of 10 is a useful first run: it exposes whether the regex matches the site's real link shape without committing the account to a full tree. Increase the limit only after inspecting the result set and credit use.

Match the final URL shape

url_regex is evaluated against absolute URLs discovered by the crawler. Anchor the expression when the crawl must stay under one host. The expression above accepts /guide/ and /reference/, but not /blog/, /login, or a different hostname.

Python's raw string notation keeps the backslashes readable. The dot in docs.example.com is escaped because an unescaped dot matches any character. Test the expression with representative URLs before sending it:

import re

pattern = re.compile(r"^https://docs\.example\.com/(guide|reference)(/.*)?$")

candidates = [
    "https://docs.example.com/guide/install",
    "https://docs.example.com/reference/api",
    "https://docs.example.com/blog/release",
]

for url in candidates:
    print(pattern.fullmatch(url) is not None, url)

The seed itself must still be a valid URL and pass the API's network safety policy. A regex is a discovery filter, not a permission to fetch an otherwise forbidden destination.

Choose subdomain behavior explicitly

follow_subdomains defaults to false. Keep it disabled when the documentation is isolated on one host. Enable it only when the site intentionally spreads its public documentation across subdomains, and widen the regex to describe those hosts. Otherwise a link to status.example.com or community.example.com can become part of the crawl without adding useful source material.

max_depth is optional. It limits traversal depth, while max_pages limits total pages and therefore provides the hard cost ceiling. Use both: depth controls shape, page count controls spend. The crawl returns Markdown in the current public contract; use /scrape or /batch when you need other output formats.

Inspect why it stopped

After the job reaches SUCCESS, read pages_crawled, stopped_reason, and each result's scrape_status. Fewer pages than max_pages is not automatically an error. completed can mean the frontier ran out of matching links. A non-success item can be a timeout, an extraction failure, or an anti-bot block.

job = wait_for_job(job_id)

print("pages:", job["pages_crawled"])
print("stop:", job["stopped_reason"])

for item in job["results"]:
    if item["scrape_status"] == "success":
        print("kept:", item["url"])
    else:
        print("not extracted:", item["url"], item["scrape_status"])

Do not solve an unexpectedly broad crawl by lowering max_pages alone. That hides the symptom and produces a partial corpus. Tighten the regex, verify subdomain behavior, then raise the budget only when the boundary is intentional. A bounded crawl is easier to review, cheaper to rerun, and safer to put in a scheduled ingestion pipeline.