A RAG pipeline indexes whatever pages you give it. The Model Context Protocol (MCP) site's sitemap listed pages from every specification revision, so a pipeline would index them all. An agent could then answer from an older revision.
A reliable fix is to list the pages and keep only the revision you need. You can parse the sitemap and homepage yourself, but sites with anti-bot protection can return a 403 or a challenge page instead.
This guide uses Decodo's /map API to list a website's pages, filter them, and check the list for changes.
TL;DR
- One POST request to Decodo's /map returns the URLs from a site's sitemap and the links on the page you submit, merged and deduplicated. On modelcontextprotocol.io, one call returned 361 URLs from the sitemap and the homepage.
- Path filters keep only the sections you need. One regex pattern reduced those 361 URLs to the 57 URLs of the 2026-07-28 specification revision.
- Fetch the pages as Markdown with the same API key. /scrape fetches each new page, and /crawl fetches a whole section with the same filters.
- Run /map on a schedule to find new and removed pages. Each run compares the new list with the saved one, so you ingest only the new pages and delete the removed ones.
What you'll need
- Python 3.11 or later
- The Requests library
- A Decodo account with the Web Data API free plan activated
What /map does
/map is the URL discovery endpoint of Decodo's Web Data API, which also includes /scrape, /search, and /crawl. It returns URLs from the site's sitemap, the links on the single page you submit, or both, already filtered by any path patterns you send.
/map has built-in anti-bot handling and runs on Decodo's proxy network, so it can map sites that block simple crawlers. The Web Data API page lists the current proxy pool size and success rate. Use /map to collect publicly available data, and follow each site's terms of service.
The price of a /map call is the same for any number of URLs it returns, and you pay only for successful requests. The billing page lists current rates, and the pricing page shows what the free plan includes.
Step 1: Set up
Copy your API key from the Web Data API playground in the Decodo dashboard. The Get code button on the same page shows the /map request for Decodo's SDKs, CLI, and MCP server.
Then create a virtual environment, install Requests, and set the key as an environment variable, which the Step 2 code reads:
python3 -m venv .venv
source .venv/bin/activate
pip install requests
export DECODO_AUTH_TOKEN="your-api-key"
Step 2: List a site's pages
Save this code as list_pages.py. Its list_pages() function calls /map once and returns the URLs and 2 metadata fields. The call_api() function sends each POST request to the API. After a timeout, a connection error, or a 429 or 5xx status code, it retries with exponential backoff.
import json
import os
import time
from urllib.parse import urlparse
import requests
API_URL = "https://data.decodo.com/v1/"
def call_api(endpoint, payload, retries=3):
headers = {"Authorization": f"Bearer {os.environ['DECODO_AUTH_TOKEN']}"}
for attempt in range(retries):
try:
resp = requests.post(API_URL + endpoint, json=payload,
headers=headers, timeout=180)
if resp.status_code == 200:
return resp.json()["results"][0]
if resp.status_code != 429 and resp.status_code < 500:
raise RuntimeError(f"{resp.status_code}: {resp.text[:200]}")
except requests.RequestException:
pass
if attempt < retries - 1:
time.sleep(2 ** attempt)
raise RuntimeError(f"No successful response after {retries} attempts")
def list_pages(url, sitemap="include", select_paths=(), exclude_paths=(),
domain_filter="domain"):
result = call_api("map", {"url": url, "sitemap": sitemap,
"select_paths": list(select_paths),
"exclude_paths": list(exclude_paths),
"domain_filter": domain_filter})
urls = [link["url"] for link in result["content"]["links"]]
return urls, result["sitemap_status"], result["map_id"]
def save(urls, url):
path = urlparse(url).netloc + ".json"
with open(path, "w") as f:
json.dump(sorted(urls), f, indent=2)
return path
if __name__ == "__main__":
site = "https://modelcontextprotocol.io"
urls, sitemap_status, map_id = list_pages(site)
print(f"{len(urls)} URLs, sitemap_status={sitemap_status}, map_id={map_id}")
print("saved to", save(urls, site))
Running python list_pages.py printed:
361 URLs, sitemap_status=include, map_id=a30421ba-49de-459d-be4e-e68346c695f4
saved to modelcontextprotocol.io.json
On a larger site, the same function returned 5,108 URLs for docs.stripe.com in one call.
sitemap_status shows which sources /map actually used. include means both, only means the sitemap, and skip means the links on the page. On a site with no sitemap, /map returns skip even when you send include. Use map_id to find the call in the dashboard's usage statistics or to ask Decodo's support about it.
domain_filter limits the list to the submitted host by default (domain). Pass subdomain to add its subdomains (e.g., blog.example.com), or all to add external sites.
Step 3: Choose the URL source
On modelcontextprotocol.io, each sitemap value returned a different list:
| sitemap | URLs | What the list contained |
|---|---|---|
include (default) |
361 | Sitemap and homepage links, merged and deduplicated |
only |
359 | Every sitemap URL, including pages from 4 older specification revisions and a draft |
skip |
19 | Only the homepage's links, with no older revisions |
Use only for the pages the site owner listed in the sitemap. Use skip for the links on the submitted page only. Use include for the largest list, then filter it.
The MCP site's own version menu showed why the sitemap was so large.
If you build your list from the site's llms.txt file, a page index for AI tools, apply the same filter. That file listed the same 359 pages as the sitemap, including older revisions.
If a site loads its navigation with JavaScript, add "JS_rendering": True to the payload in list_pages(). /map then renders the page before it collects links. The billing page lists the rate for JavaScript rendering.
Step 4: Filter and load the pages you need
The MCP site puts the revision in the path, so one pattern selects your agent's revision. Save this code as load_revision.py in the same folder as list_pages.py. It filters the list, fetches each page with one /scrape request, and saves the loaded URLs as the baseline for Step 5:
from list_pages import call_api, list_pages, save
SITE = "https://modelcontextprotocol.io"
PATTERN = "^/(specification|docs)/2026-07-28(/|$)"
def fetch_markdown(url):
result = call_api("scrape", {"url": url, "markdown": True})
return result["content"] if result["status_code"] == 200 else None
if __name__ == "__main__":
urls, _, _ = list_pages(SITE, select_paths=[PATTERN])
print(len(urls), "URLs in the 2026-07-28 revision")
loaded = []
for url in urls:
markdown = fetch_markdown(url)
if markdown is not None:
# Chunk and embed markdown, then upsert it into your vector store by url
loaded.append(url)
save(loaded, SITE)
print(len(loaded), "of", len(urls), "pages loaded")
Running python load_revision.py printed:
57 URLs in the 2026-07-28 revision
57 of 57 pages loaded
That one pattern reduced the list from 361 URLs to 57. It selects the versioned pages only. To also index unversioned sections, add them to the end of PATTERN, such as |^/extensions(/|$). To exclude a section instead, pass its pattern to exclude_paths, such as ^/(specification|docs)/draft(/|$) for the draft. For your own site, change SITE and PATTERN at the top of this file.
These 4 rules help you write patterns that match:
- Match the path. Write ^/docs/, not https://example.com/docs/.
- Anchor the end with (/|$), not /. A pattern ending in / doesn't match the revision's landing page, /specification/2026-07-28, which has no trailing slash.
- Write regex, not glob. In regex, /docs/2026* means /docs/202 followed by any number of 6s, including none, so it also matches the docs of the 2024 and 2025 revisions. /map validates each pattern and returns a 400 response that names any invalid pattern, such as the glob *.txt.
- Use anchors, groups, and alternation. /map patterns support all 3, plus the (?i) flag for case-insensitive matching.
To test a pattern in the dashboard, open Advanced settings on Decodo's Playground's Map tab. Set Sitemap to Include and paste the pattern into Select paths.
Click Map to run it and see the matching URLs.
The fetch_markdown() function sends one URL to /scrape with "markdown": True and returns that page as Markdown, which uses fewer tokens than the page's HTML. It returns None when the page responds with an error, and the script leaves that URL out of the baseline, so Step 5 tries it again.
If your pipeline uses LangChain or LlamaIndex, Decodo's DecodoLoader and DecodoWebReader load the URLs as Markdown Document objects with the same API key instead. To fetch a large section in one job, send the same select_paths to /crawl.
Step 5: Find new and removed pages on a schedule
The script uses the new list only if it passes 2 checks:
- Every URL matches PATTERN. This catches a run where your code did not apply the filter, for example, after an edit to list_pages().
- The list has at least half as many URLs as the saved one. This catches a sudden drop, such as after a site redesign.
import json
import re
from urllib.parse import urlparse
from list_pages import list_pages, save
from load_revision import PATTERN, SITE, fetch_markdown
with open(urlparse(SITE).netloc + ".json") as f:
previous = set(json.load(f))
urls, _, _ = list_pages(SITE, select_paths=[PATTERN])
current = set(urls)
if not all(re.search(PATTERN, urlparse(u).path) for u in current):
raise SystemExit("the new list has a URL outside PATTERN, keeping the baseline")
if len(current) < len(previous) / 2:
raise SystemExit(f"{len(current)} URLs vs {len(previous)} saved, keeping the baseline")
print("added: ", sorted(current - previous))
print("removed:", sorted(previous - current))
skipped = set()
for url in sorted(current - previous):
markdown = fetch_markdown(url)
if markdown is None:
print(f"skipped {url}: the page did not return 200")
skipped.add(url)
continue
print(f"fetched {url}: {len(markdown)} characters")
# Chunk and embed markdown, then upsert it into your vector store by url
# Delete the removed URLs from your vector store here
save(current - skipped, SITE)
# MCP only: list every revision date to detect a new one
all_urls, _, _ = list_pages(SITE)
revisions = {u.split("/")[4] for u in all_urls if "/specification/" in u}
print("revisions:", sorted(revisions))
latest = max(r for r in revisions if r[:1].isdigit())
if not re.search(PATTERN, f"/specification/{latest}"):
print("new revision:", latest)
Save this code as diff_runs.py. It loads the saved list (the baseline), maps the site again with the same filter, prints what changed, and fetches each added page as Markdown.
To test it, delete 2 URLs from modelcontextprotocol.io.json and run python diff_runs.py. It printed:
added: ['https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture', 'https://modelcontextprotocol.io/specification/2026-07-28']
removed: []
fetched https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture: 34494 characters
fetched https://modelcontextprotocol.io/specification/2026-07-28: 8189 characters
revisions: ['2024-11-05', '2025-03-26', '2025-06-18', '2025-11-25', '2026-07-28', 'draft']
The script skips a page that answers with an error and tries it again on the next run. If your ingestion code raises an exception, the script stops before save(), so the next run shows the same changes again.
/map normalizes each URL before it removes duplicates, so /page/, /page#section, and /page count as one URL. Use the URLs from the /map list as the keys in your vector store, so your deletes match.
When MCP publishes a new revision, the script prints new revision: with its date, so add an alert on that line. To move your agent to that revision, update PATTERN only. The next run lists the new revision's pages as added and the old ones as removed. The revision check adds one /map call to each run.
To find edits to existing pages, fetch the section again with /crawl, hash each page's Markdown, and re-ingest the pages whose hash changed.
Schedule diff_runs.py with cron or n8n. Run it from the folder that holds the baseline file, on a machine that keeps that file between runs. Set the API key as an environment variable for the job.
Use the scripts on your own site. Run load_revision.py once with your SITE and PATTERN to load your pages and save the first baseline. In diff_runs.py, delete the MCP-only revision check, or change "/specification/" and the index in split("/")[4] to match where your site keeps its versions.
Final thoughts
The 2 main decisions are the URL source and the filters, and list_pages() takes both as arguments. Coding agents can make the same /map, /scrape, and /crawl calls through Decodo's agent skills. The dashboard's Copy agent setup button copies a setup prompt for your coding agent.
Activate the free plan, which includes monthly credit and requires no credit card, and map the docs site your agent uses.



Top comments (0)