How I find all pages on a website (and the ones every tool misses)

TL;DR: Check /sitemap.xml (or the Sitemap: line in /robots.txt), run a site:example.com search, then crawl the site. Add the Wayback Machine for old URLs and Search Console if it's yours. None of these see pages behind a login or routes that only exist in your app's JavaScript. For those, crawl from a browser tab where you're already signed in.

I get asked "how many pages does this site actually have?" every time I inherit a project. The honest answer is that no single source knows. Here's the order I check things in, with the commands I use.

1. The sitemap

curl -s https://example.com/robots.txt | grep -i '^sitemap:'
curl -s https://example.com/sitemap.xml | grep -oE '<loc>[^<]+' | sed 's/<loc>//'

If the first <loc> entries point at more .xml files, you're looking at a sitemap index. Run the second command on each child. One sitemap file maxes out at 50,000 URLs or 50 MB uncompressed (sitemaps.org), so big sites always split.

What it misses: anything the owner didn't list. Apps often only list their marketing pages.

2. Google site: search

site:example.com, or site:example.com/docs for one section. Good for a quick look, useless for a count. Google's own docs say it "doesn't necessarily return all the URLs that are indexed."

3. Search Console (your own site only)

Indexing → Pages. The Page indexing report splits indexed from not-indexed URLs and gives the reason for each group. Click a reason, then Export. Example lists cap at 1,000 rows, but the not-indexed groups are where I usually find forgotten pages.

4. Crawl it

Screaming Frog is free up to 500 URLs per crawl. If you'd rather stay in the terminal:

wget --spider --recursive --level=inf --no-verbose \
  --output-file=crawl.log https://example.com/

grep -oE 'https?://[^ ]+' crawl.log | sort -u > urls.txt

wget respects robots.txt in recursive mode and doesn't run JavaScript. Keep that second part in mind.

5. The Wayback Machine for URLs that used to exist

https://web.archive.org/cdx/search/cdx?url=example.com/*&fl=original&collapse=urlkey&filter=statuscode:200

/* is a prefix match, fl=original returns just URLs, and collapse=urlkey drops duplicate captures. Expect noise. Check what still loads. Docs: CDX server API.

About "hidden pages"

Usually these are orphan pages: on the server, linked from nowhere. On your own site, diff your CMS or access logs against the crawl output. Wayback and site: sometimes surface pages that were linked once. Wordlist-guessing URLs (directory brute forcing) is for your own site or a scoped, authorised test. On someone else's site it's a good way to get blocked or worse.

Where all of this falls over: SPAs and logged-in apps

My last two projects were a React dashboard and an admin app behind SSO. Every method above returned the login page and a handful of marketing URLs.

• In a single-page app the server sends an empty shell. Routes live in JavaScript, and some are only reachable through onClick handlers instead of <a href>.
• Behind a login, a crawler without your session gets redirected on every URL. Pasting cookies into a crawler works until the session expires.

For a quick look at one rendered page, I paste this into the DevTools Console. It reads the live DOM, so JS-rendered links count:

[...new Set([...document.querySelectorAll('a[href]')].map(a => a.href))].join('\n')

For the whole app I use Netraptor, a Chrome extension I work on. Its crawler runs in my own logged-in tab: it reads the <a href> links on the start page after render, keeps same-origin ones, skips anything that looks like logout or delete, and loads each in turn while recording the API calls every page makes. It goes one level from the start page (start from a section index or use a path prefix to go deeper), and it doesn't follow #/hash routes. Free crawls 25 pages per run. I only point it at apps I build or am allowed to test.

Full guide with the Search Console and crawler details: How to find all pages on a website.