A site crawler starts with pages you choose. A backlink index starts with pages discovered across other sites. A hyperlink graph stores connections between pages, hosts, or domains. These tools share crawling technology, but they answer different questions.
If you are building an SEO workflow, choosing the wrong input can produce a perfectly valid computation that answers the wrong question.
Disclosure: I maintain crawlgraph, which queries Common Crawl hyperlink graphs. This article adapts our backlink crawling tool guide. The examples below are illustrative, not measurements of a customer's site.
Choose the evidence before the tool
| Your question | Evidence you need | Useful starting point |
|---|---|---|
| Which internal pages link to a broken URL? | Links extracted from your site's pages | A site crawler |
| Which hosts were observed linking to a competitor? | Incoming edges in a dated index or graph | A backlink index or hyperlink graph |
| Does this particular article still link to my site? | The current source page and its actual link destination | A page inspection |
| Did a publisher remove a link? | Comparable earlier and current page evidence | A change ledger plus verification |
A crawl of example.com does not enumerate incoming links from the rest of the web. It can discover outgoing links on the pages it visits. Turning that into a competitor-backlink index requires discovering other sites, fetching their pages, extracting links, and storing relationships you can query in reverse.
Common Crawl publishes hyperlink graphs that make large-scale relationship research possible without operating that whole crawl pipeline yourself. They remain dated observations with coverage limits. They are not a live check of every referring page.
Give every result a scope
Before comparing two exports, write down:
- Target: the exact hostname or domain you queried.
- Unit: source pages, hosts, or registrable domains.
- Observation: the index or graph release and any known crawl date.
- Filters: exclusions, link-type rules, and grouping choices.
- Completeness: whether the returned rows were capped or paginated.
For example, docs.example.com and www.example.com are different hosts. A domain-level export may group them together. Comparing its row count with a host-level export does not measure growth or loss.
Likewise, a result cap is part of the dataset's meaning. A row missing from the first 1,000 results is not necessarily absent from the index. Preserve the returned total and cap indicators alongside the file, rather than throwing that context away after parsing the rows.
A set difference is a research queue
Once the scopes match, a simple difference can identify candidates worth inspecting:
# Synthetic host-level observations with matching scope.
# These values are examples, not live crawl results.
earlier = {"docs.example.org", "journal.example.net", "community.example.com"}
later = {"journal.example.net", "community.example.com", "news.example.org"}
missing_observations = sorted(earlier - later)
new_observations = sorted(later - earlier)
assert missing_observations == ["docs.example.org"]
assert new_observations == ["news.example.org"]
The variable names matter. missing_observations does not establish removed_links. The earlier graph may have included a page that the later crawl did not visit. A source may have moved, blocked access, or changed its destination. A host-level edge does not identify the exact article you need to inspect.
Common Crawl's FAQ describes the sampling boundary. Keep it in your output contract so the next person using your CSV does not accidentally turn a missing observation into a claim about a publisher's behavior.
Verify before taking action
For an important candidate, find the source URL and check the final page after redirects. Inspect the actual hyperlink destination, not just a text occurrence of your domain.
A page can mention example.com in plain text without linking to it. A successful HTTP response can contain a login screen or replacement homepage. An unsuccessful request tells you that verification failed; it does not prove that an editor deleted the link.
Keep a small ledger:
| Source URL | Checked at | Final URL | Link evidence | Status | Next action |
|---|---|---|---|---|---|
| saved article URL | UTC timestamp | after redirects | capture or inspected href | present / absent on checked page / unresolved | retain / investigate / recheck |
Only label a change as a confirmed removal when you have comparable earlier evidence and a current inspection of the relevant page. Even then, the result establishes a page change, not what Google currently credits in its ranking systems.
For competitor research, the same boundary applies: a host observed linking to a competitor is a prospect to qualify, not a promise that it will link to you. Read the page, assess relevance, and check editorial requirements before contacting anyone.
The useful pipeline is discover candidates → preserve scope → inspect important pages → record a supported conclusion. That pipeline works with a commercial index, an open graph, or your own crawler, provided the evidence matches the question.
Top comments (0)