A referring domain that disappears from a new crawl snapshot is an observation to investigate. It is not proof that somebody removed a live backlink. Before sending a lost-link report, check the release identities, the comparison unit and whether either export was truncated.
I maintain crawlgraph, which uses imported Common Crawl graphs for backlink research. Here is a small offline workflow for comparing saved responses without turning missing rows into false certainty.
Keep three dates separate
A source crawl, an upstream graph publication and a tool's queryable release are different things.
| State | What it establishes |
|---|---|
| A page was captured in a crawl | That capture exists in that crawl archive |
| A graph was published upstream | That graph artifact is available from its publisher |
| A release is queryable in your tool | The tool can answer requests against its imported release |
A newer graph on Common Crawl's website does not automatically become a new dataset inside every product that uses Common Crawl. Check the upstream graph list and the tool's own release availability separately.
For a comparison, keep the original responses with their release_id. Record which release is older and which is newer. Do not infer chronology by sorting arbitrary identifier strings.
Refuse a partial comparison
Suppose your older response reports 2,000 referring domains but returns only 1,000 rows. The absent half can contain a hostname that appears in your newer export. Calling that hostname newly discovered would overstate what the two files establish.
The opposite problem also matters: a hostname from the older file can fall below a newer export's row cap while still existing in the newer observed set.
For the strict workflow below, both responses must satisfy:
returned == len(results) == total_linking_domains
The target domain must match, and release identifiers must differ. This checks the response's reported completeness. It does not independently verify the files' provenance, collection method or whole-web coverage.
Run a synthetic fixture first
The standalone Python Gist contains snapshot_diff.py and two synthetic JSON fixtures. Download its four files into one directory and run:
python3 snapshot_diff.py --older older.json.example --newer newer.json.example
The fixtures use reserved .example names and invented release identifiers. They are not measurements of customer websites.
The result contains two kinds of row:
-
observed_added: present only in the newer saved response. -
observed_absent: present only in the older saved response.
The shared hostname is omitted. Missing authority values remain empty rather than becoming zero. The script copies supplied authority values; it does not calculate a new score or convert another vendor's metric.
For real files, the command is:
python3 snapshot_diff.py --older earlier.json --newer later.json --output observations.csv
--output creates a new file and refuses to overwrite an existing one, including a symlink. Both inputs are validated before writing. The comparison runs locally: no API key, network request or lookup quota is needed to compare files you already hold.
Name the result after the evidence
A useful report sentence is:
This referring hostname was observed in release A and absent from our complete response for release B. Its live-link status has not been verified.
That wording leaves room for several explanations: changed crawl coverage, a different graph observation, a site change or an actual removed link.
The graph result is a referring hostname, not a source-page URL. To confirm a removed link, you still need to identify and inspect the relevant source page. If you cannot locate it, preserve the observation and the uncertainty rather than marking a deletion as confirmed.
A new hostname also needs context. A directory entry, an editorial recommendation and a technical resource link can suggest different actions even when they all enter a referring-domain set.
Choose the tool for the reporting job
Snapshot comparisons can support periodic backlink research. They cannot promise that a link placed today will be found tomorrow, or that every live link is represented.
If your workflow is researching referring domains and competitor overlap, try your own domain free and compare the current plans. Crawlgraph's free web preview shows five referring domains; paid core access expands results and supports capped CSV/JSON exports. If your deliverable needs continuously refreshed source-page records, evaluate that requirement before buying.
The Common Crawl release guide explains crawl, graph and import states. The API reference documents fields, release availability, quotas and the hosted snapshot-change endpoint if you prefer it to local files.
Top comments (0)