Save backlink responses once, then build a local competitor-gap report

Changing an authority threshold or sorting a backlink report should not require downloading the same data again. Save the original API responses, keep their release metadata, and run the comparison locally. The useful part is making sure the files describe comparable observations before treating an absent hostname as a gap.

I maintain crawlgraph. This tutorial uses its JSON response format. The Python CSV comparator also works with compatible referring-host exports from another source, and the included synthetic example needs no account, API key or network access.

Start with a small, reproducible fixture

The runnable Gist contains two standard-library Python scripts and three CSV fixtures. Download its six files into one directory.

The inputs use reserved .example names:

File	Referring hosts
own.csv	shared.example, ours.example
rival-a.csv	shared.example, roundup.example, guide.example
rival-b.csv	shared.example, roundup.example, review.example

Run:

python3 gap_analysis.py --your own.csv --competitor rival-a.csv --competitor rival-b.csv

Expected output:

source_domain,competitor_overlap,competitors
roundup.example,2,rival-a;rival-b
guide.example,1,rival-a
review.example,1,rival-b

The shared host disappears because it is already in your own set. roundup.example appears in both competitor sets, so its overlap is two. Repeating it ten times in one CSV would still count as one competitor.

These are test fixtures, not customer results or actual outreach prospects.

Save the response before filtering it

For a real run, keep one response per target. With your key already in an environment variable, a single request looks like this:

set -euo pipefail
set -o noclobber
: "${CG_KEY:?set CG_KEY in your shell first}"

curl --fail --silent --show-error --max-time 30 \
  -X POST https://crawlgraph.com/api/v1/backlinks \
  -H "Authorization: Bearer $CG_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"example.com","limit":1000,"sort":"authority"}' \
  > own.json

The example uses example.com; replace it with the public target you intend to research. Save separate responses for two distinct competitors as rival-a.json and rival-b.json. Three successful requests use three backlink lookups. The current free allocation is 15 backlink calls per UTC month; the hosted gap-analysis endpoint requires paid core access. The local set comparison here is ordinary Python operating on files you already have.

The command stops on an HTTP error and refuses to overwrite an existing output file. Keep the key out of the files you publish. No live requests are needed to run the synthetic fixture.

The API reference explains authentication, response fields and quotas.

Reject mismatched or partial inputs

Before making CSVs, run:

python3 export_hosts.py own.json rival-a.json rival-b.json --output-dir exports

The converter checks four things:

1. Every response has the same release_id.
2. The target domains and input filenames are distinct.
3. returned matches the number of result rows and total_linking_domains.
4. Every row contains a valid hostname, with no duplicate linking_domain rows.

It validates the inputs before writing output. A reported partial response or a mixed release stops the conversion. Existing output files also stop it.

Why reject partial responses? A competitor sample can supply a positive lead: “this host appears here.” But an incomplete own-site list cannot reliably support “this host is absent from our profile.” A set subtraction performed on two arbitrary top-1,000 slices can produce convincing-looking false gaps.

Passing these checks means the API reports no truncation for the stored responses. Common Crawl is still an observed snapshot of the web, not a complete record of every live backlink. Keep those two limits separate.

The converter writes a manifest beside the CSVs with each input filename, target domain, release and returned/total counts. Preserve the original JSON too, so another person can check how you produced the report.

Compute the gap locally

Now compare the converted files:

python3 gap_analysis.py --your exports/own.csv --competitor exports/rival-a.csv --competitor exports/rival-b.csv

The operation is:

union(competitor hosts) - own hosts

For each remaining host, the script records which competitor inputs contain it and sorts by competitor overlap, then hostname.

Normalization is deliberately limited: lowercase names, remove trailing dots and convert internationalized names to ASCII IDNA. Subdomains remain distinct. A full source-page URL is rejected rather than silently converted into a different comparison unit.

If your exports use another provider's column names, map the hostname column to source_domain first. Record that provider's observation date, filters and export cap yourself; the generic CSV comparator cannot reconstruct metadata that was never exported.

Use overlap to choose what to inspect

A host shared by several competitors is a useful place to begin inspection. It is not an automatic recommendation to contact that publisher.

Open the relevant page, check that the link is still present, and look at the page's audience and context. A software directory, an article about a competing category and a sitewide navigation link can all enter the same set while suggesting very different next steps.

The scripts handle the repeatable calculation. You still decide whether a candidate has a relevant page and a useful reason to mention your site.

You can find the wider workflow in the programmatic backlink research recipes and the underlying snapshot model in Common Crawl's web-graph documentation.