DEV Community: Petteri Pucilowski

9 domains link to every major live chat tool

Petteri Pucilowski — Mon, 02 Nov 2026 00:36:32 +0000

we compared 8 live chat tools in common crawl's domain-level link graph. 9 non-platform domains link to all 8, while 86% of linking domains appear for only one subject.

how we pulled this

for each subject domain, we pulled the top 2,000 referring domains by authority from common crawl release Apr-Jun 2026 (cc-main-2026-apr-may-jun). we intersected those lists, then removed platform, cdn, social, and other non-editorial domains from the clean overlap counts.

the filter matters. hosting providers, cdns, url shorteners and the big social networks link to almost everything, so leaving them in would produce a universal list that is technically correct and useless for outreach. the counts below are after that filter unless a column says otherwise. everything here comes from the open common crawl webgraph, so you can reproduce it without a paid backlink tool.

subject domains

domain	referring domains	cg authority
tidio.com	9,302	58
tawk.to	34,125	58
livechat.com	56,621	30
olark.com	4,408	30
drift.com	6,064	57
crisp.chat	13,823	55
chatra.com	312	30
liveagent.com	2,019	54

the overlap

across the 8 subjects, we found 11,499 unique linking domains. the clean universal set contains 9 domains.

metric	value
unique linking domains	11,499
link to one subject	86%
link to all 8	9

overlap distribution

overlap	all domains	non-platform
link to all 8	9	9
link to 7	24	21
link to 6	49	47
link to 5	77	75
link to 4	148	142
link to 3	326	318
link to 2	937	925
link to just 1	9,929	9,920

the distribution is the interesting part. most linking domains sit in the bottom row: they mention one product and never come back. the rows above it are where outreach lives, because a site that already covers several products in a category has an editorial reason to cover another one. a short universal list usually means the category is covered by a handful of directories, review sites and integration hubs rather than by a broad press base. a long one means the category has real editorial coverage, and the same sites are reachable for you.

the highest-authority sites linking to all of them are shopify.com, hubspot.com and sendpulse.com. those are the pages worth reading before you pitch anything, because they already decided this category is worth covering.

livechat.com carries the widest referring-domain profile of the set and chatra.com the narrowest, which is usually a function of age and press coverage rather than product quality. the gap between them is the practical opportunity: sites that cover the widest profile but not the narrowest are demonstrably willing to write about this category.

the category linkers

77 non-platform domains link to at least 6 of the 8 subjects. these repeat linkers are the sites most consistently connected to live chat tools.

highest-overlap category linkers

domain	subjects	cg authority	links to
shopify.com	8	80	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com, liveagent.com
hubspot.com	8	70	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com, liveagent.com
sendpulse.com	8	67	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com, liveagent.com
jivochat.com	8	51	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com, liveagent.com
saashub.com	8	51	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com, liveagent.com
elink.io	8	45	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com, liveagent.com
chatty.net	8	38	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com, liveagent.com
designbeep.com	8	37	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com, liveagent.com
podium.com	8	30	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com, liveagent.com
zendesk.com	7	69	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, liveagent.com
jotform.com	7	65	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, liveagent.com
brevo.com	7	58	tidio.com, tawk.to, livechat.com, olark.com, drift.com, chatra.com, liveagent.com
zendesk.co.uk	7	57	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, liveagent.com
databox.com	7	54	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, liveagent.com
seo.ai	7	54	tidio.com, tawk.to, livechat.com, olark.com, crisp.chat, chatra.com, liveagent.com
socialcompare.com	7	51	tidio.com, tawk.to, livechat.com, olark.com, crisp.chat, chatra.com, liveagent.com
site123.com	7	50	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, chatra.com
callbell.eu	7	48	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, liveagent.com
zendesk.com.br	7	48	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, liveagent.com
entireweb.com	7	47	tidio.com, tawk.to, livechat.com, olark.com, drift.com, crisp.chat, liveagent.com

the full data

download all 77 clean domains that link to at least 6 of the 8 subjects, ranked by overlap and authority.

download csv | download json

check any of these domains

every domain in the tables above links to its full referring-domain profile. run your own domain to see who links to you.

look up your backlinks

run this for your own niche

pick your closest competitors, compare their referring domains, and keep the sites that link to several of them but not to you. sorting that gap by overlap and authority turns the same method into a specific outreach list.

the ordering is what makes it usable. a site that links to most of your competitors and not to you is a gap with an obvious pitch. the observed overlap gives you a specific reason to contact each site instead of relying on domain authority alone. work down from the top of the overlap column and stop when the reasons start feeling thin, because a pitch you cannot justify in one sentence is not worth sending.

one caveat worth stating plainly. common crawl is a sample of the web, not a census, so treat these counts as the shape of a link profile rather than an exact total. this study measures the Apr-Jun 2026 release, so rerun the comparison against later releases before treating any movement as a trend.

find the overlap in your market

run a backlink gap analysis on the same common crawl data used for this study.

run a free gap analysis

We counted the links behind 580 first-page Google results

Petteri Pucilowski — Fri, 30 Oct 2026 19:38:18 +0000

"How many backlinks do I need to rank?" gets the same answer almost everywhere: 0 to 25 for easy keywords, 50 to 100 for hard ones. I couldn't find where those numbers came from, so we measured them.

What we did

We fixed a list of 60 keywords across six niches before pulling anything. Two had no stored results, which left 58. For each one we took the newest US Google snapshot in DataForSEO's data and the top 10 organic results, then read the referring domains for each exact URL and for its site.

The page counts come from DataForSEO rather than our own product. CrawlGraph is built on the Common Crawl web graph, and that graph is domain-level, so it can't count links to a single page.

What we found

Keyword difficulty	Pages	Median page	Median site
Low (under 15)	270	2	6,921
Medium (15 to 29)	190	9	40,851
High (30 and up)	120	48	53,899
All	580	5	25,372

The low end of the rule holds. 87% of pages on low-difficulty keywords had 25 referring domains or fewer.
The high end doesn't. Only 9 of 120 pages on hard keywords fell between 50 and 100. 53 had 25 or fewer and 49 had 100 or more.
The first result had the most links on its page, or tied for it, in only 8 of 58 keywords.

The catch

300 of the 580 results had 5 or fewer referring domains, which sounds encouraging until you look at the sites. 154 of those pages were on sites with at least 10,000 referring domains, and Reddit alone accounted for 49.

Only 17 of the 580 results came from sites with 100 or fewer referring domains, and 11 of those were on low-difficulty keywords. If your site is new, keyword choice probably matters before link count does.

The limits

We chose the keywords by hand, so this describes them, not all of Google. A zero means none found in one index, and exact-URL lookups can miss links to redirects. Snapshots and link counts weren't read on the same day. And none of this shows that links cause rankings.

The full write-up, including how to work out a number for your own keyword and a CSV of the per-keyword summary, is here: how many backlinks do you need to rank?

Two checks worth adding when you rewrite a checkout modal

Petteri Pucilowski — Wed, 28 Oct 2026 20:54:42 +0000

While rebuilding the free-preview and checkout flow in CrawlGraph, our review caught two details that are easy to miss: inherited disabled state and native email validation.

The browser APIs covered both cases. We added checks for the pending request state.

I build CrawlGraph, a backlink research and outreach tool. Here are the two checks from the interface work we just released.

1. A fieldset can disable an input without changing its attributes

Our checkout has a plan chooser. While its request is pending, the chooser's fieldset is disabled.

A focus-trap selector like this looks reasonable:

panel.querySelectorAll('button:not([disabled]), input:not([disabled])');

But an input inside a disabled fieldset can match that selector while still being disabled by the browser. It may have no disabled attribute of its own.

That matters when your trap treats the input as its last focusable element. Shift+Tab can try to focus a control that cannot receive focus. Forward Tab can leave the dialog because your calculated boundary is wrong.

Check effective state as well as visibility:

const controls = [...panel.querySelectorAll(focusableSelector)]
  .filter((element) => element.getClientRects().length > 0)
  .filter((element) => !element.matches(':disabled'));

Use a delayed checkout response, then press Tab and Shift+Tab from the close button while the request is pending.

MDN describes how disabled fieldsets affect their descendant controls.

2. Inline errors should keep the browser's email validity check

We wanted an inline error instead of the browser's validation bubble, so the rewritten form used noValidate.

That makes the submit handler responsible for validation. A check that only asks whether the string contains @ accepts user@@example.com far enough to start a request.

For an email input, the browser still exposes its validity state:

const invalid =
  !email.trim() ||
  input.validity.typeMismatch ||
  !input.validity.valid;

if (invalid) {
  showInlineError();
  input.focus();
  return;
}

// Track the valid action and start the request after this guard.

Keep the visible label, associate the error with the field, and preserve the typed address. Client validation improves the interaction; the server still validates the request independently.

The typeMismatch documentation explains the email and URL checks.

One more check for nested dialogs

Our free preview can open checkout. Escape should close the top dialog, preserve the preview beneath it, and return focus to the button that opened checkout. Background scrolling should remain locked until the final dialog closes.

We now render one responsive panel per modal instead of separate desktop and mobile copies. A shared stack owns focus and scroll behavior.

The WAI modal-dialog pattern is a useful baseline for Tab, Shift+Tab, initial focus and Escape behavior. We checked the Stripe backlink lookup against the live API and covered pending, error and already-paid checkout states with controlled responses.

The updated CrawlGraph preview lets you try up to five referring-domain rows without creating an account. Paid retrieval and CSV/JSON exports support up to 100,000 referring-domain rows per lookup. Paid users can also save outreach targets, find published contact emails and open a draft in their mail app.

The updated flow is live. Try a stripe.com lookup and follow the results into the outreach workflow.

7 domains link to every major payment gateway

Petteri Pucilowski — Mon, 26 Oct 2026 01:13:11 +0000

we compared 8 payment gateways in common crawl's domain-level link graph. 7 non-platform domains link to all 8, while 82% of linking domains appear for only one subject.

how we pulled this

subject domains

domain	referring domains	cg authority
stripe.com	100,000	78
paypal.com	100,000	79
adyen.com	18,781	57
braintreepayments.com	10,487	60
authorize.net	23,356	59
worldpay.com	4,861	54
mollie.com	18,907	55
checkout.com	3,339	55

the overlap

across the 8 subjects, we found 12,499 unique linking domains. the clean universal set contains 7 domains.

metric	value
unique linking domains	12,499
link to one subject	82%
link to all 8	7

overlap distribution

overlap	all domains	non-platform
link to all 8	13	7
link to 7	20	15
link to 6	27	23
link to 5	83	79
link to 4	162	155
link to 3	401	391
link to 2	1,519	1,503
link to just 1	10,274	10,254

the highest-authority sites linking to all of them are shopify.com, appspot.com and hubspot.com. those are the pages worth reading before you pitch anything, because they already decided this category is worth covering.

stripe.com carries the widest referring-domain profile of the set and checkout.com the narrowest, which is usually a function of age and press coverage rather than product quality. the gap between them is the practical opportunity: sites that cover the widest profile but not the narrowest are demonstrably willing to write about this category.

the category linkers

45 non-platform domains link to at least 6 of the 8 subjects. these repeat linkers are the sites most consistently connected to payment gateways.

highest-overlap category linkers

domain	subjects	cg authority	links to
shopify.com	8	80	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com, checkout.com
appspot.com	8	72	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com, checkout.com
hubspot.com	8	70	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com, checkout.com
drupal.org	8	65	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com, checkout.com
entrepreneur.com	8	65	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com, checkout.com
pymnts.com	8	60	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com, checkout.com
google.cn	8	56	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com, checkout.com
techradar.com	7	72	stripe.com, paypal.com, adyen.com, braintreepayments.com, worldpay.com, mollie.com, checkout.com
bigcommerce.com	7	70	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, checkout.com
iubenda.com	7	69	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com
wise.com	7	67	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, mollie.com, checkout.com
jotform.com	7	65	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com
woocommerce.com	7	63	stripe.com, paypal.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com, checkout.com
elementor.com	7	60	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com
freshdesk.com	7	59	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com
coda.io	7	59	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, mollie.com, checkout.com
packagist.org	7	59	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, mollie.com, checkout.com
geekflare.com	7	57	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, checkout.com
simplybook.me	7	55	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, mollie.com
netguru.com	7	54	stripe.com, paypal.com, adyen.com, braintreepayments.com, authorize.net, worldpay.com, checkout.com

the full data

download all 45 clean domains that link to at least 6 of the 8 subjects, ranked by overlap and authority.

download csv | download json

check any of these domains

every domain in the tables above links to its full referring-domain profile. run your own domain to see who links to you.

look up your backlinks

run this for your own niche

find the overlap in your market

run a backlink gap analysis on the same common crawl data used for this study.

run a free gap analysis

Your authority-score filter cannot be pasted into a different backlink export

Petteri Pucilowski — Sun, 25 Oct 2026 18:44:16 +0000

A filter such as "authority at least 50" is incomplete until you name the metric and data source. A value from one backlink provider is not a portable threshold for another provider's score, even when both fit on a 0-to-100 scale.

I maintain crawlgraph. This is the export worksheet I recommend when somebody wants to replace a backlink tool while keeping the old spreadsheet's filters.

Keep the metric's identity in the column name

These headers make different promises:

score
moz_da
ahrefs_dr
cg_authority

The generic first header hides information you need later. Keep the provider's actual metric name. Store the source, observed release or date, export limit and filters beside the file.

If the next report uses a different score, give it a different column. Do not silently rename it to the old metric because the spreadsheet formula expects that name.

For example, crawlgraph's cg_authority is a graph-derived score documented in its API contract. It is not a DA or DR conversion. The same numeric range does not supply a conversion formula.

Start with the report's purpose

An authority cutoff is often standing in for a more useful question.

Intended decision	Evidence to inspect
Is this publisher relevant to our audience?	Its content, category and actual readers
Does this site link to our competitors?	Comparable referring-domain observations
Is there a useful page to investigate?	A relevant source page and its context
Should this candidate be investigated first?	A stated prioritization rule, followed by inspection

A score can help sort a list. It does not establish those other facts by itself.

Before changing tools, write down which decision your existing threshold supports. If nobody can explain why the report uses 50 rather than 40, preserve that as an assumption to test rather than an industry requirement.

Test the candidate list, not just the totals

Choose a small sample from above and below your proposed cutoff. Open the sites and record whether each candidate is relevant and worth further investigation.

A simple worksheet is enough:

referring_hostname,metric_name,metric_value,source,release_or_date,relevance_note,next_action

The columns deliberately separate the score from the human research result. Add an actual source-page URL once you locate one; a referring hostname alone is not a page-level backlink record.

This is a way to assess your prioritization rule on the data you have. It is not a new vendor-neutral authority formula or a claim that the sampled sites represent the whole web.

Preserve unknowns

Do not replace a missing authority value with zero unless the provider explicitly defines that behavior. A missing value and a measured value of zero are different observations.

Keep truncated exports visible too. If a list contains only a top-ranked slice, it cannot establish that a lower-scored hostname has no link to the target.

For recurring reports, keep these four checks beside the import:

The metric name and source match the report definition.
The release or observation date is recorded.
Missing values remain distinguishable from measured zeros.
Returned rows and reported totals are kept separately.

Buy the fields your workflow needs

Changing the provider can change the index, observation dates, available records and metric definition at the same time. Check each before treating the new export as interchangeable with the old one.

If you need referring-domain research and competitor overlap, check a domain free and inspect crawlgraph's current plans. Its free web preview shows five referring domains. Paid core access supports expanded results and CSV/JSON exports capped at 100,000 rows per domain, with the total reported separately. It does not replace daily rank tracking, keyword research or every field in a full SEO suite.

The longer authority-metric guide covers the distinctions. Keep a provider-specific score when it is useful; keep the research decision explicit as well.

A missing backlink in a new snapshot is not a confirmed lost link

Petteri Pucilowski — Sun, 25 Oct 2026 18:42:17 +0000

A referring domain that disappears from a new crawl snapshot is an observation to investigate. It is not proof that somebody removed a live backlink. Before sending a lost-link report, check the release identities, the comparison unit and whether either export was truncated.

I maintain crawlgraph, which uses imported Common Crawl graphs for backlink research. Here is a small offline workflow for comparing saved responses without turning missing rows into false certainty.

Keep three dates separate

A source crawl, an upstream graph publication and a tool's queryable release are different things.

State	What it establishes
A page was captured in a crawl	That capture exists in that crawl archive
A graph was published upstream	That graph artifact is available from its publisher
A release is queryable in your tool	The tool can answer requests against its imported release

A newer graph on Common Crawl's website does not automatically become a new dataset inside every product that uses Common Crawl. Check the upstream graph list and the tool's own release availability separately.

For a comparison, keep the original responses with their release_id. Record which release is older and which is newer. Do not infer chronology by sorting arbitrary identifier strings.

Refuse a partial comparison

Suppose your older response reports 2,000 referring domains but returns only 1,000 rows. The absent half can contain a hostname that appears in your newer export. Calling that hostname newly discovered would overstate what the two files establish.

The opposite problem also matters: a hostname from the older file can fall below a newer export's row cap while still existing in the newer observed set.

For the strict workflow below, both responses must satisfy:

returned == len(results) == total_linking_domains

The target domain must match, and release identifiers must differ. This checks the response's reported completeness. It does not independently verify the files' provenance, collection method or whole-web coverage.

Run a synthetic fixture first

The standalone Python Gist contains snapshot_diff.py and two synthetic JSON fixtures. Download its four files into one directory and run:

python3 snapshot_diff.py --older older.json.example --newer newer.json.example

The fixtures use reserved .example names and invented release identifiers. They are not measurements of customer websites.

The result contains two kinds of row:

observed_added: present only in the newer saved response.
observed_absent: present only in the older saved response.

The shared hostname is omitted. Missing authority values remain empty rather than becoming zero. The script copies supplied authority values; it does not calculate a new score or convert another vendor's metric.

For real files, the command is:

python3 snapshot_diff.py --older earlier.json --newer later.json --output observations.csv

--output creates a new file and refuses to overwrite an existing one, including a symlink. Both inputs are validated before writing. The comparison runs locally: no API key, network request or lookup quota is needed to compare files you already hold.

Name the result after the evidence

A useful report sentence is:

This referring hostname was observed in release A and absent from our complete response for release B. Its live-link status has not been verified.

That wording leaves room for several explanations: changed crawl coverage, a different graph observation, a site change or an actual removed link.

The graph result is a referring hostname, not a source-page URL. To confirm a removed link, you still need to identify and inspect the relevant source page. If you cannot locate it, preserve the observation and the uncertainty rather than marking a deletion as confirmed.

A new hostname also needs context. A directory entry, an editorial recommendation and a technical resource link can suggest different actions even when they all enter a referring-domain set.

Choose the tool for the reporting job

Snapshot comparisons can support periodic backlink research. They cannot promise that a link placed today will be found tomorrow, or that every live link is represented.

If your workflow is researching referring domains and competitor overlap, try your own domain free and compare the current plans. Crawlgraph's free web preview shows five referring domains; paid core access expands results and supports capped CSV/JSON exports. If your deliverable needs continuously refreshed source-page records, evaluate that requirement before buying.

The Common Crawl release guide explains crawl, graph and import states. The API reference documents fields, release availability, quotas and the hosted snapshot-change endpoint if you prefer it to local files.

Before buying a backlink-tool subscription, run this three-domain test

Petteri Pucilowski — Sat, 24 Oct 2026 18:14:18 +0000

A backlink subscription is useful only if it answers the question you actually need answered. Before comparing annual plans, test your own domain and two competitors. Check whether the data produces a useful research shortlist, whether you can export it, and whether its observation dates fit your work.

I maintain crawlgraph. It sells backlink intelligence from Common Crawl, including a one-time lifetime option. This is the buying test I recommend before paying for it or another backlink tool.

1. Write down the report you need

These are different jobs:

Your next task	What the tool must provide
Find sites that mention two competitors	Comparable referring-domain sets and overlap analysis
Identify a specific link's anchor and source page	Page-level backlink records with those fields
Report links gained or lost this week	Repeated observations with dates and suitable freshness
Report keyword positions every day	A rank tracker
Put referring domains into a spreadsheet	A usable export within your required row limit

A referring-domain graph can help with the first and last tasks. It does not automatically provide the other three. A cheaper tool that omits your required fields costs you another purchase later.

2. Use three real domains

Choose your site and two competitors whose products serve the same audience. A famous company in a different category is usually a poor comparison.

Run the same lookup for all three. Save the date, dataset release where available, filters, returned row count and reported total. Then ask:

Does the result contain publishers you can plausibly learn from?
Can you distinguish the source hostname from an actual source-page URL?
Does the export include the fields your report needs?
Is the result a sample, a capped slice or the full set observed by that provider?

An absent hostname in a five-row preview is not evidence that the site has no link. It is evidence that the hostname was absent from those five rows.

If you compare competitor overlap, use comparable inputs. A partial own-site list can label an existing referring domain as a new prospect. Keep truncation and observation dates visible in the report.

3. Inspect five candidates before buying

Take five potentially relevant hosts from the result. Visit their sites and look for the context in which they discuss your category.

A software directory, an editorial comparison and a footer link can all appear as referring domains. They suggest different next steps. A hostname alone does not tell you where the link sits, whether it still exists or whether the publisher accepts submissions.

The test succeeds when you can explain why those candidates are relevant and how you would verify them. Five irrelevant rows are a reason to investigate coverage or targeting before unlocking a larger export.

4. Compare the cost of that job

Use the quote and billing terms you can actually buy. For illustration, a hypothetical $40 monthly subscription costs $120 for three monthly payments and $480 for twelve. That arithmetic says nothing about whether its features match a one-time purchase.

Compare:

Your expected number of research sessions.
The export allowance and API quota you need.
The required freshness and history.
Features you would still need to buy separately.
Refund terms and what happens to saved projects after cancellation.

For a one-off task, a monthly plan may be sensible. For repeated referring-domain research, a specialist lifetime option may be worth testing. For weekly link-loss reports, freshness can matter more than either price.

Where crawlgraph fits

Crawlgraph's free web lookup shows five referring domains. Paid core access expands the available result slice and supports CSV and JSON exports, capped at 100,000 rows per domain with the total reported separately. It also offers hosted competitor-gap analysis.

It uses imported Common Crawl graph snapshots. It does not replace keyword research, daily rank tracking, a technical site crawler or a continuously refreshed page-level backlink index. Its authority percentile is not another vendor's DA or DR score.

The free API allocation is 15 backlink calls per UTC month. Hosted gap jobs require paid core access; the current paid API allocation is 1,000 backlink calls and 50 gap jobs per UTC month. See the API documentation for the current contract.

If this is the research job you need, check your own domain free, then compare the available plans. Check the current price, plan eligibility and limits on the site before purchasing.

The longer backlink-tool cost and product-fit guide explains the tradeoffs. Common Crawl's graph documentation describes the underlying dataset.

Save backlink responses once, then build a local competitor-gap report

Petteri Pucilowski — Sat, 24 Oct 2026 17:52:01 +0000

Changing an authority threshold or sorting a backlink report should not require downloading the same data again. Save the original API responses, keep their release metadata, and run the comparison locally. The useful part is making sure the files describe comparable observations before treating an absent hostname as a gap.

I maintain crawlgraph. This tutorial uses its JSON response format. The Python CSV comparator also works with compatible referring-host exports from another source, and the included synthetic example needs no account, API key or network access.

Start with a small, reproducible fixture

The runnable Gist contains two standard-library Python scripts and three CSV fixtures. Download its six files into one directory.

The inputs use reserved .example names:

File	Referring hosts
own.csv	shared.example, ours.example
rival-a.csv	shared.example, roundup.example, guide.example
rival-b.csv	shared.example, roundup.example, review.example

Run:

python3 gap_analysis.py --your own.csv --competitor rival-a.csv --competitor rival-b.csv

Expected output:

source_domain,competitor_overlap,competitors
roundup.example,2,rival-a;rival-b
guide.example,1,rival-a
review.example,1,rival-b

The shared host disappears because it is already in your own set. roundup.example appears in both competitor sets, so its overlap is two. Repeating it ten times in one CSV would still count as one competitor.

These are test fixtures, not customer results or actual outreach prospects.

Save the response before filtering it

For a real run, keep one response per target. With your key already in an environment variable, a single request looks like this:

set -euo pipefail
set -o noclobber
: "${CG_KEY:?set CG_KEY in your shell first}"

curl --fail --silent --show-error --max-time 30 \
  -X POST https://crawlgraph.com/api/v1/backlinks \
  -H "Authorization: Bearer $CG_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"example.com","limit":1000,"sort":"authority"}' \
  > own.json

The example uses example.com; replace it with the public target you intend to research. Save separate responses for two distinct competitors as rival-a.json and rival-b.json. Three successful requests use three backlink lookups. The current free allocation is 15 backlink calls per UTC month; the hosted gap-analysis endpoint requires paid core access. The local set comparison here is ordinary Python operating on files you already have.

The command stops on an HTTP error and refuses to overwrite an existing output file. Keep the key out of the files you publish. No live requests are needed to run the synthetic fixture.

The API reference explains authentication, response fields and quotas.

Reject mismatched or partial inputs

Before making CSVs, run:

python3 export_hosts.py own.json rival-a.json rival-b.json --output-dir exports

The converter checks four things:

Every response has the same release_id.
The target domains and input filenames are distinct.
returned matches the number of result rows and total_linking_domains.
Every row contains a valid hostname, with no duplicate linking_domain rows.

It validates the inputs before writing output. A reported partial response or a mixed release stops the conversion. Existing output files also stop it.

Why reject partial responses? A competitor sample can supply a positive lead: “this host appears here.” But an incomplete own-site list cannot reliably support “this host is absent from our profile.” A set subtraction performed on two arbitrary top-1,000 slices can produce convincing-looking false gaps.

Passing these checks means the API reports no truncation for the stored responses. Common Crawl is still an observed snapshot of the web, not a complete record of every live backlink. Keep those two limits separate.

The converter writes a manifest beside the CSVs with each input filename, target domain, release and returned/total counts. Preserve the original JSON too, so another person can check how you produced the report.

Compute the gap locally

Now compare the converted files:

python3 gap_analysis.py --your exports/own.csv --competitor exports/rival-a.csv --competitor exports/rival-b.csv

The operation is:

union(competitor hosts) - own hosts

For each remaining host, the script records which competitor inputs contain it and sorts by competitor overlap, then hostname.

Normalization is deliberately limited: lowercase names, remove trailing dots and convert internationalized names to ASCII IDNA. Subdomains remain distinct. A full source-page URL is rejected rather than silently converted into a different comparison unit.

If your exports use another provider's column names, map the hostname column to source_domain first. Record that provider's observation date, filters and export cap yourself; the generic CSV comparator cannot reconstruct metadata that was never exported.

Use overlap to choose what to inspect

A host shared by several competitors is a useful place to begin inspection. It is not an automatic recommendation to contact that publisher.

Open the relevant page, check that the link is still present, and look at the page's audience and context. A software directory, an article about a competing category and a sitewide navigation link can all enter the same set while suggesting very different next steps.

The scripts handle the repeatable calculation. You still decide whether a candidate has a relevant page and a useful reason to mention your site.

You can find the wider workflow in the programmatic backlink research recipes and the underlying snapshot model in Common Crawl's web-graph documentation.

A site crawl alone cannot show who links to your competitors

Petteri Pucilowski — Fri, 23 Oct 2026 12:54:40 +0000

A site crawler starts with pages you choose. A backlink index starts with pages discovered across other sites. A hyperlink graph stores connections between pages, hosts, or domains. These tools share crawling technology, but they answer different questions.

If you are building an SEO workflow, choosing the wrong input can produce a perfectly valid computation that answers the wrong question.

Disclosure: I maintain crawlgraph, which queries Common Crawl hyperlink graphs. This article adapts our backlink crawling tool guide. The examples below are illustrative, not measurements of a customer's site.

Choose the evidence before the tool

Your question	Evidence you need	Useful starting point
Which internal pages link to a broken URL?	Links extracted from your site's pages	A site crawler
Which hosts were observed linking to a competitor?	Incoming edges in a dated index or graph	A backlink index or hyperlink graph
Does this particular article still link to my site?	The current source page and its actual link destination	A page inspection
Did a publisher remove a link?	Comparable earlier and current page evidence	A change ledger plus verification

A crawl of example.com does not enumerate incoming links from the rest of the web. It can discover outgoing links on the pages it visits. Turning that into a competitor-backlink index requires discovering other sites, fetching their pages, extracting links, and storing relationships you can query in reverse.

Common Crawl publishes hyperlink graphs that make large-scale relationship research possible without operating that whole crawl pipeline yourself. They remain dated observations with coverage limits. They are not a live check of every referring page.

Give every result a scope

Before comparing two exports, write down:

Target: the exact hostname or domain you queried.
Unit: source pages, hosts, or registrable domains.
Observation: the index or graph release and any known crawl date.
Filters: exclusions, link-type rules, and grouping choices.
Completeness: whether the returned rows were capped or paginated.

For example, docs.example.com and www.example.com are different hosts. A domain-level export may group them together. Comparing its row count with a host-level export does not measure growth or loss.

Likewise, a result cap is part of the dataset's meaning. A row missing from the first 1,000 results is not necessarily absent from the index. Preserve the returned total and cap indicators alongside the file, rather than throwing that context away after parsing the rows.

A set difference is a research queue

Once the scopes match, a simple difference can identify candidates worth inspecting:

# Synthetic host-level observations with matching scope.
# These values are examples, not live crawl results.
earlier = {"docs.example.org", "journal.example.net", "community.example.com"}
later = {"journal.example.net", "community.example.com", "news.example.org"}

missing_observations = sorted(earlier - later)
new_observations = sorted(later - earlier)

assert missing_observations == ["docs.example.org"]
assert new_observations == ["news.example.org"]

The variable names matter. missing_observations does not establish removed_links. The earlier graph may have included a page that the later crawl did not visit. A source may have moved, blocked access, or changed its destination. A host-level edge does not identify the exact article you need to inspect.

Common Crawl's FAQ describes the sampling boundary. Keep it in your output contract so the next person using your CSV does not accidentally turn a missing observation into a claim about a publisher's behavior.

Verify before taking action

For an important candidate, find the source URL and check the final page after redirects. Inspect the actual hyperlink destination, not just a text occurrence of your domain.

A page can mention example.com in plain text without linking to it. A successful HTTP response can contain a login screen or replacement homepage. An unsuccessful request tells you that verification failed; it does not prove that an editor deleted the link.

Keep a small ledger:

Source URL	Checked at	Final URL	Link evidence	Status	Next action
saved article URL	UTC timestamp	after redirects	capture or inspected href	present / absent on checked page / unresolved	retain / investigate / recheck

Only label a change as a confirmed removal when you have comparable earlier evidence and a current inspection of the relevant page. Even then, the result establishes a page change, not what Google currently credits in its ranking systems.

For competitor research, the same boundary applies: a host observed linking to a competitor is a prospect to qualify, not a promise that it will link to you. Read the page, assess relevance, and check editorial requirements before contacting anyone.

The useful pipeline is discover candidates → preserve scope → inspect important pages → record a supported conclusion. That pipeline works with a commercial index, an open graph, or your own crawler, provided the evidence matches the question.

9 domains link to every major accounting tool

Petteri Pucilowski — Mon, 19 Oct 2026 01:47:53 +0000

we compared 8 accounting tools in common crawl's domain-level link graph. 9 non-platform domains link to all 8, while 80% of linking domains appear for only one subject.

how we pulled this

subject domains

domain	referring domains	cg authority
xero.com	21,178	60
freshbooks.com	9,916	58
waveapps.com	6,408	53
sage.com	11,920	60
intuit.com	62,108	65
odoo.com	63,051	58
bench.co	2,152	53
gnucash.org	1,425	30

the overlap

across the 8 subjects, we found 11,443 unique linking domains. the clean universal set contains 9 domains.

metric	value
unique linking domains	11,443
link to one subject	80%
link to all 8	9

overlap distribution

overlap	all domains	non-platform
link to all 8	11	9
link to 7	25	20
link to 6	43	39
link to 5	114	106
link to 4	213	209
link to 3	522	516
link to 2	1,381	1,374
link to just 1	9,134	9,122

the highest-authority sites linking to all of them are shopify.com, smallbiztrends.com and jotform.com. those are the pages worth reading before you pitch anything, because they already decided this category is worth covering.

odoo.com carries the widest referring-domain profile of the set and gnucash.org the narrowest, which is usually a function of age and press coverage rather than product quality. the gap between them is the practical opportunity: sites that cover the widest profile but not the narrowest are demonstrably willing to write about this category.

the category linkers

68 non-platform domains link to at least 6 of the 8 subjects. these repeat linkers are the sites most consistently connected to accounting tools.

highest-overlap category linkers

domain	subjects	cg authority	links to
shopify.com	8	80	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co, gnucash.org
smallbiztrends.com	8	67	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co, gnucash.org
jotform.com	8	65	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co, gnucash.org
beehiiv.com	8	58	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co, gnucash.org
uschamber.com	8	58	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co, gnucash.org
geekflare.com	8	57	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co, gnucash.org
saashub.com	8	51	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co, gnucash.org
superagi.com	8	49	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co, gnucash.org
europeanbusinessreview.com	8	47	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co, gnucash.org
hubspot.com	7	70	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co
bigcommerce.com	7	70	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co
googleblog.com	7	68	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, gnucash.org
entrepreneur.com	7	65	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co
libsyn.com	7	64	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, gnucash.org
dev.to	7	63	xero.com, freshbooks.com, waveapps.com, intuit.com, odoo.com, bench.co, gnucash.org
hostinger.com	7	61	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co
pymnts.com	7	60	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co
clickup.com	7	59	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co
wondershare.com	7	55	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, gnucash.org
seo.ai	7	54	xero.com, freshbooks.com, waveapps.com, sage.com, intuit.com, odoo.com, bench.co

the full data

download all 68 clean domains that link to at least 6 of the 8 subjects, ranked by overlap and authority.

download csv | download json

check any of these domains

every domain in the tables above links to its full referring-domain profile. run your own domain to see who links to you.

look up your backlinks

run this for your own niche

find the overlap in your market

run a backlink gap analysis on the same common crawl data used for this study.

run a free gap analysis

11 domains link to every major monitoring tool

Petteri Pucilowski — Mon, 12 Oct 2026 00:49:08 +0000

we compared 8 monitoring and error tracking tools in common crawl's domain-level link graph. 11 non-platform domains link to all 8, while 81% of linking domains appear for only one subject.

how we pulled this

subject domains

domain	referring domains	cg authority
sentry.io	13,290	59
datadoghq.com	8,616	60
newrelic.com	13,994	63
grafana.com	9,459	61
dynatrace.com	5,251	56
honeycomb.io	1,444	54
betterstack.com	5,962	30
bugsnag.com	1,717	54

the overlap

across the 8 subjects, we found 11,073 unique linking domains. the clean universal set contains 11 domains.

metric	value
unique linking domains	11,073
link to one subject	81%
link to all 8	11

overlap distribution

overlap	all domains	non-platform
link to all 8	15	11
link to 7	46	41
link to 6	82	72
link to 5	132	129
link to 4	241	234
link to 3	468	467
link to 2	1,081	1,075
link to just 1	9,008	8,994

the highest-authority sites linking to all of them are dev.to, cncf.io and linuxfoundation.org. those are the pages worth reading before you pitch anything, because they already decided this category is worth covering.

newrelic.com carries the widest referring-domain profile of the set and honeycomb.io the narrowest, which is usually a function of age and press coverage rather than product quality. the gap between them is the practical opportunity: sites that cover the widest profile but not the narrowest are demonstrably willing to write about this category.

the category linkers

124 non-platform domains link to at least 6 of the 8 subjects. these repeat linkers are the sites most consistently connected to monitoring and error tracking tools.

highest-overlap category linkers

domain	subjects	cg authority	links to
dev.to	8	63	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
cncf.io	8	61	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
linuxfoundation.org	8	61	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
opentelemetry.io	8	59	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
zenn.dev	8	58	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
geekflare.com	8	57	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
solarwinds.com	8	56	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
saashub.com	8	51	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
harness.io	8	45	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
edgedelta.com	8	30	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
ministryoftesting.com	8	30	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com, bugsnag.com
wikimedia.org	7	75	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, betterstack.com, bugsnag.com
jetbrains.com	7	72	sentry.io, datadoghq.com, newrelic.com, grafana.com, honeycomb.io, betterstack.com, bugsnag.com
libsyn.com	7	64	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, bugsnag.com
digitalocean.com	7	62	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, betterstack.com, bugsnag.com
go.dev	7	61	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, bugsnag.com
computerweekly.com	7	61	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, bugsnag.com
clickup.com	7	59	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, betterstack.com
hashicorp.com	7	59	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, bugsnag.com
duo.com	7	59	sentry.io, datadoghq.com, newrelic.com, grafana.com, dynatrace.com, honeycomb.io, bugsnag.com

the full data

download all 124 clean domains that link to at least 6 of the 8 subjects, ranked by overlap and authority.

download csv | download json

check any of these domains

every domain in the tables above links to its full referring-domain profile. run your own domain to see who links to you.

look up your backlinks

run this for your own niche

find the overlap in your market

run a backlink gap analysis on the same common crawl data used for this study.

run a free gap analysis

29 domains link to every major password manager

Petteri Pucilowski — Mon, 28 Sep 2026 01:29:01 +0000

we compared 8 password managers in common crawl's domain-level link graph. 29 non-platform domains link to all 8, while 73% of linking domains appear for only one subject.

how we pulled this

subject domains

domain	referring domains	cg authority
1password.com	12,018	60
bitwarden.com	7,791	54
dashlane.com	5,498	57
lastpass.com	14,712	61
keepersecurity.com	3,886	56
nordpass.com	4,745	54
roboform.com	2,284	52
enpass.io	722	43

the overlap

across the 8 subjects, we found 9,672 unique linking domains. the clean universal set contains 29 domains.

metric	value
unique linking domains	9,672
link to one subject	73%
link to all 8	29

overlap distribution

overlap	all domains	non-platform
link to all 8	34	29
link to 7	36	34
link to 6	86	80
link to 5	159	158
link to 4	285	281
link to 3	613	603
link to 2	1,423	1,416
link to just 1	7,036	7,024

the highest-authority sites linking to all of them are mozilla.org, habr.com and smallbiztrends.com. those are the pages worth reading before you pitch anything, because they already decided this category is worth covering.

lastpass.com carries the widest referring-domain profile of the set and enpass.io the narrowest, which is usually a function of age and press coverage rather than product quality. the gap between them is the practical opportunity: sites that cover the widest profile but not the narrowest are demonstrably willing to write about this category.

the category linkers

143 non-platform domains link to at least 6 of the 8 subjects. these repeat linkers are the sites most consistently connected to password managers.

highest-overlap category linkers

domain	subjects	cg authority	links to
mozilla.org	8	82	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
habr.com	8	67	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
smallbiztrends.com	8	67	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
norton.com	8	59	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
clickup.com	8	59	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
fidoalliance.org	8	59	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
helpnetsecurity.com	8	59	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
sapo.pt	8	58	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
impress.co.jp	8	57	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
n26.com	8	56	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
pandasecurity.com	8	55	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
nero.com	8	54	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
otto.de	8	54	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
tweakers.net	8	53	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
devolutions.net	8	53	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
avg.com	8	52	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
tenorshare.com	8	52	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
saashub.com	8	51	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
wikimint.com	8	46	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io
privacy.com	8	43	1password.com, bitwarden.com, dashlane.com, lastpass.com, keepersecurity.com, nordpass.com, roboform.com, enpass.io

the full data

download all 143 clean domains that link to at least 6 of the 8 subjects, ranked by overlap and authority.

download csv | download json

check any of these domains

every domain in the tables above links to its full referring-domain profile. run your own domain to see who links to you.

look up your backlinks

run this for your own niche

find the overlap in your market

run a backlink gap analysis on the same common crawl data used for this study.

run a free gap analysis