DEV Community

Petteri Pucilowski
Petteri Pucilowski

Posted on • Originally published at crawlgraph.com

Only 5 domains link to every major AI agent framework (Common Crawl analysis)

Same experiment, third niche. Take a category's biggest players, pull every domain that links to them from Common Crawl's open hyperlink graph, and keep the ones that link to the whole set. SEO tools gave 37 universal linkers (nearly all media). CRMs gave 2 (both integration hubs). AI agent frameworks just gave 5 - and they are a different animal again.

The setup

Eight frameworks: LangChain, LlamaIndex, CrewAI, Dify, Flowise, Langflow, AutoGPT, SuperAGI. For each, we pulled referring domains ranked by authority from the Common Crawl webgraph (120M domains, 4.4B links), then intersected the eight lists.

The numbers

  • 7,508 unique domains link to at least one framework
  • 77% link to exactly one (usually LangChain: 5,949 referring domains vs ~450 for Langflow and AutoGPT)
  • 5 link to all eight: github.com, substack.com, analyticsvidhya.com, and two product sites

None of the universal linkers are tech press.

What the 7-and-6 club looks like

dev.to (yes, this site), qiita.com, zenn.dev, n8n.io, arize.com, zenml.io, langfuse.com... and then a whole crop of AI agent directories that did not exist two years ago: aiagentsdirectory.com, agenthunter.io, everydev.ai, findmyagentai.com, bestaiagents.ai. Most have single-digit-to-low authority scores. The category's shelf is still being built.

The takeaway for anyone shipping agent tooling

The niche's link graph is developer publishing platforms plus young directories, not journalists. So the play is: publish where developers already read (GitHub, Substack, dev.to, Qiita), and get listed in the agent directories while they are young and take an email. A directory with authority 20 today is not impressive. The one that becomes the geekflare of AI agents in three years is - and you got in early.

Full study with methodology, the overlap pyramid, and the downloadable CSV/JSON dataset: only 5 domains link to every major AI agent framework. The DuckDB query behind it is a public gist.

Top comments (0)