The latest articles on DEV Community by Petteri Pucilowski (@petteri_pucilowski_7ec755). https://dev.to/petteri_pucilowski_7ec755 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3958159%2F10c01a3a-4ac9-40e7-8cb4-63d6b0869fdb.jpg https://dev.to/petteri_pucilowski_7ec755 en Petteri Pucilowski Tue, 18 Aug 2026 10:06:19 +0000 https://dev.to/petteri_pucilowski_7ec755/we-had-45-subscribers-and-had-sent-them-zero-emails-25jk https://dev.to/petteri_pucilowski_7ec755/we-had-45-subscribers-and-had-sent-them-zero-emails-25jk <p>We had 45 people on our email list and had sent them, in total, zero emails.</p> <p>Not zero this month. Zero ever. I found it while checking something else, and the reason is boring enough that I suspect it is sitting in your app too.</p> <h2> How a form ends up promising something no code owns </h2> <p>Our subscribe endpoint did exactly what I wrote it to do:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="nd">@router.post</span><span class="p">(</span><span class="sh">"</span><span class="s">/newsletter/subscribe</span><span class="sh">"</span><span class="p">)</span> <span class="k">async</span> <span class="k">def</span> <span class="nf">subscribe</span><span class="p">(</span><span class="n">request</span><span class="p">:</span> <span class="n">Request</span><span class="p">,</span> <span class="n">body</span><span class="p">:</span> <span class="n">SubscribeBody</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span> <span class="bp">...</span> <span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">resend_audience_client</span><span class="p">.</span><span class="nf">add_contact</span><span class="p">(</span> <span class="n">audience_id</span><span class="o">=</span><span class="n">settings</span><span class="p">.</span><span class="n">resend_newsletter_audience_id</span><span class="p">,</span> <span class="n">email</span><span class="o">=</span><span class="n">email</span><span class="p">,</span> <span class="p">)</span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">ok</span><span class="sh">"</span><span class="p">:</span> <span class="bp">True</span><span class="p">}</span> </code></pre> </div> <p>Add the contact to the audience, return 200. That is a complete, correct, tested implementation of "subscribe". It is also the whole bug, because <em>nothing else in the system ever mails that audience except a weekly broadcast I had paused a month earlier</em>.</p> <p>Meanwhile the frontend was making promises. The exit intent popup said:</p> <blockquote> <p>get one email when the next release is live</p> </blockquote> <p>and the blog form said:</p> <blockquote> <p>one tactical post a week</p> <p>+ a free domain audit when you sign up</p> </blockquote> <p>Three promises. A per release email, a weekly post, and an audit. Grep the backend for the code that keeps any of them and you get one hit: the weekly broadcast, behind a feature flag that was set to false. The audit never existed at all. I had written it as copy, liked how it read, and shipped it.</p> <p>That is the actual failure mode. Not a bug in a function. A promise that lives in JSX with no counterpart in the backend, so no test can fail and no error can fire. The system was 100% healthy while doing nothing.</p> <h2> The first touch has to exist, and exactly once </h2> <p>The fix is a welcome email, which sounds trivial until you write down what it has to survive: a resubscribe must not send a second one, a Resend hiccup during the request must not lose the only email that subscriber ever gets, and a retry must not send two.</p> <p>I already had a <code>delayed_events</code> table for scheduling emails after a purchase, with this index:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight sql"><code><span class="k">CREATE</span> <span class="k">UNIQUE</span> <span class="k">INDEX</span> <span class="n">uq_delayed_events_email_event</span> <span class="k">ON</span> <span class="n">delayed_events</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">event_name</span><span class="p">);</span> </code></pre> </div> <p>That index is the whole dedupe. <code>schedule()</code> returns the new row id, or <code>None</code> if a row for that pair already exists. So "have we ever welcomed this address" is not a new question, it is the insert:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">async</span> <span class="k">def</span> <span class="nf">_welcome_once</span><span class="p">(</span><span class="n">email</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span> <span class="n">event_id</span> <span class="o">=</span> <span class="n">delayed_events_service</span><span class="p">.</span><span class="nf">schedule</span><span class="p">(</span> <span class="n">email</span><span class="p">,</span> <span class="n">delayed_events_service</span><span class="p">.</span><span class="n">NEWSLETTER_WELCOME_EVENT</span><span class="p">,</span> <span class="mi">0</span> <span class="p">)</span> <span class="k">if</span> <span class="n">event_id</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span> <span class="k">return</span> <span class="sh">"</span><span class="s">already_welcomed</span><span class="sh">"</span> <span class="c1"># resubscribe, ever, forever </span> <span class="n">sent</span> <span class="o">=</span> <span class="k">await</span> <span class="nf">send_newsletter_welcome_email</span><span class="p">(</span><span class="n">email</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">sent</span><span class="p">:</span> <span class="k">return</span> <span class="sh">"</span><span class="s">queued</span><span class="sh">"</span> <span class="c1"># row stays due, poller retries </span> <span class="n">delayed_events_service</span><span class="p">.</span><span class="nf">mark_sent</span><span class="p">(</span><span class="n">event_id</span><span class="p">)</span> <span class="k">return</span> <span class="sh">"</span><span class="s">sent</span><span class="sh">"</span> </code></pre> </div> <p>Sending inline matters more than it looks. A welcome email that arrives five minutes after signup is a worse welcome, and the poller interval was five minutes. So the row is a receipt, not a queue: written first, stamped after delivery, and if delivery fails it is simply left due for the poller that already exists.</p> <p><code>mark_sent</code> is guarded so a poller tick that fires in the same second cannot double send:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="n">cur</span> <span class="o">=</span> <span class="n">conn</span><span class="p">.</span><span class="nf">execute</span><span class="p">(</span> <span class="sh">"</span><span class="s">UPDATE delayed_events SET sent_at = ?, last_error = NULL </span><span class="sh">"</span> <span class="sh">"</span><span class="s">WHERE id = ? AND sent_at IS NULL</span><span class="sh">"</span><span class="p">,</span> <span class="p">(</span><span class="n">now</span><span class="p">,</span> <span class="n">event_id</span><span class="p">),</span> <span class="p">)</span> <span class="k">return</span> <span class="n">cur</span><span class="p">.</span><span class="n">rowcount</span> <span class="o">&gt;</span> <span class="mi">0</span> </code></pre> </div> <h2> Seeding a ledger so shipping the feature does not fire it </h2> <p>The popup promise was per release: our data comes from Common Crawl, which republishes its hyperlink graph about every three months, and people signed up to hear when a fresh one lands.</p> <p>So: a table keyed by release id, a poller check, one broadcast per new id. Easy, except for the first run. The moment that code deploys it finds the current release, sees no row, and mails 45 people about a graph that went live three weeks ago.</p> <p>The fix is one line in <code>ensure_tables</code>, and it is the part I would have forgotten:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="n">empty</span> <span class="o">=</span> <span class="n">conn</span><span class="p">.</span><span class="nf">execute</span><span class="p">(</span><span class="sh">"</span><span class="s">SELECT COUNT(*) FROM release_announcements</span><span class="sh">"</span><span class="p">).</span><span class="nf">fetchone</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="mi">0</span> <span class="bp">...</span> <span class="k">if</span> <span class="n">empty</span><span class="p">:</span> <span class="n">conn</span><span class="p">.</span><span class="nf">execute</span><span class="p">(</span> <span class="sh">"</span><span class="s">INSERT OR IGNORE INTO release_announcements </span><span class="sh">"</span> <span class="sh">"</span><span class="s">(release_id, status, created_at, sent_at) VALUES (?, </span><span class="sh">'</span><span class="s">seeded</span><span class="sh">'</span><span class="s">, ?, ?)</span><span class="sh">"</span><span class="p">,</span> <span class="p">(</span><span class="n">current_release_id</span><span class="p">,</span> <span class="n">now</span><span class="p">,</span> <span class="n">now</span><span class="p">),</span> <span class="p">)</span> </code></pre> </div> <p>Bootstrapping the ledger with "already handled" means only genuinely new state can trigger the side effect. Any time you add "notify on change" to a system that has been running for a while, the current value is not a change.</p> <p>The other decision worth naming: this is deliberately not behind the same flag as the weekly newsletter. I had paused a weekly column because I did not have time to write it. That is my choice to make. Cancelling a per release notification someone explicitly opted into is not the same choice, and one boolean should not silently do both.</p> <h2> The bounce nobody handled </h2> <p>While in there I pulled the last 100 sends from the Resend API. Six bounces, from two addresses. Both were junk domains someone typed into the form. Both had then received the <em>entire</em> rest of the sequence.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>2026-08-06 bounced your free CrawlGraph API key 2026-08-08 bounced the question your free API key can't answer 2026-08-12 bounced your API key works with hosted MCP </code></pre> </div> <p>One signup, three bounces, on a schedule. At my volume that is cosmetic. Bounce rate is also the single number mailbox providers use to decide whether you are a sender or a problem, and the list only grows.</p> <p>The interesting part is that the fix was already half built. Resend was posting webhooks to an endpoint I had written for inbound support email, and that handler said:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">if</span> <span class="n">payload</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">type</span><span class="sh">"</span><span class="p">)</span> <span class="o">!=</span> <span class="sh">"</span><span class="s">email.received</span><span class="sh">"</span><span class="p">:</span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">ignored</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">type</span><span class="sh">"</span><span class="p">:</span> <span class="n">payload</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">type</span><span class="sh">"</span><span class="p">)}</span> </code></pre> </div> <p>Every bounce notification Resend ever sent me hit that line and was thrown away with a cheerful 200. Now <code>email.bounced</code> and <code>email.complained</code> route to a suppression table, and permanent failures cancel anything already scheduled for that address.</p> <p>Two details worth stealing. Transient bounces do not suppress, because a full mailbox is not a reason to stop mailing someone forever. And the marketing sender reports <strong>success</strong> when it skips a suppressed address:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">if</span> <span class="n">email_suppression_service</span><span class="p">.</span><span class="nf">is_suppressed</span><span class="p">(</span><span class="n">email</span><span class="p">):</span> <span class="n">logger</span><span class="p">.</span><span class="nf">info</span><span class="p">(</span><span class="sh">"</span><span class="s">skipping %s to a suppressed address</span><span class="sh">"</span><span class="p">,</span> <span class="n">kind</span><span class="p">)</span> <span class="k">return</span> <span class="bp">True</span> <span class="c1"># terminal: the durable sender must retire the row, not retry </span></code></pre> </div> <p>Returning <code>False</code> there would be more honest looking and completely wrong: every durable sender I have treats <code>False</code> as "retry later", so a suppressed address would spin until it hit the attempt cap.</p> <h2> Ten minutes on your own funnel </h2> <p>Three queries, and you do not need my stack to run the equivalent:</p> <ol> <li> <strong>Every collection point, and what it promises.</strong> Grep the frontend for your form components and read the copy out loud. Every promise is a claim about backend behaviour.</li> <li> <strong>For each promise, the code that keeps it.</strong> If you cannot point at a function, the promise is decoration. Delete the copy or write the function.</li> <li> <strong>Your provider's recent sends, grouped by event.</strong> <code>delivered</code>, <code>bounced</code>, <code>complained</code>. Then check whether anything that bounced got mailed again afterwards.</li> </ol> <p>The thing that stung was that every dashboard was green. Signups were arriving, the API returned 200, tests passed, the provider reported no errors. There is no monitor for "the email that was never designed."</p> <p>I build <a href="https://crawlgraph.com" rel="noopener noreferrer">CrawlGraph</a>, backlink data on Common Crawl's open web graph. The free API key is 15 calls a month with no card if you want to poke at link graphs: <a href="https://crawlgraph.com/docs/api" rel="noopener noreferrer">crawlgraph.com/docs/api</a>. But mostly, go read your own signup form's copy and then go looking for the function that keeps it.</p> webdev python showdev programming Petteri Pucilowski Mon, 17 Aug 2026 02:18:26 +0000 https://dev.to/petteri_pucilowski_7ec755/12-domains-link-to-every-major-help-desk-3ela https://dev.to/petteri_pucilowski_7ec755/12-domains-link-to-every-major-help-desk-3ela <p>we compared 8 help desks in common crawl's domain-level link graph. 12 non-platform domains link to all 8, while 81% of linking domains appear for only one subject.</p> <h2> how we pulled this </h2> <p>for each subject domain, we pulled the top 2,000 referring domains by authority from common crawl release Apr-Jun 2026 (cc-main-2026-apr-may-jun). we intersected those lists, then removed platform, cdn, social, and other non-editorial domains from the clean overlap counts.</p> <p>the filter matters. hosting providers, cdns, url shorteners and the big social networks link to almost everything, so leaving them in would produce a universal list that is technically correct and useless for outreach. the counts below are after that filter unless a column says otherwise. everything here comes from the open common crawl webgraph, so you can reproduce it without a paid backlink tool.</p> <p><strong>subject domains</strong></p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>domain</th> <th>referring domains</th> <th>cg authority</th> </tr> </thead> <tbody> <tr> <td>zendesk.com</td> <td>92,417</td> <td>69</td> </tr> <tr> <td>intercom.com</td> <td>22,114</td> <td>37</td> </tr> <tr> <td>helpscout.com</td> <td>9,356</td> <td>60</td> </tr> <tr> <td>freshdesk.com</td> <td>24,015</td> <td>59</td> </tr> <tr> <td>gorgias.com</td> <td>3,165</td> <td>30</td> </tr> <tr> <td>front.com</td> <td>1,401</td> <td>48</td> </tr> <tr> <td>kayako.com</td> <td>2,111</td> <td>30</td> </tr> <tr> <td>zoho.com</td> <td>67,333</td> <td>67</td> </tr> </tbody> </table></div> <h2> the overlap </h2> <p>across the 8 subjects, we found 11,563 unique linking domains. the clean universal set contains 12 domains.</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>metric</th> <th>value</th> </tr> </thead> <tbody> <tr> <td>unique linking domains</td> <td>11,563</td> </tr> <tr> <td>link to one subject</td> <td>81%</td> </tr> <tr> <td>link to all 8</td> <td>12</td> </tr> </tbody> </table></div> <p><strong>overlap distribution</strong></p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>overlap</th> <th>all domains</th> <th>non-platform</th> </tr> </thead> <tbody> <tr> <td>link to all 8</td> <td>12</td> <td>12</td> </tr> <tr> <td>link to 7</td> <td>23</td> <td>21</td> </tr> <tr> <td>link to 6</td> <td>66</td> <td>60</td> </tr> <tr> <td>link to 5</td> <td>108</td> <td>101</td> </tr> <tr> <td>link to 4</td> <td>199</td> <td>196</td> </tr> <tr> <td>link to 3</td> <td>458</td> <td>454</td> </tr> <tr> <td>link to 2</td> <td>1,305</td> <td>1,302</td> </tr> <tr> <td>link to just 1</td> <td>9,392</td> <td>9,386</td> </tr> </tbody> </table></div> <p>the distribution is the interesting part. most linking domains sit in the bottom row: they mention one product and never come back. the rows above it are where outreach lives, because a site that already covers several products in a category has an editorial reason to cover another one. a short universal list usually means the category is covered by a handful of directories, review sites and integration hubs rather than by a broad press base. a long one means the category has real editorial coverage, and the same sites are reachable for you.</p> <p>the highest-authority sites linking to all of them are shopify.com, substack.com and hubspot.com. those are the pages worth reading before you pitch anything, because they already decided this category is worth covering.</p> <p>zendesk.com carries the widest referring-domain profile of the set and front.com the narrowest, which is usually a function of age and press coverage rather than product quality. the gap between them is the practical opportunity: sites that cover the widest profile but not the narrowest are demonstrably willing to write about this category.</p> <h2> the category linkers </h2> <p>93 non-platform domains link to at least 6 of the 8 subjects. these repeat linkers are the sites most consistently connected to help desks.</p> <p><strong>highest-overlap category linkers</strong></p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>domain</th> <th>subjects</th> <th>cg authority</th> <th>links to</th> </tr> </thead> <tbody> <tr> <td>shopify.com</td> <td>8</td> <td>80</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>substack.com</td> <td>8</td> <td>75</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>hubspot.com</td> <td>8</td> <td>70</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>zapier.com</td> <td>8</td> <td>68</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>jotform.com</td> <td>8</td> <td>65</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>webflow.io</td> <td>8</td> <td>62</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>clickup.com</td> <td>8</td> <td>59</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>activecampaign.com</td> <td>8</td> <td>59</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>tidio.com</td> <td>8</td> <td>58</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>zendesk.co.uk</td> <td>8</td> <td>57</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>crisp.chat</td> <td>8</td> <td>55</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>zight.com</td> <td>8</td> <td>52</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>github.com</td> <td>7</td> <td>81</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>salesforce.com</td> <td>7</td> <td>70</td> <td>zendesk.com, intercom.com, helpscout.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>bigcommerce.com</td> <td>7</td> <td>70</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, kayako.com, zoho.com</td> </tr> <tr> <td>sendpulse.com</td> <td>7</td> <td>67</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, kayako.com, zoho.com</td> </tr> <tr> <td>entrepreneur.com</td> <td>7</td> <td>65</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, gorgias.com, kayako.com, zoho.com</td> </tr> <tr> <td>g2.com</td> <td>7</td> <td>64</td> <td>zendesk.com, intercom.com, helpscout.com, gorgias.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>sproutsocial.com</td> <td>7</td> <td>63</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, front.com, kayako.com, zoho.com</td> </tr> <tr> <td>kinsta.com</td> <td>7</td> <td>62</td> <td>zendesk.com, intercom.com, helpscout.com, freshdesk.com, front.com, kayako.com, zoho.com</td> </tr> </tbody> </table></div> <h3> the full data </h3> <p>download all 93 clean domains that link to at least 6 of the 8 subjects, ranked by overlap and authority.</p> <p><a href="https://crawlgraph.com/api/studies/who-links-to-every-help-desk/dataset.csv" rel="noopener noreferrer">download csv</a> | <a href="https://crawlgraph.com/api/studies/who-links-to-every-help-desk/dataset.json" rel="noopener noreferrer">download json</a></p> <h3> check any of these domains </h3> <p>every domain in the tables above links to its full referring-domain profile. run your own domain to see who links to you.</p> <p><a href="https://crawlgraph.com/" rel="noopener noreferrer">look up your backlinks</a></p> <h2> run this for your own niche </h2> <p>pick your closest competitors, compare their referring domains, and keep the sites that link to several of them but not to you. sorting that gap by overlap and authority turns the same method into a specific outreach list.</p> <p>the ordering is what makes it usable. a site that links to most of your competitors and not to you is a gap with an obvious pitch. the observed overlap gives you a specific reason to contact each site instead of relying on domain authority alone. work down from the top of the overlap column and stop when the reasons start feeling thin, because a pitch you cannot justify in one sentence is not worth sending.</p> <p>one caveat worth stating plainly. common crawl is a sample of the web, not a census, so treat these counts as the shape of a link profile rather than an exact total. this study measures the Apr-Jun 2026 release, so rerun the comparison against later releases before treating any movement as a trend.</p> <h3> find the overlap in your market </h3> <p>run a backlink gap analysis on the same common crawl data used for this study.</p> <p><a href="https://crawlgraph.com/gap-analysis" rel="noopener noreferrer">run a free gap analysis</a></p> Petteri Pucilowski Mon, 10 Aug 2026 00:13:38 +0000 https://dev.to/petteri_pucilowski_7ec755/35-domains-link-to-every-major-web-host-308p https://dev.to/petteri_pucilowski_7ec755/35-domains-link-to-every-major-web-host-308p <p>we compared 8 web hosts in common crawl's domain-level link graph. 35 non-platform domains link to all 8, while 72% of linking domains appear for only one subject.</p> <h2> how we pulled this </h2> <p>for each subject domain, we pulled the top 2,000 referring domains by authority from common crawl release Apr-Jun 2026 (cc-main-2026-apr-may-jun). we intersected those lists, then removed platform, cdn, social, and other non-editorial domains from the clean overlap counts.</p> <p>the filter matters. hosting providers, cdns, url shorteners and the big social networks link to almost everything, so leaving them in would produce a universal list that is technically correct and useless for outreach. the counts below are after that filter unless a column says otherwise. everything here comes from the open common crawl webgraph, so you can reproduce it without a paid backlink tool.</p> <p><strong>subject domains</strong></p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>domain</th> <th>referring domains</th> <th>cg authority</th> </tr> </thead> <tbody> <tr> <td>hostinger.com</td> <td>36,900</td> <td>61</td> </tr> <tr> <td>siteground.com</td> <td>13,882</td> <td>60</td> </tr> <tr> <td>bluehost.com</td> <td>48,747</td> <td>61</td> </tr> <tr> <td>dreamhost.com</td> <td>36,470</td> <td>61</td> </tr> <tr> <td>wpengine.com</td> <td>100,000</td> <td>64</td> </tr> <tr> <td>cloudways.com</td> <td>13,795</td> <td>58</td> </tr> <tr> <td>kinsta.com</td> <td>13,963</td> <td>62</td> </tr> <tr> <td>namecheap.com</td> <td>16,701</td> <td>62</td> </tr> </tbody> </table></div> <h2> the overlap </h2> <p>across the 8 subjects, we found 10,077 unique linking domains. the clean universal set contains 35 domains.</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>metric</th> <th>value</th> </tr> </thead> <tbody> <tr> <td>unique linking domains</td> <td>10,077</td> </tr> <tr> <td>link to one subject</td> <td>72%</td> </tr> <tr> <td>link to all 8</td> <td>35</td> </tr> </tbody> </table></div> <p><strong>overlap distribution</strong></p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>overlap</th> <th>all domains</th> <th>non-platform</th> </tr> </thead> <tbody> <tr> <td>link to all 8</td> <td>42</td> <td>35</td> </tr> <tr> <td>link to 7</td> <td>89</td> <td>86</td> </tr> <tr> <td>link to 6</td> <td>133</td> <td>131</td> </tr> <tr> <td>link to 5</td> <td>185</td> <td>182</td> </tr> <tr> <td>link to 4</td> <td>336</td> <td>335</td> </tr> <tr> <td>link to 3</td> <td>614</td> <td>608</td> </tr> <tr> <td>link to 2</td> <td>1,432</td> <td>1,430</td> </tr> <tr> <td>link to just 1</td> <td>7,246</td> <td>7,240</td> </tr> </tbody> </table></div> <p>the distribution is the interesting part. most linking domains sit in the bottom row: they mention one product and never come back. the rows above it are where outreach lives, because a site that already covers several products in a category has an editorial reason to cover another one. a short universal list usually means the category is covered by a handful of directories, review sites and integration hubs rather than by a broad press base. a long one means the category has real editorial coverage, and the same sites are reachable for you.</p> <p>the highest-authority sites linking to all of them are wordpress.org, github.com and shopify.com. those are the pages worth reading before you pitch anything, because they already decided this category is worth covering.</p> <p>wpengine.com carries the widest referring-domain profile of the set and cloudways.com the narrowest, which is usually a function of age and press coverage rather than product quality. the gap between them is the practical opportunity: sites that cover the widest profile but not the narrowest are demonstrably willing to write about this category.</p> <h2> the category linkers </h2> <p>252 non-platform domains link to at least 6 of the 8 subjects. these repeat linkers are the sites most consistently connected to web hosts.</p> <p><strong>highest-overlap category linkers</strong></p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>domain</th> <th>subjects</th> <th>cg authority</th> <th>links to</th> </tr> </thead> <tbody> <tr> <td>wordpress.org</td> <td>8</td> <td>85</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>github.com</td> <td>8</td> <td>81</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>shopify.com</td> <td>8</td> <td>80</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>hubspot.com</td> <td>8</td> <td>70</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>iubenda.com</td> <td>8</td> <td>69</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>sendpulse.com</td> <td>8</td> <td>67</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>habr.com</td> <td>8</td> <td>67</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>smallbiztrends.com</td> <td>8</td> <td>67</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>freecodecamp.org</td> <td>8</td> <td>66</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>entrepreneur.com</td> <td>8</td> <td>65</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>smashingmagazine.com</td> <td>8</td> <td>64</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>letsencrypt.org</td> <td>8</td> <td>63</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>dev.to</td> <td>8</td> <td>63</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>digitalocean.com</td> <td>8</td> <td>62</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>sucuri.net</td> <td>8</td> <td>60</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>getresponse.com</td> <td>8</td> <td>60</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>elementor.com</td> <td>8</td> <td>60</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>searchenginejournal.com</td> <td>8</td> <td>60</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>kit.com</td> <td>8</td> <td>59</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> <tr> <td>freshdesk.com</td> <td>8</td> <td>59</td> <td>hostinger.com, siteground.com, bluehost.com, dreamhost.com, wpengine.com, cloudways.com, kinsta.com, namecheap.com</td> </tr> </tbody> </table></div> <h3> the full data </h3> <p>download all 252 clean domains that link to at least 6 of the 8 subjects, ranked by overlap and authority.</p> <p><a href="https://crawlgraph.com/api/studies/who-links-to-every-web-host/dataset.csv" rel="noopener noreferrer">download csv</a> | <a href="https://crawlgraph.com/api/studies/who-links-to-every-web-host/dataset.json" rel="noopener noreferrer">download json</a></p> <h2> run this for your own niche </h2> <p>pick your closest competitors, compare their referring domains, and keep the sites that link to several of them but not to you. sorting that gap by overlap and authority turns the same method into a specific outreach list.</p> <p>the ordering is what makes it usable. a site that links to most of your competitors and not to you is a gap with an obvious pitch. the observed overlap gives you a specific reason to contact each site instead of relying on domain authority alone. work down from the top of the overlap column and stop when the reasons start feeling thin, because a pitch you cannot justify in one sentence is not worth sending.</p> <p>one caveat worth stating plainly. common crawl is a sample of the web, not a census, so treat these counts as the shape of a link profile rather than an exact total. this study measures the Apr-Jun 2026 release, so rerun the comparison against later releases before treating any movement as a trend.</p> <h3> find the overlap in your market </h3> <p>run a backlink gap analysis on the same common crawl data used for this study.</p> <p><a href="https://crawlgraph.com/gap-analysis" rel="noopener noreferrer">run a free gap analysis</a></p> analysis data seo Petteri Pucilowski Fri, 07 Aug 2026 10:33:32 +0000 https://dev.to/petteri_pucilowski_7ec755/13-domains-link-to-every-major-no-code-tool-5136 https://dev.to/petteri_pucilowski_7ec755/13-domains-link-to-every-major-no-code-tool-5136 <p>we compared 8 no-code tools in common crawl's domain-level link graph. 13 non-platform domains link to all 8, while 76% of linking domains appear for only one subject.</p> <h2> how we pulled this </h2> <p>for each subject domain, we pulled the top 2,000 referring domains by authority from common crawl release Apr-Jun 2026 (cc-main-2026-apr-may-jun). we intersected those lists, then removed platform, cdn, social, and other non-editorial domains from the clean overlap counts.</p> <p>the filter matters. hosting providers, cdns, url shorteners and the big social networks link to almost everything, so leaving them in would produce a universal list that is technically correct and useless for outreach. the counts below are after that filter unless a column says otherwise. everything here comes from the open common crawl webgraph, so you can reproduce it without a paid backlink tool.</p> <p><strong>subject domains</strong></p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>domain</th> <th>referring domains</th> <th>cg authority</th> </tr> </thead> <tbody> <tr> <td>bubble.io</td> <td>14,655</td> <td>56</td> </tr> <tr> <td>webflow.com</td> <td>72,800</td> <td>72</td> </tr> <tr> <td>softr.io</td> <td>1,782</td> <td>29</td> </tr> <tr> <td>glideapps.com</td> <td>2,074</td> <td>30</td> </tr> <tr> <td>adalo.com</td> <td>1,889</td> <td>50</td> </tr> <tr> <td>flutterflow.io</td> <td>1,182</td> <td>44</td> </tr> <tr> <td>zapier.com</td> <td>45,819</td> <td>68</td> </tr> <tr> <td>make.com</td> <td>12,686</td> <td>60</td> </tr> </tbody> </table></div> <h2> the overlap </h2> <p>across the 8 subjects, we found 10,566 unique linking domains. the clean universal set contains 13 domains.</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>metric</th> <th>value</th> </tr> </thead> <tbody> <tr> <td>unique linking domains</td> <td>10,566</td> </tr> <tr> <td>link to one subject</td> <td>76%</td> </tr> <tr> <td>link to all 8</td> <td>13</td> </tr> </tbody> </table></div> <p><strong>overlap distribution</strong></p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>overlap</th> <th>all domains</th> <th>non-platform</th> </tr> </thead> <tbody> <tr> <td>link to all 8</td> <td>13</td> <td>13</td> </tr> <tr> <td>link to 7</td> <td>16</td> <td>15</td> </tr> <tr> <td>link to 6</td> <td>41</td> <td>36</td> </tr> <tr> <td>link to 5</td> <td>97</td> <td>95</td> </tr> <tr> <td>link to 4</td> <td>258</td> <td>251</td> </tr> <tr> <td>link to 3</td> <td>586</td> <td>583</td> </tr> <tr> <td>link to 2</td> <td>1,519</td> <td>1,514</td> </tr> <tr> <td>link to just 1</td> <td>8,036</td> <td>8,030</td> </tr> </tbody> </table></div> <p>the distribution is the interesting part. most linking domains sit in the bottom row: they mention one product and never come back. the rows above it are where outreach lives, because a site that already covers several products in a category has an editorial reason to cover another one. a short universal list usually means the category is covered by a handful of directories, review sites and integration hubs rather than by a broad press base. a long one means the category has real editorial coverage, and the same sites are reachable for you.</p> <p>the highest-authority sites linking to all of them are substack.com, jotform.com and airtable.com. those are the pages worth reading before you pitch anything, because they already decided this category is worth covering.</p> <p>webflow.com carries the widest referring-domain profile of the set and flutterflow.io the narrowest, which is usually a function of age and press coverage rather than product quality. the gap between them is the practical opportunity: sites that cover the widest profile but not the narrowest are demonstrably willing to write about this category.</p> <h2> the category linkers </h2> <p>64 non-platform domains link to at least 6 of the 8 subjects. these repeat linkers are the sites most consistently connected to no-code tools.</p> <p><strong>highest-overlap category linkers</strong></p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>domain</th> <th>subjects</th> <th>cg authority</th> <th>links to</th> </tr> </thead> <tbody> <tr> <td>substack.com</td> <td>8</td> <td>75</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>jotform.com</td> <td>8</td> <td>65</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>airtable.com</td> <td>8</td> <td>64</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>dev.to</td> <td>8</td> <td>63</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>coda.io</td> <td>8</td> <td>59</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>explodingtopics.com</td> <td>8</td> <td>59</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>framer.website</td> <td>8</td> <td>56</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>lovable.dev</td> <td>8</td> <td>53</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>taskade.com</td> <td>8</td> <td>53</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>francenum.gouv.fr</td> <td>8</td> <td>49</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>zite.com</td> <td>8</td> <td>48</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>webcatalog.io</td> <td>8</td> <td>48</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>entireweb.com</td> <td>8</td> <td>47</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>notion.site</td> <td>7</td> <td>72</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, zapier.com, make.com</td> </tr> <tr> <td>habr.com</td> <td>7</td> <td>67</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>webflow.io</td> <td>7</td> <td>62</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>monday.com</td> <td>7</td> <td>61</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, zapier.com, make.com</td> </tr> <tr> <td>formstack.com</td> <td>7</td> <td>59</td> <td>bubble.io, webflow.com, glideapps.com, adalo.com, flutterflow.io, zapier.com, make.com</td> </tr> <tr> <td>clickup.com</td> <td>7</td> <td>59</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, adalo.com, zapier.com, make.com</td> </tr> <tr> <td>beehiiv.com</td> <td>7</td> <td>58</td> <td>bubble.io, webflow.com, softr.io, glideapps.com, flutterflow.io, zapier.com, make.com</td> </tr> </tbody> </table></div> <h3> the full data </h3> <p>download all 64 clean domains that link to at least 6 of the 8 subjects, ranked by overlap and authority.</p> <p><a href="https://crawlgraph.com/api/studies/who-links-to-every-no-code-tool/dataset.csv" rel="noopener noreferrer">download csv</a> | <a href="https://crawlgraph.com/api/studies/who-links-to-every-no-code-tool/dataset.json" rel="noopener noreferrer">download json</a></p> <h2> run this for your own niche </h2> <p>pick your closest competitors, compare their referring domains, and keep the sites that link to several of them but not to you. sorting that gap by overlap and authority turns the same method into a specific outreach list.</p> <p>the ordering is what makes it usable. a site that links to most of your competitors and not to you is a gap with an obvious pitch. the observed overlap gives you a specific reason to contact each site instead of relying on domain authority alone. work down from the top of the overlap column and stop when the reasons start feeling thin, because a pitch you cannot justify in one sentence is not worth sending.</p> <p>one caveat worth stating plainly. common crawl is a sample of the web, not a census, so treat these counts as the shape of a link profile rather than an exact total. this study measures the Apr-Jun 2026 release, so rerun the comparison against later releases before treating any movement as a trend.</p> <h3> find the overlap in your market </h3> <p>run a backlink gap analysis on the same common crawl data used for this study.</p> <p><a href="https://crawlgraph.com/gap-analysis" rel="noopener noreferrer">run a free gap analysis</a></p> analysis data seo Petteri Pucilowski Mon, 06 Jul 2026 08:39:36 +0000 https://dev.to/petteri_pucilowski_7ec755/8-domains-link-to-every-major-fintech-app-common-crawl-analysis-lo5 https://dev.to/petteri_pucilowski_7ec755/8-domains-link-to-every-major-fintech-app-common-crawl-analysis-lo5 <blockquote> <p><strong>Update 2026-08-12:</strong> this post originally treated all eight as one kind of link. They are not. Five of the eight are publishing platforms, which cuts the actionable list from eight to three. Original analysis unchanged below, correction in Update: three categories, not two.</p> </blockquote> <p>Fourth niche, same experiment. Take a category's biggest players, pull every domain that links to them from Common Crawl's open hyperlink graph, keep the ones that link to the whole set. SEO tools gave 37 universal linkers (media). CRMs gave 2 (integration hubs). AI agent frameworks gave 5 (dev platforms). Fintech just gave 8 - and they are the most consistent set yet.</p> <h2> The setup </h2> <p>Eight fintech apps across payments, banking, and spend: Stripe, Wise, Revolut, Plaid, Brex, Ramp, Mercury, Chime. For each, referring domains ranked by authority from the Common Crawl webgraph (120M domains, 4.4B edges), then intersect the eight lists. Stripe is its own planet - 100,000+ referring domains, more than the other seven combined.</p> <h2> The 8 that link to all 8 </h2> <p>substack.com, entrepreneur.com, libsyn.com (podcast host), beehiiv.com, pymnts.com, thefinancialbrand.com, spocket.co, coda.io.</p> <p>Two newsletter platforms, a podcast host, a business magazine, and two dedicated fintech trade publications. Widen to the 7-of-8 club and the pattern gets sharp: github, crunchbase, ycombinator, bvp (Bessemer), contrary, builtin, bankrate, bankingdive, finovate, 11fs. Finance trade press plus startup/VC infrastructure. Almost nothing else.</p> <h2> Update: three categories, not two (2026-08-12) </h2> <p><a href="https://dev.to/mihirkanzariya">@mihirkanzariya</a> pointed out in the comments that these eight are not one kind of thing, and he is right.</p> <p>Five of them are UGC platform domains: <strong>substack.com</strong>, <strong>beehiiv.com</strong>, <strong>libsyn.com</strong>, <strong>coda.io</strong> and <strong>spocket.co</strong>. A link from those is not an editorial decision by the platform. It is one person publishing a newsletter, a podcast episode or a public doc that happens to mention the app. You cannot pitch substack.com and you cannot fill in a profile on beehiiv.com. That link shows up only because somebody already wrote about you.</p> <p>So the universal set splits three ways rather than reading as one list:</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Domain</th> <th>cg authority</th> <th>Category</th> </tr> </thead> <tbody> <tr> <td>entrepreneur.com</td> <td>66</td> <td>press, pitchable</td> </tr> <tr> <td>pymnts.com</td> <td>58</td> <td>trade press, pitchable</td> </tr> <tr> <td>thefinancialbrand.com</td> <td>54</td> <td>trade press, pitchable</td> </tr> <tr> <td>substack.com</td> <td>75</td> <td>platform, earned</td> </tr> <tr> <td>libsyn.com</td> <td>59</td> <td>platform, earned</td> </tr> <tr> <td>beehiiv.com</td> <td>58</td> <td>platform, earned</td> </tr> <tr> <td>coda.io</td> <td>53</td> <td>platform, earned</td> </tr> <tr> <td>spocket.co</td> <td>53</td> <td>platform, earned</td> </tr> </tbody> </table></div> <p>The list you can actually act on is three domains, not eight. The full study's overlap pyramid has a "non-platform" column, and for these five it is wrong: that filter stripped the obvious social platforms and stopped there, when it should have stripped the publishing platforms too. Leaving it as published rather than quietly rewriting it.</p> <p>The more useful half of the correction is what the third category is <em>for</em>. The pitch list and the profile list are things you go and do. The platform cluster is what happens after those work: if Substack and beehiiv start linking to you, the coverage you earned got read by people who then went and wrote their own thing. That makes it the best available proxy for whether the first two are landing, and it is the one part of the set you cannot buy.</p> <p>It also means the effect scales with how much a category gets written about, which puts an asterisk on the table below. SEO tools gave 37 universal linkers, and some of that is likely the same inflation rather than genuinely broader coverage. Re-running the earlier studies with platform domains split out is on the list.</p> <h2> Four niches, four shapes </h2> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Category</th> <th>Link to all 8</th> <th>Who they are</th> </tr> </thead> <tbody> <tr> <td>SEO tools</td> <td>37</td> <td>trade media + roundups</td> </tr> <tr> <td>CRMs</td> <td>2</td> <td>integration hubs</td> </tr> <tr> <td>AI agent frameworks</td> <td>5</td> <td>dev platforms + directories</td> </tr> <tr> <td>Fintech apps</td> <td>8</td> <td>finance press + VC databases</td> </tr> </tbody> </table></div> <p>The takeaway for anyone building in fintech: you do not earn links with a payments integration. You earn them by being <em>covered</em> (get into PYMNTS, The Financial Brand, Banking Dive, Tearsheet) and by being <em>tracked as a company</em> (complete Crunchbase / Sacra / BuiltIn profiles, land on VC portfolio pages). Mercury has 1,689 referring domains vs Stripe's 100,000+, yet both are reached by the same ten publications and three databases. You don't out-link Stripe; you get on the same shelf.</p> <p>Full study with the overlap pyramid and downloadable CSV/JSON: <a href="https://crawlgraph.com/blog/who-links-to-every-fintech-app" rel="noopener noreferrer">8 domains link to every major fintech app</a>. Method is category-agnostic - it's a <a href="https://crawlgraph.com/gap-analysis" rel="noopener noreferrer">backlink gap analysis</a> on Common Crawl data, free.</p> fintech data seo startup Petteri Pucilowski Fri, 03 Jul 2026 15:50:16 +0000 https://dev.to/petteri_pucilowski_7ec755/only-5-domains-link-to-every-major-ai-agent-framework-common-crawl-analysis-1e6a https://dev.to/petteri_pucilowski_7ec755/only-5-domains-link-to-every-major-ai-agent-framework-common-crawl-analysis-1e6a <p>Same experiment, third niche. Take a category's biggest players, pull every domain that links to them from Common Crawl's open hyperlink graph, and keep the ones that link to the whole set. SEO tools gave 37 universal linkers (nearly all media). CRMs gave 2 (both integration hubs). AI agent frameworks just gave 5 - and they are a different animal again.</p> <h2> The setup </h2> <p>Eight frameworks: LangChain, LlamaIndex, CrewAI, Dify, Flowise, Langflow, AutoGPT, SuperAGI. For each, we pulled referring domains ranked by authority from the Common Crawl webgraph (120M domains, 4.4B links), then intersected the eight lists.</p> <h2> The numbers </h2> <ul> <li> <strong>7,508</strong> unique domains link to at least one framework</li> <li> <strong>77%</strong> link to exactly one (usually LangChain: 5,949 referring domains vs ~450 for Langflow and AutoGPT)</li> <li> <strong>5</strong> link to all eight: github.com, substack.com, analyticsvidhya.com, and two product sites</li> </ul> <p>None of the universal linkers are tech press.</p> <h2> What the 7-and-6 club looks like </h2> <p>dev.to (yes, this site), qiita.com, zenn.dev, n8n.io, arize.com, zenml.io, langfuse.com... and then a whole crop of AI agent directories that did not exist two years ago: aiagentsdirectory.com, agenthunter.io, everydev.ai, findmyagentai.com, bestaiagents.ai. Most have single-digit-to-low authority scores. The category's shelf is still being built.</p> <h2> The takeaway for anyone shipping agent tooling </h2> <p>The niche's link graph is developer publishing platforms plus young directories, not journalists. So the play is: publish where developers already read (GitHub, Substack, dev.to, Qiita), and get listed in the agent directories while they are young and take an email. A directory with authority 20 today is not impressive. The one that becomes the geekflare of AI agents in three years is - and you got in early.</p> <p>Full study with methodology, the overlap pyramid, and the downloadable CSV/JSON dataset: <a href="https://crawlgraph.com/blog/who-links-to-every-ai-agent-framework" rel="noopener noreferrer">only 5 domains link to every major AI agent framework</a>. The DuckDB query behind it is <a href="https://gist.github.com/pucilpet/54ad13ecfd6ba3c0c54e920c24064339" rel="noopener noreferrer">a public gist</a>.</p> ai seo data opensource Petteri Pucilowski Fri, 03 Jul 2026 15:49:40 +0000 https://dev.to/petteri_pucilowski_7ec755/we-shipped-a-free-backlink-api-tier-15-callsmonth-on-common-crawl-data-4a7 https://dev.to/petteri_pucilowski_7ec755/we-shipped-a-free-backlink-api-tier-15-callsmonth-on-common-crawl-data-4a7 <p>Until this week, the CrawlGraph API required the $99 lifetime license. Now there is a free tier: <strong>15 backlink calls a month</strong> on Common Crawl's open hyperlink graph (120M domains, 4.4B links, refreshed quarterly). No card, no signup form - the key lands in your inbox.</p> <h2> Two curl commands </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code><span class="c"># 1. get a key (delivered by email)</span> curl <span class="nt">-X</span> POST https://crawlgraph.com/api/v1/free-key <span class="se">\</span> <span class="nt">-H</span> <span class="s2">"Content-Type: application/json"</span> <span class="se">\</span> <span class="nt">-d</span> <span class="s1">'{"email": "you@company.com"}'</span> <span class="c"># 2. look up any domain's backlinks</span> curl <span class="nt">-X</span> POST https://crawlgraph.com/api/v1/backlinks <span class="se">\</span> <span class="nt">-H</span> <span class="s2">"Authorization: Bearer cg_live_..."</span> <span class="se">\</span> <span class="nt">-H</span> <span class="s2">"Content-Type: application/json"</span> <span class="se">\</span> <span class="nt">-d</span> <span class="s1">'{"domain": "example.com", "limit": 10}'</span> </code></pre> </div> <p>Responses carry <code>X-RateLimit-*</code> headers so you always know where you stand. One key per email - delivery by mail is the ownership check, which is why there is no signup flow to speak of.</p> <h2> It works inside your AI agent </h2> <p>The same key powers the open-source <a href="https://github.com/pucilpet/crawlgraph-mcp" rel="noopener noreferrer">crawlgraph-mcp server</a> in Claude, Cursor, Cline, Zed, or Windsurf:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight json"><code><span class="p">{</span><span class="w"> </span><span class="nl">"mcpServers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"crawlgraph"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"npx"</span><span class="p">,</span><span class="w"> </span><span class="nl">"args"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"-y"</span><span class="p">,</span><span class="w"> </span><span class="s2">"crawlgraph-mcp"</span><span class="p">],</span><span class="w"> </span><span class="nl">"env"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"CRAWLGRAPH_API_KEY"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cg_live_..."</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre> </div> <p>Your agent can pull the backlink profile of any domain mid-conversation. The <code>gap_analysis</code> and <code>gap_outreach_targets</code> tools (the ones that turn "research my niche" into a ranked outreach list) stay part of the $99 lifetime tier, which also raises the quota to 1,000 calls + 50 gap analyses a month.</p> <h2> Why free </h2> <p>The underlying data is open - Common Crawl publishes the hyperlink graph quarterly. Gating basic lookups behind a purchase never felt right. Try the data, build something; if you need the gap tools or more volume, the upgrade is a single payment, not a subscription.</p> <p>Docs: <a href="https://crawlgraph.com/docs/api" rel="noopener noreferrer">crawlgraph.com/docs/api</a></p> api seo mcp webdev Petteri Pucilowski Mon, 01 Jun 2026 19:06:11 +0000 https://dev.to/petteri_pucilowski_7ec755/find-your-competitors-backlinks-from-inside-claude-code-free-via-mcp-1a63 https://dev.to/petteri_pucilowski_7ec755/find-your-competitors-backlinks-from-inside-claude-code-free-via-mcp-1a63 <p>Backlink prospecting is a tab-hopping chore: dashboard → export CSV → eyeball → copy domains → switch to email. It's a filter-and-rank problem, which is exactly the kind of thing an agent should do for you. So here's how to do the whole thing from inside Claude Code (or Cursor / Cline / Zed / Windsurf) with one prompt, using a free MCP server.</p> <p>No affiliation required to follow along — the data is the public <a href="https://commoncrawl.org" rel="noopener noreferrer">Common Crawl</a> webgraph, and the MCP wrapper is open source.</p> <h2> What we're building toward </h2> <p>By the end you'll be able to type this into your agent and get back a ranked, de-noised list:</p> <blockquote> <p>"Find the sites linking to competitor-a.com, competitor-b.com and competitor-c.com but not to my-site.com, and draft a short outreach email to each."</p> </blockquote> <p>The agent runs a competitor gap analysis, filters to the highest-value targets, and writes the emails — without you opening a single SEO dashboard.</p> <h2> Step 1: install the MCP server </h2> <p>The server is <a href="https://github.com/pucilpet/crawlgraph-mcp" rel="noopener noreferrer"><code>crawlgraph-mcp</code></a> on npm. It's a thin TypeScript stdio wrapper over a backlink API; nothing to clone or build.</p> <p>Add it to your MCP config (<code>claude_desktop_config.json</code>, or <code>.mcp.json</code> for Claude Code):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight json"><code><span class="p">{</span><span class="w"> </span><span class="nl">"mcpServers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"crawlgraph"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"npx"</span><span class="p">,</span><span class="w"> </span><span class="nl">"args"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"-y"</span><span class="p">,</span><span class="w"> </span><span class="s2">"crawlgraph-mcp"</span><span class="p">],</span><span class="w"> </span><span class="nl">"env"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"CRAWLGRAPH_API_KEY"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cg_live_..."</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre> </div> <p>Grab the <code>cg_live_</code> key from your account page, restart the client, and you'll see four tools appear: <code>backlinks</code>, <code>gap_analysis</code>, <code>gap_outreach_targets</code>, and <code>releases</code>.</p> <h2> Step 2: sanity-check the data </h2> <p>Before trusting it for outreach, point it at a domain you know. In your agent:</p> <blockquote> <p>"Use the backlinks tool on stripe.com, limit 5, sorted by authority."</p> </blockquote> <p>You'll get the top referring domains plus the target's own authority score:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight json"><code><span class="p">{</span><span class="w"> </span><span class="nl">"domain"</span><span class="p">:</span><span class="w"> </span><span class="s2">"stripe.com"</span><span class="p">,</span><span class="w"> </span><span class="nl">"total_linking_domains"</span><span class="p">:</span><span class="w"> </span><span class="mi">100000</span><span class="p">,</span><span class="w"> </span><span class="nl">"cg_authority"</span><span class="p">:</span><span class="w"> </span><span class="mi">78</span><span class="p">,</span><span class="w"> </span><span class="nl">"results"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"linking_domain"</span><span class="p">:</span><span class="w"> </span><span class="s2">"github.com"</span><span class="p">,</span><span class="w"> </span><span class="nl">"num_hosts"</span><span class="p">:</span><span class="w"> </span><span class="mi">2120</span><span class="p">,</span><span class="w"> </span><span class="nl">"cg_authority"</span><span class="p">:</span><span class="w"> </span><span class="mi">96</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"linking_domain"</span><span class="p">:</span><span class="w"> </span><span class="s2">"news.ycombinator.com"</span><span class="p">,</span><span class="w"> </span><span class="nl">"num_hosts"</span><span class="p">:</span><span class="w"> </span><span class="mi">880</span><span class="p">,</span><span class="w"> </span><span class="nl">"cg_authority"</span><span class="p">:</span><span class="w"> </span><span class="mi">94</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre> </div> <p>That <code>total_linking_domains</code> count is the quick gut-check: compare it to whatever your current tool reports for the same domain. If it's in the same ballpark, the graph is complete enough to prospect from.</p> <h2> Step 3: the actual play — gap analysis </h2> <p>The primitive is <code>gap_analysis</code>: give it your domain plus 2-5 competitors, and it returns every domain linking to at least one competitor but not to you, each tagged with which competitors it links to (<code>found_on</code>).</p> <p>The raw output is the <em>material</em>, not the answer — you don't want a 1,000-row dump, you want the few dozen worth emailing. That's what <code>gap_outreach_targets</code> does on top:</p> <ol> <li>keeps only domains in the gap that link to <strong>all</strong> your competitors (a site linking to all three covers your whole niche and just hasn't heard of you — the warmest possible target);</li> <li>strips platform/CDN noise (amazonaws, github.io, facebook, shorteners);</li> <li>ranks the survivors by authority.</li> </ol> <p>So you skip straight to:</p> <blockquote> <p>"Run gap_outreach_targets for my-site.com against competitor-a.com, competitor-b.com and competitor-c.com."</p> </blockquote> <p>and get back something like:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight json"><code><span class="p">{</span><span class="w"> </span><span class="nl">"priority_targets"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"linking_domain"</span><span class="p">:</span><span class="w"> </span><span class="s2">"industry-roundup.com"</span><span class="p">,</span><span class="w"> </span><span class="nl">"found_on"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"competitor-a.com"</span><span class="p">,</span><span class="s2">"competitor-b.com"</span><span class="p">,</span><span class="s2">"competitor-c.com"</span><span class="p">],</span><span class="w"> </span><span class="nl">"cg_authority"</span><span class="p">:</span><span class="w"> </span><span class="mi">73</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"linking_domain"</span><span class="p">:</span><span class="w"> </span><span class="s2">"niche-review.io"</span><span class="p">,</span><span class="w"> </span><span class="nl">"found_on"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"competitor-a.com"</span><span class="p">,</span><span class="s2">"competitor-b.com"</span><span class="p">,</span><span class="s2">"competitor-c.com"</span><span class="p">],</span><span class="w"> </span><span class="nl">"cg_authority"</span><span class="p">:</span><span class="w"> </span><span class="mi">61</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"secondary_targets"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="err">/*</span><span class="w"> </span><span class="err">link</span><span class="w"> </span><span class="err">to</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="err">*/</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"platforms_filtered"</span><span class="p">:</span><span class="w"> </span><span class="mi">26</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre> </div> <h2> Step 4: let the agent write the outreach </h2> <p>This is where doing it in an agent beats a dashboard. The list is already in context, so:</p> <blockquote> <p>"For each priority target, draft a 3-sentence outreach email: reference the kind of content they link to in my niche, and pitch my-site.com as a fit. Keep it specific, no templates."</p> </blockquote> <p>You get first-draft emails per target in the same session. You still edit them in your voice — cold outreach that goes out unedited gets ignored — but the gather → filter → rank → draft chain that used to be an afternoon is now one conversation.</p> <h2> Why 2-3 competitors, not one </h2> <p>A site linking to one competitor might be a fluke or a paid placement. A site linking to three of your competitors is a publisher who covers your category. That overlap is the qualifier — it's the difference between a prospect list and a noise list.</p> <h2> Honest limitations </h2> <ul> <li> <strong>Quarterly snapshot.</strong> Common Crawl publishes ~4×/year, so this is for one-off prospecting, not live link monitoring. For "what changed this week," use a continuous-crawl tool.</li> <li> <strong>No anchor text</strong> in the gap output (the webgraph is source→destination edges).</li> <li> <strong>Quotas:</strong> the free path covers light use; heavier use needs the lifetime tier (1,000 backlink calls + 50 gap jobs/month).</li> </ul> <h2> Why this pattern matters beyond backlinks </h2> <p>The interesting bit isn't the SEO — it's that a well-scoped MCP tool can encode a <em>workflow</em>, not just an API call. <code>gap_outreach_targets</code> doesn't mirror an endpoint; it does the filtering-and-ranking judgment you'd otherwise re-explain to the model every time. If you're building MCP servers, that's the lever: ship the composite tool that returns a decision-ready answer, not just the raw rows.</p> <p>Server's MIT and on <a href="https://github.com/pucilpet/crawlgraph-mcp" rel="noopener noreferrer">GitHub</a> (<code>npx -y crawlgraph-mcp</code>). If you try the gap play, I'd genuinely like to hear whether the priority/secondary split matches what you'd have picked by hand.</p> mcp seo ai tutorial Petteri Pucilowski Sun, 31 May 2026 21:44:12 +0000 https://dev.to/petteri_pucilowski_7ec755/i-wrapped-a-backlink-api-in-an-mcp-server-so-i-could-do-seo-gap-analysis-from-inside-claude-13mk https://dev.to/petteri_pucilowski_7ec755/i-wrapped-a-backlink-api-in-an-mcp-server-so-i-could-do-seo-gap-analysis-from-inside-claude-13mk <p>I do a fair amount of competitor backlink research, and the workflow always annoyed me: open a dashboard, run a query, export a CSV, eyeball it, copy domains into a doc, switch to email. Lots of tab-hopping for what is fundamentally a data-filtering problem an agent should handle.</p> <p>So I wrapped the backlink API I'd been using into an <a href="https://modelcontextprotocol.io" rel="noopener noreferrer">MCP</a> server. Now I stay in Claude Code (or Cursor, Cline, Zed, Windsurf) and just describe the goal. This is the build: the architecture, the four tools, and the one design decision I'm still not sure about.</p> <h2> The data source </h2> <p>The server runs on the <a href="https://commoncrawl.org" rel="noopener noreferrer">Common Crawl</a> hyperlink webgraph — about 4.4 billion edges across 120 million domains, published quarterly as Parquet. That matters for an MCP tool specifically: the data is open, so there's no scraped-proprietary-index liability in handing it to an agent, and the same query is reproducible by anyone.</p> <p>The HTTP API in front of it (<a href="https://crawlgraph.com" rel="noopener noreferrer">CrawlGraph</a>) does the heavy DuckDB work; the MCP server is a thin TypeScript stdio client over it. Keeping the server thin was deliberate — all the query cost, caching, and quota logic lives server-side, so the MCP package stays a ~300-line wrapper that's easy to audit before you hand it your API key.</p> <h2> The four tools </h2> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>backlinks → referring domains for a target, with authority scores gap_analysis → domains linking to your competitors but not to you gap_outreach_targets → the composite play (below) releases → list the Common Crawl snapshots </code></pre> </div> <p><code>backlinks</code> and <code>gap_analysis</code> map 1:1 to API endpoints. <code>gap_analysis</code> is the interesting primitive: submit your domain plus 2-5 competitors, and it returns every domain that links to at least one competitor but not to you, each tagged with a <code>found_on</code> array listing <em>which</em> competitors it links to.</p> <h2> The composite tool, and the decision I'm unsure about </h2> <p>Most API-wrapper MCP servers are pure 1:1 mappings. I added one opinionated composite tool, <code>gap_outreach_targets</code>, because the raw gap output isn't the thing you actually want — it's the raw material for the thing you want.</p> <p>What it does on top of <code>gap_analysis</code>:</p> <ol> <li> <strong>Filters to total overlap.</strong> Keep only domains whose <code>found_on</code> covers <em>every</em> competitor you listed. A site linking to one competitor might be a fluke or a paid placement. A site linking to all three is a publisher who covers your whole niche and has simply never heard of you. That overlap is the qualifier.</li> <li> <strong>Strips platform noise.</strong> <code>amazonaws.com</code>, <code>github.io</code>, <code>facebook.com</code>, CDNs, URL shorteners — they show up in every backlink profile and are never outreach targets. There's a denylist with suffix matching so subdomains get caught too.</li> <li> <strong>Ranks by authority.</strong> For the top N survivors it makes a cheap per-domain authority lookup and sorts, so the highest-value warm targets surface first. This is opt-out (<code>enrich_top: 0</code>) because each lookup costs one API call against quota. </li> </ol> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="c1">// the core filter, roughly</span> <span class="kd">const</span> <span class="nx">priority</span> <span class="o">=</span> <span class="nx">gaps</span> <span class="p">.</span><span class="nf">filter</span><span class="p">(</span><span class="nx">g</span> <span class="o">=&gt;</span> <span class="o">!</span><span class="nf">isPlatformNoise</span><span class="p">(</span><span class="nx">g</span><span class="p">.</span><span class="nx">linking_domain</span><span class="p">))</span> <span class="p">.</span><span class="nf">filter</span><span class="p">(</span><span class="nx">g</span> <span class="o">=&gt;</span> <span class="nx">g</span><span class="p">.</span><span class="nx">found_on</span><span class="p">.</span><span class="nx">length</span> <span class="o">===</span> <span class="nx">competitors</span><span class="p">.</span><span class="nx">length</span><span class="p">)</span> <span class="p">.</span><span class="nf">sort</span><span class="p">((</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">(</span><span class="nx">b</span><span class="p">.</span><span class="nx">cg_authority</span> <span class="o">??</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">cg_authority</span> <span class="o">??</span> <span class="o">-</span><span class="mi">1</span><span class="p">));</span> </code></pre> </div> <p><strong>The decision I keep going back and forth on:</strong> is a composite, opinionated tool the right call for an MCP server, or should it stay a pure API mirror and let the agent do the filtering/ranking in its own reasoning?</p> <p>Arguments for the composite tool:</p> <ul> <li>It encodes a workflow the model would otherwise have to reconstruct each time, costing tokens and inviting mistakes (I watched an agent forget to filter platforms more than once).</li> <li>It returns a small, ranked, decision-ready list instead of a 1,000-row dump the model has to chew through.</li> </ul> <p>Arguments against:</p> <ul> <li>It's a leaky abstraction. The moment someone wants a slightly different filter (2-of-3 overlap, a different noise list), they're fighting my opinion instead of composing primitives.</li> <li>It hides the platform denylist, which is a judgment call that should arguably be visible.</li> </ul> <p>I landed on "ship both" — the raw <code>gap_analysis</code> primitive <em>and</em> the composite — but I'm genuinely unsure that's not just indecision dressed up as flexibility. If you've built MCP servers, I'd like to hear where you draw the primitive-vs-composite line.</p> <h2> Using it </h2> <div class="highlight js-code-highlight"> <pre class="highlight json"><code><span class="p">{</span><span class="w"> </span><span class="nl">"mcpServers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"crawlgraph"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"command"</span><span class="p">:</span><span class="w"> </span><span class="s2">"npx"</span><span class="p">,</span><span class="w"> </span><span class="nl">"args"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"-y"</span><span class="p">,</span><span class="w"> </span><span class="s2">"crawlgraph-mcp"</span><span class="p">],</span><span class="w"> </span><span class="nl">"env"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"CRAWLGRAPH_API_KEY"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cg_live_..."</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre> </div> <p>Then the whole workflow collapses to one sentence:</p> <blockquote> <p>"Use gap_outreach_targets for mydomain.com against competitor-a.com and competitor-b.com, then draft a short outreach email to each priority target."</p> </blockquote> <p>The agent submits the gap job, polls it, filters and ranks, and writes the emails — all in one turn.</p> <h2> Honest limitations </h2> <ul> <li> <strong>Quarterly snapshot.</strong> Common Crawl publishes ~4x/year, so this is for one-off prospecting, not live link monitoring. If you need "what changed this week," it's the wrong tool.</li> <li> <strong>No anchor text in the gap result.</strong> The webgraph is <code>(src, dst)</code> edges; anchor text needs a separate WARC pass I didn't wire into the MCP.</li> <li> <strong>Authority enrichment costs calls.</strong> Each scored domain is one API call, hence the cap.</li> </ul> <p>Code is MIT, on <a href="https://github.com/pucilpet/crawlgraph-mcp" rel="noopener noreferrer">GitHub</a> and npm (<code>npx -y crawlgraph-mcp</code>). Feedback on the composite-tool question especially welcome — it's the part of the design I'm least settled on.</p> mcp ai seo typescript Petteri Pucilowski Fri, 29 May 2026 09:32:31 +0000 https://dev.to/petteri_pucilowski_7ec755/how-i-built-a-free-backlink-intelligence-tool-on-common-crawl-duckdb-367h https://dev.to/petteri_pucilowski_7ec755/how-i-built-a-free-backlink-intelligence-tool-on-common-crawl-duckdb-367h <h2> The problem </h2> <p>Backlink data is a $1.5B/year SaaS category. Ahrefs is $129/month, SEMrush is $140/month, Moz is $99/month. As an indie I needed competitor backlinks for outreach — the prospecting half of what these tools do — but I wasn't going to pay $1,548/year just for a quarterly list of domains.</p> <p>Turns out the data is already public. <a href="https://commoncrawl.org" rel="noopener noreferrer">Common Crawl</a> publishes a hyperlink graph every ~3 months containing every public link they discover. The latest release I pulled has <strong>4.4 billion edges across 120 million domains</strong> — comparable to the size of Ahrefs' index, just refreshed quarterly instead of continuously.</p> <p>This is a walkthrough of the actual stack I used to turn that public dataset into a queryable backlink lookup. Total infra cost: about $40/month for a small VPS.</p> <h2> The data: Common Crawl's webgraph </h2> <p>Common Crawl publishes the webgraph as Parquet on S3:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>s3://commoncrawl/cc-webgraph/ cc-main-2026-jan-feb-mar/ vertices/ # domain registry, with reverse-DNS string keys edges/ # (src_id, dst_id) tuples ranks/ # PageRank-equivalent + harmonic centrality per vertex </code></pre> </div> <p>The full edges table is ~120GB compressed Parquet across a few hundred files. The vertices table is ~3GB. Both are accessible without authentication — Common Crawl publishes everything as open data.</p> <h2> The query engine: DuckDB over Parquet </h2> <p>The trick that makes the whole thing economical: <strong>DuckDB can query Parquet files directly from S3 via <code>httpfs</code></strong> without downloading them. For a backlink lookup ("show me every domain linking to stripe.com"), you don't need the full graph in memory — you need columnar scans for the specific domain ID.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight sql"><code><span class="c1">-- Install the httpfs extension once</span> <span class="n">INSTALL</span> <span class="n">httpfs</span><span class="p">;</span> <span class="k">LOAD</span> <span class="n">httpfs</span><span class="p">;</span> <span class="c1">-- Resolve "stripe.com" to its vertex ID</span> <span class="k">SELECT</span> <span class="n">vertex_id</span> <span class="k">FROM</span> <span class="n">read_parquet</span><span class="p">(</span><span class="s1">'https://data.commoncrawl.org/cc-webgraph/.../vertices/*.parquet'</span><span class="p">)</span> <span class="k">WHERE</span> <span class="n">rev_domain</span> <span class="o">=</span> <span class="s1">'com.stripe'</span><span class="p">;</span> <span class="c1">-- (Common Crawl stores domains reverse-DNS: stripe.com → com.stripe)</span> <span class="c1">-- Pull every incoming edge</span> <span class="k">SELECT</span> <span class="n">src</span><span class="p">.</span><span class="n">rev_domain</span><span class="p">,</span> <span class="n">ranks</span><span class="p">.</span><span class="n">cg_authority</span> <span class="k">FROM</span> <span class="n">read_parquet</span><span class="p">(</span><span class="s1">'https://.../edges/*.parquet'</span><span class="p">)</span> <span class="n">edges</span> <span class="k">JOIN</span> <span class="n">read_parquet</span><span class="p">(</span><span class="s1">'https://.../vertices/*.parquet'</span><span class="p">)</span> <span class="n">src</span> <span class="k">ON</span> <span class="n">edges</span><span class="p">.</span><span class="n">src_id</span> <span class="o">=</span> <span class="n">src</span><span class="p">.</span><span class="n">vertex_id</span> <span class="k">JOIN</span> <span class="n">read_parquet</span><span class="p">(</span><span class="s1">'https://.../ranks/*.parquet'</span><span class="p">)</span> <span class="n">ranks</span> <span class="k">ON</span> <span class="n">edges</span><span class="p">.</span><span class="n">src_id</span> <span class="o">=</span> <span class="n">ranks</span><span class="p">.</span><span class="n">vertex_id</span> <span class="k">WHERE</span> <span class="n">edges</span><span class="p">.</span><span class="n">dst_id</span> <span class="o">=</span> <span class="o">&lt;</span><span class="n">stripe</span><span class="s1">'s vertex_id&gt; ORDER BY ranks.cg_authority DESC LIMIT 1000; </span></code></pre> </div> <p>DuckDB plans the scan, fetches only the relevant Parquet row groups via HTTP range requests, and returns in 2-30 seconds depending on how popular the target domain is.</p> <h2> The cache layer: SQLite </h2> <p>DuckDB-on-S3 is fast enough for ad-hoc queries but you don't want every user hitting Common Crawl's S3 bucket. I added a SQLite cache keyed on <code>(domain, release_id)</code>:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">get_backlinks</span><span class="p">(</span><span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="n">Backlink</span><span class="p">]:</span> <span class="n">cache_key</span> <span class="o">=</span> <span class="p">(</span><span class="n">domain</span><span class="p">,</span> <span class="nf">current_release_id</span><span class="p">())</span> <span class="n">cached</span> <span class="o">=</span> <span class="n">sqlite_db</span><span class="p">.</span><span class="nf">fetch_one</span><span class="p">(</span><span class="sh">"</span><span class="s">SELECT result FROM cache WHERE key = ?</span><span class="sh">"</span><span class="p">,</span> <span class="n">cache_key</span><span class="p">)</span> <span class="k">if</span> <span class="n">cached</span> <span class="ow">and</span> <span class="ow">not</span> <span class="nf">is_expired</span><span class="p">(</span><span class="n">cached</span><span class="p">):</span> <span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="nf">loads</span><span class="p">(</span><span class="n">cached</span><span class="p">[</span><span class="sh">"</span><span class="s">result</span><span class="sh">"</span><span class="p">])</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">duckdb_query_s3</span><span class="p">(</span><span class="n">domain</span><span class="p">)</span> <span class="n">sqlite_db</span><span class="p">.</span><span class="nf">execute</span><span class="p">(</span><span class="sh">"</span><span class="s">INSERT INTO cache (key, result, expires_at) VALUES (?, ?, ?)</span><span class="sh">"</span><span class="p">,</span> <span class="n">cache_key</span><span class="p">,</span> <span class="n">json</span><span class="p">.</span><span class="nf">dumps</span><span class="p">(</span><span class="n">result</span><span class="p">),</span> <span class="nf">now</span><span class="p">()</span> <span class="o">+</span> <span class="nf">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">30</span><span class="p">))</span> <span class="k">return</span> <span class="n">result</span> </code></pre> </div> <p>About 90% of traffic hits the cache after the first month. The DuckDB-on-S3 path stays as the cold-start handler.</p> <h2> The frontend: Next.js with SSR </h2> <p>The user-facing tool is a Next.js 14 app with App Router. The domain pages (<code>/backlinks/&lt;domain&gt;</code>) are server-rendered so they're indexable by Google — which matters because the long-tail SEO traffic ("backlinks for [specific competitor]") is a real channel.</p> <p>Per-domain pages include:</p> <ul> <li>The referring-domain list with their authority scores</li> <li>TLD breakdown (visual)</li> <li>Anchor text distribution (when available from the WARC, separate pipeline)</li> <li>CSV/JSON export</li> </ul> <p>The frontend talks to a thin FastAPI backend that owns the SQLite cache and the DuckDB connection pool.</p> <h2> Honest limitations (what this doesn't replace) </h2> <p>Common Crawl is a <strong>quarterly</strong> snapshot. The current release I'm querying is from January-March 2026, so:</p> <ul> <li> <strong>No live signal monitoring.</strong> When a new high-authority site links you on Thursday, Ahrefs surfaces it by Friday. I won't see it until the next Common Crawl release ingests (~3 months later).</li> <li> <strong>No anchor-text-velocity tracking.</strong> Same reason.</li> <li> <strong>No spam-filtering layer.</strong> Common Crawl publishes the graph as it found it. Ahrefs runs continuous re-validation and de-duplication; I don't replicate that.</li> </ul> <p>So this isn't a replacement for Ahrefs if your job is live rank-impact attribution or campaign monitoring. <strong>It is a replacement for the outreach-prospecting use case</strong> — give me a 50-domain list to pitch today — which is what indies actually need most of the time.</p> <h2> The numbers </h2> <ul> <li>Common Crawl webgraph: free (open data)</li> <li>DuckDB: free</li> <li>SQLite: free</li> <li>VPS hosting (small Hetzner box): ~$40/month</li> <li>Next.js on Vercel: included in hobby tier for now</li> <li>Domain + email: ~$15/month</li> </ul> <p>Total: under $60/month to run a tool that does ~1/30th of what Ahrefs does, for free or $99 lifetime per user. The unit economics are sharp enough that I could make this a public project — which I did.</p> <p>You can try it at <a href="https://crawlgraph.com" rel="noopener noreferrer">crawlgraph.com</a> (5 queries free, no signup). Source data and methodology are documented; if you want to build your own version on the same dataset, all the SQL above is roughly the right shape. Common Crawl's <a href="https://commoncrawl.org/get-started" rel="noopener noreferrer">getting started guide</a> is the place to begin.</p> <p>Happy to answer questions about the architecture in the comments.</p> webdev seo duckdb opensource