DEV Community

Cover image for AI crawlers found your site. Are they sending visitors back?
webdecoy
webdecoy

Posted on

AI crawlers found your site. Are they sending visitors back?

You can fix your sitemap, improve your headings, and make your content easier for machines to read. The next question is practical: are AI systems visiting, and are they sending people back?

We built a 35-second demonstration using our own site's data. I'm on the WebDecoy team; GEOMCP is our open-source audit project.

1. Start with a technical change you can verify

Use GEOMCP from your MCP-compatible editor to audit a public page and inspect the evidence behind each finding. A useful starting prompt is:

Audit https://your-site.example. Show the three most useful fixes and the evidence for each. Recheck the affected pages after the changes.

In our recorded October 17 audit, we checked 193 sitemap URLs and cleared 11 original warnings. One small example was the features page: 16 links, including one empty or generic label, became 16 links with no empty or generic labels.

That is a verified technical improvement. Keep the audit output and the deployment date so you have a concrete change to compare against later.

2. Observe crawlers where their requests arrive

A browser analytics tag only sees visits that execute it. Many crawlers never run JavaScript, so use server or edge observation to see those requests.

In WebDecoy, select your property and open AI Traffic. Choose a time window, then inspect the crawler breakdown alongside the overview. Keep AI-only crawling separate from operators that serve both search and AI.

Our seven-day snapshot showed:

Measurement Observed value
AI-only pages crawled 4,547
Additional mixed search + AI crawling 1,220
AI referral visits 7
Distinct referral landing pages 5

These are recorded counts for webdecoy.com, not a benchmark for every site. “Pages crawled” counts crawling activity; it does not mean 4,547 unique URLs.

3. Follow the return visits

Open Referrals to inspect both the referring platform and the landing page. In this snapshot, ChatGPT, Gemini and Kagi each sent two observed visits; Claude sent one.

The destination matters as much as the total. A tutorial or demo page receiving those visits gives you a specific page to improve: make its explanation clear, keep its example runnable, and give the reader an obvious next action.

Use the page tag or edge sensor to observe referral visits. A crawler request and a referred human visit are separate events. A page can also appear in an AI answer without anyone clicking it, and missing referrer information can leave a visit unattributed.

4. Build a repeatable comparison

Record the deployment date and affected URLs. Compare equal time windows and check the sensor coverage. Look at which pages are crawled, which platforms return visitors, and which pages those visitors land on. Measure your own signup or activation events separately if you want to understand what happens after arrival.

Our audit and these traffic observations do not establish that a technical fix caused a referral. They give us a way to inspect both sides of the loop and decide where to work next.

Try the GEOMCP setup and recorded audit, then connect your site to WebDecoy and inspect AI Traffic → Referrals.

Which part is hardest to measure on your site today: the crawl, the referring platform, or what visitors do after they arrive?

Top comments (0)