The latest articles on DEV Community by li huo (@lihuodev). https://dev.to/lihuodev https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4185092%2F79a6ed18-c283-414a-ab02-74244a788a19.png https://dev.to/lihuodev en li huo Wed, 14 Oct 2026 16:15:16 +0000 https://dev.to/lihuodev/i-built-a-free-crawler-audit-tool-is-your-site-invisible-to-chatgpt-claude-and-gemini-37bj https://dev.to/lihuodev/i-built-a-free-crawler-audit-tool-is-your-site-invisible-to-chatgpt-claude-and-gemini-37bj <p>ChatGPT, Claude, Gemini and Perplexity are quietly becoming a second traffic source for many websites. But before any of them can cite you, their crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider — need to actually fetch your pages.</p> <p>I built <a href="https://crawlcheck.dev" rel="noopener noreferrer">CrawlCheck</a> to answer one question in under 30 seconds: <strong>can AI crawlers see your site, or are you accidentally blocking them?</strong> It is free and needs no signup.</p> <h2> Why sites get blocked without knowing it </h2> <p>While testing the tool on real sites, three failure modes came up again and again:</p> <ol> <li> <strong>Stale CDN firewall rules.</strong> A WAF rule written years ago to stop scrapers now matches the user-agent strings that legitimate AI crawlers use. The site owner has no idea.</li> <li> <strong>Copy-pasted robots.txt.</strong> Plenty of templates floating around the web include a blanket "User-agent: * disallow-all" block or a leftover CCBot ban from the "AI panic" era.</li> <li> <strong>JS-only content.</strong> Some AI crawlers do not render JavaScript. If your key content only exists after hydration, they see an empty shell.</li> </ol> <h2> What the audit checks </h2> <ul> <li>robots.txt directives for every major AI agent</li> <li>meta robots tags and X-Robots-Tag headers</li> <li>rendered DOM vs raw HTML (to catch JS-only content)</li> <li>response codes and redirect chains</li> </ul> <h2> Per-crawler reference pages </h2> <p>We documented 55 of the most common AI and search crawlers — what each one does, which pages it hits, and how to allow or block it. For example: <a href="https://crawlcheck.dev/crawlers/gptbot/" rel="noopener noreferrer">GPTBot</a>, <a href="https://crawlcheck.dev/crawlers/claudebot/" rel="noopener noreferrer">ClaudeBot</a> and <a href="https://crawlcheck.dev/crawlers/ccbot/" rel="noopener noreferrer">CCBot</a>.</p> <p>Try the audit yourself: <a href="https://crawlcheck.dev" rel="noopener noreferrer">https://crawlcheck.dev</a> — paste a URL and get the full report in seconds.</p> <p>Feedback is welcome — which other checks would be useful to you?</p> ai webdev seo productivity