The latest articles on DEV Community by S. Trent (@hubdev). https://dev.to/hubdev https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4243547%2F84ea3403-7126-412b-96dd-e6c1c1ee655c.png https://dev.to/hubdev en S. Trent Fri, 13 Nov 2026 20:14:35 +0000 https://dev.to/hubdev/three-quick-checks-before-you-expect-ai-search-to-quote-your-page-1le2 https://dev.to/hubdev/three-quick-checks-before-you-expect-ai-search-to-quote-your-page-1le2 <p>I've been building small Claude/Cursor skills that review pages for AI-search readiness. Running them on public pages kept turning up the same three gaps. You can check all three yourself in about ten minutes with curl and view-source. You don't need a tool.</p> <h2> 1. Does robots.txt say anything about AI crawlers? </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>curl -s https://yoursite.com/robots.txt <span class="o">|</span> grep -iE <span class="s1">'gptbot|claudebot|perplexitybot|google-extended|ccbot'</span> </code></pre> </div> <p>No output means you have no AI-specific rules, so these bots follow your <code>User-agent: *</code> group. That's fine if it's on purpose. The problems I see are leftovers: a blanket <code>Disallow</code> someone added years ago, or a CDN/WAF bot rule that blocks crawlers the robots.txt says are allowed. Check both.</p> <h2> 2. Do you have an llms.txt, and is it accurate? </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>curl -s -o /dev/null -w <span class="s1">'%{http_code}\n'</span> https://yoursite.com/llms.txt </code></pre> </div> <p>llms.txt is a proposed convention, not a standard, and no AI company has promised to read it. It's still cheap to add. Put a one-line summary of what you do at the top, then short sections linking your key pages. When I checked on Nov 13, stripe.com returned 200 with exactly that layout, and anthropic.com returned 404. If you add one, keep it current. A stale llms.txt is worse than none.</p> <h2> 3. Does your structured data agree with the visible page? </h2> <p>Open view-source, search for <code>application/ld+json</code>, and compare it with what a reader sees:</p> <ul> <li> <strong>author</strong>: is it null or missing while the page names a person or company?</li> <li> <strong>dateModified</strong>: the page says "Last updated March 2026", but is that date in the JSON-LD?</li> <li> <strong>headline</strong>: does it match the H1, or carry an extra "| Brand" suffix?</li> </ul> <p>On one well-known payments explainer, all three were off: author was null, dateModified was missing even though the page showed a date, and the headline had a brand suffix. These take minutes to fix. Don't add FAQ or rating markup the page doesn't actually show.</p> <h2> What these checks won't tell you </h2> <p>None of this guarantees a citation. Nobody outside the AI companies knows exactly how sources get chosen. These checks only remove the cheap reasons a page might be skipped or misread. Whether people are asking the question your page answers, and whether you answer it well in the first paragraph, matters more.</p> <p><em>Disclosure: I make HubDev, a set of Claude/Cursor skills that run checks like these. If you'd rather have an agent score a page for you, there's a free starter scorecard (no purchase needed): <a href="https://payhip.com/b/TvxJH" rel="noopener noreferrer">https://payhip.com/b/TvxJH</a>. Paid packs exist, but nothing in this post needs them.</em></p> seo webdev ai tutorial