<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Mike</title>
    <description>The latest articles on DEV Community by Mike (@mike-at-ai-primed).</description>
    <link>https://dev.to/mike-at-ai-primed</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4246321%2F5fc02764-979d-4c97-82ab-510828e38055.png</url>
      <title>DEV Community: Mike</title>
      <link>https://dev.to/mike-at-ai-primed</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mike-at-ai-primed"/>
    <language>en</language>
    <item>
      <title>Why AI Search Crawlers Hate Single-Page Apps (and What They Actually See)</title>
      <dc:creator>Mike</dc:creator>
      <pubDate>Mon, 16 Nov 2026 03:32:43 +0000</pubDate>
      <link>https://dev.to/mike-at-ai-primed/why-ai-search-crawlers-hate-single-page-apps-and-what-they-actually-see-108n</link>
      <guid>https://dev.to/mike-at-ai-primed/why-ai-search-crawlers-hate-single-page-apps-and-what-they-actually-see-108n</guid>
      <description>&lt;p&gt;If your web app relies on client-side rendering (CSR)—like a standard React Vite SPA, Vue CLI build, or Angular app without SSR—you might be completely invisible to modern AI search engines and autonomous crawler agents.&lt;/p&gt;

&lt;p&gt;Many developers assume: &lt;em&gt;"Googlebot executes JavaScript, so ChatGPT and Perplexity probably do too."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In reality, autonomous agents and real-time AI retrieval pipelines operate under completely different latency and compute budgets than legacy search indexers.&lt;/p&gt;

&lt;p&gt;Here is why client-side SPAs struggle with AI crawlers, what these bots actually receive, and how to verify what you're serving them.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. The Real-Time Retrieval Problem
&lt;/h2&gt;

&lt;p&gt;When a user asks Perplexity, ChatGPT Search, or Claude a question requiring real-time web context, the engine needs to:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Dispatch queries to multiple URLs in parallel.&lt;/li&gt;
&lt;li&gt;Fetch the page contents.&lt;/li&gt;
&lt;li&gt;Extract clean text, facts, and structure.&lt;/li&gt;
&lt;li&gt;Synthesize the answer and stream it back to the user within &lt;strong&gt;2–3 seconds&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Spinning up a headless Chromium instance, waiting 800ms for your React bundle to download, executing your JavaScript chunks, waiting for client-side API requests to resolve, and capturing the final hydrated DOM is &lt;strong&gt;computationally expensive and too slow&lt;/strong&gt; for interactive search.&lt;/p&gt;

&lt;p&gt;As a result, most AI retrieval bots perform fast, lightweight HTTP GET requests and extract content directly from raw HTML.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. What the Bot Actually Sees
&lt;/h2&gt;

&lt;p&gt;If you ship a client-side SPA, your server returns something like this:&lt;/p&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;
html
&amp;lt;!DOCTYPE html&amp;gt;
&amp;lt;html lang="en"&amp;gt;
  &amp;lt;head&amp;gt;
    &amp;lt;meta charset="UTF-8" /&amp;gt;
    &amp;lt;title&amp;gt;My SaaS Platform&amp;lt;/title&amp;gt;
  &amp;lt;/head&amp;gt;
  &amp;lt;body&amp;gt;
    &amp;lt;div id="root"&amp;gt;&amp;lt;/div&amp;gt;
    &amp;lt;script type="module" src="/assets/index-C8a12x.js"&amp;gt;&amp;lt;/script&amp;gt;
  &amp;lt;/body&amp;gt;
&amp;lt;/html&amp;gt;


Some teams try to patch this by adding rich meta tags (og:description, description) via packages like react-helmet on the client.

However, if those tags are injected into the &amp;lt;head&amp;gt; after JavaScript mounts, the initial raw HTTP response still lacks them! The bot reads an empty &amp;lt;head&amp;gt; or default template tags before hydration ever runs.

You don't need fancy tools to run a baseline check. Open your terminal and run:


curl -A "Mozilla/5.0 (compatible; PerplexityBot/1.0)" -L https://yourdomain.com


Look at the output:


Can you read your actual value proposition, pricing, and feature headings directly in the terminal output?


Or do you just see script bundles and an empty container &amp;lt;div id="root"&amp;gt;&amp;lt;/div&amp;gt;?


If you see an empty container, search crawlers that don't execute JS are getting zero context from your domain.


Have You Tested Your Domain?

We built a free diagnostic scanner over at AI-Primed that analyzes how edge bots, AI search crawlers, and LLM parsers extract your site's data (including WAF rules, robots directives, and machine readability).

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>frontend</category>
      <category>javascript</category>
      <category>seo</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Google isn't the only Front Door anymore</title>
      <dc:creator>Mike</dc:creator>
      <pubDate>Sun, 15 Nov 2026 17:54:51 +0000</pubDate>
      <link>https://dev.to/mike-at-ai-primed/google-isnt-the-only-front-door-anymore-59il</link>
      <guid>https://dev.to/mike-at-ai-primed/google-isnt-the-only-front-door-anymore-59il</guid>
      <description>&lt;p&gt;Hey everyone! 👋&lt;/p&gt;

&lt;p&gt;Excited to join the community. I’m currently focused on how modern search engines and AI agents (ChatGPT, Perplexity, Claude) parse web apps and sites. Working on AI-Primed, a diagnostic scanner that checks site extractability, WAF blockers, and structured markup for AI crawlers.&lt;/p&gt;

&lt;p&gt;Over the past year, nearly every major tech team has been asking the same question: &lt;strong&gt;"Why isn't our product showing up in AI search answers?"&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;When companies don't get cited by tools like Perplexity, ChatGPT Search, or Claude, the default assumption is usually algorithmic:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;"Our content isn't authoritative enough."&lt;/li&gt;
&lt;li&gt;"Our domain authority is too low."&lt;/li&gt;
&lt;li&gt;"Our keywords aren't optimized for semantic embeddings."&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;While content quality certainly matters, running thousands of site diagnostics reveals that the real culprit is often much simpler and much more frustrating:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The AI crawlers never made it past the front door.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here is why web applications are silently blocking autonomous search engines without their engineering teams realizing it, and the high-level checkpoints every team should review.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. The CDN Trap: WAFs That Treat New Crawlers Like Attacks
&lt;/h2&gt;

&lt;p&gt;Most production web applications sit behind an edge layer or Web Application Firewall (Cloudflare, AWS CloudFront, Fastly, Akamai).&lt;/p&gt;

&lt;p&gt;To protect against DDoS attacks and scrapers, default security templates apply strict bot management:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;They challenge non-standard User-Agents with an interactive CAPTCHA (e.g., Turnstile or hCaptcha).&lt;/li&gt;
&lt;li&gt;They rate-limit HTTP clients that lack complete desktop browser TLS fingerprints.&lt;/li&gt;
&lt;li&gt;They silently drop requests from autonomous data center IP ranges.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;While this blocks bad actors, &lt;strong&gt;search agents aren't human browsers&lt;/strong&gt;. When an autonomous model fetches a page to synthesize a live answer, it cannot solve a visual challenge or run a complex JavaScript puzzle.&lt;/p&gt;

&lt;p&gt;If your CDN serves an HTTP 403 or an interactive challenge to AI user agents, the model gets zero text back and moves on to your competitor’s site.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. Legacy &lt;code&gt;robots.txt&lt;/code&gt; Rules That Stale-Dated
&lt;/h2&gt;

&lt;p&gt;A few years ago, many engineering teams added blanket disallow rules to prevent aggressive training scrapers from vacuuming their code repositories or blogs.&lt;/p&gt;

&lt;p&gt;The problem? Many teams haven't audited those rules since search and extraction split into separate operations.&lt;/p&gt;

&lt;p&gt;Today, there is an important distinction between:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Training scrapers&lt;/strong&gt; (crawling the web for offline model pre-training).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Search &amp;amp; retrieval agents&lt;/strong&gt; (real-time lookups to cite your brand when an end-user asks a live question).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If your &lt;code&gt;robots.txt&lt;/code&gt; uses catch-all wildcards without explicit definitions for modern search tokens, you may be telling real-time answer engines that your site is strictly off-limits.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. The Single-Page App (SPA) Content Blank
&lt;/h2&gt;

&lt;p&gt;Search engines have supported JavaScript rendering for standard SEO for years. But &lt;strong&gt;real-time AI retrieval operates under strict latency budgets&lt;/strong&gt; (often sub-second).&lt;/p&gt;

&lt;p&gt;When an AI engine performs a live web lookup to answer a user prompt, it doesn't always spin up a full headless Chromium browser to execute client-side state hydration.&lt;/p&gt;

&lt;p&gt;We built AI-Primed as a diagnostic scanner specifically to test how modern crawlers and search agents experience production domains—checking edge access, extraction barriers, and structural markup.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>seo</category>
      <category>chatgpt</category>
    </item>
    <item>
      <title>We Scanned 370k+ Websites for AI Readiness: Here’s Why Most Fail (and How to Fix It)</title>
      <dc:creator>Mike</dc:creator>
      <pubDate>Sun, 15 Nov 2026 17:33:43 +0000</pubDate>
      <link>https://dev.to/mike-at-ai-primed/we-scanned-370k-websites-for-ai-readiness-heres-why-most-fail-and-how-to-fix-it-il4</link>
      <guid>https://dev.to/mike-at-ai-primed/we-scanned-370k-websites-for-ai-readiness-heres-why-most-fail-and-how-to-fix-it-il4</guid>
      <description>&lt;p&gt;As search behavior shifts from traditional Google 10-blue-links toward AI answer engines (Perplexity, ChatGPT Search, Claude), how autonomous agents read web pages has become a critical technical consideration.&lt;/p&gt;

&lt;p&gt;Most AI agents and crawlers (such as &lt;code&gt;PerplexityBot&lt;/code&gt;, &lt;code&gt;ChatGPT-User&lt;/code&gt;, or &lt;code&gt;ClaudeBot&lt;/code&gt;) do not browse like desktop Chrome. They optimize for cost, latency, and token efficiency—which introduces failure modes that standard Lighthouse or SEO audits completely miss.&lt;/p&gt;

&lt;p&gt;After analyzing over 370,000 domains with &lt;a href="https://ai-primed.com" rel="noopener noreferrer"&gt;AI-Primed&lt;/a&gt;, here are the three most common technical reasons websites silently fail when AI agents visit.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. The Headless JS Trap (Blank HTML Shells)
&lt;/h2&gt;

&lt;p&gt;Many modern SPAs rely on client-side hydration. While Googlebot has a headless rendering queue that eventually evaluates JavaScript, most AI retrieval agents and retrieval-augmented generation (RAG) pipelines perform fast HTTP fetches without executing JavaScript.&lt;/p&gt;

&lt;p&gt;If your critical content, pricing, or product descriptions require React/Vue/Angular hydration to appear in the DOM:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The crawler receives an empty &lt;code&gt;&amp;lt;div id="root"&amp;gt;&amp;lt;/div&amp;gt;&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The assistant falls back to hallucination or reports that your site has no relevant information.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;The Fix:&lt;/strong&gt; Ensure primary identity, value propositions, and key documentation are present in the server-rendered HTML or static build output.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. Firewall &amp;amp; WAF False Positives
&lt;/h2&gt;

&lt;p&gt;Cloudflare, AWS WAF, and Akamai frequently classify AI bot user-agents under generic "automated scraper" or "bot mitigation" rules.&lt;/p&gt;

&lt;p&gt;We routinely observe sites that pass normal browser checks but return an immediate &lt;code&gt;403 Forbidden&lt;/code&gt; or a Cloudflare Turnstile / CAPTCHA challenge when an AI user-agent queries them. Because the agent cannot solve challenges, it drops the domain entirely.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Fix:&lt;/strong&gt; Check your WAF rules and &lt;code&gt;robots.txt&lt;/code&gt;. If you want search engines and answer assistants to cite your site, verify that designated bots (&lt;code&gt;ChatGPT-User&lt;/code&gt;, &lt;code&gt;PerplexityBot&lt;/code&gt;, &lt;code&gt;ClaudeBot&lt;/code&gt;) are explicitly allowed through your security layer.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Lack of Structured Machine Endpoints (&lt;code&gt;llms.txt&lt;/code&gt;)
&lt;/h2&gt;

&lt;p&gt;LLMs have finite context windows and high inference costs. When an agent has to parse complex multi-megabyte HTML full of navigation headers, SVG icons, and tracking scripts, the relevant signal is often lost.&lt;/p&gt;

&lt;p&gt;The emerging standard to solve this is &lt;code&gt;/llms.txt&lt;/code&gt;—a markdown-formatted file at your domain root that cleanly outlines what your project does, along with direct links to documentation or key resources.&lt;/p&gt;




&lt;h2&gt;
  
  
  Check Your Domain
&lt;/h2&gt;

&lt;p&gt;We built &lt;a href="https://ai-primed.com" rel="noopener noreferrer"&gt;AI-Primed&lt;/a&gt; as a free diagnostic engine to help developers inspect their site through an AI agent's eyes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tests actual bot reachability and firewall headers&lt;/li&gt;
&lt;li&gt;Inspects semantic markup and JSON-LD structured data&lt;/li&gt;
&lt;li&gt;Evaluates raw HTML extractability without client-side script execution&lt;/li&gt;
&lt;li&gt;Includes a free &lt;a href="https://ai-primed.com/tools/llms-txt-generator" rel="noopener noreferrer"&gt;&lt;code&gt;llms.txt&lt;/code&gt; generator&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Run your domain through the scanner and let me know in the comments what score you got or if you hit unexpected edge cases!&lt;/p&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>seo</category>
      <category>webdev</category>
    </item>
  </channel>
</rss>
