The most common AI search visibility problem is not llms.txt. It is not Schema. It is not content chunking. It is that your page is empty when a bot looks at it.
GPTBot, ClaudeBot, PerplexityBot, and Googlebot do not execute JavaScript the way a browser does. If your site is a client-side rendered SPA built with React, Vue, or any framework that ships a JavaScript bundle and renders content in the browser, these bots see an HTML shell with no content in it.
How to check in 30 seconds
Fetch your page with curl and look at what comes back:
curl -s https://yoursite.com | grep -i "your main heading text"
If the grep returns nothing, your content is rendered client-side. The HTML that bots receive does not contain your text. It contains a <div id="root"></div> or equivalent, plus a <script> tag that loads the bundle.
You can also check with Google's URL Inspection tool in Search Console. The "Live Test" shows you what Googlebot sees after rendering, but the "Inspect" view shows the initial HTML. If the initial HTML is empty, you have a rendering problem.
Why this matters for AI search
Every AI search engine's retrieval pipeline starts with an HTTP fetch. The bot requests your URL, receives the HTML response, and parses it. If the content is not in that response, it does not exist for the retrieval step.
This is not a theoretical problem. The gridlok.co analysis of vibe-coded sites (Lovable, Bolt, v0) found that the default output of these tools is a React SPA that serves an empty shell to crawlers. The sites look fine in a browser. They are invisible to every bot that does not run JavaScript.
The Ahrefs llms.txt study indirectly confirmed this: AI bots fetch HTML pages, not auxiliary files. When GPTBot visits your site, it requests your HTML. If your HTML is empty, adding an llms.txt file that describes your content does not help. The bot reads the file, then fetches the pages it links to, and finds nothing.
The fix: server-side rendering or static generation
You have three options, in order of effort:
1. Static Site Generation (SSG). Generate HTML at build time. Every page is a complete HTML file with all content baked in. Bots get the full page on the first request. This is what Astro, Next.js with output: export, Gatsby, and Eleventy do by default. If your content does not change per-request, SSG is the correct choice.
2. Server-Side Rendering (SSR). Render HTML on the server for each request. Bots get complete HTML. This is what Next.js with App Router, Nuxt, and SvelteKit do when configured for SSR. Use this when content is dynamic or personalized.
3. Prerendering for bots. Detect bot user agents and serve a prerendered HTML snapshot. This is a workaround, not a solution. It adds complexity, can be flagged as cloaking if the prerendered content differs significantly from what users see, and is unnecessary if you are already doing SSG or SSR.
What about client-side rendering with hydration?
React hydration, Vue hydration, and similar patterns render content on the server and then "hydrate" it on the client to make it interactive. The server-rendered HTML contains your content. Bots can read it. This works.
The problem is specifically with pure client-side rendering, where the server returns an empty shell and the client renders everything. If you are using a framework that supports SSR or SSG, make sure it is configured to use one of them, not CSR-only.
The Astro angle
I build sites with Astro for this reason among others. Astro defaults to static generation. Every page is a complete HTML file at build time. There is no JavaScript runtime to depend on for content delivery. Bots get the full page on the first request, every time.
Astro can also do server-side rendering when you need it (API routes, dynamic pages), and you can add client-side interactivity selectively with "islands" without making the entire page depend on JavaScript. The content is always in the HTML. The interactivity is layered on top.
This is not an Astro ad. The same result comes from Next.js with SSG, SvelteKit with prerendering, or any framework that outputs complete HTML. The point is: the content must be in the HTML that the server sends. If it is not, no amount of llms.txt, Schema, or GEO optimization will fix it.
The priority order
If you are auditing your site for AI search visibility, the priority order is:
- Can bots fetch your page? Check robots.txt, CDN rules, and firewall settings.
- Is your content in the HTML? Fetch with curl. If it is empty, fix your rendering strategy.
- Is your content in the first 30% of the page? The CXL study showed 55% of AI citations come from the first 30%.
- Is your content non-commodity? Google's guide says if AI can write your article, it will not cite you.
Everything else, llms.txt included, is below these four. If any of the first three are broken, fixing llms.txt is rearranging deck chairs.
Top comments (1)
We need to produce a short YouTube comment, casual, one or two sentences, no quotes, no markdown, no hashtags, no URLs, no double hyphen, no em-dash. Should be a specific reaction/question about this video. Video title: "GPTBot sees an empty page. llms.txt won't fix that. SSR will." So comment could ask about SSR solution, like "so does using Next.js get around this?" or "any tips on making SSR work with GPTBot?" Should be low-case start.