JavaScript-injected navigation can create a material discovery gap between search crawlers and AI-focused bots. In a 41-day field experiment, pages linked only through JavaScript were not discovered by GPTBot, ClaudeBot, PerplexityBot, or other AI crawlers observed in the test. Googlebot could follow JavaScript-generated links, but its coverage declined at greater crawl depths.
The experiment, conducted by SEO engineer Vinicius Stanula, is a practical warning for organizations that want content to remain accessible across conventional search and emerging AI discovery channels. A link that is absent from the raw HTML may still work perfectly for people using a browser, yet be effectively invisible to crawlers that do not render JavaScript. Stanula documented the setup and findings in his field experiment on LinkedIn.
What the crawl experiment found
The test site contained roughly 1,000 pages spread across 21 top-level categories. Eleven categories used standard HTML anchor links visible in the page source. The other 10 relied on links injected with JavaScript, which were not present in the initial HTML response.
Across 30,180 requests from 27 distinct crawlers, the split was clear. AI-oriented bots did not find hierarchy pages available only through JavaScript links. Googlebot and GoogleOther did reach JavaScript-linked content, but their traversal was not equivalent to the HTML-linked side of the site. Coverage weakened at deeper levels in the hierarchy.
| Crawler group | Discovery of JavaScript-only hierarchy pages | What the experiment indicates |
|---|---|---|
| GPTBot, ClaudeBot, PerplexityBot, and other AI-focused crawlers | No pages discovered | These bots appeared to rely on raw HTML rather than JavaScript rendering for link discovery. |
| Googlebot and GoogleOther | Pages were reached, with depth-limited coverage | Google's rendering infrastructure can process JavaScript-generated links, but discovery can still degrade deeper in a site structure. |
| After links switched to plain HTML | GPTBot found 250 new pages in 48 hours; Bingbot added 89; Google added none | Making links available in source HTML immediately changed discovery for bots that had missed the JavaScript-linked paths. |
The post-switch result is particularly useful because it tests more than a theoretical rendering difference. Once all links became plain HTML, GPTBot rapidly found 250 new pages in the first 48 hours of that period. Bingbot added 89 pages. Google added no new pages, consistent with its earlier ability to reach the JavaScript-linked areas.
Why raw HTML still matters
Modern JavaScript frameworks can deliver fast, polished user experiences, but client-side navigation should not be confused with crawler-accessible navigation. Bots generally start with the HTTP response and inspect its HTML. If an internal link is added only after a browser executes a script, a non-rendering crawler has no route to follow.
For AI crawlers, that gap has an important consequence: content may be publicly viewable but unavailable for crawler discovery through the site's internal architecture. That does not establish whether any individual AI system will use, cite, or train on a page. It does show that pages inaccessible to a crawler cannot be discovered through those JavaScript-only paths in the observed test.
Crawlability is now a governance issue
Enterprise teams increasingly manage several overlapping concerns: organic search visibility, documentation accessibility, AI assistant discovery, and controls over automated collection. The experiment suggests these should be treated as connected architecture and governance questions rather than separate marketing tasks.
A sensible policy begins by deciding which content should be discoverable and by whom. Organizations may have legitimate reasons to restrict automated access, but accidental exclusion is different from deliberate control. Teams should document where navigation is client-rendered, how critical pages are linked, and whether crawler behavior aligns with the intended policy.
For high-value public content, practical safeguards include:
- Use standard HTML anchors for essential category, product, documentation, and editorial pathways.
- Inspect delivered HTML, not only the browser-rendered page, to confirm important links are present before scripts run.
- Test crawl paths at depth, because a crawler reaching a top-level page does not guarantee full discovery of deeper content.
- Separate access policy from implementation accidents by making explicit decisions about which crawlers may access public material.
- Monitor server logs to identify which bots request pages and whether their behavior changes after technical releases.
Implications for CMS and SEO teams
The strongest lesson is not that JavaScript should be removed from websites. It is that critical internal linking needs a crawlable baseline. Server-rendered HTML, static generation, or other approaches that expose anchors in the initial response can preserve a JavaScript-enhanced experience while giving crawlers usable navigation.
This matters most for sites with large content estates. Documentation hubs, knowledge bases, ecommerce category trees, and enterprise resource centers often depend on hierarchy pages to expose large numbers of URLs. If those hierarchy links are injected only in the browser, AI bots may not discover the pages at all, while Google may discover an incomplete subset at depth.
SEO teams should also avoid treating Googlebot's behavior as a universal proxy for every crawler. The experiment demonstrates why a site can appear technically healthy in Google Search while remaining difficult for AI-focused bots to traverse. Different crawlers have different rendering capabilities, crawl budgets, and operational choices.
For businesses evaluating their presence in AI-generated answers, the immediate task is to establish a measurable baseline. Scalevise helps teams connect technical crawlability with AI discovery strategy through its AI Visibility and GEO Checker. A focused review can identify whether key pages are exposed in raw HTML, where internal-link paths break down, and which fixes deserve priority before visibility gaps affect important content. Start an AI Visibility scan.
Frequently Asked Questions
What did the 41-day JavaScript link test show?
The experiment found that GPTBot, ClaudeBot, PerplexityBot, and other AI-focused crawlers observed in the test did not discover hierarchy pages linked only through JavaScript. Googlebot and GoogleOther reached those pages, but coverage declined at deeper levels.
Why can JavaScript-only links be invisible to AI crawlers?
The observed AI crawlers appeared to fetch raw HTML without executing JavaScript for link discovery. If an anchor is not present in the initial HTML, those bots have no visible internal path to follow.
Did Googlebot crawl JavaScript-linked pages in the test?
Yes. Googlebot and GoogleOther followed JavaScript-generated links, but the experiment reported significant loss of coverage at greater crawl depths compared with plain HTML links.
What changed when the site switched links back to HTML?
GPTBot discovered 250 new pages within 48 hours of the switch, and Bingbot added 89 pages. Google added no new pages, reflecting its prior ability to reach the JavaScript-linked content.
What should CMS teams do with this finding?
CMS teams should ensure essential internal links are available as standard HTML anchors in the initial response, inspect source HTML, test deep crawl paths, and use logs to validate crawler access.
Conclusion
Stanula's experiment shows that JavaScript-only navigation can divide a site's discoverability by crawler type. Googlebot's ability to render JavaScript does not mean AI-focused bots can follow the same paths. For organizations that value reliable search and AI visibility, exposing critical links in raw HTML is a practical technical baseline, alongside deliberate policies for automated access.
Top comments (0)