DEV Community

AURA GEO
AURA GEO

Posted on

Can AI Crawlers See Your JavaScript? A 10-Minute curl Test for the HTML Bots Actually Get

#seo #webdev #ai #javascript

Your page looks perfect in Chrome. Headline, pricing, FAQ, a nice comparison table. But an AI crawler doesn't open Chrome. It sends one HTTP request, gets back whatever your server returns, and moves on.

If your important text only appears after JavaScript runs, there's a real chance that crawler saw an empty shell. This post shows how to check that in about ten minutes with curl and a couple of one-liners.

Why this matters for AI search

There are two versions of every page:

  1. The initial HTML - what the server sends in response to a request.
  2. The rendered page - what you see after scripts run, APIs return data and widgets load.

Google documents a separate rendering stage for JavaScript (JavaScript SEO basics). Other clients don't have to copy Google's capabilities. In December 2024, Vercel and MERJ published an analysis of AI crawler traffic on Vercel's network (The rise of the AI crawler). They found that OpenAI's crawlers, ClaudeBot, PerplexityBot and several others fetched JavaScript files but didn't execute them, while Gemini (via Google's infrastructure) and Applebot did render pages. Crawler behavior can change over time, so treat this as a reason to test, not as a permanent rule.

The safe assumption: if a fact isn't in the initial HTML, don't count on an AI crawler seeing it.

Step 1: Pick three pages, not your whole site

Don't start with a full crawl. Choose:

  • a service or pricing page,
  • one article or guide,
  • one product or listing page.

For each one, write down 3-5 facts that must be visible: the H1, the main description, a price or condition, a key claim, and the important internal links.

Step 2: Fetch the raw HTML 

URL="https://example.com/pricing"

curl -sL "$URL" -o raw.html
wc -c raw.html
Enter fullscreen mode Exit fullscreen mode

Ignore the file size. A 300 KB response can be all navigation and inline JSON; a 20 KB response can contain the full answer. What matters is meaning.

To see the response a specific crawler would get, you can also send its user agent string (useful when a CDN or WAF treats bots differently):

curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  -o raw_gptbot.html -w "%{http_code}\n" "$URL"
Enter fullscreen mode Exit fullscreen mode

If this returns 403, 429 or a challenge page while a normal request returns 200, you have a different problem: your firewall, not your rendering. Fix that first.

Step 3: Search for your must-have facts 

for phrase in "Pricing" "per month" "Free trial" "How it works"; do
  if grep -qi "$phrase" raw.html; then
    echo "FOUND:   $phrase"
  else
    echo "MISSING: $phrase"
  fi
done
Enter fullscreen mode Exit fullscreen mode

A match isn't the whole story. Text sitting inside a JSON blob in a <script> tag (for example __NEXT_DATA__ or a hydration payload) is technically "in the HTML," but a normal content extractor may not treat it as the main content. (The Vercel study notes that models can sometimes interpret JSON in the initial response, but you shouldn't rely on it.) Check where the match lives:

grep -o -i ".\{80\}per month.\{80\}" raw.html | head
Enter fullscreen mode Exit fullscreen mode

If every hit is inside a script, mark it as "present in data, not in markup."

Step 4: Strip the markup and read it like a bot

A quick way to see the visible text without a browser:

python3 - <<'PY'
import re, html
t = open("raw.html", encoding="utf-8", errors="ignore").read()
t = re.sub(r"(?is)<(script|style|noscript|svg)[^>]*>.*?</\1>", " ", t)
t = re.sub(r"(?s)<[^>]+>", " ", t)
t = re.sub(r"\s+", " ", html.unescape(t)).strip()
print(len(t.split()), "words")
print(t[:1500])
PY
Enter fullscreen mode Exit fullscreen mode

Read the first 1,500 characters. Does it say what the page is about and what you offer? Or is it "Loading... Menu Sign in Cookie settings"?

Step 5: Compare with the rendered page

Open the same URL in your browser, open DevTools and compare the rendered DOM with raw.html. Note which important elements only exist after load: tabs, accordions filled by API calls, pricing tables from a widget, reviews from a third-party script.

Also test direct URL access. Single-page apps sometimes work when you click from the homepage but return a generic shell (or a 404) when the route is requested directly. Crawlers always request directly.

Step 6: Fix delivery, not the content

If the useful text is missing from the initial HTML, the usual options are:

  • Server-side rendering (SSR) for pages that change often.
  • Static generation / prerendering for pages that change rarely.
  • Moving critical facts into the server-rendered template (H1, intro, price, key FAQ answers) while keeping interactive extras client-side.

Don't build a separate "bot version" with different claims. Visitors and crawlers should get the same meaning; serving different content to bots is the kind of thing that backfires.

Step 7: Save before and after

When the fix ships, re-run the same curl and the same checks. Keep both files:

curl -sL "$URL" -o "after_$(date +%F).html"
Enter fullscreen mode Exit fullscreen mode

A fix is done when the main text is present in a direct request, the page still works for humans, and the required links are there. That removes a technical barrier. It doesn't make the page relevant to every question or guarantee a citation, so track discovery and citations separately.

A quick checklist

  • [ ] Three representative pages chosen
  • [ ] Must-have facts listed per page
  • [ ] Raw HTML fetched (normal UA and crawler UA)
  • [ ] Status codes match between the two
  • [ ] Facts found in markup, not only in script data
  • [ ] Direct URL access tested for app routes
  • [ ] Fix verified with a before/after snapshot

If you want this check done automatically, AURA GEO audits the HTML your server actually returns, flags pages where little readable text comes through, and turns that into tasks you can hand to your own agent over MCP. There's a longer explainer here: how to tell whether a crawler can see the text on a JavaScript website.

Top comments (2)

Collapse
 
alexcodebytes profile image
Oleksandr •

"A 300 KB response can be all navigation and inline JSON; a 20 KB response can contain the full answer" is the most counterintuitive part of this article.

Most developers assume that bigger HTML = more content. But if the content is inside a <script> tag as a hydration payload, it's technically "in the HTML" — but not in a form that a crawler can extract as text.

The grep -o trick to check if a match is inside a script tag is a simple but powerful test. I'll be using that.

Collapse
 
citedy profile image
Dmitry Sergeev •

lol never thought a simple curl could reveal all that, what's the best way to spoof the user‑agent without