DEV Community

Ujjwal Dubey
Ujjwal Dubey

Posted on

Raw HTML vs Rendered: A Quick Check of What Crawlers Actually Read on Your Homepage

#webdev #seo #python #javascript

Owners often ask why a search engine or an AI assistant describes their business strangely, or not at all. One common, boring cause: the words on the homepage only exist after JavaScript runs. Some crawlers render pages fully. Others read the raw HTML and move on. If your headline, your services and your city are not in that raw HTML, part of the web may never see them.

Here is a quick way to compare what a person sees with what a simple crawler sees, in about thirty lines of Python.

1. Get the raw HTML text 

import re, html, requests

def visible_text(markup: str) -> str:
    markup = re.sub(r"<(script|style|noscript)[^>]*>.*?</\1>", " ", markup, flags=re.S | re.I)
    text = re.sub(r"<[^>]+>", " ", markup)
    return re.sub(r"\s+", " ", html.unescape(text)).strip()

url = "https://www.example.com/"
raw = requests.get(url, timeout=20, headers={"User-Agent": "raw-check/0.1"}).text
raw_text = visible_text(raw)
print(len(raw_text.split()), "words in raw HTML")
print(raw_text[:500])
Enter fullscreen mode Exit fullscreen mode

2. Get the rendered text

Use a headless browser for the version a person sees.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto(url, wait_until="networkidle")
    rendered_text = visible_text(page.content())
print(len(rendered_text.split()), "words after rendering")
Enter fullscreen mode Exit fullscreen mode

3. Compare the things that matter

Do not diff everything. Check a short list of phrases the business cares about.

MUST_SEE = ["YourBrand", "what you sell", "your city", "how to contact you"]

for phrase in MUST_SEE:
    in_raw = phrase.lower() in raw_text.lower()
    in_rendered = phrase.lower() in rendered_text.lower()
    print(f"{phrase!r}: raw={in_raw} rendered={in_rendered}")
Enter fullscreen mode Exit fullscreen mode

If a phrase is rendered=True but raw=False, it depends on JavaScript. That is worth raising with whoever built the site. Server-side rendering or static generation for the main marketing pages usually fixes it.

What this check cannot tell you

It tells you whether words are present, not whether they are any good. A homepage can be fully server-rendered and still fail to say what the business sells in the first screen, or say it in three different tones. That is a separate, more human question.

For that second pass, a free AI website brand audit tool that only sees public HTML can give you a fast outside read on tone and audience fit from a public URL, with no login. Treat its output as a list of things to look at, not a verdict, and fix the obvious mismatches by hand.

A small routine

Run the raw-versus-rendered check whenever the site is redesigned or the framework is upgraded. Keep the output in the repo. It takes seconds, and it catches the "our homepage is empty to crawlers" problem before it quietly costs months of visibility.

Disclosure: I work on NxFlowAI, an AI automation agency based in Mumbai, which also makes free website tools.

Top comments (0)