I run a small property site. Four languages, plain HTML, no framework, no npm dependencies, hosted on Cloudflare Pages. I have spent my working life on copy and search, so I did the usual things: structured data, hreflang, a sitemap, a generator that keeps every page's head in sync from one file.
Then I ran one command against my own listing page:
curl -s -A 'GPTBot/1.0' https://example.com/my-listing | \
python3 -c "import sys,re,html; b=sys.stdin.read(); b=b[b.index('<body'):]; \
b=re.sub(r'<script.*?</script>','',b,flags=re.S); \
print(len(re.sub(r'\s+',' ',html.unescape(re.sub(r'<[^>]+>',' ',b))).strip()))"
It counts the visible text in <body> after removing scripts. That is roughly what a crawler gets if it does not execute JavaScript.
The answer for my listing page was 23 characters. For my homepage, 17.
Where the text had gone
Nowhere surprising, once you look. The site is multilingual, and the copy for every language lives in assets/js/i18n.js. On load, app.js picks the language and fills the containers:
var descEl = document.querySelector("[data-listing-description]");
if (descEl) descEl.innerHTML = l.description + climateNoteFor(id);
That is a perfectly ordinary pattern. It also means the HTML I ship contains the layout and nothing to read.
Googlebot renders JavaScript, so Google was fine. The question is everyone else.
What the crawler did get
Not nothing. The head carried 2,891 characters of JSON-LD: name, price, number of rooms, floor area, address, coordinates, amenities. A machine could reconstruct a tidy product card from that.
And that is exactly what made it worse.
My site exists for one reason: it is a private sale, direct from the owner, and every listing states plainly what the title deed situation is — the kind of sentence an agency leaves out. That sentence lives in the prose. So a crawler without JavaScript got the price and the square metres, identical to any agency listing, and none of the disclosure the site was built around. The one thing that made the page different was the one thing it could not read.
Why no dashboard told me
Analytics did not show this, and could not. My analytics runs on JavaScript. It counts visitors whose browsers execute scripts, which by definition excludes every client that does not. The blind spot is the measurement instrument, not the site.
What did show it was Cloudflare's AI crawler log. In 24 hours up to 15 September 2026 my domain took 43 requests from AI crawlers: Applebot 20, Googlebot 14, Meta-ExternalAgent 8, GPTBot 1. They fetched pages, the sitemap, images, the CSS, both JS files, and the favicon I had published an hour earlier.
Worth noting, since it is the fashionable question: llms.txt sits in my site root, and across 23 distinct crawled paths in that window it was requested exactly zero times. Cloudflare's own readiness checklist did not mention the file either. On that day its items were robots.txt, a sitemap, content signals, markdown versions of pages, and WebMCP.
The fix, and why it was cheap
I already had a generator for the head. It takes one data file and writes it into every page between markers:
<!-- SEO:START (generated by build/generate-seo.js — do not hand-edit between markers) -->
...
<!-- SEO:END -->
The previous post is about it. The same idea works for the body. I put markers inside the containers the script already fills:
<div class="listing-description" data-listing-description><!-- BODY:listing-description --><!-- /BODY:listing-description --></div>
and taught the generator to fill them from i18n.js and listings-data.js — the same files the browser uses. That detail is the whole point: the static text cannot drift from what the script renders, because there is one source and one build step. app.js still redraws those containers on load, so the language switcher behaves exactly as before.
Results, same command as above:
| Page | Before | After (15 September 2026) |
|---|---|---|
| Homepage | 17 | 2,648 |
| Listing | 23 | 3,127 |
| About | 17 | 995 |
Total work: one script and a pass that inserted markers. No framework, no SSR runtime, no rewrite.
What I would take from this
A generator earns its keep twice. The first time when it saves you from editing twenty files by hand. The second time when a new requirement lands and the data is already in one place, so the fix is a function rather than a project.
And the general version, which is older than any of this: check what the machine receives, not what your browser draws. The page looked finished in every browser I opened. It was empty to anything that did not run my JavaScript, and it stayed that way for weeks because every tool I used to check it ran JavaScript too.
Run the command above on your own site. It takes ten seconds. If your stack renders on the server you will see a number in the thousands and can stop reading. If you see two digits, you now know something your analytics will never tell you.
The site: https://girne-apartment-sale.com/?utm_source=devto&utm_medium=article&utm_campaign=article04
Standalone, genericized version of the generator, if you want the code rather than the description: https://github.com/sergei-swh/seo-generator
Top comments (1)
We need to produce a short comment, 1-2 sentences, casual, start with lowercase, specific reaction/question about the video. No marketing, no URLs, no double hyphens. No em-dash. Use straight quotes. No markdown. Just the comment text. Video: "What AI crawlers are blind to: the text your JavaScript renders". Likely about SEO, crawlers missing content rendered by JS. The commenter could ask about using static rendering or fallback. Something like "so if i have a SPA