Our site is a set of market-data tools: screeners, heatmaps and calculators. Tables are filled by JavaScript from a small backend endpoint. For months Search Console showed "Crawled - currently not indexed" for almost every tool page, and we blamed content quality. Then we looked at two reports we had never opened, and found two plain technical bugs.
This post is from the Vunelix team and the site in question is ours. The lessons apply to any JavaScript-rendered site.
Bug 1: 96% of Googlebot's requests went to one endpoint that always failed
In Search Console, open Settings → Crawl stats → By response. Ours showed about 7.8 million Googlebot requests in 90 days, and 96% of them were "Other client error (4XX)". The example URLs were all the same path: a POST-only endpoint that our embeddable widgets call.
The reason: our widgets run on other people's sites. Googlebot renders those pages too, runs the widget script, and posts to our endpoint about 80,000 times a day. Every call was rejected (correctly, because Googlebot is not an allowed domain), so it was pure waste.
The fix was one line in robots.txt:
Disallow: /embed/render
If you have a POST endpoint that is only meant for scripts, check whether crawlers are hammering it. Crawl stats will tell you in a minute.
Bug 2: our own pages looked empty to Googlebot
We had recently moved all data requests behind a small proxy that accepts only requests that look like a real browser: it checks headers such as Sec-Fetch-Site, Sec-Fetch-Dest and Accept-Language. That is good against scrapers. But Googlebot's renderer does not send those headers, so the proxy returned 403 and every table stayed as a loading skeleton.
We found it with the one tool that shows what Google really sees:
- Search Console → URL Inspection → enter the page
- Test live URL
- View tested page → the HTML tab (this is the rendered DOM, after JavaScript)
- More info → Page resources lists every request that failed
Our stock screener had 1 table row in the rendered HTML instead of 51. We had been looking at the page in a browser, where it was fine, for weeks.
The fix: verify the bot, do not trust the User-Agent
You cannot just allow anything whose User-Agent says "Googlebot": a scraper can write that string in one line. Google documents two ways to verify a crawler:
- check the IP against the published IP range files (Google and Bing both publish JSON lists of prefixes, IPv4 and IPv6)
- or do a reverse DNS lookup, check the hostname ends in
googlebot.comorgoogle.com, then do a forward lookup and confirm it returns the same IP
Our first version used only reverse DNS. In live tests it worked most of the time, but sometimes the table was empty again. One failed or slow lookup for a fresh Google IP was cached as "not a bot" and blocked that IP for an hour. The second version checks the published IP ranges first (no DNS needed, answers instantly), uses DNS only as a fallback, and caches an uncertain result for just a minute. After that, every live test returned the full table.
A few practical points:
- Cache the IP-range files for a day, and keep using the old file if a fetch fails.
- Treat "DNS failed" and "DNS says it is not Google" as different results. Only the second is evidence.
- Log each new decision (IP, engine, method, result). When something goes wrong later, you will want to see why.
How to test your own site
Run Test live URL more than once. Our failures were intermittent, and a single good run proves nothing. Look at the rendered HTML, not at the screenshot.
You can see the result on any of our tool pages, for example the stock screener or the crypto CEX screener.
Indexing is still slow for a new site with a very large number of near-duplicate pages, and these fixes only remove our own obstacles; they do not force Google to index anything. But it is much better to know that Google sees your content than to wonder.
Top comments (0)