DEV Community: vunelix

Googlebot Saw Empty Tables on Our JavaScript Site: What We Found in Search Console

vunelix — Sat, 31 Oct 2026 08:04:54 +0000

Our site is a set of market-data tools: screeners, heatmaps and calculators. Tables are filled by JavaScript from a small backend endpoint. For months Search Console showed "Crawled - currently not indexed" for almost every tool page, and we blamed content quality. Then we looked at two reports we had never opened, and found two plain technical bugs.

This post is from the Vunelix team and the site in question is ours. The lessons apply to any JavaScript-rendered site.

Bug 1: 96% of Googlebot's requests went to one endpoint that always failed

In Search Console, open Settings → Crawl stats → By response. Ours showed about 7.8 million Googlebot requests in 90 days, and 96% of them were "Other client error (4XX)". The example URLs were all the same path: a POST-only endpoint that our embeddable widgets call.

The reason: our widgets run on other people's sites. Googlebot renders those pages too, runs the widget script, and posts to our endpoint about 80,000 times a day. Every call was rejected (correctly, because Googlebot is not an allowed domain), so it was pure waste.

The fix was one line in robots.txt:

Disallow: /embed/render

If you have a POST endpoint that is only meant for scripts, check whether crawlers are hammering it. Crawl stats will tell you in a minute.

Bug 2: our own pages looked empty to Googlebot

We had recently moved all data requests behind a small proxy that accepts only requests that look like a real browser: it checks headers such as Sec-Fetch-Site, Sec-Fetch-Dest and Accept-Language. That is good against scrapers. But Googlebot's renderer does not send those headers, so the proxy returned 403 and every table stayed as a loading skeleton.

We found it with the one tool that shows what Google really sees:

Search Console → URL Inspection → enter the page
Test live URL
View tested page → the HTML tab (this is the rendered DOM, after JavaScript)
More info → Page resources lists every request that failed

Our stock screener had 1 table row in the rendered HTML instead of 51. We had been looking at the page in a browser, where it was fine, for weeks.

The fix: verify the bot, do not trust the User-Agent

You cannot just allow anything whose User-Agent says "Googlebot": a scraper can write that string in one line. Google documents two ways to verify a crawler:

check the IP against the published IP range files (Google and Bing both publish JSON lists of prefixes, IPv4 and IPv6)
or do a reverse DNS lookup, check the hostname ends in googlebot.com or google.com, then do a forward lookup and confirm it returns the same IP

Our first version used only reverse DNS. In live tests it worked most of the time, but sometimes the table was empty again. One failed or slow lookup for a fresh Google IP was cached as "not a bot" and blocked that IP for an hour. The second version checks the published IP ranges first (no DNS needed, answers instantly), uses DNS only as a fallback, and caches an uncertain result for just a minute. After that, every live test returned the full table.

A few practical points:

Cache the IP-range files for a day, and keep using the old file if a fetch fails.
Treat "DNS failed" and "DNS says it is not Google" as different results. Only the second is evidence.
Log each new decision (IP, engine, method, result). When something goes wrong later, you will want to see why.

How to test your own site

Run Test live URL more than once. Our failures were intermittent, and a single good run proves nothing. Look at the rendered HTML, not at the screenshot.

You can see the result on any of our tool pages, for example the stock screener or the crypto CEX screener.

Indexing is still slow for a new site with a very large number of near-duplicate pages, and these fixes only remove our own obstacles; they do not force Google to index anything. But it is much better to know that Google sees your content than to wonder.

Add a live stock or crypto ticker to WordPress without an API key

vunelix — Sat, 03 Oct 2026 08:42:31 +0000

If you run a finance blog, a crypto news site or an investing newsletter on WordPress, sooner or later you want live prices on the page: a scrolling ticker in the header, a Bitcoin price next to an article, a stock screener on its own page.

The usual options are an iframe from a charting site, or a market-data API that needs a key, a server-side cache and some JavaScript you now have to maintain. This post shows a third way: a free plugin that renders the widgets from a shortcode, with no API key and no iframe.

This post is from the Vunelix team, so the plugin is our own. It sticks to the practical steps.

What you get

Vunelix Financial Widgets is a free plugin on WordPress.org with 40+ widget types:

Tickers: scrolling ticker tape, static price cards, a single-symbol card, and an inline "ticker tag" that sits inside a sentence
Screeners: crypto, stocks, forex, commodities, indices, funds, CEX and DEX pairs
Heatmaps: crypto, stock and currency
Charts: symbol overview, mini chart, seasonality
Calendars: economic calendar and dividends calendar
Tools: currency converter, cross rates, gold calculator, technical analysis gauges

Prices update live over WebSocket. Each widget renders inside Shadow DOM, so your theme's CSS cannot break it and it cannot break your theme.

Step 1: Install the plugin

In your WordPress admin go to Plugins → Add New, search for "Vunelix Financial Widgets", then Install Now and Activate.

Step 2: Verify your domain

There is no API key. Widgets are authorized by domain instead, so you sign up for free on the developer portal and add your site's domain. That is the only setup outside WordPress.

Step 3: Build a widget and copy the shortcode

Open Vunelix Widgets in the admin sidebar, pick a widget, set the options (symbols, theme, colors, language) and copy the shortcode it generates.

A scrolling crypto ticker looks like this:

[vunelix widget="ticker-tape" theme="dark" symbols="BINANCE:BTCUSD,BINANCE:ETHUSD,BINANCE:SOLUSD"]

A stock screener with 50 rows for the US market:

[vunelix widget="stock-screener" per_page="50" default_country="US"]

A USD to EUR converter with a chart:

[vunelix widget="currency-converter" from="USD" to="EUR" show_chart="true"]

Technical analysis for Apple:

[vunelix widget="technical-analysis" symbol="NASDAQ:AAPL" default_period="1d"]

Step 4: Paste it where you want it

Block editor (Gutenberg): add a Shortcode block and paste the shortcode.
Classic editor: paste it straight into the content.
Elementor, Divi, WPBakery: use the page builder's shortcode or text element.
Sidebar or footer: use a Shortcode widget in any widget area.

That is it. Widgets are lazy loaded: the JavaScript for a widget runs when it scrolls into view.

A few things worth knowing

Theme and colors. Every widget can follow your site's light or dark mode automatically, or you can force one. Positive, negative, background, text and border colors can be overridden, with separate values for light and dark.

Languages. Widgets are translated into English, Arabic, Spanish, Chinese, Russian, German, French and Japanese, with right-to-left layout for Arabic.

Several widgets on one page. Add as many as you like; each one loads on its own.

Not on WordPress? The same widgets work on any site with a script tag. The docs and a live configurator for each widget are at vunelix.com/widgets.

Where the data comes from

Prices, technical signals and chart data come from FCS API, delivered through the Vunelix backend. The plugin's readme on WordPress.org lists exactly what is sent to which service, and the JavaScript ships un-obfuscated if you want to read it.

If you try it and something does not work with your theme or page builder, the support forum on WordPress.org is the fastest way to reach us.