DEV Community: Rachel Costello

The 8 main ways JavaScript can impact SEO performance

Rachel Costello — Thu, 17 Oct 2019 09:19:22 +0000

JavaScript rendering is often a complicated and resource-intensive process, and can significantly impact a variety of different performance and user experience factors which SEO success depends on.

This is why it’s crucial to understand where these issues can occur and how they can impact your website.

Here are the 8 main things to watch out for within a JavaScript-powered website that can impact SEO performance:

Rendering speed
Main thread activity
Conflicting signals between HTML and JavaScript
Blocked scripts
Scripts in the head
Content duplication
User events
Service workers

1. Rendering speed

The process of rendering can be an expensive and strenuous process due to the different stages required to download, parse, compile and execute JavaScript. This causes significant issues when that work falls on a user’s browser or search engine crawler.

Having JavaScript-heavy pages that take a long time to process and render means that they are at risk of not being rendered or processed by search engines.

“If your web server is unable to handle the volume of crawl requests for resources, it may have a negative impact on our capability to render your pages. If you’d like to ensure that your pages can be rendered by Google, make sure your servers are able to handle crawl requests for resources.”

-Google Webmaster Central Blog

JavaScript that renders slowly will also impact your users because, with the increase of page load time, bounce rates will also rise. Nowadays a user will expect a page to load within a few seconds or less. However, getting a page that requires JavaScript rendering to load quickly enough to meet those expectations can be challenging.

Source: Think with Google

Another issue to consider is that a user’s device and CPU will usually have to do the hard work with JavaScript rendering, but not all CPUs are up for the challenge. It’s important to be aware that users will experience page load times differently depending on their device.

Just because a site appears to load quickly on a high-end phone, it doesn’t mean that this will be the case for a user accessing the same page with a lower-end phone.

“What about a real-world site, like CNN.com? On the high-end iPhone 8 it takes just ~4s to parse/compile CNN’s JS compared to ~13s for an average phone (Moto G4). This can significantly impact how quickly a user can fully interact with this site.”

-Google Web Fundamentals

Source: Google Developers

2. Main thread activity

JavaScript is single-threaded, meaning that the entire main thread is halted while JavaScript is parsed, compiled and executed. With this kind of setup, queues can form and bottlenecks can happen, meaning that the entire process of loading a page can be delayed and a search engine won't be able to see any content on the page until the scripts have been executed.

Delays within the main thread can significantly increase the time it takes to load a page for search engines, and for the page to become interactive for users, so avoid blocking main thread activity wherever possible.

Keep an eye on how many resources are being executed and where request timeouts are happening, as these can be some of the main culprits which create bottlenecks.

3. Conflicting signals between HTML and JavaScript

First impressions count with search engines, so make sure you’re giving them clear, straightforward instructions about your website in the HTML as soon as they come across the page.

Adding important meta tags using JavaScript rather than adding them in the HTML is advised against, because either Google won’t see these tags straight away because of its delayed rendering process, or other search engines won’t see them at all due to the fact that they can’t render.

All search engines will use the signals from the HTML in the initial fetch to determine crawling and indexing. Google and the few search engines that have rendering capabilities will then render pages at a later date, but if the signals served via JavaScript differ from what was initially found in the HTML, then this will contradict what the search engine has already been told about the page.

For example, if you use JavaScript to remove a robots meta tag like noindex, Google will have already seen the noindex tag in the HTML and won’t waste resources rendering a page it has been told not to include in its index. This means that the instructions to remove the noindex won’t even be seen as they’re hidden behind JavaScript which won’t be rendered in the first place.

Aim to include the most important tags and signals within the HTML where possible and make sure they’re not being altered by JavaScript. This includes page titles, content, hreflang and any other elements that are used for indexing.

“The signals you provide via JavaScript shouldn't conflict with the ones in the HTML.”

-John Mueller, Google Webmaster Hangout

4. Blocked scripts

If a script is blocked, such as in the robots.txt file, this will impact how search engines will be able to see and understand a website. Scripts that are crucial to the layout and content of a page need to be accessible so that the page can be rendered properly.

“Blocking scripts for Googlebot can impact its ability to render pages.”

-John Mueller, Google Webmaster Hangout

Source: PageSpeed Insights

This is especially important for mobile devices, as search engines rely on being able to fetch external resources to be able to display mobile results correctly.

“If resources like JavaScript or CSS in separate files are blocked (say, with robots.txt) so that Googlebot can’t retrieve them, our indexing systems won’t be able to see your site like an average user. This is especially important for mobile websites, where external resources like CSS and JavaScript help our algorithms understand that the pages are optimized for mobile.”

-Google Webmaster Central Blog

5. Scripts in the head

When JavaScript is served in the head, this can delay the rendering and loading of the entire page. This is because everything in the head is loaded as a priority before the body can start to be loaded.

“Don't serve critical JavaScript in the head as this can block rendering.”

-John Mueller, Google Webmaster Hangout

Serving JavaScript in the head is also advised against because it can cause search engines to ignore any other head tags below it. If Google sees a JavaScript tag within the contents of the head, it can assume that the body section has begun and ignore any other elements below it that were meant to be included in the head.

“JavaScript snippets can close the head prematurely and cause any elements below to be overlooked.”

-John Mueller, Google Webmaster Hangout

6. Content duplication

JavaScript can cause duplication and canonicalisation issues when it is used to serve content. This is because if scripts take too long to process, then the content they generate won’t be seen.

This can cause Google to only see boilerplate, duplicate content across a site that experiences rendering issues, meaning that Google won’t be able to find any unique content to rank pages with. This can often be an issue for Single Page Applications (SPAs) where the content dynamically changes without having to reload the page.

Here are Google Webmaster Trends Analyst, John Mueller’s thoughts on managing SPAs:

“If you're using a SPA-type setup where the static HTML is mostly the same, and JavaScript has to be run in order to see any of the unique content, then if that JavaScript can't be executed properly, then the content ends up looking the same. This is probably a sign that it's too hard to get to your unique content -- it takes too many requests to load, the responses (in sum across the required requests) take too long to get back, so the focus stays on the boilerplate HTML rather than the JS-loaded content.”

7. User events

JavaScript elements that require interactivity may work well for users, but they don’t for search engines. Search engines have a very different experience with JavaScript than a regular user.

This is because search engine bots can’t interact with a page in the same way that a human being would. They don’t click, scroll or select options from menus. Their main purpose is to discover and follow links to content that they can add to their index.

This means that any content that depends on JavaScript interactions to be generated won’t be indexed. For example, search engines will struggle to see any content hidden behind an ‘onclick’ event.

Another thing to bear in mind is that Googlebot and the other search engine crawlers clear cookies, local storage and session storage data after each page load, so this will be a problem for website owners who rely on cookies to serve any kind of personalised, unique content that they want to have indexed.

“Any features that requires user consent are auto-declined by Googlebot.”

-Google Search

8. Service workers

A service worker is a script that works in the background of the browser and on a separate thread. Service workers can run pages and provide content based on their own memory, meaning they can work offline without the server being involved.

Source: Red Hat

The benefit of using a service worker is that it decreases page load time because it doesn’t reload assets that aren’t needed. However, the issue is that Google and other search engine crawlers don’t support service workers.

The service worker can make it look like content is rendering correctly, but this may not be the case. Make sure your website and its key content still works properly without a service worker, and test your server configuration to avoid this issue.

Hopefully this guide has provided you with some new insights into the impact that JavaScript can have on SEO performance, as well as some areas that you can look into for the websites that you manage.

How search engines & social media crawlers render JavaScript

Rachel Costello — Fri, 04 Oct 2019 13:50:35 +0000

JavaScript is a widely discussed topic in the SEO community, because it can cause significant issues for search engines and other crawlers that are trying to access the pages on our sites.

The information that SEOs are gathering on the topic of JavaScript rendering should be more widely shared, as these findings will impact everyone who has a JavaScript-heavy website that they want to be visible to new users.

That’s why I’ve put together this guide to explain some of the key considerations to be aware of.

How search engines render JavaScript

From looking at this example code, a search engine like Google won’t have any idea what the page is meant to be about:

<body>
<app-root></app-root>
<script src="runtime.js"></script>
<script src="polyfills.js"></script>
<script src="main.js"></script>
</body>

The JavaScript contained within this code needs to be processed and executed so that the output code can be displayed for the client. For the contents of a JavaScript-heavy page to mean anything to a search engine or social media crawler, they need to render the page.

However, rendering is an expensive, resource-intensive process which the majority of search engine bots and social media bots struggle with. So it’s important to understand their rendering capabilities, so you can be aware of what they will struggle to see on your site.

It’s important to bear in mind that most search engines can’t render at all, and those that do have their own rendering limitations, as I’ll explain later in this article.

If your website relies on JavaScript to power its content and navigation, search engines could end up seeing a blank screen with nothing of value to crawl or index.

I’ve put together the latest updates on how the main search engines are currently equipped for rendering, as well as some key considerations for building sites that can be crawled and indexed.

Google’s rendering capabilities

Google is one of the few search engines that currently renders JavaScript, and provides a lot of documentation and resources on JavaScript best practice for search.

This means we’re able to build a pretty clear picture of what we need to do to get our websites indexed in Google’s SERPs (Search Engine Results Pages).

When Google renders, it generates markup from templates and the data available from a database or an API. The key step in this process is to get this fully generated markup, because this is what’s readable for Google’s web crawler, Googlebot.

Source: [Martin Splitt, AngularUP Conference](https://www.youtube.com/watch?v=Z9mHopLeE40)

To carry out this process, Googlebot uses a headless browser for its web rendering service (WRS). Google’s WRS used to be based on Chrome 41, which was an outdated version launched in 2015.

However, Google have now made their WRS ‘evergreen’, meaning that it will be regularly updated to run the latest version of Chrome on an ongoing basis.

This change allows Googlebot to process features that it was previously unable to, such as ES6, IntersectionObserver and Web Components.

The crawling and indexing process is usually very quick for sites that don’t rely on JavaScript, however, Google can’t crawl, render and index in one instantaneous process due to the scale of the internet and the processing power that would be required to do so.

“The internet is gigantic, that’s the problem. We see over 160 trillion documents on the web daily, so Googlebot is very busy. Computing resources, even in the cloud age, are pretty tricky to come by. This process takes a lot of time, especially if your pages are really large, we have to render a lot of images, or we have to process a lot of megabytes of JavaScript.”

-Martin Splitt, Webmaster Trends Analyst at Google

This is why Google has a two-wave indexing process. In the first wave of indexing, HTML pages are crawled and indexed and Googlebot will use a classifier to determine pages with JavaScript on them that need to be rendered.

These pages will be added to a queue to be rendered at a later date when enough resources become available, in the second wave of indexing. A page will only be added to the index in the second wave after it has been rendered.

“Google determines if pages need to be rendered by comparing content found in the initial HTML and the rendered DOM.”

-Martin Splitt, Google Webmaster Hangout

Source: [Google I/O 2018](https://www.youtube.com/watch?v=PFwUbgvpdaQ)

When resources do become available, there isn’t a specific way of prioritising the pages that will be rendered first, which means that there are no guarantees on when pages will actually be rendered after they are initially discovered by Googlebot.

What is the gap between the first and second wave of indexing then? According to Google’s Tom Greenaway and Martin Splitt during Chrome Dev Summit 2018, it could take “minutes, an hour, a day or up to a week” for Google to render content after a page has been crawled.

If your website gets stuck between these two waves of indexing, any new content you add or any changes you make to your website won’t be seen or indexed for an undetermined amount of time.

This will have the biggest impact on sites that rely on fresh search results, such as ecommerce or news sites.

“Ecommerce websites should avoid serving product page content via JavaScript.”

-John Mueller, Google Webmaster Hangout

“News sites should avoid content that requires JavaScript to load.”

-John Mueller, Google Webmaster Hangout

Bing’s rendering capabilities

Bing’s crawler allegedly does render JavaScript, but is limited in being able to process the latest browser features and render at scale.

The team at Bing recommended implementing dynamic rendering to make sure Bingbot is able to crawl and index your JavaScript-powered content and links.

“In general, Bing does not have crawling and indexing issues with web sites using JavaScript, but occasionally Bingbot encounters websites heavily relying on JavaScript to render their content, especially in the past few years. Some of these sites require far more than one HTTP request per web page to render the whole page, meaning that it is difficult for Bingbot, like other search engines, to process at scale on every page of every large website.

Therefore, in order to increase the predictability of crawling and indexing of websites relying heavily on JavaScript by Bing, we recommend dynamic rendering as a great alternative. Dynamic rendering is about detecting the search engine’s bot by parsing the HTTP request user agent, prerendering the content on the server-side and outputting static HTML, helping to minimize the number of HTTP requests needed per web page and ensure we get the best and most complete version of your web pages every time Bingbot visits your site.

When it comes to rendering content specifically for search engine crawlers, we inevitably get asked whether this is considered cloaking... and there is nothing scarier for the SEO community than getting penalized for cloaking. The good news is that as long as you make a good faith effort to return the same content to all visitors, with the only difference being that the content is rendered on the server for bots and on the client for real users, this is acceptable and not considered cloaking.”

-Fabrice Canel, Principal Program Manager at Bing

Even though Bing can render in some capacity, it isn’t able to extract and follow URLs that are contained within JavaScript.

“Don’t bury links to content inside JavaScript.”

-Bing Webmaster Guidelines

Yahoo’s rendering capabilities

Yahoo cannot currently render at all. It is recommended to make sure that content isn’t ‘hidden’ behind JavaScript, as the search engine won’t be able to render to be able to find any content generated by the script. Only content that is served within the HTML will be picked up.

You can get around this by using the <noscript> element.

“Unhide content that’s behind JavaScript. Content that’s available only through JavaScript should be presented to non-JavaScript user agents and crawlers with noscript HTML elements.”

-Yahoo Webmaster Resources

Yandex’s rendering capabilities

Yandex’s documentation explains that their search engine doesn’t render JavaScript and can’t index any content that is generated by it. If you want your site to appear in Yandex, make sure your key content is returned in the HTML upon the initial request for the page.

“Make sure that the pages return the full content to the robot. If they use JavaScript code, the robot will not be able to index the content generated by the script. The content you want to include in the search should be available in the HTML code immediately after requesting the page, without using JavaScript code. To do this, use HTML copies.”

-Yandex Support

Other search engines’ rendering capabilities

DuckDuckGo, Baidu, AOL and Ask are much less open about their rendering capabilities and lack official documentation as reference guides. The only way to find this out currently is to run tests ourselves.

In 2017, Bartosz Góralewicz ran some experiments using a test site that used different JavaScript frameworks to serve content and analysed which search engines were able to render and index the content they generated.

We can never make definitive conclusions based on the indexing of test sites alone, but the results showed that only Google and, surprisingly, Ask were able to index rendered content.

Source: [Moz](https://moz.com/blog/search-engines-ready-for-javascript-crawling)

“Bing, Yahoo, AOL, DuckDuckGo, and Yandex are completely JavaScript-blind and won’t see your content if it isn’t in the HTML.”

-Bartosz Góralewicz, CEO of Onely

Take a look at the full article covering the experiment and results to learn more about Bartosz’s conclusions.

How social media platforms render JavaScript

It’s important to know that social media and sharing platforms generally can’t render any JavaScript client-side.

“Facebook and Twitter’s crawlers can’t render JavaScript.”

-Martin Splitt, Google Webmaster Hangout

If you rely on JavaScript to serve content that would feed into Open Graph tags, Twitter Cards or even meta descriptions that would show when you share an article on Slack, for example, this content wouldn’t be able to be shown.

Make sure you pre-render, server-side render or dynamically content like featured images, titles and descriptions for crawlers like Twitterbot and Facebot, so they can display your site and its content properly.