DEV Community: SaaS.Group

Robots.Txt Files & SEO – Best Practices, and Fixes for Common Issues

SaaS.Group — Fri, 19 Aug 2022 20:46:34 +0000

Originally published on Prerender.io: Robots.Txt Files & SEO – Best Practices, and Fixes for Common Issues

Technical SEO is a well-executed strategy that factors in various on-page and off-page ranking signals to help your website rank higher in SERPs. Each SEO tactic plays into the grand scheme of boosting your page rank by ensuring web crawlers can easily crawl, rank, and index your website.

From page speed to proper title tags, there are many ranking signals that technical SEO can help with. But did you know that one of the most important files for your website’s SEO is also found on your server?

The robots.txt file is a code that tells web crawlers which pages on your website they can and cannot crawl. This might not seem like a big deal, but if your robots.txt file is not configured correctly, it can have a serious negative effect on your website’s SEO.

In this blog post, we’ll be discussing everything you need to know about robots.txt, from what is a robots.txt file in SEO to the best practices to the proper way to fix common issues.

What Is a robots.txt File & Why Is It Important in SEO?

Robots.txt file is a file located on your server that tells web crawlers which pages they can and cannot access. If a web crawler tries to crawl a page that is blocked in the robots.txt file, it will be considered a soft 404 error.

Although a soft 404 error will not hurt your website’s ranking, it is still considered an error. And too many errors on your website can lead to a slower crawl rate which can eventually hurt your ranking due to decreased crawling.

If your website has a lot of pages that are blocked by the robots.txt file, it can also lead to a wasted crawl budget. The crawl budget is the number of pages Google will crawl on your website during each visit.

Another reason why robots.txt files are important in SEO is that they give you more control over the way Googlebot crawls and indexes your website. If you have a website with a lot of pages, you might want to block certain pages from being indexed so they don’t overwhelm search engine web crawlers and hurt your rankings.

If you have a blog with hundreds of posts, you might want to only allow Google to index your most recent articles. If you have an eCommerce website with a lot of product pages, you might want to only allow Google to index your main category pages.

Configuring your robots.txt file correctly can help you control the way Googlebot crawls and indexes your website, which can eventually help improve your ranking.

What Google Says About robots.txt File Best Practices

Now that we’ve gone over why robots.txt files are important in SEO, let’s discuss some best practices recommended by Google.

Create a File Named robots.txt

The first step is to create a file named robots.txt. This file needs to be placed in the root directory of your website – the highest-level directory that contains all other files and directories on your website.

Here’s an example of proper placement of a robots.txt file: on the apple.com site, the root directory would be apple.com/.

You can create a robots.txt file with any text editor, but many CMS’ like WordPress will automatically create it for you.

Add Rules to the robots.txt File

Once you’ve created the robots.txt file, the next step is to add rules. These rules will tell web crawlers which pages they can and cannot access.

There are two types of robot.txt syntax you can add: Allow and Disallow.

Allow rules will tell web crawlers that they are allowed to crawl a certain page.

Disallow rules will tell web crawlers that they are not allowed to crawl a certain page.

For example, if you want to allow web crawlers to crawl your homepage, you would add the following rule:

Allow: /

If you want to disallow web crawlers from crawling a certain subdomain or subfolder on your blog, you use:Disallow: /

Upload the robots.txt File to Your Site

After you have added the rules to your robots.txt file, the next step is to upload it to your website. You can do this using an FTP client or your hosting control panel.

If you’re not sure how to upload the file, contact your web host and they should be able to help you.

Test Your robots.txt File

After you have uploaded the robots.txt file to your website, the next step is to test it to make sure it’s working correctly. Google provides a free tool called the robots.txt Tester in Google Search Console that you can use to test your file. It can only be used for robots .txt files that are located in the root directory of your website.

To use the robots.txt tester, enter the URL of your website into the robots.txt Tester tool and then test it. Google will then show you the contents of your robots.txt file as well as any errors it found.

Use Google’s Open-Source Robots Library

If you are a more experienced developer, Google also has an open-source robots library that you can use to manage your robots.txt file locally on your computer.

What Can Happen to Your Website’s SEO if a robots.txt File Is Broken or Missing?

If your robots.txt file is broken or missing, it can cause search engine crawlers to index pages that you don’t want them to. This can eventually lead to those pages being ranked in Google, which is not ideal. It may also result in site overload as crawlers try to index everything on your website.

A broken or missing robots.txt file can also cause search engine crawlers to miss important pages on your website. If you have a page that you want to be indexed, but it’s being blocked by a broken or missing robots.txt file, it may never get indexed.

In short, it’s important to make sure your robots.txt file is working correctly and that it’s located in the root directory of your website. Rectify this problem by creating new rules or uploading the file to your root directory if it’s missing.

Best Practices for Robots.txt Files

Now that you know the basics of robots.txt files, let’s go over some best practices. These are things you should do to make sure your file is effective and working properly.

Use a New Line for Each Directive

When you’re adding rules to your robots.txt file, it’s important to use a new line for each directive to avoid confusing search engine crawlers. This includes both Allow and Disallow rules.

For example, if you want to disallow web crawlers from crawling your blog and your contact page, you would add the following rules:

Disallow: /blog/

Disallow: /contact/

Use Wildcards To Simplify Instructions

If you have a lot of pages that you want to block, it can be time-consuming to add a rule for each one. Fortunately, you can use wildcards to simplify your instructions.

A wildcard is a character that can represent one or more characters. The most common wildcard is the asterisk (*).

For example, if you want to block all files that end in .jpg, you would add the following rule:

Disallow: /*.jpg

Use “$” To Specify the End of a URL

The dollar sign ($) is another wildcard that you can use to specify the end of a URL. This is helpful if you want to block a certain page but not the pages that come after it.

For example, if you want to block the contact page but not the contact-success page, you would add the following rule:

Disallow: /contact$

Use Each User Agent Only Once

Thankfully, when you’re adding rules to your robots.txt file, Google doesn’t mind if you use the same User-agent multiple times. However, it’s considered best practice to use each user agent only once.

Use Specificity To Avoid Unintentional Errors

When it comes to robots.txt files, specificity is key. The more specific you are with your rules, the less likely you are to make an error that could hurt your website’s SEO.

Use Comments To Explain Your robots.txt File to Humans

Despite your robots.txt files being crawled by bots, humans will still need to be able to understand, maintain and manage them. This is especially true if you have multiple people working on your website.

You can add comments to your robots.txt file to explain what certain rules do. Comments must be on their line and start with a #.

For example, if you want to block all files that end in .jpg, you could add the following comment:

Disallow: /*.jpg # Block all files that end in .jpg

This would help anyone who needs to manage your robots.txt file understand what the rule is for and why it’s there.

Use a Separate robots.txt File for Each Subdomain

If you have a website with multiple subdomains, it’s best to create a separate robots.txt file for each one. This helps to keep things organized and makes it easier for search engine crawlers to understand your rules.

Common Robots.txt File Mistakes and How To Fix Them

Understanding the most common mistakes people make with their robots.txt files can help you avoid making them yourself. Here are some of the most common mistakes and how to fix these technical SEO issues.

Missing robots.txt File

The most common robots.txt file mistake is not having one at all. If you don’t have a robots.txt file, search engine crawlers will assume that they are allowed to crawl your entire website.

To fix this, you’ll need to create a robots.txt file and add it to your website’s root directory.

Robots.txt File Not in the Directory

If you don’t have a robots.txt file in your website’s root directory, search engine crawlers won’t be able to find it. As a result, they will assume that they are allowed to crawl your entire website.

It should be a single text file name that should be not placed in subfolders but rather in the root directory.

No Sitemap URL

Your robots.txt file should always include a link to your website’s sitemap. This helps search engine crawlers find and index your pages.

Omitting the sitemap URL from your robots.txt file is a common mistake that may not hurt your website’s SEO, but adding it will improve it.

Blocking CSS and JS

According to John Mueller, you must avoid blocking CSS and JS files as Google search crawlers require them to render the page correctly.

Naturally, if the bots can’t render your pages, they won’t be indexed.

Using NoIndex in robots.txt

Since 2019, the noindex robots meta tag has been deprecated and is no longer supported by Google. As a result, you should avoid using it in your robots.txt file.

If you’re still using the noindex robots meta tag, you should remove it from your website as soon as possible.

Improper Use Of Wildcards

Using wildcards incorrectly will only result in restricting access to files and directories that you didn’t intend to.

When using wildcards, be as specific as possible. This will help you avoid making any mistakes that could hurt your website’s SEO. Also, stick to the supported wildcards, that is asterisk and dollar symbol.

Wrong File Type Extension

As the name implies, a robot.txt file must be a text file that ends in.txt. It cannot be an HTML file, image, or any other type of file. It must be created in UTF-8 format. A useful introductory resource is Google’s robot.txt guide and Google Robots.txt FAQ.

Use Robot.Txt Files Like A Pro

A robots.txt file is a powerful tool that can be used to improve your website’s SEO. However, it’s important to use it correctly.

When used properly, a robots.txt file can help you control which pages are indexed by search engines and improve your website’s crawlability. It can also help you avoid duplicate content issues.

On the other hand, if used incorrectly, a robots.txt file can do more harm than good. It’s important to avoid common mistakes and follow the best practices that will help you use your robots.txt file to its full potential and improve your website’s SEO.In addition to expertly navigating Robot.txt files, dynamic rendering with Prerender also offers the opportunity to produce static HTML for complex Javascript websites. Now you can allow faster indexation, faster response times, and an overall better user experience.

10 Tips and Tricks on How to Make a Website Mobile Friendly

SaaS.Group — Fri, 19 Aug 2022 19:42:00 +0000

Originally published on Prerender: 10 Tips and Tricks on How to Make a Website Mobile Friendly

The future of the internet is mobile.

Websites that are not mobile-friendly will soon be a thing of the past. With over half (54.4%) of all internet traffic coming from mobile devices, it is more important than ever to make sure your website looks great and functions properly on smartphones and tablets.

If your website doesn’t look good on a smartphone, your users will leave and go to a competitor’s website that does.

So what does it mean to be mobile-friendly? How can you check if your website is optimized for mobile traffic? Most importantly, why is it important that your website is mobile-friendly?

In this post, we will answer all those questions and give you some tips on how to make a website mobile-friendly.

What Does It Actually Mean To Be Mobile-Friendly?

A mobile-friendly website is easy to use on a smartphone or tablet. This means that the layout is easy to navigate, and buttons and links are appropriately sized to be used with a finger.

Mobile-friendly sites are well-optimized and use a particular design to serve the needs of mobile users. For instance, mobile-friendly websites also tend to load quickly, as users are often on the go and don’t have the patience to wait for a slow website to load.

Overall, the goal of a mobile-friendly website is to provide users with a positive experience that is optimized for the device they are using.

How To Check If Your Website Is Mobile-Friendly

Checking if your website is mobile-friendly is pretty easy due to the many approaches available. Here are some ways you can test your website’s mobile-friendliness:

Manually Change The Size Of Your Website’s Browser Window

The quickest and easiest way to test if your website is mobile-friendly is to just open it on your smartphone yourself. Simply open your website on a desktop browser and resize the window to be as small as a smartphone. If your website is easy to use and navigate, then it is most likely mobile-friendly. If not, you will need to make some changes.

Load Your Website From Different Devices

Nothing beats good ole’ fashioned testing. If you have access to different types of mobile devices, load your website on each one and see how it looks and feels. This will give you a good idea of how your website appears on different screen sizes and can help you identify any areas that need improvement.

Use Google’s Mobile-Friendly Test Tool

Google has a great free tool that will analyze your website and tell you if it is mobile-friendly. All you need to do is enter your website’s URL and the tool will do the rest.

The Mobile-Friendly Test Tool will give your website a score that categorizes how mobile-friendly your website is. It will also provide a list of current issues and actionable items so you can fix and improve your website’s mobile-friendliness.

Why Your Website Must Be Mobile-Friendly

It’s always a wise idea to understand the many benefits that you can reap by making your website mobile-friendly. After all, why put in the extra effort if there’s no real payoff?

Here are some key reasons why it’s important to have a mobile-friendly website:

Boost ROI and Conversion Rate

If your website is not mobile-friendly, you will lose potential customers and revenue. Research has shown that 1 in 2 users will leave a website that is not mobile-friendly and go to a competitor’s site.

Mobile-Friendly is a Major Ranking Factor

Google has stated that they are now using mobile-friendliness as a ranking factor in its search algorithm. This means that if your website is not mobile-friendly, you’ll appear lower in the SERPs than competitors who are, and it will be harder for people to find you online.

Mobile Users Are Increasing Every Year

As more and more people adopt smartphones and tablets, the number of mobile users is increasing every year. Over half (54.4%) of all web traffic now comes from mobile devices.

This trend is only going to continue, so it’s important to make sure your website is prepared.

Besides the many obvious benefits for you and your business, it also works in your users’ interest, which still comes back to benefit you in the long run.

Positive User Experience

A mobile-friendly website provides users with a positive experience that is optimized for their devices. This results in happy users, which leads to more recommendations and positive testimonials, which leads to more traffic and conversions for your website.

Seamless Experience Across All Devices

Having a seamless user experience from device to device is also becoming more important as people use multiple devices throughout their day.

According to Google, 90% of people move between screens across devices to accomplish a task. This means that people often start their journey on one device (usually a mobile phone) and then move to another device (usually a desktop computer) to complete it. For example, many people will casually research their choices on a mobile device before completing their purchase and checkout on their laptop.

Trust and Credibility Builder

Lastly, it also helps your users see you in a positive light. Trust and credibility go a long way when it comes to nurturing and fostering positive relationships with your users, and creating a mobile-friendly website is a step in the right direction.

10 Tips To Make Your Website Mobile-Friendly

Now that we’ve gone over what it means to be mobile-friendly and why it’s important, let’s take a look at some tips on how to make a website mobile-friendly.

1. Choose A Responsive Layout

A responsive layout is a must if you want your website to be mobile-friendly.

It allows the website to adapt and change its layout based on the device it is being viewed on. Also, with a responsive layout, side-scrolling and zooming are no longer necessary, which makes the experience much smoother for mobile users.

This means that your website will look great and be easy to use no matter what device your users are on.

2. Avoid Intrusive Pop-Ups

No one likes pop-ups, especially on mobile. They are intrusive, very difficult to close on a small screen, and just plain annoying. They’re also difficult to get rid of – a pop-up with a hard-to-see or non-existent close button is aggravating and the hallmark of a poorly-designed mobile website.

If you have pop-ups on your website, make sure they are non-intrusive and can be easily closed on a mobile device.

3. Optimize Your Mobile Pages

Today’s customers want everything to be fast and easily accessible, and if it’s not, they will quickly move on to something else.

The same goes for mobile users. If your website speed is slow and laggy, your users will leave and find a faster website.

If you want to keep your mobile users happy so they stick around, you need to make sure your website speed is up to par.

Here are a few ways to do that:

Use a content delivery network (CDN)
Optimize your images
Optimize your website code
Use browser caching
Minimize CSS files

For a full breakdown of how to optimize your website speed, check out our full guide on common technical SEO issues.

4. Use Large Font Sizes

One of the main differences between desktop and mobile devices is the screen size.

Mobile screens are much smaller, which can make it difficult to read small font sizes.

To make your website mobile-friendly, use large font sizes that are easy to read on a small screen. This will make your website much easier to use for mobile users. Once you have done this, don’t forget to test it out on a few different devices to make sure it looks good and is easy to read.

5. Stick to a Minimalist Design

Less is more when it comes to mobile design.

A minimalist design is much easier to navigate and use on a small screen. Plus, it will help your website load faster, which is always a bonus.

So, when you are designing your mobile-friendly website, stick to a minimalist design and only include the essentials such as your branding, navigation, and content.

You can always add more bells and whistles later, but for now, keep it simple.

This will make your website much easier to use on a mobile device and ensure that your users have a positive experience

6. Make Sure Button Size and Placement Work on Mobile

One of the most important things to consider when making your website mobile-friendly is button size and placement.

On a desktop, it’s easy to click on a small button or in a difficult-to-reach place, but on mobile, it’s much more difficult.

49% of mobile users use their thumbs to navigate and click on elements that are within their thumbs’ reach. That means that any element placed in the upper corners of your website will be a struggle to reach for most users.

To make your website mobile-friendly, make sure all buttons are large and placed in the middle of the screen where they are easy to reach.

And, if you want to take it a step further, you can add a “hamburger” menu icon in the upper corner of your website that will expand to show all of your navigation options when clicked.

This is a great way to make sure your mobile users can easily find what they are looking for without having to search through a long list of menu items.

7. Use the Viewport Meta Tag

The viewport meta tag is an HTML element that tells the browser how to adjust the website’s dimensions and scaling to fit the width of the device’s screen.

Without this tag, your website will not be responsive and will not adjust to different screen sizes.

To make your website mobile-friendly, you need to add the viewport meta tag to the HTML code of your website.

Once you have added the tag, be sure to test your website on different devices to make sure it is responsive and looks good on all screen sizes.

8. Use Media Queries

Now a popular technique, media queries were designed to make the responsive design simpler and more effective.

Media queries are responsive CSS code for all screen sizes. It allows you to specify different styles for different screen sizes using specific rules.

For example, you could use a media query to make sure your website’s font size is large enough to be readable on a mobile device.

Or, you could use a media query to hide certain elements of your website on mobile devices to make the design simpler and easier to use.

Media queries are a great way to make sure your website looks good and functions well on all devices.

9. Make Forms Simpler – Turn Off Autocorrect on Mobile

Another important thing to consider when making your website mobile-friendly is forms.

Forms are often long and complex, which can be difficult to fill out on a mobile device.

To make your forms simpler and easier to use on mobile, turn off the autocorrect function. This will prevent the keyboard from changing what you type and make it easier to fill out your form.

In addition, make sure all form fields are large enough to be easily clicked on mobile devices.

If you have a longer form, consider cutting down on the number of fields or breaking it up into multiple pages so that mobile users don’t have to scroll through a long list of fields.

10. Continually Test Your Website on Different Devices and Screen Sizes

Finally, as your work is never done, continually test and optimize your website on different devices and screen sizes, even after development.

Mobile devices are constantly changing and evolving as new devices and screen sizes and continuously being released, so it’s important to keep up with the latest trends.

By using the previously mentioned methods, you can test your website regularly throughout its lifespan. When you take this necessary step, you can identify and tackle any potential mobile-unfriendly issues that may arise.

You can take it one step further by testing out every user scenario and optimizing your customers’ journey. Taking this step will allow you to identify any potential bottlenecks and make the necessary changes to keep your website running smoothly.

Make Your Website Look and Perform Its Best On All Fronts

Similar to the desktop, first impressions matter on mobile.

The average user will make a judgment about your website in just a few seconds, so it’s important to make sure your website looks and acts its best.

If you follow the above tips and tricks, achieving that will be much more straightforward.Put these tips into practice one step at a time, and make your website mobile-friendly. To kick your website up a notch, factor in Javascript dynamic rendering so search engine bots can quickly process and index it. Register with Prerender for free today to get started. Your users will thank you for it, no matter their device.

SEO and Social Media: How Social Media Impacts Your SEO ROI

SaaS.Group — Mon, 27 Jun 2022 23:47:40 +0000

Post originally published on Prender.
Explaining SEO and social media’s relationship to stakeholders can be tricky because of how many different opinions exist on the topic – even among dedicated professionals. There do seem to be, however, a lot of benefits and nuances of using social media to enhance your SEO campaign ROI, although in most cases it won’t directly affect your rankings.

Today we’re digging deeper into social media’s role in SEO to see if we can provide answers once and for all.

Is Social Media a Ranking Factor?

Google admitted that social media signals aren’t a direct ranking factor for them. However, when you search for some brands on Google, you see that their social media accounts like Twitter, Instagram, and Facebook rank higher on search results than their official company website.

You can also find recent social media posts on search engine result pages (SERPs), indicating that brands can drive significant traffic to their social media accounts directly from search.

Also, features like Facebook page reviews are shown in Google search results, which can help brands create trust and authority. So, even though social media doesn’t directly impact search rankings, there is no denying that having a strong social presence gets you more visibility.

Note: One thing to keep in mind is that Bing does use social signals as a ranking factor. So even though the majority of traffic comes from Google, by optimizing our social media presence we’re also helping our website to rank higher on Bing’s search results.

What’s the Role of Social Media in SEO?

Even though you cannot get direct search engine ranking benefits from social media, there are many ways your social presence helps your website’s SEO efforts:

Social media drives referral traffic to your site.
Although getting traffic from your socials doesn’t have the same consistency as organic SEO traffic, social media can help you send targeted visitors interested in reading your content or buying your products/solutions to your website.
It increases the time your user spends on your page (Dwell time).
Generally, site visitors who are referred from social media tend to spend more time on your website, which is a very valuable engagement metric. In addition, it signals to search engines that your site is providing valuable content that the users want to engage with.
Social media creates opportunities for backlinks.
When posting on social media, you never know who will interact with your content. Even authoritative website owners can find your pages through social media posts and decide to give you a backlink if it provides value for their readers.
It might help you get new pages indexed faster.
By promoting your new articles and pages through social media, you’re creating new ways for Google to find and index your content. Social shares can also help Google find these pages and, if traffic is coming to them, it’s a clear indication for search engines that your pages are worth adding to their SERPs.
Social media provides you with more psychographic metrics about your audience.
Since people provide more information about themselves on social media, you can really gain an understanding of the beliefs, fear, and pain points of your target audience. These social media metrics help you create useful, engaging website content that caters to your visitors.

The 5 Best Social Media Platforms for SEO

To help you choose the right platform to support your SEO campaign, we’ve compiled a list of some of the best social media platforms that can help you scale your SEO efforts:

1. Pinterest
Pinterest works more or less like a search engine, even though it is a social media platform. Over 1 billion active searches are generated on Pinterest per month, making it an ideal platform for bloggers, online marketers, and eCommerce stores.

Unlike other social apps, Pinterest doesn’t limit the outreach of a post with only one outbound link (like in the case of Instagram and TikTok with the unique link in the profile).

To rank on Pinterest search results, you need to make creative pins and optimize them with the right keywords. Don’t forget to create an outbound link to your website on every pin you create. Video pins are getting more viral on Pinterest now, which is something you can leverage.

2. Quora
It must have happened to you at least once that you typed a question in Google, and the first result Google showed you were from Quora.

Quora is a social media platform where people ask questions and get the answers in real-time, making it a fantastic platform for you to build authority by providing helpful answers.

You can generate traffic directly from Google and from the platform itself. Over 500 million people visit Quora every month according to Similar Web.

To make the most out of Quora, you need to provide insightful, genuinely helpful answers. You can also add links to your website if it is relevant to the question and complements your answer.

Note: If you are new to Quora, it is better not to provide outbound links for the first few weeks.

3. LinkedIn
Although LinkedIn was designed as a professional network platform, it has become the number one choice for B2B marketers to generate leads.

If your target audience is top professionals, like CEOs, CMOs, and decision-makers, LinkedIn is the best platform on which to promote your content and create a presence.

Compared to platforms like Facebook, Instagram, and Twitter, it is easier to get visibility on LinkedIn if you have a solid content strategy.

Something to notice is that even though LinkedIn allows you to create a company page, most users prefer to follow and interact with other professionals. In other words, the best way to share and create conversations around your brand and content is to use your CEO’s profile.

4. Facebook
Facebook has over 10 Billion monthly visits %3A%202.936%20billion%20(April%202022)&text=Number%20of%20people%20who%20use,)%3A%201.960%20billion%20(April%202022)&text=Share%20of%20Facebook's%20monthly%20active,%3A%2067%25%20(April%202022))and is ranked as the 3rd most-visited website globally. It is very easy to share web content like blog posts, eCommerce products, and web pages on Facebook.

The great thing about these numbers – besides the big pool of users you can reach – is that Google continuously crawls and indexes Facebook pages as it does other pages. So your company and product pages can rank on SERPs, and it can help Google to find new content faster.

However, the best Facebook feature is its paid ads campaign. By investing some budget to promote your web content, you can reach out to new readers quickly and run CTR experiments for content optimization.

5. Twitter
Twitter drives over 6 billion monthly visits, ranking 7th most-visited website globally. Similar to Facebook, posting on Twitter can help you gain more SERP real estate thanks to Twitter carousels now being shown on result pages.

Twitter also allows you to share links in your tweets, so make sure to add a backlink to your content when promoting it on the social network. It won’t pass any link authority but it will provide relevant referral traffic to your site.

A good rule of thumb is to invest 40% of your time on Twitter talking about your industry, providing practical, useful tips, and retweeting content from relevant brands; the second 40% should be interacting with other brands and users; the last 20% should be invested in promoting your own content.

5 Strategies to Bring SEO and Social Media Together

Hopefully, by now you have a clear understanding of how social media can impact your website’s traffic and how it may fit into your overall SEO strategy.

To help you accomplish your goals, we want to share some best practices you can implement to successfully integrate SEO and social media:

1. Make Promotion a Key Part of Your Content Strategy
Promoting your content on social media has a lot of benefits, but as long as it is disjointed from the content development process, you’ll be crippling your reach.

Instead, bring SEO and social media teams together to plan SEO-relevant content and how to best promote it in a social format.

Some articles like listicles or step-by-step guides can be easy to break down into LinkedIn posts, Twitter threads, etc. If you keep the teams close, you’ll have a more cohesive marketing strategy.

2. Create a Site & Social Content Calendar
Creating a content calendar for both website and social content will provide you with foresight on what needs to be done and when. This will also allow your team to work together and better coordinate efforts.

And, because social media is an almost real-time medium, you can create seasonal content on the platform to drive traffic to old pages that become relevant at specific times of the year.

3. Make Your Web Content Easily Shareable
Optimizing on-page SEO and other technical SEO factors only makes your blog ready for landing on search results. If you want to take advantage of the power of social media, you need to make your blog posts easily shareable, too.

You want to leverage your loyal readers who love your articles and want to share them with their friends and family on social media by adding social share buttons for each popular platform. It is also a good idea to craft a compelling call-to-action to encourage users to share your content.

On the technical side, you need to pay attention to your Open Graph Protocol to avoid any problems when sharing content on social apps.

4. Repurpose Content Effectively
Creating great content is no easy feat. It takes time and creativity to generate and implement a great piece of content. To leverage it effectively, a good idea is to take the best-performing content of one platform and optimize it to be used on a different one.

For example, if a post on LinkedIn generates a lot of comments, likes, and shares, it is a great indicator that your audience wants to learn or know more about the subject and that it might be worth writing a long-form piece for your blog.

Vice versa, high-performing articles, and pages can be repurposed into social posts to bring new eyes to the pages and keep the conversation going.

5. Use Social Media To Find Keywords
Keeping an eye on the latest trends on social media like Twitter, Facebook, and Instagram can help you identify fresh keywords that can earn you huge traffic wins without having to compete with other websites. Social media can be the best place to find those low-hanging fruits.

Another source of great keywords is recurring questions your audience sends you through social media. If enough people are asking the same things over and over, it can be an indicator of a content gap you need to fill.

Wrapping UP

Sure, social media doesn’t directly affect your organic rankings, but it definitely has a place in the SEO playbook. After all, content marketing fuels both strategies, so it is only natural that both channels complement each other.

In the end, it’s not just about improving your rankings but also improving the quality of traffic you drive to your site, and platforms like Quora, Twitter, and LinkedIn can help you find and attract your ideal audience.

Build an eBay Web Scraper: How to Extract Product Data Using Javascript

SaaS.Group — Fri, 29 Apr 2022 21:51:53 +0000

Originally published on ScraperAPI.

eBay is one of the largest eCommerce platforms in the world. With over 1 billion active listings on its site, it is also one of the largest data gold mines for pricing analysis, online purchase trends, and more. However, before analyzing their data, you need to extract it.

Today, we’ll build an eBay web scraper using Node.JS and Cheerio and show you the step-by-step process behind it – from idea to execution.

Is It Legal to Scrape eBay?

Yes, scraping eBay is totally legal if you’re not accessing data behind login walls or personal data without consent.

There’s still a lot of discussion around the legality of web scraping, and the fact that there are so many conflicting interests makes it hard to find reliable information. However, as it is today, it all depends on the type of data you’re scraping, how you’re extracting it, and its end-use.

In this article and all our tutorials, we’ll only show you ethical web scraping processes, so you can be confident you can apply these strategies without repercussions.

It’s important to know and understand the legal nuances of web scraping, so we build a complete guide to ensure legal and ethical web scraping practices.

Now, let’s start coding, shall we?

Scrape eBay Product Data with Cheerio

If you’ve been following our tutorial series, by now, we’ve gone through the basics of web scraping in JavaScript and built a more complex LinkedIn scraper using a for loop and the network tab in chrome’s DevTools.

Note: You don’t need to read those first to understand this tutorial, but it might help to get a clearer picture of our thought process.

To build on top of that, we’ll create an async function to scrape the name, price, and link of 4k TVs on eBay and then export the data into a CSV using the Object-to-CSV package.

. Understanding eBay’s URL Structure
Let’s go to https://www.ebay.com/ and search for “4k smart tv” on the search bar to grab our initial URL.

It sends us to the following URL:

https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2380057.m570.l1312&_nkw=4k+smart+tv&_sacat=0

If we were to scrape just this page, we could stop at this point and start writing our code. However, we want to see how it changes when moving through the pagination to understand how we can tell our script to do the same.

At first glance, it seems like the _sacat parameter stores the page number, so if we change it, it would be enough. Because eBay uses a numbered pagination, we can just click on the “Next” button and see how the URL changes.

Here’s the resulting URL:

https://www.ebay.com/sch/i.html?_from=R40&_nkw=4k+smart+tv&_sacat=0&_pgn=2

This is quite different from what we had before. So let’s go back and see if it returns to the previous version.

https://www.ebay.com/sch/i.html?_from=R40&_nkw=4k+smart+tv&_sacat=0&_pgn=1

No, it uses the “new” version of the URL when we use the pagination. This is great! All we need to do is change the _pgn parameter, and it will move to the next page. We confirmed this by just changing the number in the addressed bar.

Awesome, we’ll use this new version as our base URL for the HTTP request and later on to allow us to scrape every page in the series.

. Testing for JavaScript
Now that we have a consistent URL structure to work with, we need to test the website’s source code and make sure the data we want is available in the HTML and not injected through JavaScript – which would change our whole approach.

Of course, we already told you we were using Cheerio, so you know we’re not dealing with JavaScript, but here’s how you can test this for any website in the future.

Go to the page you want to scrape, right-click and click on “View Page Source”.

It will show you the site’s source code before any AJAX injection. We’ll copy the name and look for it in the Page Source for the test.

And next is the price.

We did the same with a few other products just to be sure, and we could find the element every time.

This step will tell us if we can go ahead and access the data using Axios and Cheerio or if we’ll need to use a tool like ScraperAPI’s JavaScript rendering to load the JS or a headless browser like Puppeteer.

3. Sending Our HTTP Request with Axios

The fun part begins! Let’s create a new folder for our project, open it on VScode (or your favorite editor) and start it with npm init -y to create the initial package.json file. From there, we can install Axios, a great and simple tool to send HTTP requests with Node.js, with npm install axios.

To send and test the request, let’s create a new file called index.js (original, we know), require Axios at the top, and create an async function. Inside it, we’ll send our request using Axios and store the response’s data inside a variable called html for clarity.

const axios = require('axios');

(async function () {

const response = await axios('https://www.ebay.com/sch/i.html?_from=R40&_nkw=4k+smart+tv&_sacat=0&_pgn=1');

const html = await response.data;

})();

Because we’re using async, we now have access to the await operator which is “used to wait for a Promise,” making it a great tool for web scraping, as our code will be more resilient.

Let’s console.log() the html variable to verify that our request is working:

console.log(html)

Ah yes! A bunch of nonsense, as expected.

. Parsing the Raw HTML with Cheerio
Before extracting our elements, we need to parse the raw data we downloaded to give it a structure we can navigate. That’s where Cheerio comes in!

We’ll create a new variable and pass html to cheerio using cheerio.load():

const $ = cheerio.load(html);

From there, let’s head back to the website to find our selectors.

Note: Don’t forget to install Cheerio with npm install cheerio before using it.

5. Picking the Right Selectors

The first thing we’re looking for is the element that contains all the data we’re looking for. So every product seems to be contained within a card, right?

We want to grab the element that contains all cards so we can then iterate through the list and extract the information we want (name, price, and URL – to refresh your memory).

This ul element wraps all product cards, so it is a great starting point to explore the HTML structure a little bit more.

Like we thought, every product card is a li element within the ul. All we need to do is grab all the li elements and assign them to a variable, effectively creating an array we can then go through to extract the data.

For testing, let’s open the browser’s console and use the li element’s class and see what gets returned:

Unlike Python’s Scrapy, Cheerio doesn’t have a built-in shell for testing, but we can use the console to test the selectors without having to send a request every time. We did the same thing with the rest of the elements.

We’ll pick the only h3 tag inside of each element within the list for the name.

In the case of the price, it’s wrapped within a span element with the class “s-item__price.”

Lastly, for the URL we needed to do something a little different. Although the a tag had a class we could use, it was shared by other elements outside our list. Notice how it returned 64 nodes instead of 59, which is the correct number of li elements.

Technically it would still work for us because we’ll be telling our scraper to look for the element inside the list. However, just to be sure, we’ll be going up in the hierarchy and grabbing the div containing the URL and then moving down to the a tag itself like this: 'div.s-item__info.clearfix > a'.

6. Extracting eBay Data

So the logic would be like this:

// Pick all li elements

// Go through each element within the list and extract the:

// tvName, tvPrice, tvLink

Let’s put it all together now, as we already know the selectors:

const tvs = $('li.s-item.s-item__pl-on-bottom.s-item--watch-at-corner');

tvs.each((index, element) => {

   const tvName = $(element).find('h3')

   const tvPrice = $(element).find('span.s-item__price')

   const tvLink = $(element).find('div.s-item__info.clearfix > a')

})

However, we’re not done yet. We need to add a new method at the end of each string, otherwise, we’ll be getting the entire HTML information which isn’t what we want.

We want the text inside the element for the name and price, so all we need to do is add the text() method at the end. For the URL, we want the value stored inside the href attribute, so we use the attr() method and pass the attribute we want the value from.

const tvs = $('li.s-item.s-item__pl-on-bottom.s-item--watch-at-corner');

tvs.each((index, element) => {

   const tvName = $(element).find('h3').text()

   const tvPrice = $(element).find('span.s-item__price').text()

   const tvLink = $(element).find('div.s-item__info.clearfix > a').attr('href')

})

We could log each variable to the console but we would be getting a lot of messy data. Instead, let’s give it some structure before testing the scraper.

*7. Pushing the Extracted Data to an Empty Array *

This is actually quite an easy process that will help us to organize the data and making it ready to export.

First, we’ll create an empty array outside our function.

const scrapedTVs = [];

From inside the function, we can call this variable and use the push() method to add all elements to our empty array. We’ll add the following snippet of code inside tvs.each(), right after the tvLink variable:

  scrapedTVs.push({

       'productName': tvName,

       'productPrice': tvPrice,

       'productURL': tvLink,

   })

Run the test with a console.log(scrapedTVs) and see what we get:

Nothing can beat the feeling of our code working! Our data is structured and clean. In perfect shape to be exported.

8. Exporting Our Data to a CSV

Exporting data into a CSV is made super simple with the ObjectsToCsv package. Just npm i objects-to-csv and add it to the dependencies at the top.

const ObjectsToCsv = require('objects-to-csv');

ObjectsToCsv has an easy syntax:

const csv = new ObjectsToCsv(scrapedTVs)

await csv.toDisk('./test.csv', { append: true })

console.log("Saved to CSV")

To create initiate the export, we need to create a new ObjectsToCsv() instance and pass it our dataset. Then, we’ll await the promise to resolve and save the result into a CSV file by giving it the path. We’re also setting append to true (it’s false by default) because we’re going to be adding more data to it from each page of the pagination.

For testing, we’ll log “Saved to CSV” to the console:

*9. Scrape eBay’s Pagination *

We already know we can scrape all pages inside the pagination by changing the _pgn parameter in the URL. So for this project, we can implement a for loop that changes this number after every iteration.

But we also need to know when to stop. Let’s head back to the website and see how many pages the pagination has.

It caps at 21. However, if we push the URL and add a 22 in the parameter, the page still responds with a page, but it loads the last page of the series, in other words, page 21.

We now can write the three statements for the for loop and put everything inside of it:

let pageNumber = 1 //to set the initial state

pageNumber <= 21 //it'll run as long as pageNumber is less or equal to 21

pageNumber += 1 //after running, pageNumber increments by 1

Here’s how it should look like:

for (let pageNumber = 1; pageNumber <= 21; pageNumber += 1) {

}

If we put all our previous code inside this for loop (which is inside of our async function), it’ll keep running until it meets the condition and breaks. Still, there are two changes we need to make before we call it for the day.

First, we need to add the pageNumber variable inside the URL, which can be done using ${} and a backtick (`) to surround the string. Like this:

https://www.ebay.com/sch/i.html?_from=R40&_nkw=4k+smart+tv&_sacat=0&_pgn=${pageNumber}

The second change we’ll want to make is sending our request through ScraperAPI servers to handle IP rotation and headers automatically. To do so, we’ll need to create a free ScraperAPI account. It’ll provide us with an API key and the string we’ll need to add to the URL for it to work:

http://api.scraperapi.com?api_key=51e43be283e4db2a5afb62660xxxxxxx&url=https://www.ebay.com/sch/i.html?_from=R40&_nkw=4k+smart+tv&_sacat=0&_pgn=${pageNumber}

This way we can avoid any kind of anti-scraping mechanism that could block our script.

*10. eBay Web Scraper Finished Code *

Here’s the finished code ready to use:

//Dependencies

const axios = require('axios');

const cheerio = require('cheerio');

const ObjectsToCsv = require('objects-to-csv');

//Empty array

const scrapedTVs = [];

(async function () {

//The for loop will keep running until pageNumber is greater than 21

for (let pageNumber = 1; pageNumber <= 21; pageNumber += 1) {

   //Sends the request, store the data and parse it with Cheerio

   const response = await axios(`http://api.scraperapi.com?api_key=51e43be283e4db2a5afb62660xxxxxxx&url=https://www.ebay.com/sch/i.html?_from=R40&_nkw=4k+smart+tv&_sacat=0&_pgn=${pageNumber}`);

   const html = await response.data;

   const $ = cheerio.load(html);



   //Grabs all the li elements containing the product cards

   const tvs = $('li.s-item.s-item__pl-on-bottom.s-item--watch-at-corner');

  //Goes through every element inside tvs to grab the data we're looking for

   tvs.each((index, element) => {

       const tvName = $(element).find('h3').text()

       const tvPrice = $(element).find('span.s-item__price').text()

       const tvLink = $(element).find('div.s-item__info.clearfix > a').attr('href')

       //Pushes all the extracted data to our empty array

       scrapedTVs.push({

           'productName': tvName,

           'productPrice': tvPrice,

           'productURL': tvLink,

       })

   });

   //Saves the data into a CSV file

   const csv = new ObjectsToCsv(scrapedTVs)

   await csv.toDisk('./scrapedTVs.csv', { append: true })

   //If everything goes well, it logs a message to the console

   console.log('Save to CSV')

}

})();

Note: Keep in mind that you’ll need to replace the API key in the code for your own key for it to work.

Great work! You now have a fast and effective eBay web scraper ready to be deployed.

To make it even more powerful, you could use the _nkw parameter inside the URL to make it easier to add a new term to the search, and with ScraperAPI by your side, you won’t have to worry about your IP getting blocked. However, that’s something we’ll leave to your imagination.

XPath Cheat Sheet for Web Scraping: Full Guide and XPath Examples

SaaS.Group — Mon, 25 Apr 2022 22:22:54 +0000

Originally published on ScraperAPI.

XML Path Language (XPath) is a query language and a major element of the XSLT standard. It uses a path-like syntax (called path expressions) to identify and navigate nodes in an XML and XML-like document.

In web scraping, we can take advantage of XPath to find and select elements from the DOM tree of virtually any HTML document, allowing us to create more powerful parsers in our scripts.

By the end of this guide, you’ll have a solid grasp of XPath expressions and how to use them in your scripts to scrape complex websites.

Understanding XPath Syntax

Writing XPath expressions is quite simple because it uses a structure we are all well versed in. You can imagine these path expressions like the ones we use in standard file systems.

There’s a root folder, and inside it has several directories, which could also contain more folders. XPath uses the relationship between these elements to traverse the tree and find the elements we’re targeting.

For example, we can use the expression //div to select all the div elements or write //div/p to target all paragraphs inside the divs. We can do this because of the nesting nature of HTML.

Using XPath to Find Elements With Chrome Dev Tools

Let’s use an example to paint a clearer picture. Navigate to https://quotes.toscrape.com/ and inspect the page.

Now we’ll be able to see the HTML of the website and pick an element using our XPath expressions. If we want to scrape all the quotes displayed on the page, all we need to do is to press cmd + f to initiate a search and write our expression.

Note: This is a great exercise to test your expressions before spending time on your code editor and without putting any stress on the site’s server.

If we take a closer look, we can see that all quotes are wrapped inside a div with the class quote, with the text itself inside a span element with the class text, so let’s follow that structure to write our path:

XPath: //div[@class=’quote’]/span[@class=’text’]

It highlighted the first element that matches our search, and also tell us that it’s the first of 10 elements, which perfectly matches the number of quotes on the page.

Note: It would also work fine with //span[@class='text'] because there’s only one span using that class. We want to be as descriptive as possible because, in most cases, we’ll be using XPath on websites with a messier structure.

Did you notice we’re using the elements’ attributes to locate them? XPath allows us to move in any direction and almost any way through the node tree. We can target classes, IDs, and the relationship between elements.

For the previous example, we can write our path like this: //div[@class='quote']/span[1]; and still, locate the element. This last expression would translate into finding all the divs with the class quote and picking the first span element.

Now, to summarize everything we’ve learned so far, here’s the structure of the XPath syntax:

Tagname – which is the name of the HTML element itself. Think of divs, H1s, etc.
Attribute – which can be IDs, classes, and any other property of the HTML element we’re trying to locate.
Value – which is the value stored in the attribute of the HTML element.

If you’re still having a hard time with this syntax, a great place to start is understanding what data parsing is and how it works. In this article, we go deeper into the DOM and its structure, which in return will make everything about XPath click better.

When to Use XPath vs. CSS for Web Scraping

If you’ve readen any of our Beautiful Soup tutorials or Cheerio guides, you’ve noticed by now that we tend to use CSS in pretty much every project. However, it’s all due to practicality.

In real-life projects, things will be a little more complicated, and understanding both will provide you with more tools to face any challenge.

So let’s talk about the differences between XPath and CSS selectors to understand when you should use one over the other.

Advantages of CSS for Web Scraping

CSS selectors tend to be easier to write and read than XPath selectors, making them more beginner-friendly for both learning and implementation.

For comparison, here’s how we would select a paragraph with the class easy:

XPAth: //p[@class=”easy”]
CSS: p.easy

Another thing to have in mind is that when working with a website that’s structured with unique IDs and very distinct classes, using CSS would be the best bet because picking elements based on CSS selectors is more reliable.

One change to the DOM and our XPath will break, making our script very susceptible. But classes and IDs very rarely change, so you’ll be able to pick up the element no matter if its position gets altered.

Although it might be very opinionated, we consider CSS as our first option for a project, and we only move to XPath if we can’t find an efficient way to use CSS.

Advantages of XPath for Web Scraping

Unlike CSS, XPath can traverse the DOM tree up and down, giving you more flexibility when working with less structured websites. This opens many opportunities to interact with the DOM that CSS doesn’t.

An easy example is imagining that you need to pick a specific parent div from a document with 15 different divs without any class, ID, or attribute. We won’t be able to use CSS effectively because there are no good targets to handle.

However, with XPath, we can target a child element of the div we need to select and go up from there.

With //span[@class=”text”]/.. we’re making a path to find the span element with the class text and then move to the parent element of that specific span, effectively moving up the DOM.

Another great use of XPath selectors/expressions is when trying to find a by matching its text – something that can’t be done with CSS – using the contains function.

** XPath:** //*[contains(text(), ‘world’)]

In the example above, our XPath expression matches two elements because both the quote and the tag have the word “world” in them. Although we won’t probably use this function too often in web scraping, it’s a great tool in niche situations.

If you want to learn more about the differences between these two, we recommend Exadel’s guide on picking selectors for automation.

Although it’s not directly related to web scraping, there’s a lot of value in learning about automation concepts.

XPath is a powerful language needed in many cases so let’s check some common expressions you can use while web scraping.

XPath Cheat Sheet: Common Expressions for Web Scraping

So if you’re into web scraping, here’s a quick cheat sheet you can use in your daily work. Save it to your bookmarks, and enjoy!

Note: You can test every expression in the “example” column on Quotes to Scrape for extra clarity and see what you’re selecting. Except for the ID example because the website doesn’t use IDs, lol.

These are probably the most common XPath expressions you’ll be using to select elements from an HTML document. However, these are not the only ones, and we encourage you to keep learning.

For example, you can also select elements that doesn’t contain certain text by using the expression //tagName[not(contains(text(), "someText"))]. This could come in handy if the website adds some text to the elements depending on a variable like adding “out of stock” to product titles inside a category page.

We can also use the OR logic when working with a class that changes depending on a variable using //tagName[@class="class1" or @class="class2"]. Telling our scraper to select an element that has one or the other class.

In a previous entry, we were scraping some stock data, but the “price percentage change” class name changed depending on whether the price was increasing or decreasing. Because the change was consistent, we could easily implement the OR logic with XPath and make our scraper extract the value no matter which class the element was using.

XPath: //span[@class="instrument-price_change-percent_19cas ml-2.5 text-positive-main" or @class="instrument-price_change-percent_19cas ml-2.5 text-negative-main"]

XPath Web Scraper Example

Before you go, we want to share with you a script written in Puppeteer, so you can see these XPath selectors in action.

Note: Technologies like Cheerio or Beautiful Soup do not work well – and in some cases at all – with XPath, so we recommend you to use things like Scrapy for Python and Puppeteer for JavaScript whenever you need to use XPath. These are more complicated tools to begin with but you’ll be an expert in no time.

Create a new folder called “xpathproject”, open it in VScode (or your preferred editor), initiate a new Node.js project using npm init -y, and install puppeteer inside – npm install puppeteer.

Next, create a new file with whatever name you’d like (we named it index.js for simplicity) and paste the following code:

const puppeteer = require('puppeteer');

scrapedText = [];

(async () => {

const browser = await puppeteer.launch({headless: false});

const page = await browser.newPage();

await page.goto('https://quotes.toscrape.com/');

await page.waitForXPath('//div[@class="quote"]/span[1]');

let elements = await page.$x('//div[@class="quote"]/span[1]');

const elementText = await page.evaluate((...elements) => {

   return elements.map(el => el.textContent);

}, ...elements)

scrapedText.push({

   'Results': elementText

})

console.log(scrapedText);

await browser.close();

})();

If you don’t have experience with Puppeteer, then check our guide on building web scrapers using Cheerio and Puppeteer. However, if you read this script carefully you’ll see how descriptive it is.

The best part is that you can use any XPath example on the XPath cheat sheet table and replace the expressions in the script and it’ll pull the text of the elements it finds.

It’s important to notice that this web scraper is made for pulling the text inside multiple elements so it might not work to just take the title of the page, for example.

Try different combinations and play a little bit with the script. You’ll soon get the hang of XPath.

Cheerio Vs Puppeteer for Web Scraping: Picking the Best Tool for Your Project

SaaS.Group — Fri, 15 Apr 2022 22:15:40 +0000

This post was originally featured on ScraperAPI.

Cheerio vs Puppeteer: Differences and When to Use Them

Cheerio and Puppeteer are both libraries made for Node.js (a backend runtime environment for Javascript) that can be used for scraping the web. However, they have major differences that you need to consider before picking a tool for your project.

Before moving into the details for each library, here’s an overview comparison between Cheerio and Puppeteer:

Cheerio Vs Puppeteer

Cheerio was built with web scraping in mind.
Puppeteer was designed for browser automation and testing
Cheerio is a DOM parser, able to parser HTML and XML files.
Puppeteer can execute Javascript, making it able to scrape dynamic pages like single-page applications (SPAs).
Cheerio can’t interact with the site or access content behind scripts.
Puppeteer can interact with websites, accessing content behind login forms and scripts.
Cheerio has an easy learning curve thanks to its simple syntax.
Puppeteer has a steep learning curve as it has more functionalities and requires Async for better results.
Cheerio is lightning fast in comparison to Puppeteer.
Compared to Cheerio, Puppeteer is quite slow.
Cheerio makes extracting data super simple using JQuery like syntax and CSS/XPath selectors to navigate the DOM.
Puppeteer can take screenshots, submit forms and make PDFs.

Now that you have a big picture vision, let’s dive deeper into what each library has to offer and how you can use them to extract alternative data from the web.

What is Cheerio?

Cheerio is a Node.js framework that parses raw HTML and XML data and provides a consistent DOM model to help us traverse and manipulate the result data structure. To select elements, we can use CSS and XPath selectors, making navigating the DOM easier.

However, Cheerio is well known for its speed. Because Cheerio doesn’t render the website like a browser (it doesn’t apply CSS or load external resources), Cheerio is lightweight and fast. Although in small projects we won’t notice, in large scraping tasks it will become a big time saver.

What is Puppeteer?

On the other hand, Puppeteer is actually a browser automation tool, designed to mimic users’ behavior to test websites and web applications. It “provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol.”

In web scraping, Puppeteer gives our script all the power of a browser engine, allowing us to scrape pages that require Javascript execution (like SPAs), scrape infinite scrolling, dynamic content, and more.

Should You Use Cheerio or Puppeteer for Web Scraping?

Although you might already have an idea of the best scenarios, let us take all doubts out of the way. If you want to scrape static pages that don’t require any interactions like clicks, JS rendering, or submitting forms, Cheerio is the best option, but If the website uses any form of Javascript to inject new content, you’ll need to use Puppeteer.

The reasoning behind our recommendation is that Puppeteer is just overkill for static websites. Cheerio will help you scrape more pages faster and in fewer lines of code.

That said, there are multiple cases where using both libraries is actually the best solution. After all, Cheerio can make it easier to parse and select elements, while Puppeteer would give you access to content behind scripts and help you automate events like scrolling down for infinite paginations.

Building a Scraper with Cheerio and Puppeteer [Code Example]

To make this example easy to follow, we’ll build a scraper using Puppeteer and Cheerio that’ll navigate to https://quotes.toscrape.com/ and bring back all quotes and authors from page 1.

Installing Node.js, Cheerio, and Puppeteer
We’ll download Node.js from the official site and follow the instructions from the installer. Then, we’ll create a new project folder (we named it ‘cheerio-puppeteer-project’) and open it inside VScode – you can use any other editor you’d prefer. Inside your project folder, open a new terminal and type npm init -y to kickstart your project.

Open the Target Website Using Puppeteer

Now we’re ready to install our dependencies using npm install cheerio puppeteer. After a few seconds, we should be ready to go. Create a new file named ‘index.js’ and import our dependencies at the top.

_const puppeteer = require('puppeteer');

const cheerio = require('cheerio');_

Next, we’ll create an empty list named scraped_quotes to store all our results, followed by our async function, so we can have access to the await operator. Just so we don’t forget, we’ll write a browser.close() method at the of our function.

_scraped_quotes = [];

(async () => {

await browser.close();

});_

Using Puppeteer, let’s launch a new browser instance, open a new page and navigate to our target website.

_const browser = await puppeteer.launch();

const page = await browser.newPage();

await page.goto('https://quotes.toscrape.com/');_

Parsing the HTML with Cheerio

To get access to the HTML of the website, we can use evaluate and return the raw HTML data – this is an important step because Cheerio can only work with HTML or XML data, so we need to access it before being able to parse it.

_ const pageData = await page.evaluate(() => {

   return {

       html: document.documentElement.innerHTML,

   };

});_

For testing purposes, we can use console.log(pageData) to log the response to our terminal. Because we already know it works, we’ll send the raw HTML to Cheerio for parsing.

_ const $ = cheerio.load(pageData.html);_

Now we can use $ to refer to the parsed version of the HTML file for the rest of our project.

Selecting Elements with Cheerio

Before we can actually write our code, we first need to find out how the page is structured. Let’s go to the page itself on our browser and inspect the cards containing the quotes.

We can see that the elements we’re interested in are inside a div with the class quote. So we can select them and iterate through all of the divs to extract the quote text and the author.

After inspecting these elements, here are our targets:

Divs containing our target elements: $('div.quote')
Quote text: $(element).find('span.text')
Quote author: $(element).find('.author')

Let’s translate this into code:

_let quote_cards = $('div.quote');

quote_cards.each((index, element) => {

   quote = $(element).find('span.text').text();

   author = $(element).find('.author').text();

});
_
Using the text() method we can access to the text inside the element instead of returning the string of HTML.

Pushing the Scraped Data Into a Formatted List

If we console.log() our data at this point, it will be a messy chunk of text. Instead, we’ll use the empty list we created outside our function and push the data over there. To do so, add these two new lines to your script, right after your author variable:

_ scraped_quotes.push({

       'Quote': quote,

       'By': author,

   })_

Finished Code Example

Now that everything is in place, we can console.log(scraped_quotes) before closing the browser:

_//dependencies

const puppeteer = require('puppeteer');

const cheerio = require('cheerio');

//empty list to store our data

scraped_quotes = [];

//main function for our scraper

(async () => {

//launching and opening our page

const browser = await puppeteer.launch();

const page = await browser.newPage();

//navigating to a URL

await page.goto('https://quotes.toscrape.com/');

//getting access to the raw HTML

const pageData = await page.evaluate(() => {

   return {

       html: document.documentElement.innerHTML,

   };

});

//parsing the HTML and picking our elements

const $ = cheerio.load(pageData.html);

let quote_cards = $('div.quote');

quote_cards.each((index, element) => {

   quote = $(element).find('span.text').text();

   author = $(element).find('.author').text();

   //pushing our data into a formatted list

   scraped_quotes.push({

       'Quote': quote,

       'By': author,

   })

});

//console logging the results

console.log(scraped_quotes);

//closing the browser

await browser.close();

})();_

Resulting in a formatted list of data:

I hope you enjoyed this quick overview of arguably the two best web scraping tools available for Javascript/Node.js. Although in most cases you’ll want to use Cheerio over Puppeteer, for those extra complex projects Puppeteer brings the extra tools you’ll need to get the job done.

How to Scrape Stock Market Data in Python [Practical Guide, Plus Code]

SaaS.Group — Tue, 08 Mar 2022 23:25:04 +0000

This article was originally published on ScraperAPI

Whether you’re an investor tracking your portfolio, or an investment firm looking for a way to stay up-to-date more efficiently, creating a script to scrape stock market data can save you both time and energy.

In this tutorial, we’ll build a script to track multiple stock prices, organize them into an easy-to-read CSV file that will update itself with the push of a button, and collect hundreds of data points in a few seconds.

Building a Stock Market Scraper With Requests and Beautiful Soup

For this exercise, we’ll be scraping investing.com to extract up-to-date stock prices from Microsoft, Coca-Cola, and Nike, and storing it in a CSV file. We’ll also show you how to protect your bot from being blocked by anti-scraping mechanisms and techniques using ScraperAPI.

Note: The script will work to scrape stock market data even without ScraperAPI, but will be crucial for scaling your project later.

Although we’ll be walking you through every step of the process, having some knowledge of the Beautiful Soup library beforehand is helpful. If you’re totally new to this library, check out our beautiful soup tutorial for beginners. It’s packed with tips and tricks, and goes over the basics you need to know to scrape almost anything.

With that out of the way, let’s jump into the code so you can learn how to scrape stock market data.

1. Setting Up Our Project
To begin, we’ll create a folder named “scraper-stock-project”, and open it from VScode (you can use any text editor you’d like). Next, we’ll open a new terminal and install our two main dependencies for this project:

pip3 install bs4
pip3 install requests
After that, we’ll create a new file named “stockData-scraper.py” and import our dependencies to it.

import requests

from bs4 import BeautifulSoup

With Requests, we’ll be able to send an HTTP request to download the HTML file which is then passed on to BeautifulSoup for parsing. So let’s test it by sending a request to Nike’s stock page:

url = 'https://www.investing.com/equities/nike'

page = requests.get(url)

print(page.status_code)

By printing the status code of the page variable (which is our request), we’ll know for sure whether or not we can scrape the page. The code we’re looking for is a 200, meaning it was a successful request.

Success! Before moving on, we’ll pass the response stored in page to Beautiful Soup for parsing:

soup = BeautifulSoup(page.text, 'html.parser')

You can use any parser you want, but we’re going with html.parser because it’s the one we like.

Related Resource: What is Data Parsing in Web Scraping? [Code Snippets Inside]

2. Inspect the Website’s HTML Structure (Investing.com)
Before we start scraping, let’s open https://www.investing.com/equities/nike in our browser to get more familiar with the website.

As you can see in the screenshot above, the page displays the company’s name, stock symbol, price, and price change. At this point, we have three questions to answer:

Is the data being injected with JavaScript?
What attribute can we use to select the elements?
Are these attributes consistent throughout all pages?
Check for JavaScript
There are several ways to verify if some script is injecting a piece of data, but the easiest way is to right-click, View Page Source.

It looks like there isn’t any JavaScript that could potentially interfere with our scraper. Next we’ll do the W same for the rest of the information. We didn’t find any additional JavaScript we’re good to go.

Note: Checking for JavaScript is important because Requests can’t execute JavaScript or interact with the website, so if the information is behind a script, we would have to use other tools to extract it, like Selenium.

Picking the CSS Selectors
Now let’s inspect the HTML of the site to identify the attributes we can use to select the elements.

Extracting the company’s name and the stock symbol will be a breeze. We just need to target the H1 tag with class ‘text-2xl font-semibold instrument-header_title__GTWDv mobile:mb-2’.

However, the price, price change, and percentage change are separated into different spans.

What’s more, depending on whether the change is positive or negative, the class of the element changes, so even if we select each span by using their class attribute, there will still be instances when it won’t work.

The good news is that we have a little trick to get it out. Because Beautiful Soup returns a parsed tree, we can now navigate the tree and pick the element we want, even though we don’t have the exact CSS class.

What we’ll do in this scenario is go up in the hierarchy and find a parent div we can exploit. Then we can use find_all(‘span’) to make a list of all the elements containing the span tag – which we know our target data uses. And because it’s a list, we can now easily navigate it and pick those we need.

So here are our targets:

company = soup.find('h1', {'class': 'text-2xl font-semibold instrument-header_title__GTWDv mobile:mb-2'}).text

price = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[0].text

change = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[2].text

Now for a test run:

print('Loading: ', url)

print(company, price, change)

And here’s the result:

3. Scrape Multiple Stocks
Now that our parser is working, let’s scale this up and scrape several stocks. After all, a script for tracking just one stock data is likely not going to be very useful.

We can make our scraper parse and scrape several pages by creating a list of URLs and looping through them to output the data.

urls = [

'https://www.investing.com/equities/nike',

'https://www.investing.com/equities/coca-cola-co',

'https://www.investing.com/equities/microsoft-corp',

]

for url in urls:

page = requests.get(url)

soup = BeautifulSoup(page.text, 'html.parser')

company = soup.find('h1', {'class': 'text-2xl font-semibold instrument-header_title__GTWDv mobile:mb-2'}).text

price = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[0].text

change = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[2].text

print('Loading: ', url)

print(company, price, change)

Here’s the result after running it:

Awesome, it works across the board!

We can keep adding more and more pages to the list but eventually, we’ll hit a big roadblock: anti-scraping techniques.

4. Integrating ScraperAPI to Handle IP Rotation and CAPCHAs
Not every website likes to be scraped, and for a good reason. When scraping a website, we need to have in mind that we are sending traffic to it, and if we’re not careful, we could be limiting the bandwidth the website has for real visitors, or even increasing hosting costs for the owner. That said, as long as we respect web scraping best practices, we won’t have any problems with our projects, and we won’t cause the sites we’re scraping any issues.

However, it’s hard for businesses to differentiate between ethical scrapers and those that will break their sites. For this reason, most servers will be equipped with different systems like

Browser behavior profiling
CAPTCHAs
Monitoring the number of requests from an IP address in a time period
These measures are designed to recognize bots, and block them from accessing the website for days, weeks, or even forever.

Instead of handling all of these scenarios individually, we’ll just add two lines of code to make our requests go through ScraperAPI’s servers and get everything automated for us.

First, let’s create a free ScraperAPI account to access our API key and 5000 free API credits for our project.

Now we’re ready to add to our loop a new params variable to store our key and target URL and use urlencode to construct the URL we’ll use to send the request inside the page variable.

params = {'api_key': '51e43be283e4db2a5afb62660xxxxxxx', 'url': url}

page = requests.get('http://api.scraperapi.com/', params=urlencode(params))

Oh! And we can’t forget to add our new dependency to the top of the file:

from urllib.parse import urlencode

Every request will now be sent through ScraperAPI, which will automatically rotate our IP after every request, handle CAPCHAs, and use machine learning and statistical analysis to set the best headers to ensure success.

Quick Tip: ScraperAPI also allows us to scrape a dynamic site by setting ‘render’: true as a parameter in our params variable. ScraperAPI will render the page before sending back the response.

5. Store Data In a CSV File
Tos tore your data in an easy-to-use CSV file, simply add these three lines between your URL list and your loop:

file = open('stockprices.csv', 'w')

writer = csv.writer(file)

writer.writerow(['Company', 'Price', 'Change'])=

This will create a new CSV file and pass it to our writer (set in the writer variable) to add the first row with our headers.

It’s essential to add it outside of the loop, or it will rewrite the file after scraping each page, basically erasing previous data and giving us a CSV file with only the data from the last URL from our list.

In addition, we’ll need to add another line to our loop to write the scraped data:

writer.writerow([company.encode('utf-8'), price.encode('utf-8'), change.encode('utf-8')])

And one more outside the loop to close the file:

file.close()

6. Finished Code: Stock Market Data Script
You’ve made it! You can now use this script with your own API key and add as many stocks as you want to scrape:

dependencies

import requests

from bs4 import BeautifulSoup

import csv

from urllib.parse import urlencode

list of URLs

urls = [

'https://www.investing.com/equities/nike',

'https://www.investing.com/equities/coca-cola-co',

'https://www.investing.com/equities/microsoft-corp',

]

starting our CSV file

file = open('stockprices.csv', 'w')

writer = csv.writer(file)

writer.writerow(['Company', 'Price', 'Change'])

looping through our list

for url in urls:

#sending our request through ScraperAPI

params = {'api_key': '51e43be283e4db2a5afb62660fc6ee44', 'url': url}

page = requests.get('http://api.scraperapi.com/', params=urlencode(params))

#our parser

soup = BeautifulSoup(page.text, 'html.parser')

company = soup.find('h1', {'class': 'text-2xl font-semibold instrument-header_title__GTWDv mobile:mb-2'}).text

price = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[0].text

change = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[2].text

#printing to have some visual feedback

print('Loading :', url)

print(company, price, change)

#writing the data into our CSV file

writer.writerow([company.encode('utf-8'), price.encode('utf-8'), change.encode('utf-8')])

file.close()

Wrapping Up: considerations when running your stock market data scraper

You need to remember that the stock market isn’t always open. For example, if you’re scraping data from NYC’s stock exchange, it closes at 5 pm EST on Fridays and opens on Monday at 9:30 am. So there’s no point in running your scraper over the weekend. It also closes at 4 pm so that you won’t see any changes in the price after that.

Another variable to keep in mind is how often you need to update the data. The most volatile times for the stock exchange are opening and closing times. So it might be enough to run your script at 9:30 am, at 11 am, and at 4:30 pm to see how the stocks closed. Monday’s opening is also crucial to monitor as many trades occur during this time.

Unlike other markets like Forex, the stock market typically doesn’t make too many crazy swings. That said, oftentimes news and business decisions can heavily impact stock prices – take Meta shares crash or the rise of GameStop share price as examples – so reading the news related to the stocks you are scraping is vital.

We hope this tutorial helped you build your own stock market data scraper, or at least pointed you in the right direction. In a future tutorial, we’ll build on top of this project to create a real-time stock data scraper to monitor your stocks, so stay tuned for that!

10 Best Technical SEO Tools You Need for Your Next Audit

SaaS.Group — Thu, 24 Feb 2022 01:09:42 +0000

Originally published on Prerender.
You want to do routine technical website audits for the same reason you take your car to see a mechanic – to make sure it’s working as it should.

Technical SEO tools are like the wrenches and hammers the mechanic keeps in a toolbox – they help you identify any potential errors or issues with your website so that you can fix them before they cause long-term damage. Website audits can also help improve your SEO rankings and visibility. Websites that load quickly and are technically well-optimized tend to get higher search rankings and more organic traffic.

In this blog post, we will discuss the 10 best technical SEO tools you need to get the most out of your next website audit.

*What are Technical SEO tools? *

Technical SEO tools are specifically designed to help you audit your website for technical errors and issues. They can help you identify problems such as:

Broken links
Incorrect, or missing, titles and metadata
Coding errors
Fixing these issues helps you improve your site’s performance. Plus, they offer many benefits that you can use to revamp and optimize your SEO strategy.

Optimize Your SEO Strategy With Technical SEO Tools

SEO audit tools offer a wide range of benefits, including gaining insights into your website’s inner workings. Also, they help identify how your competitors are optimizing their websites so you can reverse-engineer their strategies.

Technical SEO software can also be used for keyword and backlink research. You can take what you learn about your website’s keyword optimization and backlink profile, and use it to improve your website technically to make your on-page SEO and link-building efforts more effective.

Save Time and Costs on Manual SEO Audits

Technical website tools can also assist you by saving time and hefty costs on manual technical SEO audits. Getting a technical SEO specialist to audit your website and having your developers implement those recommendations can require a team of specialists working together, and specialists don’t come cheap.

You can automate the auditing process, which will help speed up results. Many of these tools are free or offer a free trial, so you can test them out before committing to them.

Find High-Converting Keywords for Content & Content Marketing

With every SEO expert clamoring to rank for high-converting keywords, technical SEO tools can help you rank for them before they can. Technical SEO tools give you useful information like your competitor’s website performance, backlink profile, and keyword rankings. Ranking for these competitive keywords can get your content to generate more traffic.

Gauge SEO Progress & Track KPIs

Most SEO audit tools generate automated reports that track your SEO progress over time, based on a set of practical key performance indicators (KPIs). This data can help you determine whether your website is running well and what still needs improvement. It can also help you compare your website to your competitors’ sites and make strategic adjustments to stay ahead of them.

Visualize & Conceptualize Data for Analysis

Converting raw data into visual elements like charts and infographics can help you understand the data more easily and identify opportunities for improvement. After analyzing that data, you can identify and address any web issues and apply the information to enhance your strategies.

10 Best Technical SEO Tools

To give you an extra edge in your website audits, here are 10 of the best technical SEO tools that we recommend having in your toolkit. These tools will help you identify opportunities, develop a more effective SEO strategy and further optimize your website for top performance.

list of 10 best technical SEO tools

Prerender
Prerender boosts search engine rankings by effectively prerendering Javascript during the search engine crawling process. It improves your crawl budget optimization and amplifies your page speed scores.

A convenient feature of Prerender is it detects when Google is crawling your page and creates an easily-crawled version of your website. It also renders JavaScript websites built with a wide range of popular frameworks and libraries like Angular, React, and Vue.JS.

It also detects whether it’s a bot or human user who is accessing your site so it can provide the best user experience possible for human users. Afterward, it renders an HTML version for the search engine crawler and interactive JavaScript for the human, so you can optimize your crawl budget without sacrificing user experience.

Screaming Frog
Screaming Frog is a heavy-duty, yet flexible, website crawler that helps you audit your website for technical errors and issues.

Whether you have a large or small website, Screaming Frog can efficiently crawl and analyze your website in real-time. It’s an equally suitable SEO tool for beginner and experienced SEO professionals.

It helps extract and analyze data for common SEO issues, which can help you find ways to tweak your on-page SEO tactics.

Screaming Frog will let you crawl up to 500 URLs for free. Buying a license removes this limit and gives you access to more advanced features. If you have a new website, you can take advantage of its free version so you can audit your website without any additional cost, then invest in the paid license as your website grows. However, there are many Screaming Frog alternatives that you can use to meet your website’s specific technical SEO needs.

Google Search Console
Google Search Console is a free SEO audit tool offered by Google that helps you monitor your website’s performance in the SERPs. It also allows you to submit and track your website’s XML sitemaps.

This tool is a must-have for business owners who want to make sure their website is receiving optimal search engine visibility. With Google Search Console, you can identify any errors or issues that may be preventing your website from ranking higher than it could.

You can also use Google Search Console to analyze and benchmark your site’s performance and compare it to your competitors. This data can help you make necessary adjustments to stay ahead of the competition.

Google Search Console allows you to:

Submit your sitemaps to be crawled and indexed
Monitor your website’s overall performance and search traffic
Identify any indexing or crawling errors and issues with your website such as soft 404 errors
Identify keywords and search queries that are bringing traffic to your site
View the landing pages with the most traffic
Check hreflang implementation on a site

*Google Mobile-Friendly Test *
Another practical SEO tool offered by Google is the Google Mobile-Friendly Test. With this free tool, you can determine if your website is mobile-friendly.

Over the years, Google has led the movement towards mobile responsiveness in web development. That’s why they made the Google Mobile-Friendly Test to guide webmasters and technical SEO experts in giving their users a better experience on mobile devices. The tool also lets you identify any critical alerts or load time issues, and it helps you make sure your site loads and displays correctly on smartphones and tablets.

The Google Mobile-Friendly Test allows you to:

Check how visitors interact with the mobile version of your pages
See how well your pages display on different types of mobile devices
Test individual web pages or entire websites
Get feedback on what you can do to make your website more mobile-friendly
Send alerts if critical errors are found on your website
Identify issues that might cause a decrease in load times
A valuable feature of the Google Mobile-Friendly Test is that it gives you a score and breakdown of how mobile-friendly a page is. This information can help you determine where you need to make changes to improve your site’s mobile-friendliness.

Google’s Schema.org Structured Data Testing Tool
Structured data is information stored in a specific format, which allows search engines to better understand what the page is and what it’s about. With Google’s Schema.org Structured Data Testing Tool, you can identify errors in your site’s structured data markup.

It helps you ensure that your website’s structured data markup is accurate and error-free, which in turn can improve your website’s search rankings. It allows you to select the appropriate tool, the rich results test, or the schema markup validator. The rich results test tests your website to identify and preview the correct search engine-rich results. The schema markup validator approves all structured data embedded in your web pages to make sure it’s formatted correctly.

W3C Validator
The W3C Validator is a free, open-source tool that checks the markup validity of your Web documents in HTML, XHTML, SMIL, MathML, and other markup languages.

By using the W3C Validator, you can ensure that your web pages are debugged and error-free. This can help you avoid penalties that may be associated with incorrect or poorly-written markup and boost your overall search rankings.

Overall, website maintenance will be more efficient as you can solve any website code error that may arise.

Majestic
Majestic is a paid tool that helps you track and analyze your website’s backlinks.

Majestic is praised in the industry as an efficient performance tool that offers the largest crawler system and extensive backlink data repository for comprehensive backlink analysis.

By using Majestic, you can get a detailed SEO report of all of your website’s backlinks including:

Number of backlinks
Referring domains of these links
Anchor text associated with them
If you’re looking for a way to track and analyze your website’s backlinks, then be sure to try out Majestic.

*Barracuda Panguin Tool *
The Barracuda Panguin Tool is a free tool that helps you identify whether a website has been penalized by Google. This can help you determine the cause of your website’s poor rankings and take steps to fix them.

It also helps you track your website traffic when Google releases an update. With this, you can pinpoint what update penalized your website so you plan strategies to account for them.

Barracuda offers a wide range of features to help you make the best results-driven choices to optimize your site:

Mobile searches
On-site link optimization
Local and regional SEO insights
Data encryptions
Paid and organic visibility

Google Search Console XML Sitemap
An XML sitemap is a file that contains information about the pages on your website. The Google Search Console XML Sitemap is a free tool that helps you diagnose any problems with your sitemap. If an issue has been identified, you can then troubleshoot and monitor it quickly and easily without having to dig through your website’s code.

With Google Search Console XML Sitemap, you can make sure that your website’s pages are being crawled and indexed by Google, which is step 1 in improving your website’s search engine rankings. You can also receive alerts for any critical errors with your site, including indexing errors and spam. If you’re using Accelerated Mobile Pages (AMP), this technical SEO tool helps you to troubleshoot and resolve any issues that happen with them.

Botify
Botify creates the ideal mix of technical SEO, content, and search intent data to create a comprehensive keyword analysis.

Also, it gathers insights throughout each stage of the crawling and indexing process to efficiently automate any required SEO tasks.

From start to finish, Botify compiles:

Internal page rank
Broken links analysis
Page depth

Measure, Analyze and Optimize Your Website with These Top Technical SEO Tools

SEO is a complex process that requires careful measurement and analysis to improve your rankings. Using some of the technical SEO tools listed in this article can help you measure, analyze, and optimize your website for better search engine results.

How to Deal with Pagination in Python Step-By-Step Guide

SaaS.Group — Fri, 18 Feb 2022 00:20:04 +0000

Originally posted on ScraperAPI
If you’re working on a large web scraping project (like scraping product information) you have probably stumbled upon paginated pages. It’s standard practice for eCommerce and content sites to break down content into multiple pages to improve user experience. However, web scraping pagination adds some complexity to our work.

In this article, you’ll learn how to build a pagination web scraper in just a few minutes and without getting blocked by any anti-scraping techniques.

Although you can follow this tutorial with no prior knowledge, it might be a good idea to check out our Scrapy for beginners guide first for a more in-depth explanation of the framework before you get started.

Without further ado, let’s jump right into it!

Scraping a Website with Pagination Using Python Scrapy
For this tutorial, we’ll be scraping the SnowAndRock men’s hats category to extract all product names, prices, and links.

A little disclaimer- we’re writing this article using a Mac, so you’ll have to adapt things a little bit to work on PC. Other than that, everything should be the same.

TLDR: here’s a quick snippet to deal with pagination in Scrapy using the “next” button:

next_page = response.css('a[rel=next]').attrib['href']

if next_page is not None:

   yield response.follow(next_page, callback=self.parse)

Keep reading for an in-depth explanation on how to implement this code into your script, along with how to deal with pages without a next button.

Set Up Your Development Environment Before we start writing any code, we need to set up our environment to work with Scrapy, a Python library designed for web scraping. It allows us to crawl and extract data from websites, parse the raw data into a structured format, and select elements using CSS and/or XPath selectors.

First, let’s create a new directory (we’ll call it pagination-scraper) and create a python virtual environment inside using the command python -m venv venv. Where the second venv is the name of your environment – but you can call it whatever you want.

To activate it, just type source venv/bin/activate. Your command prompt should look like this:

Now, installing Scrapy is as simple as typing pip3 install scrapy – it might take a few seconds for it to download and install it.

Once that’s ready, we’ll input cd venv and create a new Scrapy project: scrapy startproject scrapypagination.

Now you can see that Scrapy kick-started our project for us by installing all the necessary files.

Setting Up ScraperAPI to Avoid Bans The hardest part of handling paginated pages is not writing the script itself, it’s how to not get our bot blocked by the server.

For that, we’ll need to create a function (or set of functions) that rotates our IP address after several attempts (meaning we also need access to a pool of IP addresses). Also, some websites use advanced techniques like CAPTCHAs and browser behavior profiling.

To save us time and headaches, we’ll use ScraperAPI, an API that uses machine learning, huge browser farms, 3rd party proxies, and years of statistical analysis to handle every anti-bot mechanism our script could encounter automatically.

Best of all, setting up ScraperAPI into our project is super easy with Scrapy:

import scrapy

from urllib.parse import urlencode

API_KEY = '51e43be283e4db2a5afb62660xxxxxxx'

def get_scraperapi_url(url):

payload = {'api_key': API_KEY, 'url': url}

proxy_url = 'http://api.scraperapi.com/?' + urlencode(payload)

return proxy_url

As you can see, we’re defining the get_scraperapi_url() method to help us construct the URL we’ll send the request to. First, we added our dependencies on the top and then added the API_KEY variable containing our API key – to get your key, just sign up for a free ScraperAPI account and you’ll find it on your dashboard.

This method will build the URL for the request for each URL our scraper finds, and that’s why we’re setting it up this way instead of the more direct way of just adding all parameters directly into the URL like this:

start_urls = ['http://api.scraperapi.com?api_key={yourApiKey}&url={URL}']

Understanding the URL Structure of the Website URL structure is pretty much unique to each website. Developers tend to use different structures to make it easier to navigate for them and, in some cases, optimize the navigation experience for search engine crawlers like Google and real users.

To scrape paginated content, we need to understand how it works and plan accordingly, and there’s no better way to do it than inspecting the pages and seeing how the URL itself changes from one page to the next.

So if we go to https://www.snowandrock.com/c/mens/accessories/hats.html and scroll to the last product listed, we can see that it uses a numbered pagination plus a next button.

This is great news, as selecting the next button on every page will be easier than cycling through each page number. Still, let’s see how the URL changes when clicking on the second page.

Here’s what we’ve found:

Page 1: https://www.snowandrock.com/c/mens/accessories/hats.html?page=0&size=48
Page 2: https://www.snowandrock.com/c/mens/accessories/hats.html?page=1&size=48
Page 3: https://www.snowandrock.com/c/mens/accessories/hats.html?page=2&size=48
Notice that the page one URL changes when you go back to the page using the navigation, changing to page=0. Although we’re going to use the next button to navigate this website’s pagination, it is not as simple in every case.

Understanding this structure will help us build a function to change the page parameter in the URL and increase it by 1, allowing us to go to the next page without a next button.

Note: not all pages follow this same structure so make sure to always check which parameters change and how.

Now that we know the initial URL for the request we can create a custom spider.

Sending the Initial Request Using the Start_Requests() Method For the initial request we’ll create a Spider class and give it the name of Pagi:

class PaginationScraper(scrapy.Spider):

name = "pagi"

Then, we define the start_requests() method:

def start_requests(self):

start_urls = ['https://www.snowandrock.com/c/mens/accessories/hats.html']

for url in start_urls:

    yield scrapy.Request(url=get_scraperapi_url(url), callback=self.parse)

Now, after running our script, it will send each new URL found to this method, where the new URL will merge with the result of the get_scraperapi_url() method, sending the request through the ScraperAPI severs and bullet-proofing our project.

Building Our Parser After testing our selectors with Scrapy Shell, these are the selectors we came up with:

def parse(self, response):

for hats in response.css('div.as-t-product-grid__item'):

    yield {

           'name': hats.css('.as-a-text.as-m-product-tile__name::text').get(),

           'price': hats.css('.as-a-price__value--sell strong::text').get(),

           'link': hats.css('a').attrib['href'],

           }

If you’re not familiar with Scrapy Shell or with Scrapy in general, it might be a good idea to check our full Scrapy tutorial where we cover all the basics you need to know.

However, we’re basically selecting all the divs containing the information we want (response.css('div.as-t-product-grid__item') and then extracting the name, the price, and product’s link.

Make Scrapy Move Through the Pagination Great! We have the information we need from the first page, now what? Well, we’ll need to tell our parser to find the new URL somehow and send it to the start_requests() method we defined before. In other words, we need to find an ID or class we can use to get the link inside the next button.

Technically we could use the class ‘.as-a-btn.as-a-btn--pagination as-m-pagination__item’ but lucky for us, there’s a better target: rel=next. It won’t get confused with any other selectors and picking an attribute with Scrapy is simple.

next_page = response.css('a[rel=next]').attrib['href']

if next_page is not None:

    yield response.follow(next_page, callback=self.parse)

Now it will iterate between pages until there are no more pages in the pagination – so we don’t need to set any other stop mechanism.

If you’ve been following along, your file should look like this:

import scrapy

from urllib.parse import urlencode

API_KEY = '51e43be283e4db2a5afb62660xxxxxx'

def get_scraperapi_url(url):

payload = {'api_key': API_KEY, 'url': url}

proxy_url = 'http://api.scraperapi.com/?' + urlencode(payload)

return proxy_url

class PaginationScraper(scrapy.Spider):

name = "pagi"

def start_requests(self):

start_urls = ['https://www.snowandrock.com/c/mens/accessories/hats.html']

for url in start_urls:

   yield scrapy.Request(url=get_scraperapi_url(url), callback=self.parse)

def parse(self, response):

for hats in response.css('div.as-t-product-grid__item'):

   yield {

          'name': hats.css('.as-a-text.as-m-product-tile__name::text').get(),

          'price': hats.css('.as-a-price__value--sell strong::text').get(),

          'link': hats.css('a').attrib['href'],

          }

next_page = response.css('a[rel=next]').attrib['href']

if next_page is not None:

   yield response.follow(next_page, callback=self.parse)

It is now ready to run!

Dealing With Pagination Without Next Button
So far we’ve seen how to build a web scraper that moves through pagination using the link inside the next button – remember that Scrapy can’t actually interact with the page so it won’t work if the button has to be clicked in order for it to show more content.

However, what happens when it isn’t an option? In other words, how can we navigate a pagination without a next button to rely on.

Here’s where understanding the URL structure of the site comes in handy:

Page 1: https://www.snowandrock.com/c/mens/accessories/hats.html?page=0&size=48
Page 2: https://www.snowandrock.com/c/mens/accessories/hats.html?page=1&size=48
Page 3: https://www.snowandrock.com/c/mens/accessories/hats.html?page=2&size=48
The only thing changing between URLs is the page parameter, which increases by 1 for each next page. What does it mean for our script? Well, first of all, we’ll have to change the way we’re sending the initial request by adding a new variable:

class PaginationScraper(scrapy.Spider):

name = "pagi"

page_number = 1

start_urls = ['http://api.scraperapi.com?api_key=51e43be283e4db2a5afb62660xxxxxxx&url=https://www.snowandrock.com/c/mens/accessories/hats.html?page=0&size=48']

In this case we’re also using the direct cURL structure of ScraperAPI because we’re just changing a parameter- meaning there’s no need to construct a whole new URL. This way every time it changes, it will still send the request through ScraperAPI’s servers.

Next, we’ll need to change our condition at the end to match the new logic:

next_page = 'http://api.scraperapi.com?api_key=51e43be283e4db2a5afb62660xxxxxxx&url=https://www.snowandrock.com/c/mens/accessories/hats.html?page=' + str(PaginationScraper.page_number) + '&size=48'

if PaginationScraper.page_number < 6:

    PaginationScraper.page_number += 1

    yield response.follow(next_page, callback=self.parse)

What’s happening here is that we’re accessing the page_number variable from the PaginationScraper() method to replace the value of the page parameter inside the URL.

Afterwards, it will check if the value of page_number is less than 6 – because after page 5 there are no more results.

As long as the condition is met, it will increase the page_number value by 1 and send the URL to be parsed and scraped, and so on until the page_number is 6 or more.

Here’s the full code to scrape paginated pages without a next button:

import scrapy

class PaginationScraper(scrapy.Spider):

name = "pagi"

page_number = 1

start_urls = ['http://api.scraperapi.com?api_key=51e43be283e4db2a5afb62660xxxxxxx&url=https://www.snowandrock.com/c/mens/accessories/hats.html?page=0&size=48']

def parse(self, response):

for hats in response.css('div.as-t-product-grid__item'):

   yield {

          'name': hats.css('.as-a-text.as-m-product-tile__name::text').get(),

          'price': hats.css('.as-a-price__value--sell strong::text').get(),

          'link': 'https://www.snowandrock.com/' + hats.css('a').attrib['href']

          }

next_page = 'http://api.scraperapi.com?api_key=51e43be283e4db2a5afb62660fc6ee44&url=https://www.snowandrock.com/c/mens/accessories/hats.html?page=' + str(PaginationScraper.page_number) + '&size=48'

if PaginationScraper.page_number < 6:

   PaginationScraper.page_number += 1

   yield response.follow(next_page, callback=self.parse)

Wrapping Up
Whether you’re compiling real estate data or scraping eCommerce platforms like Etsy, dealing with pagination will be a common occurrence and you need to be prepared to get creative.

Alternative data has become a must-have for almost every industry in the world, and having the ability to create complex and efficient scrapers will give you a huge competitive advantage.

2022 SEO Statistics: Industry Stats From Video to Local SEO

SaaS.Group — Tue, 01 Feb 2022 04:29:54 +0000

SEO is constantly changing 100s of times in 100s of little ways. It’s considered one of the most challenging digital marketing disciplines because of its complexity and scope.

SEO is a highly technical discipline at its heart, but it incorporates the creative and social aspects of marketing as well. SEO professionals need to be good not just at optimizing websites, but also at content creation and relationship-building. It’s not enough to have a website that just runs fast and looks good, you also have to provide content that’s valuable to your audience, and have a backlink profile that demonstrates that other websites trust you.

To keep you up to date on what’s going on in SEO for 2022, we have gathered some of the most important SEO statistics available. Staying informed about SEO trends will help you make the best website you can, provide the best user experience possible, and keep the organic traffic coming in.

General SEO Stats and Facts

We’ll start with a top-level view on the value of SEO as a marketing strategy. These statistics are a solid foundation that you can use to build and develop an effective SEO strategy and execute it efficiently.

Google is the most popular search engine in the market. In 2021, 79 % of the total desktop search traffic came from Google. The next big search engine is Bing at 7.27%, Baidu accounts for 6.55%, Yahoo is at 5.06%, and the remaining 2.2% is spread from AOL and other small players
About 70% of small companies don’t have an SEO strategy in place. Businesses that are not investing in SEO are missing out on a chance to increase their online visibility and ranking on the first page
Quality content and off-page SEO are still considered the most significant aspects of SEO success. Several authoritative sources claim that the combination of these two factors is the effective way to rank content in search engines
50% of all organic search queries consist of four or more words. If your website does not include long-tail keywords, you’re missing out on vital web traffic that can affect your conversion rates
Around 61% of marketers believe that SEO is significant for online success. Ongoing SEO optimization is the most effective way to increase traffic and grow your business online
Google crawls 100 billion pages in a month
46% of businesses have reported allocating $10,000 or more of their marketing budget for link building. Companies using SEO best practices have better chances to get on search engine pages and run successful campaigns
Long-tail keywords have a higher click-through-rate (CTR), about 2.5x more than head terms (queries with one or two words)
Businesses that have online reviews are trusted more by customers. Around 88% of customers trust the businesses that have online reviews. Social reviews on platforms such as Google Business Profile, Facebook, and Yelp are effective trust and conversion boosters
URLs on page 2 of Google SERPs only get 6% of clicks. This means if you are not ranking on the first page of Google, you have less than a 1% chance of receiving organic clicks from users. This highlights the importance of getting your website on the first page of Google
In 2020, 50 billion web pages were indexed webpages in Google search
97% of customers research a company using the internet
About 35% of companies allocate more than $1000 per month to their link-building efforts. The top-ranking websites of search engines are constantly working to retain their positions by adding high authority and relevant links to their websites every month
Around 1.88 billion websites exist in the market today
65% of digital marketers claim that link building is the most challenging aspect of SEO. Link building is one of the most unpredictable and challenging aspect of SEO because it’s the one that web masters have the least control over
70% of clicks in Google search results go to organic results, with only 30% going to Google Ads listings. Even after Google’s best result, users are still able to distinguish between the paid and organic listing in the search results

Search Engine Statistics

SEO is a field that studies the way that search engines work, first and foremost. This means not just understanding what search engines think is important, but how they decide that.

The first result in Google’s Organic search has an average click-through-rate (CTR) of 31.7%. This CTR decreases as you go down the page, with the 2nd result having a CTR of 24.71%, the 3rd result having a CTR of 18.6%, and the 4th result having a CTR of 13.6%
Relevant search results influence 39% of purchase decisions. In other words, users are more likely to make a purchase from websites they see on their first page of organic results
Around 12.29% of search queries find featured snippets in their search results. Featured snippets are popular among users as they allow them to quickly obtain the information they’re searching for
About 3.5 billion searches take place on Google every day
Around 34.4% of searches on mobile and 61.5% of searches on desktop result in absolutely no clicks

Video SEO Statistics

Video SEO refers to optimizing videos to rank higher in search, increasing each video’s organic viewership, and maximizing its potential reach. It mostly refers to YouTube videos, but can also include other video platforms such as Vimeo and Dailymotion. With these helpful video SEO statistics, you can use your video marketing efforts to augment your SEO strategy.

The first page videos on YouTube have an average length of 14 minutes and 50 seconds
Posts that contain videos gain a 157% boost in organic traffic and backlinks. Multimedia content is best – a mixture of text and videos enhances your content and makes it more engaging for your users
Engagement metrics such as the number of comments, shares, likes and view count has a strong correlation with higher video rankings on YouTube
However, there was no link between YouTube rankings and a keyword-optimized video description
Videos that gain new subscribers tend to rank higher on YouTube.
68.2% of first-page YouTube results are HD videos.
YouTube is the second most viewed website on the internet

Local SEO Statistics

ics
Businesses with a strong local presence in a geographic region are more likely to earn consumer trust. For many industries like retail and service-based industries, people are more likely to gravitate towards a local business with a strong online reputation than a franchise without much visibility in their region.

Here are a few facts and figures that help demonstrate that.

You will find a Yelp page in the top 5 results for up to 92% of search queries containing a city location and a business category
76% of people who search on their smartphones for something nearby, visit the same business within a day
46% of all searches on Google are for a local service or a local business
28% of all local searches result in a purchase
25% of websites that belong to small businesses do not have an H1 tag
A survey found that 87% of customers read online reviews for businesses during a local search.
Around 93% of customers use the internet to find a business in their local area

Mobile SEO Statistics

Google rolled out mobile-first indexing March 2018 to account for the rise of web traffic on mobile devices. Mobile SEO, or the practice of optimizing websites for search on mobile devices, has become more important than ever as a result . To help you get started, we’ve gathered some insightful mobile SEO stats to help you effectively target mobile users.

58% of all Google searches happen on mobile devices
87% of smartphone owners conduct a search on mobile at least once a day
30% of all mobile searches are specific to a location
51% of smartphone users found a new company or product while searching on mobile.
Voice searches account for 20% of the queries on mobile
The organic click-through rate on mobile is approximately 50% less when compared to desktop.
The very first organic search result on mobile receives around 27.26% of clicks which is higher than 19.3% of clicks received on desktop
84% of all “near me” queries take place on mobile
Around 67% of smartphone users are more likely to buy from businesses that have a location-optimized mobile website
Paid clicks on mobile devices now account for almost 62% of Google’s paid ads

Link Building Statistics

When it comes to link building, quality is more important than quantity. You will get more value out of 5 good, strong, authoritative backlinks than 50 spammy backlinks with no traffic. The challenge of finding those strong, quality backlinks consistently and at scale is what makes link building one of the hardest SEO tactics to plan and execute.

With this in mind, here are a few link building statistics to help give you an edge in your next outreach campaign.

Content with at least 1,000 words or more gets more links and social media engagement than shorter-form content
70% of digital marketers use Ahrefs as a tool for link building
17% of SEO professionals say they only use free SEO tools
The average cost of a paid guest post is $77.80.
46% of marketers spend $10,000 or more on linkbuilding
A quality backlink costs an average of $800-$1,000 to acquire
Around 53% of marketers consider guest posting as an effective tool for link building.
More than 63% of marketers and businesses prefer to outsource their linking building routines to a third party
34% of marketers report that their website has a Moz domain authority of 40 or higher
66.31% of web pages have no backlinks
65% of marketers say that linkbuilding is the hardest SEO tactic to execute

Keyword Statistics

Keyword research is the backbone of SEO campaigns. This is where the actual campaign begins. Therefore, it is critical to understand keywords and their importance. These SEO statistics will educate you about the keyword trends, and help you understand how people search on Google.

70.87% of keywords that are searched more than 10,000 times monthly consist of only one or two words.
Around 8% of search queries are phrased as questions on search engines.
92.42% of keywords only have ten or fewer monthly searches.
13.5% of keywords that have ten or less searches per month consist of only one or two words
0.16% of the most popular keywords are behind 60.67% of all searches taking place on the internet.

Search Engine Trends

SEO is constantly evolving, and new trends emerge every day. To stay on top of your SEO game, you must be aware of the search engine trends and incorporate them into your SEO strategy. These Search engine SEO facts will give you the current state of a search engine, and help you optimize your campaigns to maximize the results.

75% of search engine users never go beyond the first page of Google results
Around 21% of people surfing the internet click on more than one search result
Organic traffic accounts for more than 40% of revenue across Media and Entertainment, Retail, Hospitality, Technology/internet
SEMrush reported that users visit 3 to 3.5 pages every time they land on a website from search engines.
Ahrefs conducted a study that found that around 90.63% of content gets no traffic from Google.
Domains ranking in the top 3 spots have an average bounce rate of 49%
Wordstream reported that brand affinity helped boost the CTRs by 2 to 3 times.
More than 86,000 searches happen on Google every second.
The 1st to 3rd ranking pages on SERPs generate a clickthrough rate of 36%.
Google made more than 3,620 improvements to search in 2018.
Around 18% of people searching the internet enter a new query before they click on any result for the original query.
99.58% of featured snippets rank in 1 to 10 positions on Google.
Organic methods drive around 300% more traffic to websites when compared to social media.
The major search engines, YouTube, Bing, Google and Yahoo account for a total of 70.6% of all website traffic.
Searchmetrics reported that the average time a user spends on a website in the top 10 SERP results is 3 minutes and 10 seconds.
Google reported that 1.1 billion smartphone users will be use voice search every week
OC&C reported that voice search has a 20% market share of all Google Searches in the US.

Voice Search Stats

Voice searches have started to replace traditional text-based searches. They have made it easier for people to search the internet and have reduced the need for typing every query. These voice search SEO stats will get you started on an effective SEO strategy.

Voice search contains an average of only 29 words
The average Google voice search result receives around 44 tweets and 1,199 Facebook shares
The average page that results in Google voice search contains around 212 words
ComScore predicted that search queries would take up half of the searches by 2020
Mobile voice searches are 3 times more likely to be for something local when compared to text searches
Voice searches load in 4.6 seconds, which is 52% faster when compared to the load time of the average page.
9th grade is the average reading level of a voice search result
Around 40.7% of all answers that are displayed in a voice search are pulled from a featured snippet.
Over 47 million people use Alexa and Google smart speakers in the US.
Websites that have a strong link authority are more likely to rank well in voice search.
HTTPS websites have 70.4% of voice search result pages.
While the global average of voice search is 31.3%, schema pages is 36.4% of voice search
75% of voice search results are pulled from any of the top 3 desktop ranking pages for any given query.

Content Marketing Stats

Content is king, in marketing, and it’s just as true in SEO as anywhere else. Your website is highly unlikely to rank for the keywords you want without high-quality content that’s well-written, well-researched, and provides value to your audience.

These statistics will help you understand the relationship between SEO and content marketing, and the thinking that goes into a content marketing strategy.

Simply updating the existing content with fresh images and written content has the potential to increase your organic traffic by 111.3%
About 57% of SEO experts agree that content creation and marketing are the best way to drive results
Around 65% of searchers unveil that relevance is the most important factor to curate a successful SEO campaign
60% of marketers are publishing content once a day. Despite knowing the power of content marketing, only 60% of marketers are posting content once or more every week
The average 1st-page result in Google contains around 1890 words of content. This means detailed articles and landing pages have the potential of establishing you as the industry leader
Content creation can also improve indexation rates by more than 434%. Websites that are regularly updated are loved by Google. The latest SEO stats show that creating regular blog content will result in 434% higher indexation rates when compared to the websites that do not update their content on a regular basis

Technical SEO Statistics

Technical SEO is the process of optimizing a website from a performance standpoint. You need a website to be techically well-optimized for the same reason you need an engine for your car to run.

Technical SEO gets highly complicated at the granular level, but even a basic understanding of what makes a website run well can help you make your website significantly better. It’s also useful for understanding what search users look for from a user experience and technical standpoint.

Mobile users are more likely to purchase from businesses that offer custom features for their location
The chances of bounce rates increase if the page load speed is increased from one to three seconds. It automatically goes up by around 90% if the load speed increases to five seconds
Your website conversion rates gradually fall by about 4.42% with each second of load time
The average attention span of a human is 8 seconds and dropping. Attention spans have decreased from 12 seconds in 2000 to 8 seconds in 2018. This means that you now have even less time to capture someone’s attention with your website or content
The first 5 seconds of page-load time have a massive effect on your conversion rates. These are times when people are making up their minds whether to stay on the site or not. If you take too long, they’ll leave without seeing your content
In the recent survey, about 48% of the website owners surveyed said they would be willing to drop animation and video for faster load times on their sites

SEO Is Fundamental For Success

SEO is a multi-faceted discipline that requires analytical skills as well as creative thinking to be successful. It begins with anticipating how your users expect to find you on Google, and accounting for that with content that accounts for that while providing them with a good user experience.

Your website is the face of your business on the internet, so it’s important to make sure that it represents who you are and what you’re capable of. Sign up for Prerender, and make your website the best it can possibly be.

Step by Step Guide to Scrape Google Using Python & Scrapy

SaaS.Group — Thu, 23 Dec 2021 20:28:54 +0000

Collecting Customer Feedback Data to Inform Your Marketing
In the modern shopping experience, it is common for consumers to look for product reviews before deciding on a purchase.

With this in mind, a powerful application for a Google SERPs scraper is to collect reviews and customer feedback from your competitor’s products to understand what’s working and what’s not working for them.

It can be to improve your product, find a way to differentiate yourself from the competition, or to know which features or experiences to highlight in your marketing.

Keep this in mind because we’ll be building our scraper around this issue exactly.

https://www.scraperapi.com/blog/scrape-data-google-search-results/

Technical SEO Advice from an Industry Veteran

SaaS.Group — Wed, 01 Dec 2021 03:17:32 +0000

It’s not that often that you meet experienced marketers who are nice people and also good at their jobs at the same time.

Dave Davies is an SEO veteran we featured in our 25 Technical SEO Experts on Twitter roundup who has been in the industry for longer than almost anyone. Davies has been writing about SEO topics as a contributor to Search Engine Journal and Search Engine Watch for over a decade. He is the founder of Beanstalk Marketing and is currently the Lead SEO at Weights & Biases.

Apart from being a skilled SEO professional, Davies is also knowledgeable about web development and machine learning topics. As such, Davies has a more intimate understanding of the relationship between internet users and search engines better than nearly anyone else in the field today.

Davies isn’t just an SEO expert with technical chops either – he loves sharing his knowledge and using his experience to make the industry better for everyone. That coupled with his affable personality and sense of humor make him widely respected in the SEO world.

We sat down with Davies to ask him about technical SEO, the relationship between Google and smaller brands, and where he thinks the next core algorithm update might have in store. Here’s what he has to say.

I. Google’s official stance is that Googlebot can crawl and index Javascript without any issues. The available studies out there show that although technically true, it takes them longer and uses more resources – meaning Javascript SPAs exhaust their crawl budget quickly.

You’ve been in the SEO industry longer than almost anyone. What is your opinion on this?

They are wrong.

I am right now working for a company that has a SPA website that uses prerendering and let me tell you, whenever the slightest thing goes wrong I see it very clearly in the rankings and the caches.

I noticed just a couple of months ago a hiccup at Google with prerendering, which was followed pretty closely with a lag in the coverage reports and a form to submit indexing bugs.

In short, I do think they’re working on it – but they’re still a ways off from this being true, and I’m not sure that the solution will ever be crawling.

II. In recent years it’s become harder and harder for small businesses and startups to get visibility on Google SERPs because of algorithm changes that favor established brands that already have an audience and a web presence.

What can Google do to better support smaller businesses and startups and be their advocates?

While I understand the context of the question and sentiment, when I really think about it I’m not sure that’s true.

Yes, when we’re fighting battles with national brands on their turf (national SERPs) it does often come to this, but Google is giving local businesses a lot of new tools and visibility options. The national brands can play there, as applicable – but it’s a lot harder for them to stand out and they don’t seem as favored by traditional metrics.

So if small businesses focus on local markets, which many do, they have serious advantages if they know how to take them. For smaller businesses tackling national markets against sites like Amazon and Walmart, it is true they’ll be fighting an uphill battle.

They need to find a sub-niche to start, where keywords are easier and start there. In that context, not a lot has changed over the years.

III. Many SEO professionals make the mistake of making the Google gods happy at the expense of user experience.

This is a fundamentally flawed approach because Google’s mission statement focuses on the user – to provide the user with the best possible result for a given query.

How do we solve for the user instead? How do we make that user-first mentality the conventional wisdom in SEO?

I have a very short answer to this question because I think we often make it more difficult than it has to be.

Create the content the user wants. Deliver it in the format they want it in. And make sure Google understands that you’ve done that.

To expand a twitch:

Create the content the user wants – Think about the user as the person entering the query, not your customer. Think about all the things a person entering that query might be looking for and deliver as many as you can while keeping the content clean. With that, you maximize the probability that you will satisfy a user, and that’s what Google wants you to do.

Deliver it in the format they want it in – If they want a video, give them a video. They all want it fast. They all want it secure. They all want to be able to access it on any device from any location. Give people what they want, and you’ll be ahead of the next rule Google throws at you.

And make sure Google understands that you’ve done that – Make sure you link between your pages logically, add schema where applicable, etc. You’ve done the work for the user, do a bit more to make sure Google understands it, and you’ll be well on your way.

IV. Even if we take Google’s word for it that their web crawler can crawl and render Javascript, there’s no guarantee that websites made using Javascript frameworks will be well-optimized for both users and search engines.

What is the single most important thing that webmasters and technical SEO experts can do to make sure their Javascript web applications are well-optimized for search?

Monitor. Monitor. Monitor.

Set up alerts on key pages to run daily and alert you to an unexpected drop.

Manually check pages not just with a crawler, but inspect the cache and inspect the code produced by testing your URL in Google Search Console – see how it renders. Check a variety of pages and page types. Just because one part of the page is fine, doesn’t necessarily mean it all is.

Beyond that, make sure you have a good dev and good technology.

V. You’ve been covering Google core algorithm updates on Search Engine Journal for years.

What do you anticipate the next core algorithm update to focus on, and why? What’s missing in the way Google ranks and categorizes web pages that isn’t there already?

This question really got me thinking.

I think as far as core updates go, the next series will likely focus on infrastructure and keeping an increasingly complex arrangement of pieces working together.

We’re watching MUM starting to get used in the wild, and we’ve heard about LamDA. We’ve read about KELM and the potential it has in creating a more reliable and “honest” picture of the world.

What we don’t read a lot about (mainly because it’s boring and we don’t want to) is the technology behind it. KELM would add verified facts to a picture of the world Google has created from a different system (MUM, for example). Great, but how do you get those two parts communicating and sharing information?

This is, to me, the biggest of their challenges and why I suspect it will be the focus on their core updates for the foreseeable future.

I’ve started reading some of the papers on some of the technologies behind the technologies we hear about.

How ByT5 can improve understanding content in a noisy environment (where noise may be something like misspelled words on social media, etc) by moving away from tokens and working byte-to-byte which required a lot to overcome the hurdle of ballooning the compute time.

Or how Google FLAN improves zero-shot NLP across domains (where domains are not sites, but rather tasks) so a system trained on classifying sentiment (for example) can be used to improve a translation model with little additional training required for the new task.

This, in my mind, is what the core updates need to deal with.

VI. Many web developers lack even a basic understanding of SEO. That creates problems down the line when SEO problems become ignored or buried under legacy code which makes them harder to diagnose and fix.

As an SEO veteran with web development credentials, what do you think we can do to bridge that gap? How can web developers make sure that an SEO infrastructure is in place from the moment they begin development? On the other side, what can marketing teams do to make the developers’ jobs easier?

I honestly believe it’s a two-way street.

When I was cutting my teeth I used Dreamweaver 4 to put content into tables and upload them page-by-page. I learned a lot on my way, but the pace of change in dev and SEO meant I needed to choose a path and I was never a developer so I stuck with SEO.

Yes I can still throw together a decent WordPress site, and probably edit the themes without breaking anything, but I wouldn’t consider myself even an intermediate dev. And it’s great that I know that.

That history and ability though, I think makes me a bit better than some at understanding how to communicate with developers.

I can’t count the number of times I’ve outlined my needs and how to solve a problem to a capable developer, only to have it bite me in the butt when they followed by instructions to unexpected results.

Now I isolate what the problem is, describe and send screenshots of how I know and how I’ll know when it’s fixed to the developer, and while I might include a potential fix I found – I try to be clear that it is for illustrative purposes only.

9 times out of 10, if you are working with a good developer they’ll be able to think of solutions you never would and often solve additional problems you might not have known you had.

Respect them, respect their knowledge and they will respect yours.