DEV Community: Steven van Vessum

How to remove URLs from Google (quickly)

Steven van Vessum — Thu, 19 Nov 2020 14:05:19 +0000

We've all been there probably — Google indexed URLs, which should have NEVER been indexed.

Maybe it a staging environment got indexed, a site got hacked and a bunch of dodgy pages popped up in Google Search, sensitive information like an acquisition or merger was accidentally published.

Or, you're going all Marie Kondo on your content and you've started removing and merging old content.

Regardless of the situation, you want to dictate what picture Google paints. Right away.

Because every situation is different, the steps you need to to take to get Google to remove your URLs are different.

I've identified the most common reasons to remove URLs from Google, and documented what to do in great detail.

Check out the article here: How to Remove URLs from Google Search in a Snap!

A brief rant about meta tags and SEOs

Steven van Vessum — Wed, 28 Oct 2020 13:49:31 +0000

Most SEOs throw around the term meta tags incorrectly.

They mean meta elements most of the time.

While meta tags belong to the HTML basics, and SEO basics with that, it's important to get your lingo right.

Why?

Because when SEOs complain that they're not being understood or respected by developers, it may be because they're (literally) not using the right language to explain what they need.

I know that sometimes was the case with me, in my consulting days. And rightfully so, I was corrected.

I put together an article explaining what meta tags are and aren't, and describe the most important ones from an SEO perspective.

Check it out here: Meta Tags in SEO: the Ultimate Guide

Nofollow, UGC or Sponsored? The Link Attribute Explained

Steven van Vessum — Wed, 01 Jul 2020 11:59:57 +0000

Although the whole nofollow attribute game changed significantly in the world of SEO, it still remains an important factor to consider when creating links, both within your website and from other sites to yours.

This also applies to the new attribute values: sponsored and UGC.

The nofollow link attribute value—also known as the link builders’ most tolerated enemy—was first introduced in 2005 to tackle spammy links within comments and message boards that were aimed to game Google's system.

Since its implementation, Google required all unnaturally created links to be marked with the nofollow link attribute so that it could understand the true origin of the link.

Apart from indicating the link’s origin, it also signals search engines that they should neither follow the link to the target page nor pass any authority.

However, this world’s biggest search engine’s adamant approach towards unnatural links has changed significantly. In September 2019, Google announced that they wouldn’t consider the rel="nofollow" link attribute as an undeniable directive, and would start treating it as a mere hint instead.

They implemented this change in March 2020 and as part of their new policy, they introduced two new attribute values: rel="sponsored" for paid and sponsored links (such as affiliate backlinks) and rel="ugc" for user-generated links coming from comments, forums or message boards. While it may seem that this makes making money with blogging is harder, that's not the case if you're creative enough.

The new attribute values can be combined with other values, such as the nofollow itself.

When to use nofollow link attribute

However, despite the current changes, the core principles of using the nofollow attribute value remain and it pays off to remind ourselves why and when we should keep on using it.

In general, we should use the nofollow link attribute when a link hasn’t occurred naturally or isn’t being earned via genuine interest. It should be implemented in these examples:

Cases where you need to link, but don’t want to be associated * with the link target

Widgets
Certification badges
Press releases

In addition, the new “sponsored” link attribute value should indicate any sponsored or paid links, while “UGC” should mark user-generated content links in forums, message boards, and comments. Bear in mind that links without nofollow should be the result of genuine interest only.

Piqued your interest? We've explained other uses for these link attributes in Nofollow link and rel=nofollow explained.

Using Google Search Console to Find and Fix Index Coverage Errors

Steven van Vessum — Mon, 11 May 2020 14:34:42 +0000

I'm sure most here have heard about Google Search Console. But did you know Google reports back how they fared during crawling and indexing your site, in great detail?

All this information is made accessible through the Index Coverage Report.
The report helps you identify which pages have been successfully indexed and/or if your directives towards search engines work well.

To evaluate the performance of the pages, the Index Report can result in four different statuses — valid, valid with warnings, excluded and error.

The Valid URL status reflects a page that has been successfully indexed. The pages can be either indexed and submitted in the XML sitemap, or index without being submitted. If the latter occurs, you should check if the URL should be indexed and if so add them to your XML sitemap. If not, make sure you implement the robots noindex directive.

Optionally, you can exclude the URLs in your robots.txt if they can cause crawl budget issues.

The Valid with warnings status means that the pages that have been indexed have some issues that need checking. The status contains one type only — “Indexed, though blocked by robots.txt”.

This means that Google has indexed these URLs, despite being blocked by your robots.txt file.

Under normal circumstances, Google wouldn’t have indexed these URLs. But for some reason, it found links to these URLs and thus went ahead and indexed them anyway.

In this case, you should review these URLs, update your robots.txt, and possibly apply robots noindex directives.

Status Excluded means that search engines picked up a clear signal that the pages shouldn’t be indexed. It contains these 15 different types:

Alternative page with proper canonical tag
Blocked by page removal tool
Blocked by robots.txt
Blocked due to unauthorized request (401)
Crawl anomaly
Crawled - currently not indexed
Discovered - currently not indexed
Duplicate without user-selected canonical
Duplicate, Google chose different canonical than user
Duplicate, submitted URL not selected as canonical
Excluded by ‘noindex’ tag
Not found (404)
Page removed because of legal complaint
Page with redirect
Soft 404

The Error status means that pages couldn’t be indexed for some reason, and contains the following eight types:

Redirect error
Server error (5xx)
Submitted URL blocked by robots.txt
Submitted URL marked ‘noindex’
Submitted URL seems to be a Soft 404
Submitted URL returns unauthorized request (401)
Submitted URL has crawl issue
Submitted URL not found (404)

Keeping an eye on each of these statuses and their types will allow you to zoom in on specific issues Google has found on your site, and thus improve the overall performance.

The descriptions in Google Search Console are notoriously cryptic, so I wrote a massive guide on how what they mean, and what action to take.

If this starter has piqued your interested, read more about How to Find and Fix Index Coverage Errors in Google Search Console.