Quickly index these web pages with IndexNow

Introduction

IndexNow is a protocol for rapidly indexing web pages on search engines. It was created by Bing and Yandex, and is now an open-source project.
Created in 2021, it is now supported by several search engines, including Bing, Yandex, Seznam, Naver and Yep.

How does it work?

The IndexNow protocol is based on the Ping protocol, which notifies search engines when a new article is published. IndexNow goes one step further, notifying search engines when a web page is updated.

To use IndexNow, simply send an HTTP POST request to the URL https://www.bing.com/indexnow (or any other search engine with an /indexnow API):

POST https://www.bing.com/indexnow HTTP/1.1
Content-Type: application/json; charset=utf-8

{
  "host": "thomasbnt.dev",
  "key": "0e11834b62ad4f089b83b91666a61105",
  "keyLocation": "https://thomasbnt.dev/0e11834b62ad4f089b83b91666a61105.txt",
  "urlList": [
    "https://thomasbnt.dev/about",
    "https://thomasbnt.dev/blog",
    "https://thomasbnt.dev/uses",
    "https://thomasbnt.dev/contact",
    "https://thomasbnt.dev/terms"
  ]
}

Fields description

• host : the domain name of the website.
• key: the public key generated for the website.
• keyLocation: the URL of the public key.
• urlList : the list of URLs to be indexed.

See the list of search engines supporting IndexNow →

Why use IndexNow?

IndexNow enables rapid indexing of web pages on search engines, which can be useful for websites that frequently publish content. This helps to improve the visibility of web pages on search engines and attract more traffic.

Implement IndexNow on my website

To implement IndexNow on your website, you can use a plugin like IndexNow WordPress Plugin, or send HTTP requests manually.
There are also hosting providers that support IndexNow, such as Cloudflare, WordPress.com, Shopify, etc. You can find the list directly on the site.

Let's go step by step

1. Generate a public key and add it to your public/ folder on your website, this will create a link like this https://thomasbnt.dev/0e11834b62ad4f089b83b91666a61105.txt (here 0e11834b62ad4f089b83b91666a61105 is the public key).
2. Update your website with the key, or use a plugin that will do it for you.
3. Send an HTTP POST request to the IndexNow API URL of the search engine you wish to use, with the public key and the list of URLs to be indexed. In this case, we're using Bing.

POST https://www.bing.com/indexnow HTTP/1.1
Content-Type: application/json; charset=utf-8

The request body should look like this:

{
  "host": "thomasbnt.dev",
  "key": "0e11834b62ad4f089b83b91666a61105",
  "keyLocation": "https://thomasbnt.dev/0e11834b62ad4f089b83b91666a61105.txt",
  "urlList": [
      "https://thomasbnt.dev/about",
      "https://thomasbnt.dev/blog",
      "https://thomasbnt.dev/uses",
      "https://thomasbnt.dev/contact",
      "https://thomasbnt.dev/terms"
  ]
}

1. You should receive a 200 OK response if the request was successfully sent. Note: I use the HTTP client HTTPie to send HTTP requests.
2. Check that the pages have been indexed by consulting your website logs or by using search engine tools such as Bing's Bing Webmaster Tools.
3. Congratulations, your web pages are now quickly indexed on search engines! 🎉

Useful links

• IndexNow official website
• IndexNow documentation
• Introducing IndexNow
• List of search engines supporting IndexNow
• IndexNow WordPress Plugin
• Cloudflare supporte IndexNow

Comments

[Chris Dolan]

Nice walkthrough. A few things we only learned by running IndexNow in production that might save someone an afternoon:

Don’t expect only 200. A first submission with a new key often comes back 202 - “received, key validation pending” - which is fine and not an error. The ones to watch for are 403 (key not valid), 422 (a URL that isn’t on the host, or a key that doesn’t match) and 429 (slow down). It’s worth logging the status per submission, not just “sent”.

A 200 only means the notice was accepted. It isn’t a promise the pages get crawled or indexed - check Bing Webmaster Tools before celebrating.

One key per site, not one key everywhere. We announced several sites with a single shared key. Bing refused it with 403 “key not valid” on every submission - even though the key file returned 200 with exactly the right key, to a browser and to Bingbot. Yandex accepted the same submissions (202). A fresh key generated for that one host was accepted the same day. Each host, subdomains included, should have its own key file.

Mind where the key file lives. If it isn’t at the site root (say, a WordPress media folder), declare it with keyLocation - but then it only vouches for URLs under that same folder. Root is simplest.

[Thomas Bonnet]

Good to know that about the sharing key !