DEV Community

Rost
Rost

Posted on • Originally published at glukhov.org

IndexNow explained - notify search engines when you publish

#hugo #seo #devops #selfhosting

Static sites and blogs change whenever you deploy. Search engines that support IndexNow can learn about those changes without waiting for the next blind crawl.

This page covers why that matters, what the protocol does, and how to wire it into a real workflow, including patterns you can reuse in your own automation or a small Go CLI.

Why use IndexNow on a static or Hugo site

If you deploy Hugo to S3 or similar you already ship HTML and a sitemap.xml. Crawlers will eventually read the sitemap but timing is not under your control. After a migration or a batch of new posts you care about fresh indexing more than "sometime next week."

IndexNow is a push channel. You POST a list of canonical URLs you care about. Participating engines (including Microsoft Bing and others listed on indexnow.org) can prioritize fetching those URLs. It does not replace good URLs redirects or internal linking but it closes the loop between git push and search engine awareness.

What IndexNow does

At a high level each submission is an HTTPS POST with JSON like this conceptually:

  • host - your site hostname (for example www.example.com)
  • key - your pre-generated secret string
  • keyLocation (optional) - full URL of the verification file if it is not at the default path
  • urlList - one or more absolute URLs on that host you want to signal

Engines reject bad keys wrong hosts or malformed payloads. Success is usually HTTP 200 or 202 depending on the endpoint.

You can read the full rules and partner list on the official site. The important mental model is domain ownership proof via a text file plus explicit URL list not keywords or page content.

How to prepare your site

Key file and hostname

  1. Pick a key - a long random string (treat it like a secret).
  2. Publish https://your-domain/<key>.txt with only the key as the body (one line).
  3. Use the same key in your CLI or automation when you POST.
  4. Submit only URLs on that host that you want recrawled (new posts updated pages or redirects targets).

After you move many URLs at once you may want to batch-notify paths. IndexNow accepts multiple URLs in one request subject to each engine’s limits.

Ways to submit URLs

  • Manual POST - fine for debugging use curl with JSON.
  • Plugins - some CMS and hosting panels include IndexNow toggles.
  • Your deploy script - after hugo and upload call a small binary with the list of changed URLs or your sitemap URL.

For a Hugo workflow the natural triggers are "after build" or "after sync to bucket." Pass full HTTPS URLs that match your live site including www vs apex if that is what you serve.

A small Go CLI (optional)

Features you might implement

A minimal Go command-line tool fits IndexNow well because the payload is a small JSON POST and you can wire it into deploy scripts. A typical design includes:

  • Single or multiple URLs as positional arguments
  • --sitemap to fetch a sitemap.xml and submit every <loc> (with optional --limit)
  • Several engines in parallel via --engines (for example indexnow for the global aggregator, or per-provider endpoints)
  • Flags or environment variables such as INDEXNOW_KEY, INDEXNOW_WEBSITE_URL, and INDEXNOW_ENGINES
  • Verbose output with -v for debugging 403 or 422 responses

Build with go build or go install, install the binary on your PATH, then call it after publish:

indexnow --key YOUR_KEY --website https://www.example.com https://www.example.com/new-post/
Enter fullscreen mode Exit fullscreen mode

For a full site refresh after deploy you can pass --sitemap with your public sitemap URL. Document response codes and engine lists in your own README, and keep a publish-then-index shell snippet next to whatever triggers your static deploy.

The Best LLMs for OpenCode - tested locally post used "implement an IndexNow notifier in Go" as a coding benchmark - useful if you want to see how different models handle the same spec and structured tasks.

Practical tips

  • Prefer the indexnow engine target when you want one submission to fan out through the shared infrastructure (see searchengines.json and mirror that list in your own client if you support multiple endpoints).
  • 429 means slow down. 403 usually means key or host mismatch. Fix the key file location or hostname first.
  • IndexNow does not replace 301 redirects when you rename paths. Notify after redirects are live.

See also

Top comments (0)