IndexNow in Go: tell search engines about new pages in one request

When a maker's product goes live on RadarMakers, we want search engines to know about the new page without waiting for a crawler to find it. IndexNow is a small protocol for exactly that: you send the URLs that changed, and the participating search engines are told about them. This post shows our Go client, the key file it needs, and what the protocol actually promises.

What IndexNow asks for

The IndexNow documentation describes two things.

A key you can prove you own. The key is 8 to 128 characters of letters, numbers and dashes. You host it in a UTF-8 text file, by default named after the key and placed at the root of the site, for example https://example.com/your-key.txt, and the file contains the key. The documentation strongly recommends the root location. A file elsewhere on the host has to be named in each request with keyLocation, and then only covers URLs under its own folder.

A request listing the changed URLs. A single URL can go in a GET query string, but several URLs go in one JSON POST with the host, the key, an optional keyLocation and a urlList of up to 10,000 URLs.

The FAQ lists a global endpoint, https://api.indexnow.org/indexnow, next to the endpoints of individual search engines, and says you may submit to only one of them, because a submission is shared with all IndexNow enabled search engines.

The client

// internal/indexnow/client.go, excerpt
var validKey = regexp.MustCompile(`^[a-zA-Z0-9-]{8,128}$`)

// ... Client declaration omitted

func New(siteURL, key string) *Client {
    if key == "" {
        return nil
    }
    u, err := url.Parse(siteURL)
    if err != nil || u.Host == "" || u.User != nil || (u.Scheme != "https" && u.Scheme != "http") || !validKey.MatchString(key) {
        log.Print("indexnow: invalid public site URL or key format; disabled")
        return nil
    }
    return &Client{origin: u.Scheme + "://" + u.Host, host: u.Host, key: key,
        endpoint: "https://api.indexnow.org/indexnow", http: &http.Client{Timeout: 5 * time.Second}}
}

Without a key, or with a key or site URL that does not look right, New returns nil. Both methods handle a nil receiver, so the rest of the code calls the client the same way whether IndexNow is configured or not. A development machine without a key simply sends nothing.

Submit builds the payload. It only accepts local absolute paths and turns them into full URLs on our own origin, so nothing else can end up in the list:

// internal/indexnow/client.go, excerpt
urls := make([]string, 0, len(paths))
for _, path := range paths {
    if !strings.HasPrefix(path, "/") || strings.HasPrefix(path, "//") || strings.Contains(path, "\\") {
        return fmt.Errorf("expected a local absolute path")
    }
    urls = append(urls, c.origin+path)
}
payload := struct {
    Host        string   `json:"host"`
    Key         string   `json:"key"`
    KeyLocation string   `json:"keyLocation"`
    URLList     []string `json:"urlList"`
}{c.host, c.key, c.origin + "/" + c.key + ".txt", urls}

The request goes out as POST with Content-Type: application/json; charset=utf-8, the header the documentation shows. A response of 200 or 202 counts as success; anything else is an error.

Fire and forget, on purpose

The repositories call Notify, not Submit:

// internal/indexnow/client.go, comment omitted
func (c *Client) Notify(paths ...string) {
    if c == nil {
        return
    }
    paths = append([]string(nil), paths...)
    go func() {
        ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()
        if err := c.Submit(ctx, paths); err != nil {
            log.Printf("indexnow: %v", err)
        }
    }()
}

Three choices are packed into those lines:

• It runs in its own goroutine, so publishing a product never waits for a search engine.
• It uses a fresh context with a 5-second timeout, not the request's context, so the ping does not depend on the publishing request being cancelled or finished.
• It logs failures and does not retry. The sitemap is another way for search engines to find the page, and a later qualifying event, such as a product being set live again, sends another submission. We chose not to add a retry queue for this.

The paths slice is copied before the goroutine starts, so a caller that reuses its slice cannot change what gets sent.

Calling it after a successful write

The product repository calls Notify only after the database write succeeded and only when the product is live:

// internal/repository/product.go, excerpt
err := r.db.QueryRowContext(ctx, query+" RETURNING slug", status, id).Scan(&slug)
if err == nil && status == model.StatusLive {
    r.indexNow.Notify("/product/" + url.PathEscape(slug))
}

The same pattern covers a product created as live and a product whose status is set back to live. Community posts do it when a post ends up published after it is created, updated or approved. Each event sends its own single URL; the client can take several paths in one call, but we do not collect separate events into a batch.

Serving the key file from Next.js

The site itself is a Next.js app, and the key is only known at runtime, from an environment variable. The file is answered by src/proxy.ts, whose matcher only covers top-level .txt paths:

// src/proxy.ts, imports omitted
export function proxy(request: NextRequest) {
  const pathname = request.nextUrl.pathname;
  if (!isIndexNowKeyPath(pathname, process.env.INDEXNOW_KEY)) return NextResponse.next();
  return indexNowKeyResponse(pathname, process.env.INDEXNOW_KEY);
}

export const config = { matcher: ["/:file([^/]+\\.txt)"] };

// src/lib/indexnow-key.ts, comment omitted
export function isIndexNowKeyPath(pathname: string, key: string | undefined): boolean {
  return !!key && /^[a-zA-Z0-9-]{8,128}$/.test(key) && pathname === `/${key}.txt`;
}

export function indexNowKeyResponse(pathname: string, key: string | undefined): Response {
  if (!isIndexNowKeyPath(pathname, key)) return new Response(null, { status: 404 });
  return new Response(key, {
    headers: { "Content-Type": "text/plain; charset=utf-8", "Cache-Control": "no-store" },
  });
}

Only the exact path for the configured key answers, with the key as plain text. Any other .txt path passes through to the rest of the app. We checked the live site: the key path answers 200 with text/plain; charset=utf-8, no-store and a body equal to the key, and a made-up key path answers 404.

Testing it

The client tests point the endpoint at a local httptest server. One test calls Notify with two paths and checks the exact JSON that arrives, including the keyLocation and both full URLs. Another makes the server answer 429 and checks that Submit returns an error, checks that a cancelled context fails, and checks that https://elsewhere.test, //elsewhere.test and /\elsewhere.test are refused as paths. A third checks that a client created without a key, or with a key that is too short or contains a slash, is nil, and that calling Notify and Submit on it is safe.

What it does not promise

The IndexNow FAQ is direct about this. A 200 only means the search engine received the URLs; the first request may get a 202 while the key is verified. Submitting a URL does not guarantee that it is indexed. Each search engine decides for itself, and every submitted URL counts toward the site's crawl quota. The FAQ also says IndexNow is meant for recently added, updated or deleted URLs, recommends using it together with a sitemap, and advises against resubmitting unchanged URLs.

Takeaways

• Validate the key and the site URL once, and return a nil client when IndexNow is not configured, so callers never need a feature flag.
• Notify after the database write succeeds, from a goroutine with its own short timeout.
• Accept only local paths and build full URLs on your own origin.
• Serve the key file at the root with the key as plain text, and nothing else on that route.
• Keep your sitemap. IndexNow is a hint about what changed, not a replacement.