DEV Community

Cover image for My IndexNow submissions returned 202 for two weeks. None of them worked.
Sam_tj
Sam_tj

Posted on

My IndexNow submissions returned 202 for two weeks. None of them worked.

#seo #webdev #github #debugging

I have a small static site on GitHub Pages. It is served from a project subpath:

https://shantj.github.io/sproutguard/
Enter fullscreen mode Exit fullscreen mode

Twelve times over about two weeks, I pinged IndexNow to get the pages crawled. Every single
time I logged a success, because that is what came back:

api.indexnow.org  ->  202
bing.com/indexnow ->  202
yandex.com/indexnow -> 202
Enter fullscreen mode Exit fullscreen mode

Twelve rounds later, site:shantj.github.io/sproutguard returned zero results on Bing. Not
"ranked badly" — not present. That is the point where I stopped trusting my own log.

The bug

IndexNow authenticates you by fetching a key file from your host. Unless you say otherwise,
it looks in exactly one place:

https://<host>/<key>.txt
Enter fullscreen mode Exit fullscreen mode

Note what is missing: the subpath. My key file is at /sproutguard/<key>.txt, because that
is the only place a GitHub Pages project site can put a file. The domain root belongs to my
user page, which does not exist:

https://shantj.github.io/<key>.txt   -> 404
https://shantj.github.io/            -> 404
Enter fullscreen mode Exit fullscreen mode

So the verification fetch 404s, and the submission is rejected. The fix is one field,
keyLocation, which tells IndexNow where the key actually lives.

The part that cost me two weeks

I could have caught this on day one if I had read the response instead of the status code.
Here is the same payload, submitted twice, differing only by the presence of keyLocation:

POST https://www.bing.com/indexnow
  without keyLocation -> 403 {"errorCode":"UserForbiddedToAccessSite",
                              "message":"User is unauthorized to access the site.
                                         Please verify the site using the key and try again"}
  with    keyLocation -> 200
Enter fullscreen mode Exit fullscreen mode

Unambiguous. A 403 with an error message naming the exact problem.

So why did my log say 202 for twelve rounds? Because of this:

POST https://yandex.com/indexnow
  without keyLocation -> 202 {"success":true}
  with    keyLocation -> 202 {"success":true}
Enter fullscreen mode Exit fullscreen mode

Yandex returns 202 {"success":true} for a submission that Bing rejects outright. IndexNow
is a shared protocol with independent implementations, and they disagree about what a rejection
looks like. My scripts submitted to three endpoints, treated 2xx as success, and one endpoint
was always willing to give me one. Somewhere in there Bing's 403 started reading to me as rate
limiting — I even wrote "stop resubmitting, we're being throttled" in my notes and reduced my
submissions. I had the error in front of me and interpreted it as noise.

The corrected request 

payload = {
    "host": "shantj.github.io",
    "key": key,
    "keyLocation": f"https://shantj.github.io/sproutguard/{key}.txt",  # <- the fix
    "urlList": urls,
}
Enter fullscreen mode Exit fullscreen mode

Result:

key file OK at https://shantj.github.io/sproutguard/<KEY>.txt
submitting 10 URLs
  OK   https://api.indexnow.org/indexnow -> 200
  OK   https://www.bing.com/indexnow     -> 200
  OK   https://yandex.com/indexnow       -> 202
3/3 endpoints accepted
Enter fullscreen mode Exit fullscreen mode

Two things I'd bake into any submitter

1. Verify the key file before submitting, not after wondering. Four lines, and it turns an
invisible auth failure into a loud one:

resp = urllib.request.urlopen(key_location)
assert resp.status == 200 and resp.read().decode().strip() == key
Enter fullscreen mode Exit fullscreen mode

2. Write down which endpoint's status you are actually trusting. "3/3 returned 2xx" felt
like triple confirmation. It was one honest answer, one wrong answer, and one endpoint I had
mentally reclassified as broken. Aggregating across implementations that disagree about
failure semantics doesn't give you redundancy — it gives you a system where the most
permissive endpoint decides what you believe.

I also found my sitemap.xml had a lastmod frozen at a date five rounds stale while the
pages themselves kept changing. Same family of mistake: a signal I generated once, never
re-derived, and kept trusting. It's now computed from git commit dates.

If you're on GitHub Pages

You are affected if your URL has a repo name in it (user.github.io/repo/) rather than being
a bare user or custom domain. Cloudflare Pages preview URLs, docs served from /docs, and
anything behind a path-based reverse proxy have the same shape of problem.

Quick check — if this 404s, you need keyLocation:

curl -sI https://<your-host>/<your-key>.txt | head -1
Enter fullscreen mode Exit fullscreen mode

The submitter I ended up with is here,
including the negative-control test that asserts a submission without keyLocation still
gets rejected — because a fix you can't watch fail isn't one you can trust.

Context, since it's relevant to why I was submitting so often: I build
SproutGuard, a screen-time app, and these were its
content pages. Two weeks of "successful" indexing pings that indexed nothing is a fairly
expensive way to learn to read the response body.

Top comments (0)