DEV Community

Cover image for Check Which Pages Google Has Indexed With the URL Inspection API
Nesim Razon
Nesim Razon

Posted on Originally published at nesimapps.com

Check Which Pages Google Has Indexed With the URL Inspection API

Search Console reports only show pages that already earned impressions, so they cannot tell you which pages Google has not indexed at all. The URL Inspection API can. I used it to check all 381 URLs in my sitemap from a script: 329 were indexed and 52 were not. Here is how the check works, what the answers mean and what I did next.

Why the normal reports are not enough

The performance report lists queries and pages that got shown in search. A page with zero impressions is simply missing from it, and that is the page you most want to know about. For a site that is only a few weeks old, most pages are in that group.

How the API call works

The API takes two values: the URL you want to inspect and the property it belongs to. It answers with the index status, the coverage state in words, the last crawl time, the canonical URL Google chose and whether crawling was allowed. I read the sitemap file, sent each URL and kept the coverage state.

One detail cost me a failed first try: the property value must match how the site is registered. A domain property is written with a special prefix, and using the plain address returned a permission error saying the URL is not part of the property. A service account added as a restricted user of the property can run the inspections.

Be gentle with the quota

The API has a daily limit per site and a per-minute limit. My script reuses a cache and only re-checks a page after about a week, so a weekly run stays far below the limits, and a first run for a few hundred URLs finishes in several minutes with a few parallel requests.

What I found

Out of 381 URLs, 329 were “Submitted and indexed”. The other 52 split into three groups: 33 that Google did not know at all, 13 “Discovered, currently not indexed” and 6 “Crawled, currently not indexed”. By language the English pages were the best covered, and the Turkish ones had the most gaps. The pages Google did not know were mostly the newest posts and guides.

The three groups need different actions. “Unknown” means Google has not found the page yet, so help it find it. “Discovered” means it knows the address but has not crawled it. “Crawled, not indexed” means it looked and chose not to index, which is a content quality signal, and there I need to improve the page.

What I did about it

  • Resubmitted the sitemap in Search Console, after which it reported all 381 pages as discovered.
  • Made sure internal links and the blog index reach the newest pages.
  • Plan to re-run the same check a week later and compare the counts.
  • For the “crawled, not indexed” pages, plan to add real depth rather than request indexing again.

Make it part of a dashboard

Because the check is a plain script with a cache, it is easy to put on a dashboard: indexed over checked, a split by language and a short list of pages that are not indexed. A weekly number tells you whether your publishing is reaching the index.

FAQ

Can I check indexing for many pages at once?

Yes. The URL Inspection API checks one URL per request, so a script can loop over a sitemap, as long as it respects the documented daily and per-minute limits.

What is the difference between “discovered” and “crawled” but not indexed?

Discovered means Google knows the address but has not crawled it yet. Crawled means Google fetched the page and decided not to index it, usually a quality signal.

Why did I get a permission error?

The property value probably did not match how the site is registered, for example a plain URL instead of the domain property form.

Does resubmitting a sitemap guarantee indexing?

No. It helps Google discover URLs. Indexing still depends on Google’s decision about each page.

Top comments (0)