DEV Community

Emirhan Güven
Emirhan Güven

Posted on

Google crawled 230 pages of my site and indexed 2. Here is how I diagnosed it.

#seo #webdev #googlecloud #python

My marketing site has 230 URLs in its sitemaps. Google has indexed 2 of them.

Not "pending". Not "blocked". Crawled, fetched successfully, canonical accepted, and then declined. This post is the diagnostic path I took, the two measurement mistakes I made along the way, and the Search Console API calls that finally produced a straight answer.

The setup

The site is a Vite + React SPA prerendered at build time by Puppeteer. Every route ships as a static HTML file with the content already in the markup, served by nginx. Not SSR, but for a crawler it is indistinguishable: the body arrives full.

The obvious hypotheses, in the order everyone tries them:

  1. Google cannot render the JavaScript.
  2. robots.txt is blocking something.
  3. Canonicals point somewhere else.
  4. The pages are thin or duplicated.
  5. There is a manual action.

All five were wrong. Here is how each died.

Killing the easy hypotheses

Fetch the page as Googlebot and check the markup directly:

curl -sS -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://example.com/ -o home.html

grep -o '<title>[^<]*</title>' home.html
grep -o '<link rel="canonical"[^>]*>' home.html
grep -c '<h1' home.html
Enter fullscreen mode Exit fullscreen mode

381 KB of HTML, one <h1>, correct canonical, index, follow. Rendering was never the problem.

For internal links, count real anchors rather than trusting a framework's router:

hrefs = re.findall(r'<a\s[^>]*href="([^"]+)"', html)
internal = [h for h in hrefs if h.startswith('/')]
print(len(set(internal)), html.count('nofollow'))
# 84 unique internal links, 0 nofollow
Enter fullscreen mode Exit fullscreen mode

That matters because a React app that navigates with onClick handlers instead of <a href> is invisible to a crawler's link graph. This one was fine.

The Search Console API is the only source of truth

Everything above tells you what you serve. It says nothing about what Google decided. For that there are exactly two useful endpoints.

URL Inspection — per-URL verdict, up to 2000 calls/day:

from googleapiclient.discovery import build

service = build('searchconsole', 'v1', credentials=creds)
result = service.urlInspection().index().inspect(body={
    'inspectionUrl': 'https://example.com/pricing',
    'siteUrl': 'sc-domain:example.com',
}).execute()

status = result['inspectionResult']['indexStatusResult']
print(status['verdict'], '|', status['coverageState'])
print(status['robotsTxtState'], status['pageFetchState'], status['lastCrawlTime'])
Enter fullscreen mode Exit fullscreen mode

Run it over a representative sample rather than one URL. Mine came back like this:

/                      PASS    | Submitted and indexed
/pricing               NEUTRAL | Crawled - currently not indexed
/features              NEUTRAL | Crawled - currently not indexed
/telegram-crm          NEUTRAL | Crawled - currently not indexed
/compare/x-vs-y        NEUTRAL | Crawled - currently not indexed
/guides/some-guide     NEUTRAL | Crawled - currently not indexed
Enter fullscreen mode Exit fullscreen mode

Every one of them: robotsTxtState: ALLOWED, pageFetchState: SUCCESSFUL, lastCrawlTime within the last two weeks, googleCanonical matching userCanonical.

That combination is unambiguous. Google arrived, fetched a 200, parsed the page, agreed with my canonical, and chose not to index. There is no technical defect to fix, because nothing failed.

The aggregate view confirmed it: 2 indexed, 185 not indexed — 99 "Discovered, currently not indexed" and 86 "Crawled, currently not indexed".

Two measurement mistakes worth stealing

1. quick_ratio will tell you your pages are duplicates when they are not

Template-generated pages (/compare/a-vs-b, /use-cases/industry) are the obvious suspects for thin content. So I measured similarity:

import difflib
ratio = difflib.SequenceMatcher(None, page_a, page_b).quick_ratio()
# 95.9%, 90.9%, 88.5% ...
Enter fullscreen mode Exit fullscreen mode

Damning, apparently. Except quick_ratio() compares character frequency multisets, not sequences. Two unrelated English documents of similar length score high on it by construction. It is a cheap upper bound designed to skip expensive comparisons, not a similarity metric.

The honest measurement is to strip the shared chrome and count what survives:

from collections import Counter

# how many pages does each line of text appear on?
counts = Counter()
for lines in docs.values():
    for line in set(lines):
        counts[line] += 1

for url, lines in docs.items():
    unique = [l for l in lines if counts[l] == 1]
    total_w  = sum(len(l.split()) for l in lines)
    unique_w = sum(len(l.split()) for l in unique)
    print(url, f"{unique_w}/{total_w} = {unique_w*100//total_w}% unique")
Enter fullscreen mode Exit fullscreen mode

Result: 90-95% unique per page, 1400-1950 words of it, with only ~35 words of shared boilerplate (nav, CTA strip, footer). The pages were not the problem. Had I stopped at quick_ratio I would have spent a week rewriting content that was already fine.

2. site: in a search box is not an index count

num=100 no longer works on Google. If you count results on the first page you are counting ten. Paginate with start= and stop when a page returns fewer than a full set, or just read the Page Indexing report, which is authoritative.

What the answer actually was

With the technical hypotheses dead, one report explained everything:

External links: Total 10   (3 domains)
Internal links: Total 0
Enter fullscreen mode Exit fullscreen mode

Ten inbound links from three domains, all to the homepage. Meanwhile a backlink tool reported 197 links — but more than half came from a single PBN spam network hammering one keyword-stuffed anchor. Google had simply not counted them. The tool's number was noise; Search Console's was the signal.

So: a site with real content, clean markup, correct canonicals, valid sitemaps, no manual action, and effectively zero independent references, publishing 230 pages into one of the most saturated categories on the web.

"Crawled, currently not indexed" is not a bug report. It is a verdict. Google read the pages and decided the index did not need them. More crawl budget does not fix that, and neither does resubmitting a sitemap — I have two months of flat data to prove it.

The part that generalises

  • If pageFetchState is SUCCESSFUL and robotsTxtState is ALLOWED, stop debugging your stack. The answer is not in your code.
  • Check the Links report before rewriting content. Zero external links explains more indexing problems than any rendering bug.
  • Backlink tools and Search Console disagree, and for "does Google count this link", Search Console wins.
  • Verify a claimed fix against production before repeating it. Three broken URLs I was still citing from a month-old crawl had already been redirected; I only found out because I re-checked with curl instead of trusting my notes.

The uncomfortable conclusion is that for a new domain in a crowded category, indexing is downstream of being referenced by someone other than yourself. That is a distribution problem wearing a technical costume.

I work on CRM Solid, an omnichannel CRM. The numbers above are from its marketing site, which is a live and ongoing example of the problem.

Top comments (0)