DEV Community

Aliaksandr Bandziuk
Aliaksandr Bandziuk

Posted on

Perfect technical SEO, and Google still skipped half my site

On 18 September I opened a new site to indexing. 160 pages, three languages, Next.js, built from scratch. Twenty-eight days later Google had 81 pages in the index and 79 outside it, 71 of them sitting in the status every developer eventually meets: “Discovered — currently not indexed”.

I am writing this up because the usual advice for that status is a list of technical fixes, and on this site every one of those fixes was already in place before the first crawl.

Here is what shipped on day one. A sitemap with real lastmod values, not a build timestamp rewritten on every deploy. Schema.org markup on each page type rather than a single Organization block on the homepage. IndexNow pings fired on publish, so Bing and Yandex learn about a URL the moment it exists. FAQ answers rendered into HTML on the server, because an answer that only appears after hydration is an answer a crawler may never see. Core Web Vitals in the green: LCP 1.8s, INP 110ms, CLS 0.003. One honest caveat on that last line, since I am asking you to trust the rest: the field data behind it came from nine views, which is a sample, not a measurement.

What that bought: indexing started fast. First impressions appeared on day two. 70 of the 160 pages were in the index within nine days. For a domain registered weeks earlier, that is the technical setup doing its job.

What it did not buy: the other half. Three weeks in, 71 URLs were still discovered and uncrawled, and nothing on the site explains why. robots.txt is clean, the server answers quickly, the markup validates, the pages are linked from the sitemap and from each other. The status is not a defect report. It means Google knows the URL exists and has decided it is not worth the fetch. Crawl budget follows signals of trust, and this domain has none: zero backlinks, zero mentions, zero branded demand.

The traffic numbers for the same window are small enough to print in full. 1,068 impressions and 6 clicks over 28 days, a CTR of 0.56%, average position 14.9 across devices. First clicks arrived on day ten. Impressions climbed to 110 on the best day and fell back to 17 by the end of the period.

Three findings worth more than the totals.

Positions are not traffic. 250 impressions landed at positions 3 to 8 and produced zero clicks. Being on the first screen means nothing when the answer sits above you in an AI Overview and the brand below it is unknown.

Bots are a bigger share of a new site’s analytics than anyone budgets for. Microsoft Clarity recorded 161 sessions and separately excluded 107 bot sessions. That is 40% of everything that reached the counter. An unfiltered report would have shown 268 sessions and a far healthier-looking month.

Assistants show up before rankings do. In those 161 sessions, Google sent 11 referrals and ChatGPT sent 7. The volumes are tiny and I would not build a strategy on them, but the channel was live on a site that had not yet reached the first page for anything.

IndexNow deserves its own line, since it is the fix most often recommended for slow indexing. It worked exactly as documented and Bing picked the site up: 56 impressions between 20 September and 11 October, and zero clicks. Fast notification is not demand.

The full export, with the Search Console, Bing and Clarity screenshots behind every number, is here: https://bandziuk.com/blog/discovered-currently-not-indexed-new-site-data

If you want the other half of the story — how the site itself is structured, and which of those technical decisions were deliberate rather than defaults — that is written up as a case study: https://bandziuk.com/portfolio/consultant-website-design-retail-analytics

Next month I repeat the measurement and test one thing only: whether the uncrawled pages move once external signals exist. If they do not, the technical explanation was never the explanation

Top comments (1)

Collapse
 
omyvnss profile image
Om Yaduvanshi •

the 'discovered — currently not indexed' bucket is doing exactly what it says. google found the pages and has no reason to crawl them yet. people read it as a verdict when it's a queue.

the bot-share point is the one that stays with me. a new site has no returning visitors, so the crawl share looks enormous and every early number has to be read twice. most tools leave the reader-plus-crawler mix to you and call it a dashboard.

one question on the re-measure: will you watch whether the uncrawled pages move on their own, or do they ride along once something links them? that split tells you whether the variable is the page or the path to it.