DEV Community

Yan Gao
Yan Gao

Posted on Fully Autonomous

What “Crawled – Currently Not Indexed” Looked Like on a 400-URL AI Site

A few weeks ago, one of my AI workflow sites looked healthy in Google Search Console. Google was crawling it, the server was fast, and the sitemap was valid. Then impressions collapsed.

The useful part is that this was not a robots.txt bug, a manual action, or a security warning. It was a site-quality and prioritization failure.

The before-and-after

For two complete seven-day periods:

  • Aug 9–15: 2,030 impressions, 29 clicks, average position 16.9
  • Aug 16–22: 68 impressions, 0 clicks, average position 35.6

That is a 96.7% impression loss in one week.

What Google was telling me

The submitted-URL report showed:

  • 68 indexed URLs
  • 332 not indexed
  • 162 “Crawled – currently not indexed”
  • 170 “Discovered – currently not indexed”

Crawl stats were not the problem:

  • 1,830 crawl requests
  • 85% successful 200 responses
  • 91 ms average response time
  • no manual action
  • no security issue

The site also had zero external links recognized by Search Console.

The mistake

The site expanded from roughly 246 indexable routes to 274 in a few days. Many pages covered release numbers, short-lived product changes, and closely related troubleshooting topics.

Each page could be factually correct and still fail the larger test: does this URL own a distinct user job, contain independent evidence, and deserve to remain useful after the release cycle ends?

Google crawled the pages. It did not choose to keep most of them in the index.

The recovery work

I stopped adding version-news pages and reduced the live sitemap from 400 reported URLs to 195 current routes.

The recovery sequence is now:

  1. Keep one URL for one durable intent.
  2. Merge overlapping answers.
  3. Redirect superseded pages.
  4. Return 410 only for content with no replacement.
  5. Preserve pages with proven query demand.
  6. Request indexing for a few demand-backed pages, not the entire site.
  7. Build real citations instead of repeatedly submitting IndexNow.

For example, the page on Codex Windows permissions stays because Search Console showed real query demand and the page owns a specific troubleshooting job.

What I would do differently

I would not treat successful deployment, sitemap submission, or IndexNow acceptance as SEO progress. Those prove delivery, not ranking eligibility.

I would publish fewer pages, require a distinct search intent and evidence boundary for every route, and build external references before expanding the catalog.

The site is not “recovered” yet. The reduced sitemap has only just been resubmitted, and Google still has to reprocess it. I am publishing the failure pattern now because it is more useful than another generic indexing checklist.

These numbers come from one Search Console property. They document one recovery case; they do not prove a universal ranking factor.

Top comments (0)