DEV Community

Discussion on: My website (idoks.ai) not getting crawled or indexed properly

Collapse
 
pascal_cescato_692b7a8a20 profile image
Pascal CESCATO • Edited

Maybe it's because of the x-robots-tag: noindex, follow in your sitemap headers - even if there's a follow instruct, the noindex may cause problems. Don't you think so?

Collapse
 
ui_projects_ad6066e71d4ad profile image
UI Projects

I had checked with the SiteGround team, and that’s not an issue. The x-robots-tag header is generated by the Yoast SEO plugin.

Collapse
 
pascal_cescato_692b7a8a20 profile image
Pascal CESCATO

You're rigth, the x-robots-tag: noindex, follow on your sitemap isn’t really a problem - Google doesn’t index sitemap files anyway. I checked your headers, and there’s no noindex sent from the server on regular pages, so that’s fine.

If only the homepage is indexed, it’s more likely about how Google’s crawling the site - maybe it’s still new, or the internal links/content don’t give enough signals yet. I’d check Search Console (“Crawled – not indexed”) and maybe submit a few key pages manually. It usually resolves once Google builds more trust.

Thread Thread
 
ui_projects_ad6066e71d4ad profile image