DEV Community

UI Projects
UI Projects

Posted on

My website (idoks.ai) not getting crawled or indexed properly

Hi,
My website https://idoks.ai is not being crawled or indexed properly by Google. Only the homepage appears in search results.

Sitemap and robots.txt are valid:

Sitemap: https://idoks.ai/sitemap_index.xml
Robots: https://idoks.ai/robots.txt
I have checked with hosting (SiteGround), and the site is live and accessible. Can someone please check if there’s any issue preventing Googlebot from crawling my pages?

Thanks,
Muruganantham Chandran

Top comments (4)

Collapse
 
pascal_cescato_692b7a8a20 profile image
Pascal CESCATO • Edited

Maybe it's because of the x-robots-tag: noindex, follow in your sitemap headers - even if there's a follow instruct, the noindex may cause problems. Don't you think so?

Collapse
 
ui_projects_ad6066e71d4ad profile image
UI Projects

I had checked with the SiteGround team, and that’s not an issue. The x-robots-tag header is generated by the Yoast SEO plugin.

Collapse
 
pascal_cescato_692b7a8a20 profile image
Pascal CESCATO

You're rigth, the x-robots-tag: noindex, follow on your sitemap isn’t really a problem - Google doesn’t index sitemap files anyway. I checked your headers, and there’s no noindex sent from the server on regular pages, so that’s fine.

If only the homepage is indexed, it’s more likely about how Google’s crawling the site - maybe it’s still new, or the internal links/content don’t give enough signals yet. I’d check Search Console (“Crawled – not indexed”) and maybe submit a few key pages manually. It usually resolves once Google builds more trust.

Thread Thread
 
ui_projects_ad6066e71d4ad profile image