DEV Community

Cover image for Sitemap couldn't fetch in Google Search Console - fix
1

Sitemap couldn't fetch in Google Search Console - fix

To set up your site to be indexed by Google Search, you need to have a robots.txt file and a sitemap.xml file in your site.

Here’s how I set up my own sitemap and robots file using Next.js. My solution takes TypeScript files and converts them into .txt and .xml files, which are then placed in the public directory of the site.

robot.ts
This is the code for my robots.txt file:

import { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: "/",
    },
    sitemap: "https://www.thook.xyz/sitemap.xml",
    host: "https://www.thook.xyz",
  };
}
Enter fullscreen mode Exit fullscreen mode

sitemap.ts
Below is the code for generating my sitemap.xml file dynamically:

import { MetadataRoute } from "next";

export default function sitemap(): MetadataRoute.Sitemap {
  const routes: MetadataRoute.Sitemap = [
    {
      url: "",
      changeFrequency: "monthly",
      priority: 1,
    },
    {
      url: "/search",
      changeFrequency: "monthly",
      priority: 0.5,
    },
  ];

  return routes.map(({ url, ...rest }) => ({
    ...rest,
    url: `https://thook.xyz${url}`,
    lastModified: new Date(),
  }));
}
Enter fullscreen mode Exit fullscreen mode

The Problem 🤔

While my robots.txt worked flawlessly, my sitemap.xml kept returning an error:

sitemap unsuccessful

It seemed like Google Search Console had cached an old version of my sitemap, and nothing I did seemed to fix the issue.

The Solution 🚀

To work around this, I created a new sitemap2.xml file and placed it in the public directory of my Next.js application:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">
    <url>
        <loc>https://thook.xyz/</loc>
        <lastmod>2025-01-01T11:39:05.531Z</lastmod>
        <changefreq>yearly</changefreq>
        <priority>1</priority>
        <xhtml:link rel="home" href="https://thook.xyz/" />
    </url>
    <url>
        <loc>https://thook.xyz/search</loc>
        <lastmod>2025-01-01T11:39:05.531Z</lastmod>
        <changefreq>monthly</changefreq>
        <priority>0.8</priority>
        <xhtml:link rel="search" href="https://thook.xyz/search" />
    </url>
</urlset>
Enter fullscreen mode Exit fullscreen mode

And just like that, Booom!💥.
Jan 1st problem solved ✅.

sitemap successful

Lessons Learned 📚

Here’s what I learned about using Google Search Console:

  • Be patient: Google Search Console can take time to crawl and update your site. Don’t panic.
  • Try a fresh name: If you’re facing persistent errors, create a new sitemap with a different name that Google hasn’t cached yet.

Thanks for reading.

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read full post →

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay