When building a website, developers usually focus on application logic, performance, security, and user experience. But there is another important technical layer that developers should not overlook: how search engines discover the URLs on a website.
This is where an XML sitemap becomes useful.
An XML sitemap provides search engines with a structured list of URLs that you want them to discover and consider. It does not guarantee that every URL will be indexed, and it is not a direct ranking factor, but it can make URL discovery more efficient.
For developers working with Next.js, React, Laravel, ASP.NET Core, WordPress, eCommerce platforms, or custom applications, understanding how sitemaps work can prevent a number of technical SEO problems.
What Is an XML Sitemap?
An XML sitemap is an XML file that contains URLs from a website. A basic sitemap looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="[http://www.sitemaps.org/schemas/sitemap/0.9](http://www.sitemaps.org/schemas/sitemap/0.9)">
<url>
<loc>[https://example.com/](https://example.com/)</loc>
<lastmod>2026-09-01</lastmod>
</url>
<url>
<loc>[https://example.com/about](https://example.com/about)</loc>
<lastmod>2026-08-25</lastmod>
</url>
<url>
<loc>[https://example.com/blog](https://example.com/blog)</loc>
<lastmod>2026-08-30</lastmod>
</url>
</urlset>
The most important element is , which contains the URL.
The element can indicate when the page was last meaningfully modified.
A sitemap is not a database of every URL your application can generate. Ideally, it represents the important URLs that you want search engines to discover and potentially index.
Why Do Search Engines Need Sitemaps?
Search engines can discover URLs through many different mechanisms, especially links.
For example:
Homepage
↓
Category
↓
Product
If your website has a good internal linking structure, crawlers can often discover a large portion of the site naturally.
But real-world websites are not always that simple.
Consider an application with:
50,000 product pages
10,000 blog posts
Multiple category levels
Dynamically generated URLs
Frequently changing inventory
Pages that aren't heavily linked internally
A sitemap provides another structured source of URL information.
This is especially useful for large, new, frequently updated, or complex websites.
A Sitemap Does Not Guarantee Indexing
This is an important distinction.
Putting this URL into your sitemap:
does not mean:
Google must index this page.
A sitemap is primarily a discovery signal.
Search engines still need to evaluate the URL and determine whether it should be crawled, indexed, and shown in search results.
So developers should avoid thinking of a sitemap as an indexing command.
A better mental model is:
Sitemap
↓
URL Discovery
↓
Crawling
↓
Processing
↓
Indexing decision
↓
Search results
Each stage has its own requirements.
What Should Go Into a Sitemap?
A good sitemap should contain URLs that are:
Important
Canonical
Accessible
Intended for indexing
Returning an appropriate HTTP response
For example:
https://example.com/
https://example.com/products
https://example.com/products/laptop
https://example.com/blog/technical-seo
The exact URLs depend on the architecture of your application.
What Should Not Go Into a Sitemap?
One of the most common problems I've seen with automatically generated sitemaps is that they contain URLs that shouldn't be there.
Examples include:
404 URLs
Redirect URLs
Duplicate URLs
Non-canonical URLs
Noindex pages
Temporary URLs
Internal search URLs
Tracking parameter URLs
For example:
https://example.com/product?id=123
https://example.com/product?id=123&utm_source=google
https://example.com/product?id=123&sort=price
If these are simply different variations of the same canonical page, you probably don't want all of them in your sitemap.
The Developer's Problem: Automatically Generated Sitemaps
This is where things get interesting.
Many modern applications generate sitemaps automatically.
For example, you might have:
const products = await db.product.findMany({
where: {
published: true
}
});
Then generate:
/products/product-1
/products/product-2
/products/product-3
This seems straightforward.
But what happens when:
A product is deleted?
A product becomes unpublished?
The slug changes?
The database contains an invalid record?
The URL starts returning 404?
The page becomes noindex?
A redirect is introduced?
Your sitemap generator can continue producing URLs that shouldn't be there.
That's why sitemap generation and sitemap validation should be treated as two separate concerns.
Sitemap Generation vs Sitemap Validation
Think about it this way:
Database
↓
Sitemap Generator
↓
sitemap.xml
↓
Sitemap Validator / Crawler
↓
Problems detected
Generating the sitemap answers:
"Which URLs should I put into the sitemap?"
Validation answers:
"Are those URLs actually healthy?"
Both are important.
How to Check an XML Sitemap
For a small site, you can open the sitemap in a browser and inspect it manually.
For example:
https://example.com/sitemap.xml
But this approach doesn't scale.
Imagine manually checking 10,000 URLs.
That's where a sitemap crawler becomes useful.
A sitemap crawler can take the URLs listed in your sitemap and inspect their responses.
For example, you may discover:
200 OK
200 OK
301 Redirect
200 OK
404 Not Found
200 OK
That immediately gives you information about the health of the URLs listed in your sitemap.
If you're looking for a simple way to inspect sitemap URLs, the Blaze Solutions Sitemap Crawler can be used to crawl a sitemap:
https://blazesolutions.info/tools/sitemap-crawler
The important point isn't the specific tool—it's the workflow:
Generate → Crawl → Identify problems → Fix → Recheck
HTTP Status Codes Matter
When validating sitemap URLs, HTTP status codes are particularly useful.
200 OK
Usually indicates that the URL successfully returns a page.
GET /blog/example
200 OK
This is generally what you want for a normal indexable page.
301 / 308 Redirect
The URL redirects somewhere else.
GET /old-page
301 → /new-page
If the final URL is the canonical page, consider putting the final URL in the sitemap instead.
404 Not Found
The resource doesn't exist.
GET /deleted-page
404 Not Found
A deleted page generally shouldn't remain in your sitemap.
5xx Errors
Server-side errors can indicate application or infrastructure problems.
500 Internal Server Error
503 Service Unavailable
These deserve investigation, especially if they affect important URLs.
XML Sitemap and Canonical URLs
Sitemaps and canonical URLs should generally agree with each other.
Suppose you have:
https://example.com/article
https://example.com/article?ref=twitter
If the canonical version is:
your sitemap should normally contain the canonical URL.
You want your technical SEO signals to be consistent:
Sitemap
↓
Canonical URL
Internal Link
↓
Canonical URL
Canonical Tag
↓
Canonical URL
Consistency reduces unnecessary ambiguity.
XML Sitemap and robots.txt
robots.txt and sitemap.xml solve different problems.
A sitemap tells crawlers about URLs you want them to discover.
A robots.txt file provides crawler instructions.
For example:
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
The sitemap can be referenced directly from robots.txt.
One important point for developers:
Do not assume that adding a URL to a sitemap overrides a robots.txt restriction.
They are separate mechanisms with different purposes.
Dynamic Websites Need Extra Attention
Static websites are relatively simple.
Dynamic applications are different.
Consider an eCommerce application where product pages are generated from a database.
Today:
/products/phone-x
returns:
200 OK
Tomorrow, the product is deleted.
If your sitemap generator doesn't account for that change, the sitemap may still contain:
/products/phone-x
which now returns:
404 Not Found
This is why dynamic sitemap generation should use the same business rules that determine whether a page is publicly available.
For example:
const products = await db.product.findMany({
where: {
status: "published",
isDeleted: false
}
});
The exact implementation depends on your application, but the principle is simple:
Don't generate sitemap URLs from data that shouldn't produce public pages.
Sitemap Indexes for Large Websites
Large websites don't necessarily need one massive sitemap file.
You can split your sitemap into multiple files.
For example:
sitemap.xml
sitemap-products.xml
sitemap-blog.xml
sitemap-categories.xml
A sitemap index can reference the individual files.
This approach makes large websites easier to manage.
For example:
https://example.com/sitemap-products.xml
https://example.com/sitemap-blog.xml
For large applications, separating sitemap generation by content type can also simplify debugging.
A Practical Sitemap Monitoring Workflow
If I were maintaining a large web application, I wouldn't treat the sitemap as a one-time SEO task.
I'd make it part of the technical monitoring process.
A simple workflow could look like this:
- Generate sitemap ↓
- Validate XML ↓
- Crawl sitemap URLs ↓
- Check HTTP status ↓
- Check canonical consistency ↓
- Remove invalid URLs ↓
- Deploy ↓
- Monitor Search Console
This is especially useful after:
Website migrations
CMS migrations
URL structure changes
Large database updates
Product imports
Removing thousands of pages
Changing routing logic
Common Sitemap Mistakes Developers Should Avoid
Mistake 1: Generating Every Database URL
Not every database record should become a public URL.
Filter your data before generating the sitemap.
Mistake 2: Ignoring HTTP Status Codes
A URL existing in your database doesn't mean the HTTP endpoint works.
Always consider the actual response.
Mistake 3: Including Redirects
If a URL redirects permanently to another URL, the final URL is usually a better sitemap candidate.
Mistake 4: Ignoring Canonicalization
Make sure sitemap URLs generally match your canonical URLs.
Mistake 5: Updating lastmod on Every Build
If your application runs a deployment every night, that doesn't mean every page was modified.
Only update lastmod when meaningful content changes.
Mistake 6: Treating Sitemap Submission as an SEO Strategy
Submitting a sitemap is useful, but it isn't a replacement for:
Good content
Internal linking
Technical SEO
Performance optimization
Search intent
Backlinks
A good website architecture
A Simple Technical SEO Checklist
Before considering your sitemap production-ready, check:
[ ] XML is valid
[ ] URLs are absolute
[ ] HTTPS is used where appropriate
[ ] URLs return expected responses
[ ] No unnecessary redirects
[ ] No 404 URLs
[ ] Canonical URLs are used
[ ] Noindex pages are excluded
[ ] Important pages are included
[ ] lastmod values are accurate
[ ] robots.txt references the sitemap
[ ] Sitemap is submitted to Search Console
For larger sites, automate as many of these checks as possible.
Final Thoughts
An XML sitemap is a small part of a web application, but it sits at an interesting intersection between development and technical SEO.
For developers, the key lesson is that a sitemap isn't just an XML file.
It's a representation of your application's public URL architecture.
If the application changes, the sitemap needs to change with it.
If products are deleted, URLs should disappear.
If routes change, the sitemap should reflect the new routes.
If pages become non-indexable, they shouldn't continue appearing as important sitemap URLs.
A healthy sitemap is therefore the result of both good application architecture and good technical SEO practices.
And if your sitemap contains hundreds or thousands of URLs, don't rely entirely on manual inspection. Crawl it, check the responses, identify anomalies, and fix the underlying application or content issues.
That's a much more reliable approach than simply generating sitemap.xml and forgetting about it..
Top comments (0)