DEV Community: Asikul Islam

Understanding XML Sitemaps: A Developer's Guide to Better URL Discovery

Asikul Islam — Wed, 09 Sep 2026 14:04:34 +0000

When building a website, developers usually focus on application logic, performance, security, and user experience. But there is another important technical layer that developers should not overlook: how search engines discover the URLs on a website.

This is where an XML sitemap becomes useful.

An XML sitemap provides search engines with a structured list of URLs that you want them to discover and consider. It does not guarantee that every URL will be indexed, and it is not a direct ranking factor, but it can make URL discovery more efficient.

For developers working with Next.js, React, Laravel, ASP.NET Core, WordPress, eCommerce platforms, or custom applications, understanding how sitemaps work can prevent a number of technical SEO problems.

What Is an XML Sitemap?

An XML sitemap is an XML file that contains URLs from a website. A basic sitemap looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="[http://www.sitemaps.org/schemas/sitemap/0.9](http://www.sitemaps.org/schemas/sitemap/0.9)">
    <url>
        <loc>[https://example.com/](https://example.com/)</loc>
        <lastmod>2026-09-01</lastmod>
    </url>
    <url>
        <loc>[https://example.com/about](https://example.com/about)</loc>
        <lastmod>2026-08-25</lastmod>
    </url>
    <url>
        <loc>[https://example.com/blog](https://example.com/blog)</loc>
        <lastmod>2026-08-30</lastmod>
    </url>
</urlset>

The most important element is , which contains the URL.

The element can indicate when the page was last meaningfully modified.

A sitemap is not a database of every URL your application can generate. Ideally, it represents the important URLs that you want search engines to discover and potentially index.

Why Do Search Engines Need Sitemaps?

Search engines can discover URLs through many different mechanisms, especially links.

For example:

Homepage
↓
Category
↓
Product

If your website has a good internal linking structure, crawlers can often discover a large portion of the site naturally.

But real-world websites are not always that simple.

Consider an application with:

50,000 product pages
10,000 blog posts
Multiple category levels
Dynamically generated URLs
Frequently changing inventory
Pages that aren't heavily linked internally

A sitemap provides another structured source of URL information.

This is especially useful for large, new, frequently updated, or complex websites.

A Sitemap Does Not Guarantee Indexing

This is an important distinction.

Putting this URL into your sitemap:

https://example.com/page

does not mean:

Google must index this page.

A sitemap is primarily a discovery signal.

Search engines still need to evaluate the URL and determine whether it should be crawled, indexed, and shown in search results.

So developers should avoid thinking of a sitemap as an indexing command.

A better mental model is:

Sitemap
↓
URL Discovery
↓
Crawling
↓
Processing
↓
Indexing decision
↓
Search results

Each stage has its own requirements.

What Should Go Into a Sitemap?

A good sitemap should contain URLs that are:

Important
Canonical
Accessible
Intended for indexing
Returning an appropriate HTTP response

For example:

https://example.com/
https://example.com/products
https://example.com/products/laptop
https://example.com/blog/technical-seo

The exact URLs depend on the architecture of your application.

What Should Not Go Into a Sitemap?

One of the most common problems I've seen with automatically generated sitemaps is that they contain URLs that shouldn't be there.

Examples include:

404 URLs
Redirect URLs
Duplicate URLs
Non-canonical URLs
Noindex pages
Temporary URLs
Internal search URLs
Tracking parameter URLs

For example:

https://example.com/product?id=123
https://example.com/product?id=123&utm_source=google
https://example.com/product?id=123&sort=price

If these are simply different variations of the same canonical page, you probably don't want all of them in your sitemap.

The Developer's Problem: Automatically Generated Sitemaps

This is where things get interesting.

Many modern applications generate sitemaps automatically.

For example, you might have:

const products = await db.product.findMany({
where: {
published: true
}
});

Then generate:

/products/product-1
/products/product-2
/products/product-3

This seems straightforward.

But what happens when:

A product is deleted?
A product becomes unpublished?
The slug changes?
The database contains an invalid record?
The URL starts returning 404?
The page becomes noindex?
A redirect is introduced?

Your sitemap generator can continue producing URLs that shouldn't be there.

That's why sitemap generation and sitemap validation should be treated as two separate concerns.

Sitemap Generation vs Sitemap Validation

Think about it this way:

Database
↓
Sitemap Generator
↓
sitemap.xml
↓
Sitemap Validator / Crawler
↓
Problems detected

Generating the sitemap answers:

"Which URLs should I put into the sitemap?"

Validation answers:

"Are those URLs actually healthy?"

Both are important.

How to Check an XML Sitemap

For a small site, you can open the sitemap in a browser and inspect it manually.

For example:

https://example.com/sitemap.xml

But this approach doesn't scale.

Imagine manually checking 10,000 URLs.

That's where a sitemap crawler becomes useful.

A sitemap crawler can take the URLs listed in your sitemap and inspect their responses.

For example, you may discover:

200 OK
200 OK
301 Redirect
200 OK
404 Not Found
200 OK

That immediately gives you information about the health of the URLs listed in your sitemap.

If you're looking for a simple way to inspect sitemap URLs, the Blaze Solutions Sitemap Crawler can be used to crawl a sitemap:

https://blazesolutions.info/tools/sitemap-crawler

The important point isn't the specific tool—it's the workflow:

Generate → Crawl → Identify problems → Fix → Recheck

HTTP Status Codes Matter

When validating sitemap URLs, HTTP status codes are particularly useful.

200 OK

Usually indicates that the URL successfully returns a page.

GET /blog/example
200 OK

This is generally what you want for a normal indexable page.

301 / 308 Redirect

The URL redirects somewhere else.

GET /old-page
301 → /new-page

If the final URL is the canonical page, consider putting the final URL in the sitemap instead.

404 Not Found

The resource doesn't exist.

GET /deleted-page
404 Not Found

A deleted page generally shouldn't remain in your sitemap.

5xx Errors

Server-side errors can indicate application or infrastructure problems.

500 Internal Server Error
503 Service Unavailable

These deserve investigation, especially if they affect important URLs.

XML Sitemap and Canonical URLs

Sitemaps and canonical URLs should generally agree with each other.

Suppose you have:

https://example.com/article
https://example.com/article?ref=twitter

If the canonical version is:

https://example.com/article

your sitemap should normally contain the canonical URL.

You want your technical SEO signals to be consistent:

Sitemap
↓
Canonical URL

Internal Link
↓
Canonical URL

Canonical Tag
↓
Canonical URL

Consistency reduces unnecessary ambiguity.

XML Sitemap and robots.txt

robots.txt and sitemap.xml solve different problems.

A sitemap tells crawlers about URLs you want them to discover.

A robots.txt file provides crawler instructions.

For example:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

The sitemap can be referenced directly from robots.txt.

One important point for developers:

Do not assume that adding a URL to a sitemap overrides a robots.txt restriction.

They are separate mechanisms with different purposes.

Dynamic Websites Need Extra Attention

Static websites are relatively simple.

Dynamic applications are different.

Consider an eCommerce application where product pages are generated from a database.

Today:

/products/phone-x

returns:

200 OK

Tomorrow, the product is deleted.

If your sitemap generator doesn't account for that change, the sitemap may still contain:

/products/phone-x

which now returns:

404 Not Found

This is why dynamic sitemap generation should use the same business rules that determine whether a page is publicly available.

For example:

const products = await db.product.findMany({
where: {
status: "published",
isDeleted: false
}
});

The exact implementation depends on your application, but the principle is simple:

Don't generate sitemap URLs from data that shouldn't produce public pages.

Sitemap Indexes for Large Websites

Large websites don't necessarily need one massive sitemap file.

You can split your sitemap into multiple files.

For example:

sitemap.xml
sitemap-products.xml
sitemap-blog.xml
sitemap-categories.xml

A sitemap index can reference the individual files.

This approach makes large websites easier to manage.

For example:

https://example.com/sitemap-products.xml

https://example.com/sitemap-blog.xml

For large applications, separating sitemap generation by content type can also simplify debugging.

A Practical Sitemap Monitoring Workflow

If I were maintaining a large web application, I wouldn't treat the sitemap as a one-time SEO task.

I'd make it part of the technical monitoring process.

A simple workflow could look like this:

Generate sitemap ↓
Validate XML ↓
Crawl sitemap URLs ↓
Check HTTP status ↓
Check canonical consistency ↓
Remove invalid URLs ↓
Deploy ↓
Monitor Search Console

This is especially useful after:

Website migrations
CMS migrations
URL structure changes
Large database updates
Product imports
Removing thousands of pages
Changing routing logic
Common Sitemap Mistakes Developers Should Avoid
Mistake 1: Generating Every Database URL

Not every database record should become a public URL.

Filter your data before generating the sitemap.

Mistake 2: Ignoring HTTP Status Codes

A URL existing in your database doesn't mean the HTTP endpoint works.

Always consider the actual response.

Mistake 3: Including Redirects

If a URL redirects permanently to another URL, the final URL is usually a better sitemap candidate.

Mistake 4: Ignoring Canonicalization

Make sure sitemap URLs generally match your canonical URLs.

Mistake 5: Updating lastmod on Every Build

If your application runs a deployment every night, that doesn't mean every page was modified.

Only update lastmod when meaningful content changes.

Mistake 6: Treating Sitemap Submission as an SEO Strategy

Submitting a sitemap is useful, but it isn't a replacement for:

Good content
Internal linking
Technical SEO
Performance optimization
Search intent
Backlinks
A good website architecture
A Simple Technical SEO Checklist

Before considering your sitemap production-ready, check:

[ ] XML is valid
[ ] URLs are absolute
[ ] HTTPS is used where appropriate
[ ] URLs return expected responses
[ ] No unnecessary redirects
[ ] No 404 URLs
[ ] Canonical URLs are used
[ ] Noindex pages are excluded
[ ] Important pages are included
[ ] lastmod values are accurate
[ ] robots.txt references the sitemap
[ ] Sitemap is submitted to Search Console

For larger sites, automate as many of these checks as possible.

Final Thoughts

An XML sitemap is a small part of a web application, but it sits at an interesting intersection between development and technical SEO.

For developers, the key lesson is that a sitemap isn't just an XML file.

It's a representation of your application's public URL architecture.

If the application changes, the sitemap needs to change with it.

If products are deleted, URLs should disappear.

If routes change, the sitemap should reflect the new routes.

If pages become non-indexable, they shouldn't continue appearing as important sitemap URLs.

A healthy sitemap is therefore the result of both good application architecture and good technical SEO practices.

And if your sitemap contains hundreds or thousands of URLs, don't rely entirely on manual inspection. Crawl it, check the responses, identify anomalies, and fix the underlying application or content issues.

That's a much more reliable approach than simply generating sitemap.xml and forgetting about it..

Hello DEV Community 👋 — My Journey from Software Development to AI Automation

Asikul Islam — Mon, 07 Sep 2026 10:03:39 +0000

Hello DEV Community 👋 — My Journey from Software Development to AI Automation

This is my first post on DEV.to, so I thought I would start with a simple introduction.

I'm a software developer and AI automation engineer who has spent more than a decade building software, solving business problems, and exploring new technologies.

Over the years, I've worked with different technologies and frameworks, but one thing has remained constant: I'm always curious about how technology can make people's work easier.

From Software Development to AI

My professional journey started primarily around Microsoft technologies, including .NET, ASP.NET, ASP.NET MVC, ASP.NET Core, and Microsoft SQL Server.

Like many developers, I spent years building applications, designing databases, debugging production issues, optimizing systems, and working with teams to turn business requirements into working software.

But the technology landscape has changed dramatically.

AI is no longer something that exists only in research papers or experimental projects. It is becoming part of the way we build software, automate workflows, communicate with customers, analyze information, and run businesses.

That shift caught my attention.

I started exploring how AI could be combined with traditional software engineering and workflow automation to build systems that don't just respond to users, but can actually perform useful tasks.

That led me deeper into AI automation.

Why AI Automation?

I believe the most interesting opportunity isn't simply putting an AI chatbot on a website.

The real opportunity is connecting AI with business processes.

For example:

Finding and qualifying leads
Researching prospects
Generating personalized outreach
Processing documents
Automating repetitive administrative tasks
Monitoring competitors
Creating and optimizing content
Connecting different business applications
Building AI-powered internal assistants
Automating multi-step workflows

When these pieces are connected properly, AI becomes more than a chat interface.

It becomes part of the workflow.

And that's what I'm interested in building.

What I'm Working On

Currently, I'm exploring and building AI-powered automation systems using technologies such as:

AI APIs and LLMs
n8n workflow automation
Next.js
React
ASP.NET Core
SQL Server and MySQL
REST APIs
Cloud services
Webhooks
Business process automation

One area I'm particularly interested in is the concept of AI Employees.

Instead of thinking of AI only as a tool that answers questions, I like thinking about AI agents as digital workers that can be given a specific responsibility.

For example:

An AI Sales Employee that can research prospects, qualify leads, prepare personalized messages, and keep the sales pipeline updated.

There are still many challenges to solve—reliability, context management, security, human oversight, cost, and integration with existing systems.

That's what makes it interesting.

Why I'm Joining DEV.to

I've been working in software development for many years, but this is my first post on DEV.to.

I joined because I want to become more involved in the developer community and share what I learn along the way.

I don't want this profile to be just a place where I publish polished tutorials.

I also want to share the real development process:

Things that worked
Things that didn't work
Architecture decisions
Debugging stories
AI experiments
Automation workflows
Development lessons
Useful tools
Real-world problems and solutions

Sometimes the most valuable lesson comes from a problem that took several hours to solve.

So I'll try to document those experiences too.

What You'll Find Here

Going forward, I plan to write about topics such as:

AI & Automation

Practical AI automation, AI agents, LLM integrations, and business workflows.

Software Development

Architecture, APIs, backend development, databases, performance, and full-stack development.

n8n & Workflow Automation

Real-world automation workflows and integrations.

AI-Powered Products

Lessons from building and experimenting with AI-based products.

SEO & Developer Tools

I'm also interested in building useful tools that solve everyday problems for developers, marketers, and website owners.

A Developer's Journey Never Really Ends

One thing I've learned after spending many years in software development is that you never really "finish" learning.

The technologies change.

The frameworks change.

The tools change.

The problems change.

And now AI is changing the way we think about software itself.

That's exciting.

I'm looking forward to learning, building, experimenting, and sharing the journey here.

If you're also working with AI, automation, software development, or developer tools, I'd love to connect and learn from your experience.

This is just the beginning.

Hello, DEV Community! 👋

I'm glad to finally be here.

introduction #ai #automation #webdev #programming

DEV Community: Asikul Islam

Understanding XML Sitemaps: A Developer's Guide to Better URL Discovery

What Is an XML Sitemap?

Hello DEV Community 👋 — My Journey from Software Development to AI Automation

Hello DEV Community 👋 — My Journey from Software Development to AI Automation

From Software Development to AI

Why AI Automation?

What I'm Working On

Why I'm Joining DEV.to

What You'll Find Here

A Developer's Journey Never Really Ends

Tags

introduction #ai #automation #webdev #programming