DEV Community: Asikul Islam

JWT Explained: How JSON Web Tokens Work

Asikul Islam — Tue, 15 Sep 2026 09:15:32 +0000

If you've worked with REST APIs, ASP.NET Core, Node.js, React, Next.js, or modern authentication systems, you've probably encountered JWT (JSON Web Token).

A JWT often looks like a long, unreadable string:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.signature

But what's actually inside it, and how does JWT authentication work?

What Is JWT?

JWT stands for JSON Web Token.

It is a compact format for representing claims that can be transferred between systems. JWTs are commonly used for authentication and authorization in web applications and APIs.

A simplified authentication flow looks like this:

User
  ↓
Login
  ↓
Authentication Server
  ↓
JWT
  ↓
Client
  ↓
API Request + Bearer Token
  ↓
API Server
  ↓
Token Validation
  ↓
Protected Resource

After a successful login, the server issues a token. The client then sends that token when accessing protected resources.

For example:

Authorization: Bearer <JWT>

The API validates the token before trusting its claims.

The 3 Parts of a JWT

A typical signed JWT consists of three parts:

HEADER.PAYLOAD.SIGNATURE

Each part has a different purpose.

1. Header

The header usually contains information about the token type and signing algorithm.

{
  "alg": "HS256",
  "typ": "JWT"
}

For example, HS256 indicates HMAC using SHA-256.

2. Payload

The payload contains claims.

{
  "sub": "12345",
  "name": "John Doe",
  "role": "admin",
  "iat": 1789474800,
  "exp": 1789478400
}

Common claims include:

sub — Subject
iss — Issuer
aud — Audience
exp — Expiration time
iat — Issued at
nbf — Not before
jti — JWT ID

Applications can also define custom claims.

3. Signature

The signature helps the server verify that the signed token has not been modified.

Conceptually, the signature is created from the encoded header and payload using a cryptographic algorithm and an appropriate secret or key.

This allows the server to detect tampering.

Is a JWT Encrypted?

Not necessarily.

This is one of the most important things to understand about JWT.

A normal signed JWT uses Base64URL encoding for its header and payload. Encoding does not make the information secret.

For example, if a JWT contains:

{
  "role": "admin"
}

someone who possesses the token can generally decode that information.

Therefore, don't put passwords, secret keys, or unnecessary sensitive information inside a normal JWT payload.

Remember:

Encoding ≠ Encryption

Decoding vs Verifying a JWT

Another common misconception is that decoding a JWT means the token is valid.

It doesn't.

Decoding allows you to inspect the contents of the header and payload.

Verification checks whether the signature is valid and whether the token meets the application's validation requirements.

For example, a decoded token might contain:

{
  "role": "admin"
}

That doesn't prove that the server should trust the role claim.

The server must properly validate the token before using its claims for authentication or authorization.

How to Decode a JWT

A JWT can be separated into its three components using the . character:

HEADER.PAYLOAD.SIGNATURE

The header and payload can then be Base64URL-decoded.

For example, the payload might reveal:

{
  "sub": "12345",
  "name": "John",
  "role": "user",
  "exp": 1789478400
}

You can do this manually during development, but a JWT decoder is much more convenient when debugging APIs.

If you need to inspect a token quickly, you can use the BlazeSolutions JWT Decoder to decode and inspect the JWT header and payload.

Important: Never paste production tokens containing sensitive information into an online tool unless you understand the privacy and security implications.

How to Check JWT Expiration

One of the most useful claims when debugging authentication problems is exp.

For example:

{
  "sub": "12345",
  "exp": 1789478400
}

The exp claim represents the token's expiration time.

If an API suddenly starts returning:

401 Unauthorized

checking the expiration claim is a useful first step.

However, an expired token isn't the only possible reason for a 401 response. Signature validation, issuer, audience, clock differences, authentication configuration, and other factors can also cause authentication failures.

JWT Access Tokens and Refresh Tokens

Modern authentication systems often use both access tokens and refresh tokens.

An access token is typically short-lived and is used to access protected APIs.

A refresh token can be used to obtain a new access token when the existing access token expires.

A simplified flow looks like this:

Login
  ↓
Access Token + Refresh Token
  ↓
Access Token Expires
  ↓
Refresh Token
  ↓
New Access Token

The exact implementation depends on the authentication architecture and protocol being used.

Common JWT Security Mistakes

JWT itself doesn't automatically make an authentication system secure. The implementation matters.

Some common mistakes include:

Trusting decoded claims

A decoded payload should not automatically be considered trustworthy.

Always perform proper token validation on the server.

Putting sensitive information in the payload

Avoid storing passwords, secret keys, or unnecessary confidential information in a normal JWT.

Using unnecessarily long-lived access tokens

If an access token is compromised, a long lifetime can increase the potential impact.

Skipping issuer and audience validation

If your application depends on iss and aud, make sure those claims are properly validated.

Accepting unexpected signing algorithms

Configure the server to accept only the algorithms appropriate for your authentication system rather than blindly trusting token-provided algorithm information.

JWT Debugging Checklist

When troubleshooting a JWT authentication problem, check:

Token structure — Does it contain three expected sections?
Header — Is the expected signing algorithm being used?
Payload — Are the required claims present?
Expiration — Has the token expired?
Issuer — Is the token from the expected issuer?
Audience — Is it intended for your API?
Signature — Can the server successfully validate it?
Authorization — Does the authenticated user have the required permissions?

This simple checklist can save a lot of time when debugging API authentication.

Final Thoughts

JWT is relatively simple once you understand its structure:

JWT
├── Header
├── Payload
└── Signature

The most important distinction to remember is:

A JWT can be decoded without being verified.

Understanding that difference helps avoid many common authentication and security mistakes.

If you want a deeper explanation of JWT structure, authentication flow, claims, expiration, access and refresh tokens, security considerations, and practical debugging, read the full guide:

JWT: How It Works, Structure, Authentication, and How to Decode a JWT

For quickly inspecting a JWT during development:

BlazeSolutions JWT Decoder

What JWT-related issue have you encountered recently — expired tokens, invalid signatures, 401 Unauthorized, or something else?

Caesar Cipher Explained: How It Works, Encryption, Decryption, and Examples

Asikul Islam — Sun, 13 Sep 2026 13:14:48 +0000

If you've ever been curious about how encryption works at a basic level, the Caesar Cipher is a great place to start.

It is one of the simplest classical encryption techniques. The idea is straightforward: shift each letter in a message by a fixed number of positions in the alphabet.

For example, with a shift of 3:

A → D
B → E
C → F

So, HELLO becomes KHOOR.

The Caesar Cipher isn't secure enough for modern applications, but it is an excellent way to understand fundamental concepts such as encryption, decryption, substitution ciphers, keys, modular arithmetic, and brute-force attacks.

In this article, we'll explore how it works and why it remains relevant for learning cryptography.

What Is a Caesar Cipher?

A Caesar Cipher is a type of substitution cipher where every letter in the plaintext is replaced by another letter a fixed number of positions away in the alphabet.

The number of positions is called the shift or key.

Consider the alphabet:

ABCDEFGHIJKLMNOPQRSTUVWXYZ

With a shift of 3, the mapping becomes:

ABCDEFGHIJKLMNOPQRSTUVWXYZ
DEFGHIJKLMNOPQRSTUVWXYZABC

Therefore:

A → D
B → E
C → F
...
X → A
Y → B
Z → C

The alphabet wraps around when the shift goes beyond Z.

How Caesar Cipher Encryption Works

Let's encrypt HELLO using a shift of 3.

We move each character three positions forward:

H → K
E → H
L → O
L → O
O → R

The encrypted result is KHOOR.

Plaintext: HELLO
Shift: 3
Ciphertext: KHOOR

The process is deterministic. If you encrypt the same text with the same shift, you'll always get the same ciphertext.

How Decryption Works

Decryption simply reverses the process.

If KHOOR was encrypted using a shift of 3, move every character three positions backward:

K → H
H → E
O → L
O → L
R → O

The original message is recovered: HELLO.

In simple terms:

Encryption → shift forward
Decryption → shift backward

Understanding the Shift Value

The shift value controls how far each character moves.

Shift	'A' becomes
1	B
2	C
3	D
5	F
10	K
13	N
25	Z

A shift of 0 doesn't change the text. Because the English alphabet contains 26 letters, a shift of 26 also produces the original text. This is why implementations commonly use modulo 26.

The Mathematics Behind Caesar Cipher

The Caesar Cipher becomes particularly interesting when we represent letters as numbers:

A = 0, B = 1, C = 2, ..., Z = 25

Encryption Formula

$$E(x) = (x + k) \pmod{26}$$

Where:

$x$ is the numerical value of the character
$k$ is the shift value
$E(x)$ is the encrypted value

Decryption Formula

$$D(x) = (x - k) \pmod{26}$$

The modulo operation provides the wraparound behavior. For example, if $Z = 25$ and $k = 3$:

$$(25 + 3) \pmod{26} = 28 \pmod{26} = 2$$

Since $2 = \text{C}$, $Z \to C$.

A Simple Caesar Cipher Algorithm

A basic implementation can follow these steps:

Read the input text.
Choose a shift value.
Iterate through every character.
Check whether the character is alphabetic.
Convert the character into a numerical position.
Apply the shift.
Use modulo 26 for wraparound.
Convert the result back to a character.
Preserve spaces and punctuation.

Pseudocode

function caesarCipher(text, shift):
    result = ""

    for each character in text:
        if character is a letter:
            convert character to alphabet position
            apply shift
            wrap using modulo 26
            convert back to letter
        else:
            keep character unchanged

        append character to result

    return result

The same basic algorithm can be used for both encryption and decryption by changing the direction of the shift.

Example With a Sentence

Let's encrypt ATTACK AT DAWN using a shift of 3.

The characters transform as follows:

A → D
T → W
T → W
A → D
C → F
K → N

The complete result is: DWWDFN DW GDZQ

Notice that spaces remain unchanged. This is a common design choice when implementing simple Caesar Cipher tools.

What About Uppercase and Lowercase?

A good implementation should decide how to handle both uppercase and lowercase characters.

For example, Hello World could become Khoor Zruog while preserving capitalization. Characters that aren't part of the alphabet—such as spaces, numbers, and punctuation—can generally be left unchanged.

What Is ROT13?

ROT13 is a special version of the Caesar Cipher that uses a shift of 13.

For example, HELLO becomes URYYB.

Applying ROT13 again produces the original text:

$$\text{URYYB} \xrightarrow{\text{ROT13}} \text{HELLO}$$

This works because $13 + 13 = 26$.

ROT13 has been used for lightweight text obfuscation and puzzles, but it should not be considered secure encryption.

Can Caesar Cipher Be Cracked?

Yes—and that's one of the most important things to understand about it.

1. Brute-Force Attack

The standard Caesar Cipher has a very small number of possible shifts (only 25 non-trivial shifts). An attacker can simply try:

Shift 1
Shift 2
Shift 3
...
Shift 25

And inspect the results. For a computer, trying all possible Caesar shifts is trivial.

2. Frequency Analysis

Natural languages have predictable character frequencies. Some letters (like E, T, and A) occur much more frequently than others in English.

Because the Caesar Cipher only shifts letters rather than changing their frequency relationships, those patterns remain visible. This makes the cipher particularly weak against statistical analysis.

Why Caesar Cipher Is Not Secure

The Caesar Cipher was useful historically, but it doesn't provide the security properties required by modern applications.

Major weaknesses include:

Very small key space
Vulnerable to easy brute-force attacks
Vulnerable to frequency analysis
Predictable substitution patterns
No protection against modern cryptanalysis

⚠️ Warning: Never use a Caesar Cipher to protect passwords, API keys, financial information, authentication tokens, or confidential business data.

Modern applications should use established cryptographic algorithms (such as AES) and trusted implementations rather than classical ciphers.

Caesar Cipher vs. Modern Encryption

Feature	Caesar Cipher	Modern Cryptography
Type	Classical substitution	Modern cryptographic algorithms
Key space	Very small (25 keys)	Extremely large ($2^{128}$ or higher)
Brute-force resistance	Very low	Designed to be computationally infeasible
Frequency analysis	Highly vulnerable	Highly resistant
Modern security	No	Yes (when properly implemented)
Best use	Education and puzzles	Real-world data security

Try a Caesar Cipher Online

If you want to experiment with different shift values, an online tool can make the process easier than manually shifting every character.

You can use the BlazeSolutions Caesar Cipher Tool to experiment with encoding and decoding directly in your browser.

It can be useful for:

Testing different shift values
Practicing encryption and decryption
Checking code examples
Learning classical cryptography
Experimenting with cipher logic

Building a Caesar Cipher Yourself

If you're a developer learning a programming language, implementing a Caesar Cipher is an ideal beginner project to practice:

String manipulation
Character encoding (ASCII/Unicode)
Loops and conditionals
Modular arithmetic
Functions and input validation

You can implement this algorithm in almost any language, including:

JavaScript / TypeScript
Python
C# / Java
Go / PHP / C++

Common Mistakes When Implementing Caesar Cipher

Forgetting Wraparound: Failing to handle $Z \to A$ correctly.
Handling Negative Shifts Incorrectly: Decryption requires moving backward, so negative modulo behavior needs careful handling depending on the programming language.
Modifying Non-Alphabet Characters: Accidentally shifting spaces, numbers, or punctuation.
Losing Letter Case: Failing to preserve uppercase and lowercase distinctions.
Assuming It Provides Security: Treating a Caesar Cipher as usable modern encryption.

Frequently Asked Questions

What is a Caesar Cipher?

A Caesar Cipher is a classical substitution cipher that shifts each letter by a fixed number of positions in the alphabet.

What is a Caesar Cipher key?

The key is the integer value representing how many positions each character is shifted.

What is the most common Caesar Cipher shift?

A shift of 3 is traditionally associated with Julius Caesar's original use.

How do you decrypt a Caesar Cipher?

Move every encrypted character backward by the same shift value used during encryption.

Is Caesar Cipher secure?

No. It is extremely easy to brute-force and should never be used for sensitive information.

Is ROT13 the same as a Caesar Cipher?

Yes, ROT13 is a specific Caesar Cipher implementation that uses a fixed shift of 13.

Can a Caesar Cipher encrypt numbers?

A standard Caesar Cipher operates only on alphabetic characters. Extending it to numbers requires defining a custom character set or separate mapping rules.

What is the Caesar Cipher used for today?

It is primarily used for education, programming exercises, puzzles, and learning fundamental cryptography concepts.

Final Takeaway

The Caesar Cipher is simple, old, and insecure—but that simplicity is exactly what makes it valuable for learning.

By implementing or experimenting with a Caesar Cipher, you can master foundational concepts that apply throughout computer science:

Substitution techniques
Encryption and decryption mechanisms
Modular arithmetic applications
Cryptanalysis principles (brute-force and frequency analysis)

For educational experiments, the Caesar Cipher is a great starting point. For protecting real-world data, always rely on modern, well-tested cryptographic algorithms.

PDF to JPG Conversion: Best Practices, DPI Settings & Format Selection

Asikul Islam — Fri, 11 Sep 2026 10:18:09 +0000

As developers, content managers, and designers, we frequently encounter workflows where multi-page PDFs need to be converted into image formats. Whether you are generating website thumbnail previews, preparing assets for presentation slides, or building document pipeline tools, understanding the technical nuances of PDF-to-image rendering is key.

In this quick guide, we’ll break down the core mechanics of PDF to JPG conversion, compare JPG vs. PNG, and look at optimal DPI settings.

1. Vector vs. Raster: What Happens During Conversion?

PDF (Vector & Structured): Contains scalable vector shapes, selectable text, layers, and interactive elements (like clickable links). It stays sharp at any zoom level.
JPG (Raster Grid): Converts every visual element into a static grid of pixels. Once converted, text is flattened and interactive features are lost.

If you want to quickly test how vector pages render into raster images on the web, you can try this online PDF to JPG converter.

2. Choosing the Right DPI (Resolution)

DPI (Dots Per Inch) determines pixel density, directly impacting output clarity and file size:

DPI	Best For	Trade-offs
72 DPI	Web thumbnails, fast loading previews	Low resolution; text blurriness when zoomed
150 DPI	Slide decks, general digital docs, email assets	Sweet spot: Great balance of quality & size
300 DPI	High-res printing, fine technical schematics	High visual clarity; significantly larger file size

3. PDF to JPG vs. PDF to PNG

Choosing the right format depends heavily on your document’s content:

Use JPG when:
- File size optimization is critical for performance.
- The PDF contains photographs, heavy gradients, or complex imagery.
Use PNG when:
- The PDF contains fine text, technical blueprints, or wireframes.
- You require lossless pixel accuracy or alpha-channel transparency.

4. Key Developer & Technical Takeaways

Flat Assets: Interactivity (hyperlinks, form fields) does not survive rasterization into JPG/PNG.
Scanned PDFs & OCR: Converting a scanned PDF to JPG outputs the raw visual scan. It does not extract text. If you need searchable text, run an OCR tool before or after image generation.
Avoid Re-compression: JPG uses lossy compression. Always keep the original PDF as the master source file to avoid degradation from repeated edits.

What's Your Preferred Workflow?

How do you handle PDF rendering or image conversions in your projects? Do you rely on CLI utilities (like ImageMagick / pdf2image), API services, or web tools? Let’s discuss in the comments below!

How to Use AI for SEO: A Practical Guide

Asikul Islam — Thu, 10 Sep 2026 06:48:04 +0000

Artificial Intelligence is changing the way SEO teams research, create, optimize, and measure content.

AI can process large amounts of search and website data quickly, but it works best as an SEO productivity and analysis tool—not a replacement for human expertise.

Want the complete guide?
Read the full article at the Canonical URL mentioned below.

What Is AI in SEO?

AI in SEO means using technologies such as machine learning and natural language processing to:

Analyze search data
Understand search intent
Discover keyword opportunities
Optimize website content
Identify technical SEO issues
Analyze competitors
Monitor SEO performance
Automate repetitive SEO tasks

How AI Can Be Used for SEO

1. Keyword Research

AI can analyze:

Search queries
Related keywords
Long-tail opportunities
Search volume and competition
Semantic relationships

Instead of targeting only broad keywords like "SEO", AI can help identify more specific queries such as "how to improve SEO for a small business website."

2. Search Intent Analysis

AI can help classify search intent into:

Informational
Navigational
Commercial
Transactional

Understanding intent helps you create content that matches what users actually want.

3. Content Creation

AI can assist with:

Topic ideas
Content outlines
First drafts
FAQs
Content expansion
Meta descriptions

However, publishing large amounts of generic AI-generated content is not a reliable SEO strategy.

Human expertise, originality, accuracy, and useful insights still matter.

4. Content Optimization

AI can analyze existing pages and identify:

Content gaps
Weak sections
Readability problems
Missing topics
Internal linking opportunities
Metadata issues

This makes AI particularly useful for content refresh and optimization.

5. Competitor Analysis

AI can help analyze competitors for:

Keyword coverage
Content topics
Page structure
Backlink patterns
Search visibility
Content gaps

The goal should not be to copy competitors, but to discover opportunities to provide something better and more useful.

6. Technical SEO Audits

AI-powered tools can help identify:

Broken links
Duplicate content
Canonical issues
Indexing problems
Missing metadata
Structured-data issues
Performance problems

AI can also help prioritize issues based on their potential SEO impact.

7. Internal Linking

AI can analyze your website content and recommend relevant internal links between related pages.

For large websites, this can save significant manual effort.

8. Automated SEO Audits

AI can combine technical, content, and performance data to create prioritized recommendations.

For example:

🔴 Critical: Important pages not indexed
🟠 High: Missing title tags
🟡 Medium: Internal linking opportunities
🟢 Low: Minor content improvements

9. SEO Analytics

AI can help investigate changes in:

Impressions
Click-through rate
Rankings
Organic traffic
Conversions

Instead of simply showing a traffic drop, AI can help identify possible reasons behind the change.

10. SEO Forecasting

Historical SEO data can be used to estimate potential outcomes.

For example, AI can help compare whether a strategy should focus on:

Publishing new content
Updating existing pages
Improving internal links
Fixing technical issues

Forecasts are estimates, not guarantees.

11. Voice & Conversational Search

Search is becoming more conversational.

AI-assisted SEO can help optimize for:

Natural-language questions
FAQ content
Direct answers
Conversational phrases
Question-based searches

12. AI Search Experiences

Search is evolving beyond traditional blue links.

AI-generated search experiences can summarize information from multiple sources.

This means SEO increasingly involves making your:

Brand
Content
Expertise
Research
Evidence

visible and trustworthy across both traditional search and AI-driven search experiences.

Benefits of AI in SEO

AI can provide several advantages:

⚡ Faster research — Analyze large amounts of data quickly
🚀 Higher productivity — Automate repetitive SEO tasks
📈 Better scalability — Manage larger websites efficiently
🎯 Improved targeting — Match content with search intent
🔍 Faster diagnostics — Identify SEO problems sooner
💰 Lower operational effort — Reduce repetitive manual work

Limitations of AI in SEO

AI also has important limitations:

AI can generate inaccurate information
AI-generated content can become repetitive or generic
Search algorithms continue to evolve
Human expertise is still essential
AI cannot automatically guarantee rankings
Original research and real-world experience remain valuable

The Future of AI + SEO

The future of SEO will likely combine AI automation with human strategic thinking.

AI can handle:

Research
Data analysis
Monitoring
Reporting
Repetitive optimization tasks

Humans should continue to focus on:

Strategy
Creativity
Brand positioning
Original research
Expertise
Quality control

Final Thoughts

AI is not replacing SEO.

It is changing how SEO is performed.

The strongest approach is not to use AI simply to publish more content. Instead, use AI to:

Work faster
Analyze more data
Discover opportunities
Automate repetitive tasks
Improve existing content
Make better SEO decisions

The winning combination is AI efficiency + human expertise + high-quality content + technical SEO.

Read the Full Guide

This DEV.to article provides a shorter overview of the topic.

For the complete guide with detailed explanations and practical examples, please visit the Canonical URL of this article.

Understanding XML Sitemaps: A Developer's Guide to Better URL Discovery

Asikul Islam — Wed, 09 Sep 2026 14:04:34 +0000

When building a website, developers usually focus on application logic, performance, security, and user experience. But there is another important technical layer that developers should not overlook: how search engines discover the URLs on a website.

This is where an XML sitemap becomes useful.

An XML sitemap provides search engines with a structured list of URLs that you want them to discover and consider. It does not guarantee that every URL will be indexed, and it is not a direct ranking factor, but it can make URL discovery more efficient.

For developers working with Next.js, React, Laravel, ASP.NET Core, WordPress, eCommerce platforms, or custom applications, understanding how sitemaps work can prevent a number of technical SEO problems.

What Is an XML Sitemap?

An XML sitemap is an XML file that contains URLs from a website. A basic sitemap looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="[http://www.sitemaps.org/schemas/sitemap/0.9](http://www.sitemaps.org/schemas/sitemap/0.9)">
    <url>
        <loc>[https://example.com/](https://example.com/)</loc>
        <lastmod>2026-09-01</lastmod>
    </url>
    <url>
        <loc>[https://example.com/about](https://example.com/about)</loc>
        <lastmod>2026-08-25</lastmod>
    </url>
    <url>
        <loc>[https://example.com/blog](https://example.com/blog)</loc>
        <lastmod>2026-08-30</lastmod>
    </url>
</urlset>

The most important element is , which contains the URL.

The element can indicate when the page was last meaningfully modified.

A sitemap is not a database of every URL your application can generate. Ideally, it represents the important URLs that you want search engines to discover and potentially index.

Why Do Search Engines Need Sitemaps?

Search engines can discover URLs through many different mechanisms, especially links.

For example:

Homepage
↓
Category
↓
Product

If your website has a good internal linking structure, crawlers can often discover a large portion of the site naturally.

But real-world websites are not always that simple.

Consider an application with:

50,000 product pages
10,000 blog posts
Multiple category levels
Dynamically generated URLs
Frequently changing inventory
Pages that aren't heavily linked internally

A sitemap provides another structured source of URL information.

This is especially useful for large, new, frequently updated, or complex websites.

A Sitemap Does Not Guarantee Indexing

This is an important distinction.

Putting this URL into your sitemap:

https://example.com/page

does not mean:

Google must index this page.

A sitemap is primarily a discovery signal.

Search engines still need to evaluate the URL and determine whether it should be crawled, indexed, and shown in search results.

So developers should avoid thinking of a sitemap as an indexing command.

A better mental model is:

Sitemap
↓
URL Discovery
↓
Crawling
↓
Processing
↓
Indexing decision
↓
Search results

Each stage has its own requirements.

What Should Go Into a Sitemap?

A good sitemap should contain URLs that are:

Important
Canonical
Accessible
Intended for indexing
Returning an appropriate HTTP response

For example:

https://example.com/
https://example.com/products
https://example.com/products/laptop
https://example.com/blog/technical-seo

The exact URLs depend on the architecture of your application.

What Should Not Go Into a Sitemap?

One of the most common problems I've seen with automatically generated sitemaps is that they contain URLs that shouldn't be there.

Examples include:

404 URLs
Redirect URLs
Duplicate URLs
Non-canonical URLs
Noindex pages
Temporary URLs
Internal search URLs
Tracking parameter URLs

For example:

https://example.com/product?id=123
https://example.com/product?id=123&utm_source=google
https://example.com/product?id=123&sort=price

If these are simply different variations of the same canonical page, you probably don't want all of them in your sitemap.

The Developer's Problem: Automatically Generated Sitemaps

This is where things get interesting.

Many modern applications generate sitemaps automatically.

For example, you might have:

const products = await db.product.findMany({
where: {
published: true
}
});

Then generate:

/products/product-1
/products/product-2
/products/product-3

This seems straightforward.

But what happens when:

A product is deleted?
A product becomes unpublished?
The slug changes?
The database contains an invalid record?
The URL starts returning 404?
The page becomes noindex?
A redirect is introduced?

Your sitemap generator can continue producing URLs that shouldn't be there.

That's why sitemap generation and sitemap validation should be treated as two separate concerns.

Sitemap Generation vs Sitemap Validation

Think about it this way:

Database
↓
Sitemap Generator
↓
sitemap.xml
↓
Sitemap Validator / Crawler
↓
Problems detected

Generating the sitemap answers:

"Which URLs should I put into the sitemap?"

Validation answers:

"Are those URLs actually healthy?"

Both are important.

How to Check an XML Sitemap

For a small site, you can open the sitemap in a browser and inspect it manually.

For example:

https://example.com/sitemap.xml

But this approach doesn't scale.

Imagine manually checking 10,000 URLs.

That's where a sitemap crawler becomes useful.

A sitemap crawler can take the URLs listed in your sitemap and inspect their responses.

For example, you may discover:

200 OK
200 OK
301 Redirect
200 OK
404 Not Found
200 OK

That immediately gives you information about the health of the URLs listed in your sitemap.

If you're looking for a simple way to inspect sitemap URLs, the Blaze Solutions Sitemap Crawler can be used to crawl a sitemap:

https://blazesolutions.info/tools/sitemap-crawler

The important point isn't the specific tool—it's the workflow:

Generate → Crawl → Identify problems → Fix → Recheck

HTTP Status Codes Matter

When validating sitemap URLs, HTTP status codes are particularly useful.

200 OK

Usually indicates that the URL successfully returns a page.

GET /blog/example
200 OK

This is generally what you want for a normal indexable page.

301 / 308 Redirect

The URL redirects somewhere else.

GET /old-page
301 → /new-page

If the final URL is the canonical page, consider putting the final URL in the sitemap instead.

404 Not Found

The resource doesn't exist.

GET /deleted-page
404 Not Found

A deleted page generally shouldn't remain in your sitemap.

5xx Errors

Server-side errors can indicate application or infrastructure problems.

500 Internal Server Error
503 Service Unavailable

These deserve investigation, especially if they affect important URLs.

XML Sitemap and Canonical URLs

Sitemaps and canonical URLs should generally agree with each other.

Suppose you have:

https://example.com/article
https://example.com/article?ref=twitter

If the canonical version is:

https://example.com/article

your sitemap should normally contain the canonical URL.

You want your technical SEO signals to be consistent:

Sitemap
↓
Canonical URL

Internal Link
↓
Canonical URL

Canonical Tag
↓
Canonical URL

Consistency reduces unnecessary ambiguity.

XML Sitemap and robots.txt

robots.txt and sitemap.xml solve different problems.

A sitemap tells crawlers about URLs you want them to discover.

A robots.txt file provides crawler instructions.

For example:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

The sitemap can be referenced directly from robots.txt.

One important point for developers:

Do not assume that adding a URL to a sitemap overrides a robots.txt restriction.

They are separate mechanisms with different purposes.

Dynamic Websites Need Extra Attention

Static websites are relatively simple.

Dynamic applications are different.

Consider an eCommerce application where product pages are generated from a database.

Today:

/products/phone-x

returns:

200 OK

Tomorrow, the product is deleted.

If your sitemap generator doesn't account for that change, the sitemap may still contain:

/products/phone-x

which now returns:

404 Not Found

This is why dynamic sitemap generation should use the same business rules that determine whether a page is publicly available.

For example:

const products = await db.product.findMany({
where: {
status: "published",
isDeleted: false
}
});

The exact implementation depends on your application, but the principle is simple:

Don't generate sitemap URLs from data that shouldn't produce public pages.

Sitemap Indexes for Large Websites

Large websites don't necessarily need one massive sitemap file.

You can split your sitemap into multiple files.

For example:

sitemap.xml
sitemap-products.xml
sitemap-blog.xml
sitemap-categories.xml

A sitemap index can reference the individual files.

This approach makes large websites easier to manage.

For example:

https://example.com/sitemap-products.xml

https://example.com/sitemap-blog.xml

For large applications, separating sitemap generation by content type can also simplify debugging.

A Practical Sitemap Monitoring Workflow

If I were maintaining a large web application, I wouldn't treat the sitemap as a one-time SEO task.

I'd make it part of the technical monitoring process.

A simple workflow could look like this:

Generate sitemap ↓
Validate XML ↓
Crawl sitemap URLs ↓
Check HTTP status ↓
Check canonical consistency ↓
Remove invalid URLs ↓
Deploy ↓
Monitor Search Console

This is especially useful after:

Website migrations
CMS migrations
URL structure changes
Large database updates
Product imports
Removing thousands of pages
Changing routing logic
Common Sitemap Mistakes Developers Should Avoid
Mistake 1: Generating Every Database URL

Not every database record should become a public URL.

Filter your data before generating the sitemap.

Mistake 2: Ignoring HTTP Status Codes

A URL existing in your database doesn't mean the HTTP endpoint works.

Always consider the actual response.

Mistake 3: Including Redirects

If a URL redirects permanently to another URL, the final URL is usually a better sitemap candidate.

Mistake 4: Ignoring Canonicalization

Make sure sitemap URLs generally match your canonical URLs.

Mistake 5: Updating lastmod on Every Build

If your application runs a deployment every night, that doesn't mean every page was modified.

Only update lastmod when meaningful content changes.

Mistake 6: Treating Sitemap Submission as an SEO Strategy

Submitting a sitemap is useful, but it isn't a replacement for:

Good content
Internal linking
Technical SEO
Performance optimization
Search intent
Backlinks
A good website architecture
A Simple Technical SEO Checklist

Before considering your sitemap production-ready, check:

[ ] XML is valid
[ ] URLs are absolute
[ ] HTTPS is used where appropriate
[ ] URLs return expected responses
[ ] No unnecessary redirects
[ ] No 404 URLs
[ ] Canonical URLs are used
[ ] Noindex pages are excluded
[ ] Important pages are included
[ ] lastmod values are accurate
[ ] robots.txt references the sitemap
[ ] Sitemap is submitted to Search Console

For larger sites, automate as many of these checks as possible.

Final Thoughts

An XML sitemap is a small part of a web application, but it sits at an interesting intersection between development and technical SEO.

For developers, the key lesson is that a sitemap isn't just an XML file.

It's a representation of your application's public URL architecture.

If the application changes, the sitemap needs to change with it.

If products are deleted, URLs should disappear.

If routes change, the sitemap should reflect the new routes.

If pages become non-indexable, they shouldn't continue appearing as important sitemap URLs.

A healthy sitemap is therefore the result of both good application architecture and good technical SEO practices.

And if your sitemap contains hundreds or thousands of URLs, don't rely entirely on manual inspection. Crawl it, check the responses, identify anomalies, and fix the underlying application or content issues.

That's a much more reliable approach than simply generating sitemap.xml and forgetting about it..

Hello DEV Community 👋 — My Journey from Software Development to AI Automation

Asikul Islam — Mon, 07 Sep 2026 10:03:39 +0000

Hello DEV Community 👋 — My Journey from Software Development to AI Automation

This is my first post on DEV.to, so I thought I would start with a simple introduction.

I'm a software developer and AI automation engineer who has spent more than a decade building software, solving business problems, and exploring new technologies.

Over the years, I've worked with different technologies and frameworks, but one thing has remained constant: I'm always curious about how technology can make people's work easier.

From Software Development to AI

My professional journey started primarily around Microsoft technologies, including .NET, ASP.NET, ASP.NET MVC, ASP.NET Core, and Microsoft SQL Server.

Like many developers, I spent years building applications, designing databases, debugging production issues, optimizing systems, and working with teams to turn business requirements into working software.

But the technology landscape has changed dramatically.

AI is no longer something that exists only in research papers or experimental projects. It is becoming part of the way we build software, automate workflows, communicate with customers, analyze information, and run businesses.

That shift caught my attention.

I started exploring how AI could be combined with traditional software engineering and workflow automation to build systems that don't just respond to users, but can actually perform useful tasks.

That led me deeper into AI automation.

Why AI Automation?

I believe the most interesting opportunity isn't simply putting an AI chatbot on a website.

The real opportunity is connecting AI with business processes.

For example:

Finding and qualifying leads
Researching prospects
Generating personalized outreach
Processing documents
Automating repetitive administrative tasks
Monitoring competitors
Creating and optimizing content
Connecting different business applications
Building AI-powered internal assistants
Automating multi-step workflows

When these pieces are connected properly, AI becomes more than a chat interface.

It becomes part of the workflow.

And that's what I'm interested in building.

What I'm Working On

Currently, I'm exploring and building AI-powered automation systems using technologies such as:

AI APIs and LLMs
n8n workflow automation
Next.js
React
ASP.NET Core
SQL Server and MySQL
REST APIs
Cloud services
Webhooks
Business process automation

One area I'm particularly interested in is the concept of AI Employees.

Instead of thinking of AI only as a tool that answers questions, I like thinking about AI agents as digital workers that can be given a specific responsibility.

For example:

An AI Sales Employee that can research prospects, qualify leads, prepare personalized messages, and keep the sales pipeline updated.

There are still many challenges to solve—reliability, context management, security, human oversight, cost, and integration with existing systems.

That's what makes it interesting.

Why I'm Joining DEV.to

I've been working in software development for many years, but this is my first post on DEV.to.

I joined because I want to become more involved in the developer community and share what I learn along the way.

I don't want this profile to be just a place where I publish polished tutorials.

I also want to share the real development process:

Things that worked
Things that didn't work
Architecture decisions
Debugging stories
AI experiments
Automation workflows
Development lessons
Useful tools
Real-world problems and solutions

Sometimes the most valuable lesson comes from a problem that took several hours to solve.

So I'll try to document those experiences too.

What You'll Find Here

Going forward, I plan to write about topics such as:

AI & Automation

Practical AI automation, AI agents, LLM integrations, and business workflows.

Software Development

Architecture, APIs, backend development, databases, performance, and full-stack development.

n8n & Workflow Automation

Real-world automation workflows and integrations.

AI-Powered Products

Lessons from building and experimenting with AI-based products.

SEO & Developer Tools

I'm also interested in building useful tools that solve everyday problems for developers, marketers, and website owners.

A Developer's Journey Never Really Ends

One thing I've learned after spending many years in software development is that you never really "finish" learning.

The technologies change.

The frameworks change.

The tools change.

The problems change.

And now AI is changing the way we think about software itself.

That's exciting.

I'm looking forward to learning, building, experimenting, and sharing the journey here.

If you're also working with AI, automation, software development, or developer tools, I'd love to connect and learn from your experience.

This is just the beginning.

Hello, DEV Community! 👋

I'm glad to finally be here.

introduction #ai #automation #webdev #programming