<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Maria </title>
    <description>The latest articles on DEV Community by Maria  (@marianadeem08).</description>
    <link>https://dev.to/marianadeem08</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4255922%2F05f679de-d8ef-47ed-9cf7-35105de1eb6f.png</url>
      <title>DEV Community: Maria </title>
      <link>https://dev.to/marianadeem08</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/marianadeem08"/>
    <language>en</language>
    <item>
      <title>How to Make a Small Website Crawlable: Sitemap, robots.txt and Internal Links</title>
      <dc:creator>Maria </dc:creator>
      <pubDate>Fri, 20 Nov 2026 15:18:54 +0000</pubDate>
      <link>https://dev.to/marianadeem08/how-to-make-a-small-website-crawlable-sitemap-robotstxt-and-internal-links-52fl</link>
      <guid>https://dev.to/marianadeem08/how-to-make-a-small-website-crawlable-sitemap-robotstxt-and-internal-links-52fl</guid>
      <description>&lt;p&gt;&lt;em&gt;A practical checklist for developers who built a site and now wonder why Google is not showing it.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In this tutorial, we’ll be seeing how the XML sitemap, robots.txt, and your internal linking structure help decide that for your website. Weeks pass, and when you are trying to find your site through search, it does not show up. Do not rush to re-write your content. First, do a simpler check – can search engines find and access your pages at all?&lt;/p&gt;

&lt;p&gt;In this tutorial, we will see how your XML sitemap, robots.txt, and the internal linking of your small website decide this for you. There is no need for any plugin or premium tool; just an easy-to-use text editor is all that is needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Crawling, indexing and ranking are different steps
&lt;/h2&gt;

&lt;p&gt;There is a sequence of steps involved in the process of working of search engines. The first step is that the search engine finds the URL; the second step involves crawling the website. It then decides whether to include the URL in its indexes or not, and finally it ranks the site. Crawlability problems will hinder this process either at step one or step two, and hence no matter how good the content may be, it won’t serve any purpose for the site.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. The XML sitemap: a list of pages you want found
&lt;/h2&gt;

&lt;p&gt;A sitemap is a file, usually at &lt;code&gt;/sitemap.xml&lt;/code&gt;, that lists the URLs you want search engines to know about. It is a hint, not a command, but for new or small sites it speeds up discovery a lot. Google explains how it works in its &lt;a href="https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview" rel="noopener noreferrer"&gt;sitemap overview&lt;/a&gt;, and the format itself is defined at &lt;a href="https://www.sitemaps.org/protocol.html" rel="noopener noreferrer"&gt;sitemaps.org&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;A minimal example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight xml"&gt;&lt;code&gt;&lt;span class="cp"&gt;&amp;lt;?xml version="1.0" encoding="UTF-8"?&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;urlset&lt;/span&gt; &lt;span class="na"&gt;xmlns&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"http://www.sitemaps.org/schemas/sitemap/0.9"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;url&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;loc&amp;gt;&lt;/span&gt;https://example.com/&lt;span class="nt"&gt;&amp;lt;/loc&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;lastmod&amp;gt;&lt;/span&gt;2026-11-20&lt;span class="nt"&gt;&amp;lt;/lastmod&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;/url&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;url&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;loc&amp;gt;&lt;/span&gt;https://example.com/mens-jackets/&lt;span class="nt"&gt;&amp;lt;/loc&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;lastmod&amp;gt;&lt;/span&gt;2026-11-18&lt;span class="nt"&gt;&amp;lt;/lastmod&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;/url&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/urlset&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Rules that make your life easier:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Only list pages you want indexed: 200 status codes with no redirection or blocking.&lt;/li&gt;
&lt;li&gt;Don’t forget your protocol and domain: &lt;a href="https://example.com/page" rel="noopener noreferrer"&gt;https://example.com/page&lt;/a&gt; and not just &lt;code&gt;/page&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Be honest about &lt;code&gt;lastmod&lt;/code&gt; – update it only when it really changes.&lt;/li&gt;
&lt;li&gt;A single sitemap file can contain up to 50,000 URLs or 50MB of uncompressed data; small sites will never reach such numbers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Submit it using Search Console, as well as refer to it in your robots.txt (shown below).&lt;/p&gt;

&lt;h2&gt;
  
  
  2. robots.txt: tell crawlers where not to waste time
&lt;/h2&gt;

&lt;p&gt;The robots.txt file lives at the root of your domain (&lt;a href="https://example.com/robots.txt" rel="noopener noreferrer"&gt;https://example.com/robots.txt&lt;/a&gt;). It tells crawlers which paths they should not request. Google's &lt;a href="https://developers.google.com/search/docs/crawling-indexing/robots/intro" rel="noopener noreferrer"&gt;robots.txt introduction&lt;/a&gt; is a good reference.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search

Sitemap: https://example.com/sitemap.xml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Three common mistakes:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Robots.txt is not for privacy protection. The excluded URL may be included in search results due to links from other pages pointing to it. For exclusion from indexation, apply noindex meta tag or password protection, but allow crawlers to access the page to recognize the tag.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Do not block CSS and JavaScript necessary for rendering the page. They are used by crawlers to analyze your page structure.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Look for &lt;code&gt;Disallow: /&lt;/code&gt; directive. Quite often everything is blocked on the staging website, and the developers forget about it when launching. One single line can exclude your website.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  3. Internal links: the roads between your pages
&lt;/h2&gt;

&lt;p&gt;Crawlers move through your site by following links. If a page has no link pointing to it, it is an &lt;strong&gt;orphan page&lt;/strong&gt;, and it may only be found through the sitemap, if at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The secret of good internal linking lies in your habits:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Each significant page must be no further than a couple of clicks away from the homepage.&lt;/li&gt;
&lt;li&gt;Don't create non-clickable links with a JavaScript function – Google explicitly recommends that links created in this way may go unnoticed.&lt;/li&gt;
&lt;li&gt;Anchor text should be meaningful. "Men's shearling jackets" provides much more information than just "click here".&lt;/li&gt;
&lt;li&gt;Connect similar pages, e.g. a category page to its products, a product page to its category, a guide to the relevant pages.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="c"&gt;&amp;lt;!-- Crawlable --&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;a&lt;/span&gt; &lt;span class="na"&gt;href&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"/mens-jackets/"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Men's leather jackets&lt;span class="nt"&gt;&amp;lt;/a&amp;gt;&lt;/span&gt;

&lt;span class="c"&gt;&amp;lt;!-- Easy to miss --&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;span&lt;/span&gt; &lt;span class="na"&gt;onclick&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"location.href='/mens-jackets/'"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Men's leather jackets&lt;span class="nt"&gt;&amp;lt;/span&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  A real-world example: a small online store
&lt;/h2&gt;

&lt;p&gt;Small online stores show all three of these problems in one place. Take a leather jacket shop like &lt;a href="https://celebsjacket.com" rel="noopener noreferrer"&gt;celebsjacket.com&lt;/a&gt;, a store I work on from the SEO side. A typical structure looks like this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;HOME PAGE&lt;/li&gt;
&lt;li&gt;Pages of categories (men’s jackets, women’s jackets, shearling, bombers)&lt;/li&gt;
&lt;li&gt;Page for each individual jacket&lt;/li&gt;
&lt;li&gt;Utility pages, such as cart, checkout and search&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;The application of three tools looks as follows:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sitemap:&lt;/strong&gt; include the homepage, category and product pages. Do not include cart, checkout and internal search results.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;robots.txt:&lt;/strong&gt; disallow the paths of cart, checkout and search so that crawlers concentrate on pages that can really rank.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Internal links:&lt;/strong&gt; all products must be linked from the page of corresponding category via a normal &lt;code&gt;&amp;lt;a href&amp;gt;&lt;/code&gt; link, and categories must be linked from the main navigation. Products which can be accessed only after filtering or using a JS widget are orphan pages.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This model suits practically any small store, blog and portfolio. Replace “jackets” by “articles” or “projects” and you will still have the same checklist.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to check your own site in 10 minutes
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Open &lt;a href="https://yourdomain.com/robots.txt" rel="noopener noreferrer"&gt;https://yourdomain.com/robots.txt&lt;/a&gt; in a browser and read it line by line.&lt;/li&gt;
&lt;li&gt;Open &lt;a href="https://yourdomain.com/sitemap.xml" rel="noopener noreferrer"&gt;https://yourdomain.com/sitemap.xml&lt;/a&gt; and confirm it loads and lists the right URLs. From a terminal you can also run &lt;code&gt;curl -I https://yourdomain.com/sitemap.xml&lt;/code&gt; and look for a 200 status.&lt;/li&gt;
&lt;li&gt;In &lt;a href="https://search.google.com/search-console" rel="noopener noreferrer"&gt;Google Search Console&lt;/a&gt;, submit the sitemap and run &lt;strong&gt;URL Inspection&lt;/strong&gt; on an important page to see whether it is indexed and when it was last crawled.&lt;/li&gt;
&lt;li&gt;Search &lt;code&gt;site:yourdomain.com&lt;/code&gt; in Google for a rough idea of how many pages are indexed.&lt;/li&gt;
&lt;li&gt;Click through your own navigation. If a page needs a search box or a script to reach it, add a normal link.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Common mistakes at a glance
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Redirects and blocked URLs on a sitemap&lt;/li&gt;
&lt;li&gt;Important pages without any internal links pointing to them&lt;/li&gt;
&lt;li&gt;Navigation based completely on click events in JavaScript code&lt;/li&gt;
&lt;li&gt;Blocking essential CSS and JS files&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Wrap-up
&lt;/h2&gt;

&lt;p&gt;Crawlability is the boring aspect of SEO, but it is the backbone – a page that cannot be crawled cannot be ranked. Correctly configured sitemaps, robot.txt and proper internal links usually solve issues for small websites.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>tutorial</category>
      <category>beginners</category>
    </item>
  </channel>
</rss>
