Serving sitemap.xml and robots.txt with ASP.Net core MVC

Esty (c) — Thu, 14 May 2020 08:36:41 +0000

Robots.txt is a text file which is used to tell Search Engines which URLs are allowed or disallowed for them to index. The file must be located at https://www.example.com/robots.txt.

Some example of robots.txt:

Blocking all content from all Search Engines:
User-agent: *
Disallow: /

Allow all content for all Search Engines:
User-agent: *
Allow: /

Sitemap.xml is an XML file that contains all URLs of a website that you want Search Engines to index. Sitemap XML file’s helps Search Engines find and index site quickly. The sitemap.xml file default location is: https://www.example.com/sitemap.xml
Example of sitemap file:

<?xml version="1.0" encoding="utf-8"?>    
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">    
  <url>    
    <loc>https://www.codingwithesty.com/</loc>    
    <lastmod>2020-05-07T13:32:30+06:00</lastmod>    
    <changefreq>daily</changefreq>    
    <priority>1.0</priority>    
  </url>    
  <url>    
    <loc>https://www.codingwithesty.com/search-engine-optimization-library-for-dot-net-code-developers</loc>    
    <lastmod>2020-05-07T13:32:30+06:00</lastmod>    
    <changefreq>daily</changefreq>    
    <priority>0.8</priority>    
  </url>    
</urlset>

In this article we will learn how to create simtemal.xml file and routing simtemal.xml in asp.net core. We also create routing for robots.txt file.

Package Installation

Create a new Asp.Net core project. Go to Nuget Package Manger install AspNetCore.SEOHelper package

Create sitemap.xml file:

AspNetCore.SEOHelper provides SitemapNode class for each URL and CreateSitemapXML
() helps to create sitemap.xml file. Copy to following code in your controller:

public class HomeController : Controller
    {
        private readonly ILogger<HomeController> _logger;
        private readonly IWebHostEnvironment _env;

        public HomeController(ILogger<HomeController> logger, IWebHostEnvironment env)
        {
            _logger = logger;
            _env = env;
        }


        public string Index()
        {
            var list = new List<SitemapNode>();

            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.8, Url = "https://codingwithesty.com/serilog-mongodb-in-asp-net-core", Frequency = SitemapFrequency.Daily });
            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.8, Url = "https://codingwithesty.com/logging-in-asp-net-core", Frequency = SitemapFrequency.Yearly });
            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.7, Url = "https://codingwithesty.com/robots-txt-in-asp-net-core", Frequency = SitemapFrequency.Monthly });
            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.5, Url = "https://codingwithesty.com/versioning-asp.net-core-apiIs-with-swagger", Frequency = SitemapFrequency.Weekly });
            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.4, Url = "https://codingwithesty.com/configuring-swagger-asp-net-core-web-api", Frequency = SitemapFrequency.Never });

            new SitemapDocument().CreateSitemapXML(list, _env.ContentRootPath);
            return View();
        }

    }

CreateSitemapXML Method takes two parameters, the first parameter is SitemapNode list and the second parameter is the directory path where the sitemap.xml file will be stored. In this case we provide root directory of our project.

Routing sitemap.xml

We have already created sitemap.xml file in our asp.net core project. Now we want to route this sitemap file like: https://example.com/sitemap.xml

Add app.UseXMLSitemap middleware to the Configure method

   public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            app.UseStaticFiles();

            app.UseXMLSitemap(env.ContentRootPath);


           app.UseRouting();
            app.UseEndpoints(endpoints => { endpoints.MapControllerRoute( name: "default", pattern: "{controller=Home}/{action=Index}/{id?}"); });
        }

Now run your application and navigate to https://hostname:port/sitemap.xml

Routing robots.txt

Routing robots.txt in similar to previous one. Just add this app.UseRobotsTxt middleware to your configure method.

  public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            app.UseStaticFiles();
            app.UseRobotsTxt(env.ContentRootPath);

            app.UseRouting();
            app.UseEndpoints(endpoints => { endpoints.MapControllerRoute( name: "default", pattern: "{controller=Home}/{action=Index}/{id?}"); });
        }

UseRobotsTxt takes a directory path parameter. If there is no robots.txt file in the directory, it will show default text disallow all

Now run your application and navigate to https://hostname:port/robots.txt

Thanks a lot for reading. Write in the comment box in case you have any questions.

Reference