The middleware

Hi everybody,

I hope, you are feeling good today. Also, I'm good thanks. In this post, I'm gonna talk about the importance of system tracing. The main subject, I want to tell you in this post will be outside of the classical methods. Maybe you ask me what are you talking about? I'm talking about designing a custom log framework using different tech stacks. Today, I will create a new library, after that implemented a basic API. Here is the most important content is meaning of architecture

Here is the I used technology stacks

• Dotnet Core More Information
• MediatR
• MassTransit
• RabbitMq

Why do we need tracing of the system?

It does not make sense that I do say specific reasons. If we have domain size does not matter, we should trace them. I would like to specify, as the complexity of the domain increases, monitoring and interpreting the system decrease.

I have seen that in small and medium-sized systems, logs are monitored without being written to databases and even kept in text files. It might seem to like normal but here is the biggest problem are it will be hard to identify the problem. Let's give an example. Just think about a domain.

This project about customer information. I created 2 API and 3 worker service. Each worker services applications provide a different service. Like register, login, notification, etc ...
Also, each service has its own service log like warning, error, or information. Let's assume, every API and worker service write request and response logs write to disk. Any service stopped working after that I have to check the service logs. Firstly open the txt log files. After that search it with ctrl+f. This is a very bad situation. You do not have to spend a lot of effort.

What should I do?

Some companies buy software or some companies write their own solutions after that to implement. Let's give an example Nlog, Elmah, Graylog, SeriLog, DataDog.

In this post, I would like to explain the following titles.

What is Middleware?

More Information

What is MediatR?

MediatR is an open-source project and an implementation of the mediator design pattern. The mediator design pattern controls how a set of objects communicate and helps to reduce the number of dependencies among these objects that you must manage. In the mediator design pattern, objects don’t communicate with one another directly, but through a mediator. This article presents a discussion of how we can use MediatR in ASP.Net Core applications.
More Information

What is MassTransit?

More Information

Let's open Visual Studio after that let's write some code.

Project Design and Packages

I would like to create a framework that name is 'RequestTracer'. I designed 2 layers. Here is the folder structure.

RequestTracer.Core layer

Install the below nugget packages.

Let's examine the codes in the file.

• Commands: It's about the MediatR request parameter and the return type is bool.

using MediatR;
using RequestTracer.Core.Dtos;

namespace RequestTracer.Core.Commands
{
    public class LogCommand : IRequest<bool>
    {
        public LogDto LogDtos { get; set; }
    }
}

• Dtos: It's about request and response parameters. Referenced to Commands.

using System;

namespace RequestTracer.Core.Dtos
{
    public class LogDto
    {
        public Guid Id { get { return Guid.NewGuid(); } }

        public DateTime Date { get { return DateTime.UtcNow; } }

        public string ApplicationName { get; set; }

        public string Environment { get; set; }

        public LogDetail LogDetail { get; set; }
    }
}

namespace RequestTracer.Core.Dtos
{
    public class LogDetail
    {
        public string Host { get; set; }

        public string Method { get; set; }

        public string Protocol { get; set; }

        public string Schema { get; set; }

        public string Path { get; set; }

        public string QueryStringParameter { get; set; }

        public int StatusCode { get; set; }

        public string ContentType { get; set; }

        public long? ContentLength { get; set; }

        public string ExecutionTime { get; set; }
    }
}

• Handlers: It's about the MediatR. Every command has a handler. Every environment creates its own exchange.

using System;
using System.Threading;
using System.Threading.Tasks;
using MassTransit;
using MediatR;
using Microsoft.Extensions.Configuration;
using RequestTracer.Core.Commands;

namespace RequestTracer.Core.Handlers
{
    public class LogCommandHandler : IRequestHandler<LogCommand, bool>
    {
        private readonly IBus _bus;
        private readonly IConfiguration _configuration;
        private readonly ISendEndpoint _sendEndpoint;

        public LogCommandHandler(IBus bus, IConfiguration configuration)
        {
            _bus = bus;
            _configuration = configuration;

            var environment = Environment.GetEnvironmentVariables()["ASPNETCORE_ENVIRONMENT"].ToString();

            _sendEndpoint = _bus.GetSendEndpoint(
                new Uri($"{_configuration.GetSection("RabbitMqSettings:server").Value}/{configuration.GetSection("ApplicationName").Value}-{environment}-Logs")).GetAwaiter().GetResult();
        }

        public async Task<bool> Handle(LogCommand request, CancellationToken cancellationToken)
        {
            await _sendEndpoint.Send(request);

            return true;
        }
    }
}

• Middlewares: It's about request and response custom pipeline. Every single request goes through here without errors cause I only trace information logs.

using System;
using System.Diagnostics;
using System.Threading.Tasks;
using MediatR;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;

namespace RequestTracer.Core.Middlewares
{
    public class LogMiddleware
    {
        private readonly RequestDelegate _next;
        private readonly IMediator _mediator;
        private readonly IConfiguration _configuration;
        private readonly ILogger<LogMiddleware> _logger;

        public LogMiddleware(RequestDelegate next,
                             IMediator mediator,
                             IConfiguration configuration,
                             ILogger<LogMiddleware> logger)
        {
            _next = next ?? throw new ArgumentNullException(nameof(next));
            _mediator = mediator ?? throw new ArgumentNullException(nameof(next));
            _configuration = configuration ?? throw new ArgumentNullException(nameof(_configuration));
            _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        }

        public async Task Invoke(HttpContext httpContext)
        {
            try
            {
                var stopwatch = new Stopwatch();
                stopwatch.Start();
                await _next(httpContext);
                stopwatch.Stop();

                await SendRequest(httpContext, stopwatch);
            }
            catch (Exception ex)
            {
                _logger.LogError($"LogMiddleware {ex}");

                if (httpContext.Response.HasStarted)
                {
                    _logger.LogWarning("The response has already started");
                    throw;
                }
            }
        }

        private async Task SendRequest(HttpContext httpContext,Stopwatch stopwatch)
        {
            var request = httpContext.Request;
            var response = httpContext.Response;

            await _mediator.Send(new Commands.LogCommand()
            {
                LogDtos = new Dtos.LogDto()
                {
                    ApplicationName = _configuration.GetSection("ApplicationName").Value,
                    Environment = Environment.GetEnvironmentVariables()["ASPNETCORE_ENVIRONMENT"].ToString(),
                    LogDetail = new Dtos.LogDetail()
                    {
                        Host = request.Host.HasValue ? request.Host.ToString() : string.Empty,
                        Method = request.Method,
                        Protocol = request.Protocol,
                        Schema = request.Scheme,
                        Path = request.Path,
                        QueryStringParameter = request.QueryString.HasValue == true ? request.QueryString.Value : string.Empty,
                        StatusCode = response.StatusCode,
                        ContentType = response.ContentType,
                        ContentLength = request.ContentLength,
                        ExecutionTime = $"Execution time of {stopwatch.ElapsedMilliseconds} ms"
                    }
                }
            });
        }
    }
}

Another one is an extension class here is the code.

using Microsoft.AspNetCore.Builder;
using RequestTracer.Core.Middlewares;

namespace RequestTracer.Core
{
    public static class MiddlewareExtensions
    {
        public static IApplicationBuilder UseCustomMiddleware(this IApplicationBuilder builder)
        {
            return builder.UseMiddleware<LogMiddleware>();
        }
    }
}

RequestTracer.Infrastructure.IoC:

Install the below nugget packages.

DependencyContainer.cs

This class referenced to API. At the same time, this class related to setup information like dependency injections.

using System;
using MassTransit;
using MediatR;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using RequestTracer.Core.Commands;

namespace RequestTracer.Infrastructure.IoC
{
    public static class DependencyContainer
    {
        public static void RegisterService(IServiceCollection services, IConfiguration configuration)
        {
            #region IoC layer
            services.AddMediatR(typeof(LogCommand));

            //mass transit configuration
            services.AddMassTransit(x =>
            {
                x.AddBus(provider => Bus.Factory.CreateUsingRabbitMq(config =>
                {
                    config.UseHealthCheck(provider);
                    config.Host(new Uri(configuration.GetSection("RabbitMqSettings:host").Value), "/",
                    h =>
                    {
                        h.Heartbeat(TimeSpan.FromSeconds(30));
                        h.Username(configuration.GetSection("RabbitMqSettings:username").Value);
                        h.Password(configuration.GetSection("RabbitMqSettings:password").Value);
                    });

                }));
            });

            services.AddMassTransitHostedService();

            #endregion

            #region Database Layer
            #endregion

            #region Application Layer
            #endregion
        }
    }
}

API configuration

I designed a Web API that name is "Customer.API". The important is, settings must be defined in the API layer for RabbitMq to work. Be careful! We have 3 different worker services and 2 web API. Each service has its own log queue. The distinctive thing is the name of the apps. If you define worker1,worker2, and worker3, you have 3 different queues. Evert worker service has its own environment variables like development, pre-production, or production. It means a different exchange.

"ApplicationName" :  "Customer.Api",
  "RabbitMqSettings": {
    "host": "rabbitmq://localhost",
    "server": "rabbitmq://localhost:15672",
    "username": "burki",
    "password": "123"
  }

Please reference all layers to CustomerAPI application then open the Startup.cs. Let's define configurations.

  public void ConfigureServices(IServiceCollection services)
        {
            services.AddControllers();
            services.AddSwaggerGen(c =>
            {
                c.SwaggerDoc("v1", new OpenApiInfo { Title = "CustomerApi", Version = "v1" });
            });

            DependencyContainer.RegisterService(services, Configuration);
        }

  public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
                app.UseSwagger();
                app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "MonitorApi.Demo v1"));
            }

            app.UseHttpsRedirection();

            app.UseRouting();

            app.UseAuthorization();

            _ = app.UseCustomMiddleware(); 

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();
            });
        }

Install RabbitMQ

1. Run the command

docker run -d --hostname my-rabbit --name myrabbit -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=123456 -p 5672:5672 -p 15672:15672 rabbitmq:3-management

After that start the RabbitMQ docker file with this command

docker run imageid

Check the status

docker ps

1. Open the browser go to the

http://localhost:15672/
username: admin
password: 123456

I would like to create a new account. I'll set up username: burki and password: 123 and give permission. For this information, please open the Admin tab.

Here is the config result.

PS: If you do not declare a virtual host you'll get The agent is stopped or has been stopped, no additional provocateurs can be created error

Looks good. Let's test it.

First of all, open the run the CustomerAPI and make some requests. Here is my controller class code

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using CustomerApi;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;

namespace MonitorApi.Demo.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class CustomerController : ControllerBase
    {
        private readonly ILogger<CustomerController> _logger;

        public CustomerController(ILogger<CustomerController> logger)
        {
            _logger = logger;
        }

        [HttpGet]
        public IEnumerable<Customer> GetCustomers()
        {
            var customers = new List<Customer>();
            customers.Add(new Customer()
            {
                Id = 1,
                FirsName = "Burak",
                LastName = "Seyhan",
                City = "Ankara",
                EmailAddress = "[e-mail]",
                Fax = string.Empty,
                Phone = "444"
            });
            customers.Add(new Customer()
            {
                Id = 2,
                FirsName = "User_FirstName",
                LastName = "User_LastName",
                City = "User_City",
                EmailAddress = "[e-mail]",
                Fax = string.Empty,
                Phone = "User_PhoneNumber"
            });

            return customers;
        }
    }
}

Did you see it? I do not care about RabbitMQ configurations or another thing. I just only created a Web API method. I only care about appsettings.Development.json file that's all.

Run the CustomerAPI then request GetCustomers() method.

When you request any method, our middleware catches all information and connects with the RabbitMQ here is the connection status.

Looks good go head 🤟🏻

Please open the Exchange tab.

Click the "Customer.Api-Development-Logs" then you'll see a new window after that I'll configure it to the binding so choose to queue but make sure the queue is defined. Open Queue tab then creates a new queue and I called 'Customer.Api-Development-Logs'. Here is my queue.

After that bind Exchange to Queue.

Now let's make a new request.

My request was reflected here

I haven't written a consumer yet. So let's look at the output you want, made manually. Again open the Queue tab then click the Get Messages button.

Add new Order API then configure the appsettings.Development.json file same with previous.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "ApplicationName": "Order.Api",
  "RabbitMqSettings": {
    "host": "rabbitmq://localhost",
    "server": "rabbitmq://localhost:15672",
    "username": "burki",
    "password": "123"
  }
}

Let's get test it.

When you make a request Order API exchange created.

Then create a queue then bind it.

Here is the payload.

Every service has its own middleware. So Every middleware creates a single exchange. If I run the all workers you can see the worker exchanges.

As a result,

Logging mechanisms are separated by the application-based queue. In this way, the metrics of the requests and responses of each service can be easily analyzed within itself.
Consumer applications to be written in addition to these applications and payload information can be easily displayed on the web screen interface.

Repository

Thank you for reading.