DEV Community

loading...
Cover image for The middleware

The middleware

bseyhan profile image bseyhan ・9 min read

Hi everybody,

I hope, you are feeling good today. Also, I'm good thanks. In this post, I'm gonna talk about the importance of system tracing. The main subject, I want to tell you in this post will be outside of the classical methods. Maybe you ask me what are you talking about? I'm talking about designing a custom log framework using different tech stacks. Today, I will create a new library, after that implemented a basic API. Here is the most important content is meaning of architecture

Here is the I used technology stacks

Why do we need tracing of the system?

It does not make sense that I do say specific reasons. If we have domain size does not matter, we should trace them. I would like to specify, as the complexity of the domain increases, monitoring and interpreting the system decrease.

I have seen that in small and medium-sized systems, logs are monitored without being written to databases and even kept in text files. It might seem to like normal but here is the biggest problem are it will be hard to identify the problem. Let's give an example. Just think about a domain.

Screen Shot 2020-12-21 at 2.31.18 PM

This project about customer information. I created 2 API and 3 worker service. Each worker services applications provide a different service. Like register, login, notification, etc ...
Also, each service has its own service log like warning, error, or information. Let's assume, every API and worker service write request and response logs write to disk. Any service stopped working after that I have to check the service logs. Firstly open the txt log files. After that search it with ctrl+f. This is a very bad situation. You do not have to spend a lot of effort.

What should I do?

Some companies buy software or some companies write their own solutions after that to implement. Let's give an example Nlog, Elmah, Graylog, SeriLog, DataDog.

In this post, I would like to explain the following titles.

What is Middleware?

More Information

What is MediatR?

MediatR is an open-source project and an implementation of the mediator design pattern. The mediator design pattern controls how a set of objects communicate and helps to reduce the number of dependencies among these objects that you must manage. In the mediator design pattern, objects don’t communicate with one another directly, but through a mediator. This article presents a discussion of how we can use MediatR in ASP.Net Core applications.
More Information

What is MassTransit?

More Information

Let's open Visual Studio after that let's write some code.

Project Design and Packages

I would like to create a framework that name is 'RequestTracer'. I designed 2 layers. Here is the folder structure.

Screen Shot 2020-12-21 at 3.14.18 PM

RequestTracer.Core layer

Install the below nugget packages.

Screen Shot 2020-12-21 at 3.15.39 PM

Let's examine the codes in the file.

  • Commands: It's about the MediatR request parameter and the return type is bool.
using MediatR;
using RequestTracer.Core.Dtos;

namespace RequestTracer.Core.Commands
{
    public class LogCommand : IRequest<bool>
    {
        public LogDto LogDtos { get; set; }
    }
}

Enter fullscreen mode Exit fullscreen mode
  • Dtos: It's about request and response parameters. Referenced to Commands.
using System;

namespace RequestTracer.Core.Dtos
{
    public class LogDto
    {
        public Guid Id { get { return Guid.NewGuid(); } }

        public DateTime Date { get { return DateTime.UtcNow; } }

        public string ApplicationName { get; set; }

        public string Environment { get; set; }

        public LogDetail LogDetail { get; set; }
    }
}
Enter fullscreen mode Exit fullscreen mode
namespace RequestTracer.Core.Dtos
{
    public class LogDetail
    {
        public string Host { get; set; }

        public string Method { get; set; }

        public string Protocol { get; set; }

        public string Schema { get; set; }

        public string Path { get; set; }

        public string QueryStringParameter { get; set; }

        public int StatusCode { get; set; }

        public string ContentType { get; set; }

        public long? ContentLength { get; set; }

        public string ExecutionTime { get; set; }
    }
}
Enter fullscreen mode Exit fullscreen mode
  • Handlers: It's about the MediatR. Every command has a handler. Every environment creates its own exchange.
using System;
using System.Threading;
using System.Threading.Tasks;
using MassTransit;
using MediatR;
using Microsoft.Extensions.Configuration;
using RequestTracer.Core.Commands;

namespace RequestTracer.Core.Handlers
{
    public class LogCommandHandler : IRequestHandler<LogCommand, bool>
    {
        private readonly IBus _bus;
        private readonly IConfiguration _configuration;
        private readonly ISendEndpoint _sendEndpoint;

        public LogCommandHandler(IBus bus, IConfiguration configuration)
        {
            _bus = bus;
            _configuration = configuration;

            var environment = Environment.GetEnvironmentVariables()["ASPNETCORE_ENVIRONMENT"].ToString();

            _sendEndpoint = _bus.GetSendEndpoint(
                new Uri($"{_configuration.GetSection("RabbitMqSettings:server").Value}/{configuration.GetSection("ApplicationName").Value}-{environment}-Logs")).GetAwaiter().GetResult();
        }

        public async Task<bool> Handle(LogCommand request, CancellationToken cancellationToken)
        {
            await _sendEndpoint.Send(request);

            return true;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode
  • Middlewares: It's about request and response custom pipeline. Every single request goes through here without errors cause I only trace information logs.
using System;
using System.Diagnostics;
using System.Threading.Tasks;
using MediatR;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;

namespace RequestTracer.Core.Middlewares
{
    public class LogMiddleware
    {
        private readonly RequestDelegate _next;
        private readonly IMediator _mediator;
        private readonly IConfiguration _configuration;
        private readonly ILogger<LogMiddleware> _logger;

        public LogMiddleware(RequestDelegate next,
                             IMediator mediator,
                             IConfiguration configuration,
                             ILogger<LogMiddleware> logger)
        {
            _next = next ?? throw new ArgumentNullException(nameof(next));
            _mediator = mediator ?? throw new ArgumentNullException(nameof(next));
            _configuration = configuration ?? throw new ArgumentNullException(nameof(_configuration));
            _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        }

        public async Task Invoke(HttpContext httpContext)
        {
            try
            {
                var stopwatch = new Stopwatch();
                stopwatch.Start();
                await _next(httpContext);
                stopwatch.Stop();

                await SendRequest(httpContext, stopwatch);
            }
            catch (Exception ex)
            {
                _logger.LogError($"LogMiddleware {ex}");

                if (httpContext.Response.HasStarted)
                {
                    _logger.LogWarning("The response has already started");
                    throw;
                }
            }
        }

        private async Task SendRequest(HttpContext httpContext,Stopwatch stopwatch)
        {
            var request = httpContext.Request;
            var response = httpContext.Response;

            await _mediator.Send(new Commands.LogCommand()
            {
                LogDtos = new Dtos.LogDto()
                {
                    ApplicationName = _configuration.GetSection("ApplicationName").Value,
                    Environment = Environment.GetEnvironmentVariables()["ASPNETCORE_ENVIRONMENT"].ToString(),
                    LogDetail = new Dtos.LogDetail()
                    {
                        Host = request.Host.HasValue ? request.Host.ToString() : string.Empty,
                        Method = request.Method,
                        Protocol = request.Protocol,
                        Schema = request.Scheme,
                        Path = request.Path,
                        QueryStringParameter = request.QueryString.HasValue == true ? request.QueryString.Value : string.Empty,
                        StatusCode = response.StatusCode,
                        ContentType = response.ContentType,
                        ContentLength = request.ContentLength,
                        ExecutionTime = $"Execution time of {stopwatch.ElapsedMilliseconds} ms"
                    }
                }
            });
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Another one is an extension class here is the code.

using Microsoft.AspNetCore.Builder;
using RequestTracer.Core.Middlewares;

namespace RequestTracer.Core
{
    public static class MiddlewareExtensions
    {
        public static IApplicationBuilder UseCustomMiddleware(this IApplicationBuilder builder)
        {
            return builder.UseMiddleware<LogMiddleware>();
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

RequestTracer.Infrastructure.IoC:

Install the below nugget packages.

Screen Shot 2020-12-21 at 3.16.13 PM

DependencyContainer.cs

This class referenced to API. At the same time, this class related to setup information like dependency injections.

using System;
using MassTransit;
using MediatR;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using RequestTracer.Core.Commands;

namespace RequestTracer.Infrastructure.IoC
{
    public static class DependencyContainer
    {
        public static void RegisterService(IServiceCollection services, IConfiguration configuration)
        {
            #region IoC layer
            services.AddMediatR(typeof(LogCommand));

            //mass transit configuration
            services.AddMassTransit(x =>
            {
                x.AddBus(provider => Bus.Factory.CreateUsingRabbitMq(config =>
                {
                    config.UseHealthCheck(provider);
                    config.Host(new Uri(configuration.GetSection("RabbitMqSettings:host").Value), "/",
                    h =>
                    {
                        h.Heartbeat(TimeSpan.FromSeconds(30));
                        h.Username(configuration.GetSection("RabbitMqSettings:username").Value);
                        h.Password(configuration.GetSection("RabbitMqSettings:password").Value);
                    });

                }));
            });

            services.AddMassTransitHostedService();

            #endregion

            #region Database Layer
            #endregion

            #region Application Layer
            #endregion
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

API configuration

I designed a Web API that name is "Customer.API". The important is, settings must be defined in the API layer for RabbitMq to work. Be careful! We have 3 different worker services and 2 web API. Each service has its own log queue. The distinctive thing is the name of the apps. If you define worker1,worker2, and worker3, you have 3 different queues. Evert worker service has its own environment variables like development, pre-production, or production. It means a different exchange.

"ApplicationName" :  "Customer.Api",
  "RabbitMqSettings": {
    "host": "rabbitmq://localhost",
    "server": "rabbitmq://localhost:15672",
    "username": "burki",
    "password": "123"
  }
Enter fullscreen mode Exit fullscreen mode

Please reference all layers to CustomerAPI application then open the Startup.cs. Let's define configurations.

  public void ConfigureServices(IServiceCollection services)
        {
            services.AddControllers();
            services.AddSwaggerGen(c =>
            {
                c.SwaggerDoc("v1", new OpenApiInfo { Title = "CustomerApi", Version = "v1" });
            });

            DependencyContainer.RegisterService(services, Configuration);
        }

Enter fullscreen mode Exit fullscreen mode
  public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
                app.UseSwagger();
                app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "MonitorApi.Demo v1"));
            }

            app.UseHttpsRedirection();

            app.UseRouting();

            app.UseAuthorization();

            _ = app.UseCustomMiddleware(); 

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();
            });
        }
Enter fullscreen mode Exit fullscreen mode

Install RabbitMQ

  1. Run the command

docker run -d --hostname my-rabbit --name myrabbit -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=123456 -p 5672:5672 -p 15672:15672 rabbitmq:3-management

Screen Shot 2020-12-21 at 8.37.51 PM

After that start the RabbitMQ docker file with this command

docker run imageid

Screen Shot 2020-12-21 at 8.39.40 PM

Check the status

docker ps

Screen Shot 2020-12-21 at 8.42.58 PM

  1. Open the browser go to the

http://localhost:15672/
username: admin
password: 123456

Screen Shot 2020-12-21 at 8.44.42 PM

I would like to create a new account. I'll set up username: burki and password: 123 and give permission. For this information, please open the Admin tab.

Screen Shot 2020-12-21 at 8.47.49 PM

Here is the config result.

Screen Shot 2020-12-21 at 9.10.13 PM

PS: If you do not declare a virtual host you'll get The agent is stopped or has been stopped, no additional provocateurs can be created error

Looks good. Let's test it.

First of all, open the run the CustomerAPI and make some requests. Here is my controller class code

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using CustomerApi;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;

namespace MonitorApi.Demo.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class CustomerController : ControllerBase
    {
        private readonly ILogger<CustomerController> _logger;

        public CustomerController(ILogger<CustomerController> logger)
        {
            _logger = logger;
        }

        [HttpGet]
        public IEnumerable<Customer> GetCustomers()
        {
            var customers = new List<Customer>();
            customers.Add(new Customer()
            {
                Id = 1,
                FirsName = "Burak",
                LastName = "Seyhan",
                City = "Ankara",
                EmailAddress = "burakseyhan8@gmail.com",
                Fax = string.Empty,
                Phone = "444"
            });
            customers.Add(new Customer()
            {
                Id = 2,
                FirsName = "User_FirstName",
                LastName = "User_LastName",
                City = "User_City",
                EmailAddress = "user@gmail.com",
                Fax = string.Empty,
                Phone = "User_PhoneNumber"
            });

            return customers;
        }
    }
}

Enter fullscreen mode Exit fullscreen mode

Did you see it? I do not care about RabbitMQ configurations or another thing. I just only created a Web API method. I only care about appsettings.Development.json file that's all.

Run the CustomerAPI then request GetCustomers() method.

Screen Shot 2020-12-21 at 9.15.27 PM

When you request any method, our middleware catches all information and connects with the RabbitMQ here is the connection status.

Screen Shot 2020-12-21 at 9.16.58 PM

Looks good go head 🀟🏻

Please open the Exchange tab.

Screen Shot 2020-12-21 at 9.17.58 PM

Click the "Customer.Api-Development-Logs" then you'll see a new window after that I'll configure it to the binding so choose to queue but make sure the queue is defined. Open Queue tab then creates a new queue and I called 'Customer.Api-Development-Logs'. Here is my queue.

Screen Shot 2020-12-21 at 9.31.16 PM

After that bind Exchange to Queue.

Screen Shot 2020-12-21 at 9.32.29 PM

Now let's make a new request.

Screen Shot 2020-12-21 at 9.34.35 PM

My request was reflected here

Screen Shot 2020-12-21 at 9.33.28 PM

I haven't written a consumer yet. So let's look at the output you want, made manually. Again open the Queue tab then click the Get Messages button.

Screen Shot 2020-12-21 at 9.37.28 PM

Add new Order API then configure the appsettings.Development.json file same with previous.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "ApplicationName": "Order.Api",
  "RabbitMqSettings": {
    "host": "rabbitmq://localhost",
    "server": "rabbitmq://localhost:15672",
    "username": "burki",
    "password": "123"
  }
}
Enter fullscreen mode Exit fullscreen mode

Screen Shot 2020-12-21 at 10.16.03 PM

Let's get test it.

When you make a request Order API exchange created.
Screen Shot 2020-12-21 at 10.23.16 PM

Screen Shot 2020-12-21 at 10.04.06 PM

Then create a queue then bind it.

Screen Shot 2020-12-21 at 10.06.35 PM

Here is the payload.

Screen Shot 2020-12-21 at 10.07.11 PM

Every service has its own middleware. So Every middleware creates a single exchange. If I run the all workers you can see the worker exchanges.

Screen Shot 2020-12-21 at 10.23.16 PM

As a result,

Logging mechanisms are separated by the application-based queue. In this way, the metrics of the requests and responses of each service can be easily analyzed within itself.
Consumer applications to be written in addition to these applications and payload information can be easily displayed on the web screen interface.

Repository

Thank you for reading.

Discussion (0)

pic
Editor guide