DEV Community: Mohsen Esmailpour

Transaction middleware in ASP.NET Core

Mohsen Esmailpour — Fri, 15 Oct 2021 16:12:46 +0000

Sometimes to fulfill a business use case in an HTTP request you need to save data to the database on each step of processing the request and if a part fails, you have to roll back previous parts. In this article, I'm going to show you how middleware can be used to create transaction and commit changes to the database implicitly if no exception is thrown. Let's assume we are using SQL database, Dapper micro ORM and repository pattern for hiding data access layer.

Let's get started by creating connection provider class:

public class SqlConnectionProvider
{
    private readonly IDbConnection _connection;
    private IDbTransaction _transaction;

    public SqlConnectionProvider(string connectionString)
    {
        _connection = new SqlConnection(connectionString);
    }

    public IDbConnection GetDbConnection => _connection;

    public IDbTransaction GetTransaction => _transaction;

    public IDbTransaction CreateTransaction()
    {
        if (_connection.State == ConnectionState.Closed)
            _connection.Open();

        _transaction = _connection.BeginTransaction();

        return _transaction;
    }
}

We use SqlConnectionProvider to create SqlConnection object and inject it to repositories and besides SqlConnection, it's responsible for creating transaction too. In ConfigureServices in Startup class we need to register SqlConnectionProvider:

services.AddScoped((_) => new SqlConnectionProvider(Configuration.GetConnectionString("Default")));

Now it's time to create transaction middleware. Here is the definition of middleware. I'm going to add the necessary codes to handle transaction step by step.

public class DbTransactionMiddleware
{
    private readonly RequestDelegate _next;

    public DbTransactionMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task Invoke(HttpContext httpContext, SqlConnectionProvider connectionProvider)
    {
    }
}

First, we don't want to open transaction for GET request. normally for GET request we just fetch data and in POST, PUT and DELETE we modify data.

// For HTTP GET opening transaction is not required
if (httpContext.Request.Method.Equals("GET", StringComparison.CurrentCultureIgnoreCase))
{
    await _next(httpContext);
    return;
}

The below code is straightforward. We open transaction, calling the next middleware and then commit transaction and finally, disposing transaction.

IDbTransaction transaction = null;

try
{
    transaction = connectionProvider.CreateTransaction();

    await _next(httpContext);

    transaction.Commit();
}
finally
{
    transaction?.Dispose();
}

Also we need pass to pass transaction to repository, otherwise we face this error:

"ExecuteNonQuery requires the command to have a transaction when the connection assigned to the command is in a pending local transaction. The Transaction property of the command has not been initialized."

Let's assume we have a todo repository:

public class TodoItemRepository : ITodoItemRepository
{
    private readonly SqlConnectionProvider _connectionProvider;
    private readonly IDbConnection _connection;

    public TodoItemRepository(SqlConnectionProvider connectionProvider)
    {
        _connectionProvider = connectionProvider;
        _connection = connectionProvider.GetDbConnection;
    }

    public Task<int> AddTodoItemAsync(TodoItem todoItem)
    {
        const string command = "INSERT INTO TodoItems (Title, Note, TodoListId) VALUES (@Title, @Note, @TodoListId)";
        var parameters = new DynamicParameters();
        parameters.Add("Title", todoItem.Title, DbType.String);
        parameters.Add("Note", todoItem.Note, DbType.String);
        parameters.Add("TodoListId", todoItem.TodoListId, DbType.Int32);

        // Passing transaction to ExecuteAsync method
        return _connection.ExecuteAsync(command, parameters, _connectionProvider.GetTransaction);
    }

    public Task<IEnumerable<TodoItem>> GetTodoItemsAsync()
    {
        return _connection.ExecuteScalarAsync<IEnumerable<TodoItem>>("SELECT * FROM TodoItems");
    }
}

Sometimes it's not necessary to open a transaction. We can open transaction more accurately by decorating action with an attribute.

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TransactionAttribute : Attribute
{
}

And decorating our API action with Transaction attribute:

[Transaction]
[HttpPost("todo-item")]
public async Task<IActionResult> Post(...)
{
    ...
}

And in transaction middleware we can check if action is decorated with Transaction attribute or not:

public class DbTransactionMiddleware
{
    private readonly RequestDelegate _next;

    public DbTransactionMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task Invoke(HttpContext httpContext, SqlConnectionProvider connectionProvider)
    {
        // For HTTP GET opening transaction is not required
        if (httpContext.Request.Method.Equals("GET", StringComparison.CurrentCultureIgnoreCase))
        {
            await _next(httpContext);
            return;
        }

        // If action is not decorated with TransactionAttribute then skip opening transaction
        var endpoint = httpContext.Features.Get<IEndpointFeature>()?.Endpoint;
        var attribute = endpoint?.Metadata.GetMetadata<TransactionAttribute>();
        if (attribute == null)
        {
            await _next(httpContext);
            return;
        }

        IDbTransaction transaction = null;

        try
        {
            transaction = connectionProvider.CreateTransaction();

            await _next(httpContext);

            transaction.Commit();
        }
        finally
        {
            transaction?.Dispose();
        }
    }
}

public static class MiddlewareExtensions
{
    public static IApplicationBuilder UseDbTransaction(this IApplicationBuilder app)
        => app.UseMiddleware<DbTransactionMiddleware>();
}

And final step is registering middleware just before UsEndpoints middleware:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    ...

    app.UseRouting();

    app.UseAuthorization();

    app.UseDbTransaction();

    app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllers();
    });
}

If you are using Entity Framework, you can open and commit transaction in this way:

IDbContextTransaction transaction = null;
try
{
    transaction = await dbContext.Database.BeginTransactionAsync();

    await _next(httpContext);

    await transaction.CommitAsync();
}
finally
{
    if (transaction != null)
        await transaction.DisposeAsync();
}

You can find the source code for this walkthrough on Github.

Extending ProblemDetails - Add error code to ValidationProblemDetails

Mohsen Esmailpour — Sun, 03 Oct 2021 11:31:05 +0000

A while back in one project I faced a situation that required returning custom error code from API to the client. In the client app, the user after completing the registration process had to confirm his phone number. For some users, no text message was sent at that time and they could not do the phone verification step. Later, the user after login encountered an unverified phone number error and had to be redirected to the phone number verification page, but it was difficult for the client to understand when to redirect the user to the page because the server only returned an error message.

Since ASP.NET Core 2.2, I followed RFC 7807 specification standard to format errors in web API responses by using ProblemDetails.

The constructor of ValidationProblemDetails class accepts ModelStateDictionary and Dictionary<string, string[]> that the key field is property name and value is a list of error messages and the dictionary serialized into error field:

{
    "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
    "title": "One or more validation errors occurred.",
    "status": 400,
    "errors": {
        "User": [
            "The user phone number is not verified."
        ]
    }
}

I'm going to change error field to this:

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "errors": [
    {
      "code": 100,
      "message": "The user phone number is not verified."
    }
  ]
}

Let's begin by creating custom ValidationProblemDetails class:

using Microsoft.AspNetCore.Mvc;

public class CustomValidationProblemDetails : ValidationProblemDetails
{
    public CustomValidationProblemDetails()
    {
    }

    [JsonPropertyName("errors")]
    public new IEnumerable<ValidationError> Errors { get; } = new List<ValidationError>();
}

ValidationProblemDetails has an Error property that is IDictionary<string, string[]> and we replace this property with our version to add code error.

public class ValidationError
{
    public int Code { get; set; }

    public string Message { get; set; }
}

Constructor of ValidationProblemDetails accepts ModelStateDictionary and we need to convert it to list of ValidationError:

public CustomValidationProblemDetails(IEnumerable<ValidationError> errors)
{
    Errors = errors;
}

public CustomValidationProblemDetails(ModelStateDictionary modelState)
{
    Errors = ConvertModelStateErrorsToValidationErrors(modelState);
}

private List<ValidationError> ConvertModelStateErrorsToValidationErrors(ModelStateDictionary modelStateDictionary)
{
    List<ValidationError> validationErrors = new();

    foreach (var keyModelStatePair in modelStateDictionary)
    {
        var errors = keyModelStatePair.Value.Errors;
        switch (errors.Count)
        {
            case 0:
                continue;

            case 1:
                validationErrors.Add(new ValidationError { Code = null, Message = errors[0].ErrorMessage });
                break;

            default:
                var errorMessage = string.Join(Environment.NewLine, errors.Select(e => e.ErrorMessage));
                validationErrors.Add(new ValidationError { Message = errorMessage });
                break;
        }
    }

    return validationErrors;
}

Now it's time to create custom ProblemDetailsFactory to create CustomValidationProblemDetails when we want to return bad request response:

public class CustomProblemDetailsFactory : ProblemDetailsFactory
{
    public override ProblemDetails CreateProblemDetails(HttpContext httpContext, int? statusCode = null, string title = null,
        string type = null, string detail = null, string instance = null)
    {
        var problemDetails = new ProblemDetails
        {
            Status = statusCode,
            Title = title,
            Type = type,
            Detail = detail,
            Instance = instance,
        };

        return problemDetails;
    }

    public override ValidationProblemDetails CreateValidationProblemDetails(HttpContext httpContext,
        ModelStateDictionary modelStateDictionary, int? statusCode = null, string title = null, string type = null,
        string detail = null, string instance = null)
    {
        statusCode ??= 400;
        type ??= "https://tools.ietf.org/html/rfc7231#section-6.5.1";
        instance ??= httpContext.Request.Path;

        var problemDetails = new CustomValidationProblemDetails(modelStateDictionary)
        {
            Status = statusCode,
            Type = type,
            Instance = instance
        };

        if (title != null)
        {
            // For validation problem details, don't overwrite the default title with null.
            problemDetails.Title = title;
        }

        var traceId = Activity.Current?.Id ?? httpContext?.TraceIdentifier;
        if (traceId != null)
        {
            problemDetails.Extensions["traceId"] = traceId;
        }

        return problemDetails;
    }
}

And final step is registering CustomProblemDetailsFactory in ConfigureServices method of Startup class:

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();

    services.AddTransient<ProblemDetailsFactory, CustomProblemDetailsFactory>();
}

So far we haven't set the Code property of ValidationError and it's value will be null for data annotation validation or you can set default value when converting ModelStateDictionary to list of ValidationError:

validationErrors.Add(new ValidationError { Code = "100", Message = errors[0].ErrorMessage });

In part three of What Every ASP.NET Core Web API project needs - Exception handling middleware series, I have created a domain exception class and also handling domain exception in ExceptionHandlingMiddleware and convert it to bad request response. I added the Code property to domain exception class to pass code:

public class DomainException : Exception
{
    public DomainException(string message, string code = null)
        : base(message)
    {
        Code = code;
    }

    public string Code { get; }
}

Back to my scenario, in the login method, I want to check if the phone number of the user is not confirmed throw a domain exception and catch in middleware and return bad request:

public async Task<IActionResult> Login(LoginViewModel model)
{
    var user = await _userManager.FindByEmailAsync(model.Email);
    if (!user?.PhoneConfirmed)
        throw new DomainException("The user phone number is not verified.", Code: "120");
}

And finally in exception handling middleware catch exception and convert to CustomValidationProblemDetails:

public async Task Invoke(HttpContext context)
{
    try
    {
        await _next(context);
    }
    catch (Exception ex)
    {
        await HandleExceptionAsync(context, ex);
    }
}

private async Task HandleExceptionAsync(HttpContext context, Exception ex)
{
    string result;

    if (ex is DomainException e)
    {
        var problemDetails = new CustomValidationProblemDetails(
            new List<ValidationError>
            {
                new() { Code = e.Code, Message = e.Message }
            }
        )
        {
            Type = "https://tools.ietf.org/html/rfc7231#section-6.5.1",
            Title = "One or more validation errors occurred.",
            Status = (int)HttpStatusCode.BadRequest,
            Instance = context.Request.Path,
        };
        context.Response.StatusCode = (int)HttpStatusCode.BadRequest;
        result = JsonSerializer.Serialize(problemDetails);
    }
...

You can find the source code for this walkthrough on Github.

Leaky abstraction and clean architecture template

Mohsen Esmailpour — Tue, 29 Jun 2021 12:32:37 +0000

According to Wikipedia in software development, a leaky abstraction is an abstraction that leaks details that it is supposed to abstract away. The term "leaky abstraction" was popularized in 2002 by Joel Spolsky. An earlier paper by Kiczales describes some of the issues with imperfect abstractions and presents a potential solution to the problem by allowing for the customization of the abstraction itself.

As systems become more complex, software developers must rely upon more abstractions. Each abstraction tries to hide complexity, letting a developer write software that "handles" the many variations of modern computing.
However, this law claims that developers of reliable software must learn the abstraction's underlying details anyway.

During the past year, I've seen several implementations of clean architecture that inspired form Jason Taylor Clean Architecture Solution Template and all of them have a common interface, IApplicationDbContext. This interface aims to hide the underlying data access technology that is being used but when you look at the interface you notice that the interface is coupled to Entity Framework Core. I also can guess Jason Taylor assumed that you are always using EF Core and it never going to change.

Let's see leaked abstraction in action. I want to implement a unit test for UpdateTodoListCommandHandler and assume it is a real-world project and we some logic inside the Handle method but we don't implement integration for each scenario. Here is the actual implementation of :

namespace CleanArchitecture.Application.TodoLists.Commands.UpdateTodoList
{
    public class UpdateTodoListCommand : IRequest
    {
        public int Id { get; set; }

        public string Title { get; set; }
    }

    public class UpdateTodoListCommandHandler : IRequestHandler<UpdateTodoListCommand>
    {
        private readonly IApplicationDbContext _context;

        public UpdateTodoListCommandHandler(IApplicationDbContext context)
        {
            _context = context;
        }

        public async Task<Unit> Handle(UpdateTodoListCommand request, CancellationToken cancellationToken)
        {
            var entity = await _context.TodoLists.FindAsync(request.Id);
            if (entity == null)
            {
                throw new NotFoundException(nameof(TodoList), request.Id);
            }

            entity.Title = request.Title;
            await _context.SaveChangesAsync(cancellationToken);

            return Unit.Value;
        }
    }
}

I want to ensure when the entity does not exist, NotFoundException is thrown.

[Test]
public void UpdateTodoList_WhenEntityNotExist_ThrowsException()
{
    // Arrange
    var list = new List<TodoList>();
    var queryable = list.AsQueryable();

    var dbSet = new Mock<DbSet<TodoList>>();
    dbSet.As<IQueryable<TodoList>>().Setup(m => m.Provider).Returns(queryable.Provider);
    dbSet.As<IQueryable<TodoList>>().Setup(m => m.Expression).Returns(queryable.Expression);
    dbSet.As<IQueryable<TodoList>>().Setup(m => m.ElementType).Returns(queryable.ElementType);
    dbSet.As<IQueryable<TodoList>>().Setup(m => m.GetEnumerator()).Returns(() => queryable.GetEnumerator());
    dbSet.Setup(d => d.FindAsync(It.IsAny<object[]>())).ReturnsAsync((object[] id) => list.SingleOrDefault(t => t.Id == (int)id[0]));

    var dbContext = new Mock<IApplicationDbContext>();
    dbContext.SetupGet(d => d.TodoLists).Returns(dbSet.Object);

    var sut = new UpdateTodoListCommandHandler(dbContext.Object);
    var command = new UpdateTodoListCommand { Id = 1, Title = "Test" };

    // Act
    var exception = Assert.ThrowsAsync<NotFoundException>(() => sut.Handle(command, new CancellationToken()));

    // Assert
    Assert.NotNull(exception);
}

As you can see I tried to mock IApplicationDbContext and the interface should help to hide underlying implementation but to implement such a test you should know how to mock DbSet. If you use the ApplicationDbContext class instead of IApplicationDbContext interface, the result will be the same and same amount of code is need to mock ApplicationDbContext class.

Instead, we can use repository pattern and hide underlying technology and mock it easily.
Lets replace IApplicationDbContext with ITodoListRepository.

public interface ITodoListRepository
{
    Task<TodoList> GetByIdAsync(int id);

    Task UpdateAsync(TodoList todoList);
}

And the handler:

public class UpdateTodoListCommandHandler : IRequestHandler<UpdateTodoListCommand>
{
    private readonly ITodoListRepository _repository;

    public UpdateTodoListCommandHandler(ITodoListRepository repository)
    {
        _repository = repository;
    }

    public async Task<Unit> Handle(UpdateTodoListCommand request, CancellationToken cancellationToken)
    {
        var entity = await _repository.GetByIdAsync(request.Id);
        if (entity == null)
        {
            throw new NotFoundException(nameof(TodoList), request.Id);
        }

        entity.Title = request.Title;
        await _repository.UpdateAsync(entity);

        return Unit.Value;
    }
}

Now we can refactor the test method:

[Test]
public void UpdateTodoList_WhenEntityNotExist_ThrowsException()
{
    // Arrange
    var list = new List<TodoList>();

    var repository = new Mock<ITodoListRepository>();
    repository.Setup(r => r.GetByIdAsync(It.IsAny<int>())).ReturnsAsync((int id) => list.SingleOrDefault(t => t.Id == id));

    var command = new UpdateTodoListCommand { Title = "Test", Id = 1 };
    var sut = new UpdateTodoListCommandHandler(repository.Object);

    // Act
    var exception = Assert.ThrowsAsync<NotFoundException>(async () => await sut.Handle(command, new CancellationToken()));

    // Assert
    Assert.NotNull(exception);
    Assert.AreEqual(typeof(NotFoundException), exception.GetType());
}

How to extend identity UserManager class

Mohsen Esmailpour — Sun, 06 Jun 2021 21:28:20 +0000

Recently I've answered a question on StackOverflow website and the question was how to create an overloaded method for both AddToRoleAsync() and IsInRoleAync() to look for the role by its ID and not by name. I this post I show how to extend UserManager class in the way it is implemented and access some internals of UserManager.

First way to extend the UserManager is using extension methods. In some cases you don't need to create another class to extend UserManager and by implementing some extensions methods to achieve what you want.



public static class UserManagerExtensions
{
    public static Task<ApplicationUser> FindByNameAsync(this UserManager<ApplicationUser> userManager, string name)
    {
        return userManager?.Users?.FirstOrDefaultAsync(um => um.UserName == name);
    }

    public static Task<ApplicationUser> FindByCardIDAsync(this UserManager<ApplicationUser> userManager, string cardId)
    {
        return userManager?.Users?.FirstOrDefaultAsync(um => um.CardId == cardId);
    }

    ....
}

In some cases like create an overloaded method for AddToRoleAsync() to add a role to a user by role ID and not by role name, the extension method doesn't work and it requires a custom implementation of UserManager class. Lets spin up a new class that inherits from the built-in UserManager:



public class ApplicationUserManager : UserManager<IdentityUser>
{
    public ApplicationUserManager(
        IUserStore<IdentityUser> store,
        IOptions<IdentityOptions> optionsAccessor,
        IPasswordHasher<IdentityUser> passwordHasher,
        IEnumerable<IUserValidator<IdentityUser>> userValidators,
        IEnumerable<IPasswordValidator<IdentityUser>> passwordValidators,
        ILookupNormalizer keyNormalizer,
        IdentityErrorDescriber errors,
        IServiceProvider services,
        ILogger<UserManager<IdentityUser>> logger)
        : base(store, optionsAccessor, passwordHasher, userValidators, passwordValidators, keyNormalizer, errors, services, logger)
    {
    }
}

You can generate the constructor automatically with Visual Studio by using the Generate Constructor Quick Action (by pressing CTRL-ENTER).

The first parameter of constructor is store and for implementing another method to add role to a user by identifier we need that.



public class ApplicationUserManager : UserManager<IdentityUser>
{
    private readonly UserStore<IdentityUser, IdentityRole, ApplicationDbContext, string, IdentityUserClaim<string>,
        IdentityUserRole<string>, IdentityUserLogin<string>, IdentityUserToken<string>, IdentityRoleClaim<string>> 
        _store;

    public ApplicationUserManager(
        IUserStore<IdentityUser> store,
        ...)
    {
        _store = (UserStore<IdentityUser, IdentityRole, ApplicationDbContext, string, IdentityUserClaim<string>,
            IdentityUserRole<string>, IdentityUserLogin<string>, IdentityUserToken<string>, IdentityRoleClaim<string>>)store;
    }
}

Before explaining why I cast the interface of _store to generic UserStore class with 9 parameters, let's look at the implementation AddToRoleAsync method (in such a case as good practice always check the source code of identity to get an idea to implement what you want)

Inside AddToRoleAsync implementation FindRoleAsync method is called and I checked FindRoleAsync implementation:

Inside FindRoleAsync Role property is used to fetch a role by name and again I check Role definition:

I noticed I cannot use the Role property to the get role by identifier because it's private property but I noticed there is a public property of DbContext that I can use. So that's why I cast IUserStor interface to UserStore class to have access to Context property.

It's time to implement AddToRoleAsync() and IsInRoleAync():



public virtual async Task<IdentityResult> AddToRoleByRoleIdAsync(IdentityUser user, string roleId)
{
    ThrowIfDisposed();

    if (user == null)
        throw new ArgumentNullException(nameof(user));

    if (string.IsNullOrWhiteSpace(roleId))
        throw new ArgumentNullException(nameof(roleId));

    if (await IsInRoleByIdAsync(user, roleId, CancellationToken))
        return IdentityResult.Failed(ErrorDescriber.UserAlreadyInRole(roleId));

    _store.Context.Set<IdentityUserRole<string>>().Add(new IdentityUserRole<string> { RoleId = roleId, UserId = user.Id });

    return await UpdateUserAsync(user);
}

public async Task<bool> IsInRoleByIdAsync(IdentityUser user, string roleId, CancellationToken cancellationToken = default(CancellationToken))
{
    cancellationToken.ThrowIfCancellationRequested();
    ThrowIfDisposed();

    if (user == null)
        throw new ArgumentNullException(nameof(user));

    if (string.IsNullOrWhiteSpace(roleId))
        throw new ArgumentNullException(nameof(roleId));

    var role = await _store.Context.Set<IdentityRole>().FindAsync(roleId);
    if (role == null)
        return false;

    var userRole = await _store.Context.Set<IdentityUserRole<string>>().FindAsync(new object[] { user.Id, roleId }, cancellationToken);
    return userRole != null;
}

I used _store.Context.Set<IdentityRole>() to find by a role by identifier.

Next step is to register ApplicationUserManager:



services.AddIdentity<IdentityUser, IdentityRole>(options => options.SignIn.RequireConfirmedAccount = false)
   .AddEntityFrameworkStores<ApplicationDbContext>()
   .AddUserManager<ApplicationUserManager>() // Add ApplicationUserManager
   .AddDefaultTokenProviders()
   .AddDefaultUI();

The last thing I want to mention is that if you have extended IdentityUser, IdentityRole, ... classes, replace them with your own implementation.

Some Docker commands make your life easier

Mohsen Esmailpour — Thu, 27 May 2021 18:16:51 +0000

In this post, I show some useful docker commands that you may be less familiar with them and they make your life easier as a developer. Lets begin with build command:

docker build -f path/to/file/Dockerfile -t image-name .

`-f` lets you specify the path to `Dockerfile`.

Tag or rename image

docker tag old-image-name new-image-name

You Can use image id rather than image name:

docker tag 0e5574283393 new-image-name

Save image:

docker save image-name -o output-image-name

If the image name contains `/`:

docker save image-name/image-name > output-image-name

Load image:

docker load --input output-image-name

Remove images that name starting with `myimage`

docker rmi $(docker image ls 'myimage*' -q)

Delete images whose names starting with `myimage`.

Delete all untagged images

docker rmi $(docker images -f "dangling=true" -q)

Delete all images (warning)

docker rmi -f $(docker images -q)

Start all stopped containers

docker start $(docker ps -aq)

Stop all running containers

docker stop $(docker ps -q)

Delete all containers

docker rm $(docker ps -aq)
or
docker container stop $(docker container ls –aq) && docker system prune –af ––volumes

Delete any resources — images, containers, volumes, and networks — that are dangling (warning)

docker system prune

Delete any stopped containers and all unused images (warning)

docker system prune -a

Copy file to container

docker cp ./path/to/file.ext container-name:/path/to/destination/inside/container

Build an image from running container

docker commit container-name image-name:tag

Mount file to a container

docker run -v /path/to/file/appsettings.json:/app/appsettings.json --name my-image image-name

Get into a Docker container's shell

docker exec -it container-name /bin/bash

Display volumes with mounted path

docker ps --format 'table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.Mounts}}'

Delete volumes whose names starting with `myvloume`.

docker volume rm $(docker volume ls --filter name=myvloume -q)

Delete dangling volumes

docker volume ls -f dangling=true

Build docker-compose from several files

docker-compose -f docker-compose.yml -f docker-compose.dev.yml build

Scale a service

docker-compose scale service-name=2

Command that contains $ could be executed on windows command prompt (cmd), use PowerShell instead

How to scale Hangfire with docker

Mohsen Esmailpour — Mon, 24 May 2021 21:05:56 +0000

Hangfire is an open-source framework that helps you to create, process and manage your background jobs and an easy way to perform fire-and-forget, delayed and recurring jobs inside .NET applications. This post does not cover the basics of Hanfgire so read this article to learn the basics of Hangfire.

This post covers:

How to config Hangfire to have multiple queues
How to config worker count
How to scale Hangfire worker service

I'm going to implement a sample project that contains a producer service that produces and exposes messages via web API and a consumer service which every second fetch messages from the producer service and enqueue messages into Hangfire. Each message has a different priority to process, so we enqueue each message to different queues based on priority and a worker service that processes messages and can be scaled out. You can skip steps 1-10 if you already have experience with Hangfire.

Producer Service

Imagine you need to call an external service and get a substantial amount of messages from that external service and queue messages to process. Our producer service mimics that external service behavior that provides messages.

Step 1 - Create a class library project and name it JobQueue.Shared
Step 2 - Add Message model class to the class library project ```csharp

public class MessageModel
{
public Guid MessageId { get; set; }

public DateTime CreateDate { get; set; }

public string Category { get; set; }

public object Payload { get; set; }

}

- Step 3 - Create ASP.NET Core Web API project and name it `JobQueue.ProducerService`
- Step 4 - Reference shared class library project to producer service project

- Step 5 - Create message producer

Let's create a message producer class that generates a random message. I used an excellent library [Bogus](https://github.com/bchavez/Bogus) to generate fake data for testing purposes.
```csharp


internal class MessageGenerator
{
    private static readonly string[] Categories = { "express", "normal" };
    private static readonly Faker<MessageModel> Faker;

    static MessageGenerator()
    {
        var random = new Random();

        Faker = new Faker<MessageModel>()
            .StrictMode(false)
            .RuleFor(p => p.Category, f => f.PickRandom(Categories))
            .RuleFor(p => p.MessageId, f => f.Random.Guid())
            .RuleFor(p => p.CreateDate, f => f.Date.Between(DateTime.Now.AddSeconds(-random.Next(1, 5)), DateTime.Now));
    }

    public static IEnumerable<MessageModel> GenerateMessages()
    {
        return Faker.Generate(100);
    }
}

Let's save generated message in a message store.



internal class MessageStore
{
    private readonly List<MessageModel> _store = new();
    private static readonly MessageStore _instance = new();

    private MessageStore()
    {
    }

    public static MessageStore Instance => _instance;

    public int Count => _store.Count;

    public void AddMessages(IEnumerable<MessageModel> messages)
    {
        _store.AddRange(messages);
    }

    public IEnumerable<MessageModel> GetMessages(int count)
    {
        var message = _store.Take(count).ToList();
        _store.RemoveRange(0, message.Count);

        return message;
    }
}

Let's create a background service that periodically generates messages and save them into messages. I used hosted service in ASP.NET Core to achieve this aim.



internal class MessageProducerHostedService : IHostedService, IDisposable
{
    private Timer _timer;

    public Task StartAsync(CancellationToken cancellationToken)
    {
        _timer = new Timer(SeedData, null, TimeSpan.Zero, TimeSpan.FromSeconds(1));

        return Task.CompletedTask;
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _timer?.Change(Timeout.Infinite, 0);

        return Task.CompletedTask;
    }

    public void Dispose()
    {
        _timer?.Dispose();
    }

    private void SeedData(object state)
    {
        if (MessageStore.Instance.Count > 2000)
            return;

        var messages = MessageGenerator.GenerateMessages();
        MessageStore.Instance.AddMessages(messages);
    }
}

Step 6 - Create an API to expose messages ```csharp

[ApiController]
[Route("api/v1/[controller]")]
public class MessagesController : ControllerBase
{
[HttpGet]
public IEnumerable Get()
{
return MessageStore.Instance.GetMessages(new Random().Next(50, 200));
}
}


####Consumer Service
- Step 7 - Create ASP.NET Core Web API project and name it `JobQueue.ConsumerService`

- Step 8 - Install `Hangfire.AspNetCore` and `HangFire.Redis.StackExchange` nuget packages

- Step 9 - Implement a background service that periodically fetches messages from the producer service and enqueue into a queue
```csharp


public class MessageReceiverHostedService : IHostedService
{
    private readonly CancellationTokenSource _cts;
    private readonly IServiceProvider _serviceProvider;

    public MessageReceiverHostedService(IServiceProvider serviceProvider, ILogger<MessageReceiverHostedService> logger)
    {
        _serviceProvider = serviceProvider;
        _cts = new CancellationTokenSource();
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        await Task.Factory.StartNew(() => FetchMessagesAsync(_cts.Token), cancellationToken);
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _cts.Cancel();

        return Task.CompletedTask;
    }

    private async Task FetchMessagesAsync(CancellationToken cancellationToken)
    {
        while (true)
        {
            using var scope = _serviceProvider.CreateScope();
            var httpClient = scope.ServiceProvider.GetRequiredService<JobHttpClient>();
            var messages = await httpClient.GetJobMessagesAsync(cancellationToken);

            if (!messages.Any())
                continue;

            var categories = messages.GroupBy(m => m.Category).ToList();

            Parallel.ForEach(categories, category =>
            {
                Enqueue(category.Key, category.ToList());
            });

            await Task.Delay(TimeSpan.FromSeconds(5), cancellationToken);

            if (cancellationToken.IsCancellationRequested)
                break;
        }
    }

    private void Enqueue(string queueName, List<MessageModel> messages)
    {
        var client = new BackgroundJobClient();
        var state = new EnqueuedState(queueName);

        foreach (var message in messages.OrderBy(o => o.CreateDate))
        {
            Expression<Action> action = queueName == "express"
                ? () => MessageProcessor.ProcessExpressMessageAsync(message, message.MessageId)
                : () => MessageProcessor.ProcessNormalMessageAsync(message, message.MessageId);
            client.Create(action, state);
        }
    }
}

Each message has a Category property that identifies the priority of it. We have two categories, express and normal and express has a higher priority. We need two queues for express and normal categories.

One more thing I want to mention is that in this background service I didn't use the timer. Read this article to find out more on this case.

Step 8 - Add new class MessageProcessor to JobQueue.Shared project ```csharp

public class MessageProcessor
{
[Queue("express")]
[DisplayName("JobId: {1}")]
[AutomaticRetry(Attempts = 3)]
public static async Task ProcessExpressMessageAsync(MessageModel message, Guid messageId)
{
await Task.Delay(TimeSpan.FromSeconds(new Random().Next(1, 4)));
}

[Queue("normal")]
[DisplayName("JobId: {1}")]
[AutomaticRetry(Attempts = 3)]
public static async Task ProcessNormalMessageAsync(MessageModel message, Guid messageId)
{
    await Task.Delay(TimeSpan.FromSeconds(new Random().Next(1, 4)));
}

}

We have to methods for processing messages from each queue. One limitation of Hangfire is that you cannot use a method to process jobs from multiple queues (or at least I'm not aware of it but it can be done by implementing a custom `Queue` attribute).

- Step 9 - Reference shared class library project to consumer service project

- Step 10 - Add Hangfire dashboard to consumer service
Hangfire has a dashboard that enables you to monitor the jobs and their statuses. It also allows you to manually trigger available jobs.
Open `Startup.cs` class and add Hangfire dependencies to `ConfigureServices` method:
```csharp


public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddHostedService<MessageReceiverHostedService>();

    services.AddHangfire(configuration => configuration
        .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
        .UseSimpleAssemblyNameTypeSerializer()
        .UseRecommendedSerializerSettings()
        .UseRedisStorage(_redis));
}

And in configure method:



public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    ...

    app.UseEndpoints(endpoints =>
    {
        endpoints.MapDefaultControllerRoute();
        endpoints.MapHangfireDashboard();
    });
}

Run producer project and navigate to <localhost>/hangfire to view the dashboard.

Worker Service

Step 11 - Create Worker Service project and name it JobQueue.WorkerService
Step 12 - Open appsettings.json file and the following configuration: ```json

"Hangfire": [
{
"QueueName": "express",
"WorkerCount": 5
},
{
"QueueName": "normal",
"WorkerCount": 2
}
],

Background jobs are processed by a [dedicated pool of worker](https://docs.hangfire.io/en/latest/background-processing/configuring-degree-of-parallelism.html) threads that run inside the Hangfire server subsystem. For each queue, we can config the number of workers.

- Step 13 - Add new class `HangfireQueueSetting` to bind configuration:
```csharp


internal class HangfireQueueSetting
{
    public string QueueName { get; set; }

    public int WorkerCount { get; set; }
}

Step 14 - Install Hangfire.AspNetCore and HangFire.Redis.StackExchange nuget packages
Step 15 - Open Program.cs class and add Hangfire dependencies ```csharp

public class Program
{
private static ConnectionMultiplexer _redis;

public static void Main(string[] args)
{
    CreateHostBuilder(args).Build().Run();
}

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureServices((hostContext, services) =>
        {
            _redis = ConnectionMultiplexer.Connect(hostContext.Configuration.GetConnectionString("RedisConnection"));

            services.AddHangfire(configuration => configuration
                .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
                .UseSimpleAssemblyNameTypeSerializer()
                .UseRecommendedSerializerSettings()
                .UseRedisStorage(_redis));

            var queueSettings = hostContext.Configuration.GetSection("Hangfire").Get<List<HangfireQueueSetting>>();
            foreach (var setting in queueSettings)
            {
                services.AddHangfireServer(options =>
                {
                    options.ServerName = $"{Environment.MachineName}:{setting.QueueName}";
                    options.Queues = new[] { setting.QueueName };
                    options.WorkerCount = setting.WorkerCount;
                });
            }
        });

}

- Step 16 - Reference shared class library project to producer service project

Now run all projects and navigate to Hangfire dashboard and you could see jobs are processing:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/916njnmii6sb6vqgsdu5.jpg)

Click on the `Servers` tab and you can see we have two queues and with different worker count:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pl2hl21uhaoob0yh82mo.jpg)

####Dockerizing Services
```yml


version: '3.4'

networks:
  service_network:

services:
  redis:
    image: "redis"
    ports:
      - 6379:6379
    networks:
      - service_network

  consumerservice:
    image: ${DOCKER_REGISTRY-}jobqueueconsumerservice
    container_name: consumerservice
    ports:
      - 9000:80
    networks:
      - service_network
    build:
      context: .
      dockerfile: JobQueue.ConsumerService/Dockerfile
    environment:
      - ConnectionStrings__RedisConnection=redis:6379
      - JobApi__BaseAddress=http://producerservice

  producerservice:
    image: ${DOCKER_REGISTRY-}jobqueueproducerservice
    container_name: producerservice
    build:
      context: .
      dockerfile: JobQueue.ProducerService/Dockerfile
    networks:
      - service_network

  workerservice:
    image: ${DOCKER_REGISTRY-}workerservice
    networks:
      - service_network
    build:
      context: .
      dockerfile: JobQueue.WorkerService/Dockerfile
    environment:
      - ConnectionStrings__RedisConnection=redis:6379
      - Hangfire__0__WorkerCount=10
      - Hangfire__1__WorkerCount=5

We can configure worker count via docker-compose file by passing values through environment:



Hangfire__0__WorkerCount=10

Let's run projects via docker-compose:

run docker-compose build
run docker-compose up -d
run docker-compose scale workerservice=2
To access the job dashboard, enter http://localhost:9000 address in the browser

You can find the source code for this walkthrough on Github.

EF Core multiple Database providers

Mohsen Esmailpour — Tue, 11 May 2021 15:13:04 +0000

Most of the time you don't need to have multiple database providers, for example, you start with the SQL Server provider and there is no need to switch to another database provider.
In my recent project, we have to add multiple providers to support different deployment environments. Some customers prefer to use Windows Server and SQL Server database and others prefer to use Linux and MySQL or PostgreSQL.

In this walkthrough, I show how to add SQL Server and PostgresSQL providers and configure them. The default provider is SQL Server and the alternative provider is PostgreSQL.

Step 1 - Create a project

Create ASP.NET Core Web API

Step 2 - Create Default DbContext

Add Data folder to the root of the project
Add a WeatherDbContext to Data folder:

public abstract class WeatherDbContext : DbContext
{
    protected readonly IConfiguration Configuration;

    protected WeatherDbContext(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public DbSet<WeatherForecast> WeatherForecasts { get; set; }
}

MsSqlDbContext is our default DbContext and we use this default DbContext every in code. MsSqlDbContext contains DbSets and shared functionalities that we want use among all DbContext classes.

Step 3 - Create SQL Server DbContext

Add a MsSqlDbContext to Data folder:

public class MsSqlDbContext : WeatherDbContext
{
    public MsSqlDbContext(IConfiguration configuration) : base(configuration)
    {
    }

    protected override void OnConfiguring(DbContextOptionsBuilder options)
    {
        options.UseSqlServer(Configuration.GetConnectionString("MsSqlConnection"));
    }
}

Instead of configuring SQL Server provider in ConfigureService method in Startup class:

services.AddDbContext<YourDbContext>(options => { options.UseSqlServer(Configuration.GetConnectionString("SomeConnection"); })

I configure provider in OnConfiguring method of MsSqlDbContext:

protected override void OnConfiguring(DbContextOptionsBuilder options)
{
    options.UseSqlServer(Configuration.GetConnectionString("MsSqlConnection"));
}

Step 4 - Create PosgreSQL DbContext

Add a PostgresDbContext class to Data folder:

public class PostgresDbContext : WeatherDbContext
{
    public PostgresDbContext(IConfiguration configuration)
        : base(configuration)
    {
    }

    protected override void OnConfiguring(DbContextOptionsBuilder options)
    {
        options.UseNpgsql(Configuration.GetConnectionString("PostgreSqlConnection"));
    }
}

Step 5 - Registring DbContexts

So far we have two DbContexts but we want to choose a preferred database provider in the configuration file appsettings.json.

Add following key/valu to appsettings.json:

"DatabaseProvider": "MsSql"

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();

    switch (Configuration["DatabaseProvider"])
    {
        case "MsSql":
            services.AddDbContext<WeatherDbContext,MsSqlDbContext>();
            break;

        case "PostgreSql":
            services.AddDbContext<WeatherDbContext,PostgresDbContext>();
            break;
    }
}

Step 6 - Migrations

Let's generate a migration for SQL Server database.

Add Migrations\MsSql folders to Data folder
In appsettings.json file set "MsSql" value for "DatabaseProvider" key
Run the following command for generating migration SQL Server database:

Add-Migration InitialDbMsSql -Context MsSqlDbContext -OutputDir Data\Migrations\MsSql

Let's generate a migration for PostgreSQL database.

Add Migrations\PostgreSql folders to Data folder
In appsettings.json file set "PostgreSql" value for "DatabaseProvider" key
Run the following command for generating migration SQL Server database:

Add-Migration InitialDbPostgres -Context PostgresDbContext -OutputDir Data\Migrations\PostgreSql

Step 4 - Entity Configuration

Here is the WeatherForecast class:

public class WeatherForecast
{
    public int Id { get; set; }

    public DateTime Date { get; set; }

    public int TemperatureC { get; set; }

    public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);

    public string Summary { get; set; }
}

Consider the following configuration:

internal class WeatherForecastConfiguration : IEntityTypeConfiguration<WeatherForecast>
{
    public void Configure(EntityTypeBuilder<WeatherForecast> builder)
    {
        builder.Property(wf => wf.TemperatureC).HasColumnType("tinyint");
        builder.Property(wf => wf.Date).HasColumnType("datetime");
        builder.Property(wf => wf.Summary).HasColumnType("nvarchar(512)").IsRequired(false);
        builder.Ignore(wf => wf.TemperatureF);
    }
}

In the above configuration, I used tinyint and nvarchar are supported in SQL Server but not supported in PostgreSQL. In such a case you can move the configuration to OnModelCreating method DbContext:

public class MsSqlDbContext : DbContext
{
    ...
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<WeatherForecast>().Property(wf => wf.TemperatureC).HasColumnType("tinyint");
        modelBuilder.Entity<WeatherForecast>().Property(wf => wf.Date).HasColumnType("datetime");
        modelBuilder.Entity<WeatherForecast>().Property(wf => wf.Summary).HasColumnType("nvarchar(512)").IsRequired(false);
    }
}

And for PostgresDbContext:

public class PostgresDbContext : WeatherDbContext
{
    ...
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<WeatherForecast>().Property(wf => wf.TemperatureC).HasColumnType("samllint");
        modelBuilder.Entity<WeatherForecast>().Property(wf => wf.Date).HasColumnType("timestamp(3)");
        modelBuilder.Entity<WeatherForecast>().Property(wf => wf.Summary).HasColumnType("varchar(512)").IsRequired(false);
    }
}

I prefer to move specific configuration from OnModelCreating to configuration classes and makeOnModelCreating method clean with help of reflection but you can keep it as it is.
To move configurations:

Add Configurations folder to Data folder for common configuration of database providers
Add WeatherForecastSharedConfiguration class to Configurations folder:

internal class WeatherForecastSharedConfiguration : IEntityTypeConfiguration<WeatherForecast>
{
    public void Configure(EntityTypeBuilder<WeatherForecast> builder)
    {
        builder.Ignore(wf => wf.TemperatureF);
    }
}

Add MsSql folder to Configurations folder
Add WeatherForecastMsSqlConfiguration class to MsSql folder:

internal class WeatherForecastMsSqlConfiguration : IEntityTypeConfiguration<WeatherForecast>
{
    public void Configure(EntityTypeBuilder<WeatherForecast> builder)
    {
        builder.Property(wf => wf.TemperatureC).HasColumnType("tinyint");
        builder.Property(wf => wf.Date).HasColumnType("datetime");
        builder.Property(wf => wf.Summary).HasColumnType("nvarchar(512)").IsRequired(false);
    }
}

Add PostgreSql folder to Configurations folder
Add WeatherForecastMsSqlConfiguration class to PostgreSql folder:

internal class WeatherForecastPostgresConfiguration : IEntityTypeConfiguration<WeatherForecast>
{
    public void Configure(EntityTypeBuilder<WeatherForecast> builder)
    {
        builder.Property(wf => wf.TemperatureC).HasColumnType("samllint");
        builder.Property(wf => wf.Date).HasColumnType("timestamp(3)");
        builder.Property(wf => wf.Summary).HasColumnType("varchar(512)").IsRequired(false);
    }
}

Add the following codes to WeatherDbContext (default DbContext) to load shared and provider-specific configuration:

protected void ApplyConfiguration(ModelBuilder modelBuilder, string[] namespaces)
{
    var methodInfo = (typeof(ModelBuilder).GetMethods()).Single((e =>
        e.Name == "ApplyConfiguration" &&
        e.ContainsGenericParameters &&
        e.GetParameters().SingleOrDefault()?.ParameterType.GetGenericTypeDefinition() == typeof(IEntityTypeConfiguration<>)));

    foreach (var configType in typeof(MsSqlDbContext).GetTypeInfo().Assembly.GetTypes()

        .Where(t => t.Namespace != null &&
                    namespaces.Any(n => n == t.Namespace) &&
                    t.GetInterfaces().Any(i => i.IsGenericType &&
                                               i.GetGenericTypeDefinition() == typeof(IEntityTypeConfiguration<>))))
    {
        var type = configType.GetInterfaces().First();
        methodInfo.MakeGenericMethod(type.GenericTypeArguments[0]).Invoke(modelBuilder, new[]
        {
            Activator.CreateInstance(configType)
        });
    }
}

In OnModelCreating method of MsSqlDbContext:

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    var namespaces = new[] { "EfMultipleProviders.Data.Configurations",
        "EfMultipleProviders.Data.Configurations.MsSql" };
    ApplyConfiguration(modelBuilder, namespaces);
}

Load shared configuration from EfMultipleProviders.Data.Configurations namespace and SQL Server configuration from EfMultipleProviders.Data.Configurations.MsSql namespace.

And for PsotgresDbContext:

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    var namespaces = new[] { "EfMultipleProviders.Data.Configurations",
        "EfMultipleProviders.Data.Configurations.PostgreSql" };
    ApplyConfiguration(modelBuilder, namespaces);
}

Do not use modelBuilder.ApplyConfigurationsFromAssembly() method to load configuration.
Inject WeatherDbContext into your repositories, services or controller not MsMsqlDbContext or PostgresSqlDbContext

You can find the source code for this walkthrough on Github.

Create a dashboard to visualize application logs in Kibana

Mohsen Esmailpour — Thu, 29 Apr 2021 20:21:46 +0000

In this article, I'm going to show you how to create a dashboard in Kibana to visualize application logs, and of course if you are using Elasticsearch to store your application logs.

Step 1- Setup Elasticsearch and Kibana

I use docker to run an instance of Elastic and Kibana.

Create docker-compose.yml file and add the following content:

version: "3.0"
services:
  elasticsearch:
    container_name: es-container
    image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
    environment:
      - xpack.security.enabled=false
      - "discovery.type=single-node"
    networks:
      - es-net
    ports:
      - 9200:9200
  kibana:
    container_name: kb-container
    image: docker.elastic.co/kibana/kibana:7.12.0
    environment:
      - ELASTICSEARCH_HOSTS=http://es-container:9200
    networks:
      - es-net
    depends_on:
      - elasticsearch
    ports:
      - 5601:5601
networks:
  es-net:
    driver: bridge

Run thedocker-compose up command Now you should have access to Elasticsearch via http://localhost:9200/ and Kibana through http://localhost:5601/

Step 2- Setup ASP.NET Core Web API project

This step is optional if you are not a .NET developer, just create an application store some logs into Elasticsearch.

Open Visual Studio and create a Web API project
Install Serilog.AspNetCore and Serilog.Sinks.Elasticsearch nuget packages
Open Program.cs file and modify CreateHostBuilder method and Add UseSerilog extension method:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        })
        .UseSerilog((hostingContext, loggerConfiguration) =>
            loggerConfiguration.ReadFrom.Configuration(hostingContext.Configuration));

Open appsettings.json file and get rid of the logging section:

"Logging": {
  "LogLevel": {
    "Default": "Information",
    "Microsoft": "Warning",
    "Microsoft.Hosting.Lifetime": "Information"
  }
}

Add Serilog configuration section to appsettings.json file:

"Serilog": {
  "MinimumLevel": {
    "Default": "Debug",
    "Override": {
      "Microsoft": "Information",
      "System": "Information"
    }
  },
  "Enrich": [ "FromLogContext" ],
  "WriteTo": [
    {
      "Name": "Elasticsearch",
      "Args": {
        "nodeUris": "http://localhost:9200",
        "indexFormat": "demo-api-{0:yyyy.MM}",
        "autoRegisterTemplate": true,
        "autoRegisterTemplateVersion": "ESv7"
      }
    }
  ]
}

Open Startup class and in the Configure method add the following code:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    ....
    app.UseSerilogRequestLogging();

Run the API project and make several API call through Swagger

Step 3- Create index in Kibana

In Kibana dashboard from the menu go to Management -> Stack Management
In Stack Management page and in Kibana section click on Index Patterns
In Index Pattern click on Create index pattern button
In Create index pattern page you should see your index pattern demo-api-2021.04 name (if your logs are saved successfully in Elasticsearch) and in Index pattern name input enter your index name demo-api-* and click Next button
In the next step in the Time field drop down select @timestamp and click on Create index pattern button
From the menu go to Analytics -> Discover page and select your index to view logs

Step 4- Create Visualizer

Before creating a dashboard you need to create several visualizers to add your log dashboard.
From the menu click on Analytics -> Visualize library then click on Create new visualization
From New visualization popup window click on Aggregation based (easiest way to create visualizer is using Lens but after creating visualizer aggregation based method, you can create other visualizers very quickly)

Total logs metric visualizer

Click on Metric visualizer
After choosing Metric visualizer then click on your index demo-api-*
By now I have 100 logs
You just need to save the visualizer, click on the save button name it Total logs

Log level metric visualizer

Follow the above steps and create another metric visualizer
In Buckets section click on Add then Split group
From Aggregation dropdown select Terms
After selecting Terms in Field dropdown select level.raw and then click the Update button at the bottom Now you have all log levels count and save the visualizer and name it Log levels count

Log level pie visualizer

Create a new visualizer and choose Pie
In Buckets section click on Add then Split slices
In Aggregation dropdown select Terms
In Field dropdown select level.raw
Click the Update button at the bottom
Save the visualizer name it Log levels percentage You can click on Options and change the appearance of the pie You can also change the color of each level by clicking on the level label

Vertical bar visualizer

Create a new visualizer and choose Vertical bar
In Buckets section click on Add then X-axis
In Aggregation dropdown select Date Histogram
Click the Update button at the bottom
Save the visualizer name it Total logs bar

Table visualizer

Create a new visualizer and choose Table
In Buckets section click on Add then Split rows
In Aggregation dropdown select Terms
In Field dropdown select fields.SourceContext.raw
Click the Update button at the bottom
Add another Split rows
In Aggregation dropdown select Terms
In Field dropdown select level.raw
Save the visualizer name it Log level sources

Error log level search

From the Analytic menu click on Discover
In KQL input enter level.raw:"Error
In Date filter choose Today
Save the search name it Today error level search

It's time to create a dashboard.

From the Analytic menu click on Dashboard
Click on Create new dashboard button
From the right sidebar click all visualizer we have created previously
From Types dropdown click on Saved search then click on Today error level search
Click on Save button and save the dashboard

In the end, you can drag and drop or resize visualizers and arrange visualizers as you like.

Pomodoro - A technique to rescue your eyes as a software developer

Mohsen Esmailpour — Thu, 22 Apr 2021 20:23:48 +0000

A career in software or computing can involve years of staring at a computer screen for extended periods. The consequences to your vision, which can affect not only your productivity as a software developer but also your quality of life in general, should be taken seriously.

The most important part of a software job is to work with computers all time and look at the screen consistently. The screen is bright and can cause eye troubles over a period of time. We stare at the screen all the time especially when we want to find a bug, searching for a solution to fix the bug, or improving the software and we do R&D a lot.

One of the best techniques you can do for your eyes when working on a computer is to remember the 20/20/20 rule – which means taking a break every 20 minutes and stare at an object 20 feet away for 20 seconds.

Another technique is Pomodoro. The Pomodoro Technique is a time management method developed by Francesco Cirillo in the late 1980s.[1] The technique uses a timer to break down work into intervals, traditionally 25 minutes in length, separated by short breaks. Each interval is known as a pomodoro, from the Italian word for 'tomato', after the tomato-shaped kitchen timer that Cirillo used as a university student.

The technique has been widely popularized by dozens of apps and websites providing timers and instructions. Closely related to concepts such as timeboxing and iterative and incremental development used in software design, the method has been adopted in pair programming contexts.

There are six steps in the original technique:

Decide on the task to be done.
Set the pomodoro timer (traditionally to 25 minutes).[1]
Work on the task.
End work when the timer rings and put a checkmark on a piece of paper.
If you have fewer than four checkmarks, take a short break (3–5 minutes) and then return to step 2; otherwise continue to step 6.
After four pomodoros, take a longer break (15–30 minutes), reset your checkmark count to zero, then go to step.

One of the best software for adopting Pomodoro is TeamViz Pomodoro app.

Since 2015 I,m using TeamViz Pomodoro app and I have logs of all my works until now.

TeamViz Pomodoro app has clients for Windows, Mac, iOS and Android and you can sync your work logs across all devices you are using.

After Download and install the app follow these steps:

Step 1: (optional) create a new named list, for example, “Sprint 1. market research”. The “list” here can also be a project or an epic in agile methodology.
Step 2: Add your tasks to the currently selected list. After input text, press “ENTER” to confirm.
Step 3: The default estimation to a task is 1 Pomodoro timer. You can change it by double-clicking the number.
Step 4: Hover onto the task, you’ll see some small tools. Hover your cursor onto the tool, there will be some tooltips. You can click the timer button to add it to today’s list. This is useful before you start your work every morning. You can take a review of existing tasks and add tasks by priority based on importance and level of urgency.
Step 5: After you marked a task as “to do today”, the tasks will appear on the lane of “To Do Today”, hover your cursor on the task there, there will be a small “play” button. Click the button there to start the timer. So once you made a plan for the whole day, you can now focus only on today’s tasks.

Step 6: Once the timer started, the main window will automatically minimize after 3 seconds. You can now focus on your tasks and avoid interruption in the Pomodoro timer.

At the end:

Make sure your workspace is properly lighted
Adjust the brightness and contrast of your screen
Remember to blink
Take breaks
Have yearly eye exams
Read this great article The definitive guide of software engineers for eye care

20 useful Visual Studio extensions

Mohsen Esmailpour — Tue, 13 Apr 2021 22:00:30 +0000

This article is about introducing several Visual Studio extensions that boost your development productivity and make your life as a developer easier (at least they made my life easier).

1- CodeMaid

CodeMaid is one of my favorite extensions that without it I really cannot write code. CodeMaid features are:

Now let's see some features in action.

CodeMaid can clean up an opened file after saving it. After installing CodeMain options:

In the Cleaning section click on General then check the Automatically run cleanup on file save.
Again in the Cleaning section click on Visual Studio then uncheck the Skip during automatic cleanup on save. Let's save the file:

After saving file CodeMaid:

Removed unused namespaces
Sorted namespaces alphabetically
Removed extra blank lines

Another thing I really like is inserting a single line between properties.

In the Cleaning section click on Insert then click on single line properties. After saving the file: Before inserting a single line between properties class was less readable.

CodeMaid can reorganize class member types in the following order:

After organizing untidy class:

Last but not least, I'm no fan of using #region and I always avoid using it but if you are using region a lot, CodeMaid can add reign name to the end of the region #endregion:

Watch this video to learn more about CodeMaid:

2- Open Command Line

Open Command Line adds a new command to the project context menu that will open a command prompt on the project's path. If the solution node is selection in Solution Explorer, then a console will open at the root of the .sln file.

You can access the command by hitting ALT+Space as well.
You may change this shortcut in the Options Window under Environment -> Keyboard
Look for the command ProjectAndSolutionContextMenus.Project.OpenCommandLine.Default Most of the time I use this extension to do Git staff via command line instead of the Visual Studio Git interface.

3- Solution Error Visualizer

Solution Error Visualizer adds error, warning, and message squiggles to Solution Explorer. The information is fetched from the Error List and directly displayed in the Solution Explorer tree view. Hovering over the nodes with squiggles brings up a pop-up with the error/warning/message information.

You can filter the Solution Explorer to just files with warnings/errors/messages using the filter dropdown in the Solution Explorer toolbar. You can create a new Solution Explorer view, filter it down to Errors and use it as a dedicated Error window.

4- Viasfora

Viasfora extension adds many useful features for code (C#, C/C++, JS, VB, F#, SQL, TypeScript, Python, PowerShell, R, JSON, Fortran) as well as editing XML files.

Rainbow Braces
Keyword Highlighting
XML Improvements

5- Add New File

Add New File is for easily adding new files to any project. Simply hit Shift+F2 to create an empty file in the selected folder or in the same folder as the selected file.

Easily create any file with any file extension
Create files starting with a dot like .gitignore
Create deeper folder structures easily if required
Create folders when the entered name ends with a /

6- VSColorOutput | Output Enhancer

VSColorOutput can change the color of a line emitted to the output window based on specified rules. The rules consist of regular expressions. Rules map to classifications which in turn map to colors.

Output Enhancer is an extension that adds styling to the Visual Studio output window. For now, it supports coloring for build/build order output:

Build/rebuild result messages
Publish result messages
Compiler errors and warnings
npm and bower errors and warnings

Debug output:

Trace warning and error messages
"Exception thrown" messages

7- Visual Studio Spell Checker

Visual Studio Spell Checker is a Visual Studio editor extension that checks the spelling of comments, strings, and plain text as you type or interactively with a tool window. It is based largely on the spell checker extension originally created by Noah Richards, Roman Golovin, and Michael Lehenbauer.

8- Learn the Shortcut

Learn the Shortcut shows how easy you can make the same action using only the keyboard. Displays the keyboard shortcut for any command that you execute to help you learn the shortcuts you need the most.
It is also helpful for presentations where you want to display to the audience what keyboard shortcuts you are using.

Status bar
Output Window

9- Open in Visual Studio Code

Open in Visual Studio Code is for those times where you have a project open in Visual Studio and you want to be able to quickly open it in Visual Studio Code.

10- Better Comments

Better Comments is a Visual Studio extension that gives you the ability to customize the font and opacity of your comments independently of the editor's font settings. It also adds four additional comment classifications, each classification with its own customizable foreground.

Additional comment classifications. Important, Question, Task, and Crossed.
Customizable foreground for each comment classification.
Customize the font settings and opacity of your comments.
Works with C#, F#, VB.NET, C/C++, JavaScript, Python, HTML, and XAML.
You can use your own custom tokens.

11- GhostDoc Community

GhostDoc is a Visual Studio extension for developers who need to Generate XML Comments from source code using customizable templates, maintain clean and up-to-date documentation, produce help documentation in multiple formats, use intelligent source code Spell Checker in Visual Studio, and more.
GhostDoc also helps to automate the generation of clean, helpful documentation of your clean code, building it, and packaging it up all without human intervention.

Automatically generate a starting point for your help documentation
Visual editing for your help docs
Produce help documentation in multiple formats
Find auto-generated comments - edit or remove the bulk created docs
Ensure code documentation quality
Maintain clean and up-to-date code documentation

12- Trailing Whitespace Visualizer

Trailing Whitespace Visualizer
will highlight and remove any trailing whitespace on any line in any editor in Visual Studio. This makes it really easy to get rid of those annoying invisible characters.
You can very easily delete all the trailing whitespace in a file by executing the Delete Horizontal White Space command found in Edit -> Advanced or by using the shortcut key combination Ctrl+K, Ctrl+\.
You can change the background color easily from the Tools -> Options -> Fonts and Colors menu. The item is called "Trailing Whitespace".

13- File Icons

File Icons adds icons for files that are not recognized by Solution Explorer

14- File Nesting

File Nesting automatically nest files based on the file name and enables developers to nest and unnest any file manually.

Manually nest files
Manually un-nest files
Auto-nesting based on naming conventions
Option to enable auto-nesting when files are added or renamed
Options to specify which naming conventions to apply
Keyboard shortcut for manual nesting (Ctrl+Alt+N)

15- Bundler and Minifier

Bundler and Minifier let's you configure bundling and minification of JS, CSS and HTML files.

Bundles CSS, JavaScript or HTML files into a single output file
Saving a source file triggers re-bundling automatically
Support for globbing patterns
MSBuild support for CI scenarios supported
Minify individual or bundled CSS, JavaScript and HTML files
Minification options for each language is customizable
Shows a watermark when opening a generated file
Task Runner Explorer integration
Command-line support
Shortcut to update all bundles in the solution
Suppress output file generation
Convert to Gulp

16- Image Optimizer

Image Optimizer
uses industry standard tools to optimize any JPEG, PNG and Gifs - including animated Gifs. It can do both lossy and lossless optimization.
Adds a right-click menu to any folder and image in Solution Explorer that let's you automatically optimize all PNG, GIF and JPEG files in that folder.

Optimizes PNGs (uses Zopfli compression)
Optimizes GIFs
Optimizes animated GIFs
Optimizes JPGs (uses MozJPEG)
Works on single images files or entire folders
Copy any image as base64 dataURI to clipboard

17- Color Themes for Visual Studio

Color Themes for Visual Studio installs additional color themes for Visual Studio.
Similar to the Color Theme Editor, but without the editor - just the themes.
Includes all the themes available in the Color Theme Editor, plus additional themes contributed by the community.

18- Indent Guides

Indent Guides displays indent and page width guides in Visual Studio text editor windows.
Guides can be displayed at the indent specified in your settings, regardless of tabs or spaces, or wherever text has been indented to. Page width markers can be displayed at a fixed location and will change color when code extends past them.

Here are three styles of guides: solid, dotted and dashed, available in thin and thick varieties and customizable color. The default is dotted grey, as shown in the image. Each indent level can have a different style and color.

19- Match Margin

Match Margin highlights all the textual matches of token at the caret location in the Editor.
As an example, when the focus is on a variable, all usages of that variable are highlighted and a marker is placed on the vertical scrollbar to indicate the position on the page where that variable is used.

20- WakaTime

WakaTime automatically generates metrics, insights, and time tracking from your programming activity.

Paid

1 - Resharper
2 - OzCode
3 - Supercharger
4 - NDepend
5 - GhostDoc Enterprise
6 - Tabs Studio

What every ASP.NET Core Web API project needs - Part 6 - IServiceCollection Extension

Mohsen Esmailpour — Wed, 07 Apr 2021 21:06:27 +0000

The 5th part of this series was about Polly and this article is about writing extension method for IServiceCollection. Honestly not every Web API project needs a service collection extension and if you are ok with the long and messy Startup class, you can finish up reading this article.

So far I have added and configured several packages to the cool-webapi project and in the future articles I will add more packages and configuration and the Startup class will become a large class.

Let's clean up ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers().AddDataAnnotationsLocalization();
    services.AddLocalization(options => options.ResourcesPath = "Resources");

    var supportedCultures = new List<CultureInfo> { new("en"), new("fa") };
    services.Configure<RequestLocalizationOptions>(options =>
    {
        options.DefaultRequestCulture = new RequestCulture("fa");
        options.SupportedCultures = supportedCultures;
        options.SupportedUICultures = supportedCultures;
    });

    services.Configure<RouteOptions>(options => { options.LowercaseUrls = true; });

    services.AddApiVersioning(options =>
    {
        // reporting api versions will return the headers "api-supported-versions" and "api-deprecated-versions"
        options.ReportApiVersions = true;
    });

    services.AddVersionedApiExplorer(options =>
    {
        // add the versioned api explorer, which also adds IApiVersionDescriptionProvider service
        // note: the specified format code will format the version as "'v'major[.minor][-status]"
        options.GroupNameFormat = "'v'VVV";

        // note: this option is only necessary when versioning by url segment. the SubstitutionFormat
        // can also be used to control the format of the API version in route templates
        options.SubstituteApiVersionInUrl = true;
    });

    services.AddTransient<IConfigureOptions<SwaggerGenOptions>, ConfigureSwaggerOptions>();
    services.AddSwaggerGen(options =>
    {
        // add a custom operation filter which sets default values
        options.OperationFilter<SwaggerDefaultValues>();
        options.OperationFilter<SwaggerLanguageHeader>();

        // JWT Bearer Authorization
        options.AddSecurityDefinition("Bearer", new OpenApiSecurityScheme
        {
            Description = "JWT Authorization header using the Bearer scheme. Example: \"Authorization: Bearer {token}\"",
            Name = "Authorization",
            In = ParameterLocation.Header,
            Type = SecuritySchemeType.ApiKey
        });
        options.AddSecurityRequirement(new OpenApiSecurityRequirement
                    {
            {
                new OpenApiSecurityScheme
                {
                    Reference = new OpenApiReference
                    {
                        Type = ReferenceType.SecurityScheme,
                        Id = "Bearer"
                    },
                    Scheme = "oauth2",
                    Name = "Bearer",
                    In = ParameterLocation.Header,
                },
                new List<string>()
            }
                    });

        var xmlFile = $"{Assembly.GetExecutingAssembly().GetName().Name}.xml";
        var xmlPath = Path.Combine(AppContext.BaseDirectory, xmlFile);
        options.IncludeXmlComments(xmlPath);
    });

    var weatherSettings = new WeatherSettings();
    Configuration.GetSection("WeatherSettings").Bind(weatherSettings);
    services.AddSingleton(weatherSettings);

    var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(10);

    services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
        .AddTransientHttpErrorPolicy(policy => policy.WaitAndRetryAsync(3, _ => TimeSpan.FromSeconds(2)))
        .AddTransientHttpErrorPolicy(policy => policy.CircuitBreakerAsync(6, TimeSpan.FromSeconds(5)))
        .AddPolicyHandler(request =>
        {
            if (request.Method == HttpMethod.Get)
                return timeoutPolicy;

            return Policy.NoOpAsync<HttpResponseMessage>();
        });
}

To get started add new class ServiceCollectionExtensions to the Extensions folder.

Let's move localization configuration from ConfigureServices to an extension method:

public static class ServiceCollectionExtensions
{
    public static IServiceCollection AddAndConfigureLocalization(this IServiceCollection services)
    {
        services.AddLocalization(options => options.ResourcesPath = "Resources");

        var supportedCultures = new List<CultureInfo> { new("en"), new("fa") };
        services.Configure<RequestLocalizationOptions>(options =>
        {
            options.DefaultRequestCulture = new RequestCulture("fa");
            options.SupportedCultures = supportedCultures;
            options.SupportedUICultures = supportedCultures;
        });

        return services;
    }
}

API versioning configuration:

public static IServiceCollection AddAndConfigureApiVersioning(this IServiceCollection services)
{
    services.Configure<RouteOptions>(options => { options.LowercaseUrls = true; });

    services.AddApiVersioning(options =>
    {
        // reporting api versions will return the headers "api-supported-versions" and "api-deprecated-versions"
        options.ReportApiVersions = true;
    });

    services.AddVersionedApiExplorer(options =>
    {
        // add the versioned api explorer, which also adds IApiVersionDescriptionProvider service
        // note: the specified format code will format the version as "'v'major[.minor][-status]"
        options.GroupNameFormat = "'v'VVV";

        // note: this option is only necessary when versioning by url segment. the SubstitutionFormat
        // can also be used to control the format of the API version in route templates
        options.SubstituteApiVersionInUrl = true;
    });

    return services;
}

Swagger:

public static IServiceCollection AddAndConfigureSwagger(this IServiceCollection services)
{
    services.AddTransient<IConfigureOptions<SwaggerGenOptions>, ConfigureSwaggerOptions>();
    services.AddSwaggerGen(options =>
    {
        // add a custom operation filter which sets default values
        options.OperationFilter<SwaggerDefaultValues>();
        options.OperationFilter<SwaggerLanguageHeader>();

        // JWT Bearer Authorization
        options.AddSecurityDefinition("Bearer", new OpenApiSecurityScheme
        {
            Description = "JWT Authorization header using the Bearer scheme. Example: \"Authorization: Bearer {token}\"",
            Name = "Authorization",
            In = ParameterLocation.Header,
            Type = SecuritySchemeType.ApiKey
        });
        options.AddSecurityRequirement(new OpenApiSecurityRequirement
        {
            {
                new OpenApiSecurityScheme
                {
                    Reference = new OpenApiReference
                    {
                        Type = ReferenceType.SecurityScheme,
                        Id = "Bearer"
                    },
                    Scheme = "oauth2",
                    Name = "Bearer",
                    In = ParameterLocation.Header,
                },
                new List<string>()
            }
        });

        var xmlFile = $"{Assembly.GetExecutingAssembly().GetName().Name}.xml";
        var xmlPath = Path.Combine(AppContext.BaseDirectory, xmlFile);
        options.IncludeXmlComments(xmlPath);
    });

    return services;
}

Weather HTTP client:

public static IServiceCollection AddAndConfigureWeatherHttpClient(this IServiceCollection services, IConfiguration configuration)
{
    var weatherSettings = new WeatherSettings();
    configuration.GetSection("WeatherSettings").Bind(weatherSettings);
    services.AddSingleton(weatherSettings);

    var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(10);

    services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
        .AddTransientHttpErrorPolicy(policy => policy.WaitAndRetryAsync(3, _ => TimeSpan.FromSeconds(2)))
        .AddTransientHttpErrorPolicy(policy => policy.CircuitBreakerAsync(6, TimeSpan.FromSeconds(5)))
        .AddPolicyHandler(request =>
        {
            if (request.Method == HttpMethod.Get)
                return timeoutPolicy;

            return Policy.NoOpAsync<HttpResponseMessage>();
        });

    return services;
}

And here is the ConfigureServices method after clean up :

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers().AddDataAnnotationsLocalization();

    services.AddAndConfigureLocalization();

    services.AddAndConfigureApiVersioning();

    services.AddAndConfigureSwagger();

    services.AddAndConfigureWeatherHttpClient(Configuration);
}

You can find the source code for this walkthrough on Github.

What every ASP.NET Core Web API project needs - Part 5 - Polly

Mohsen Esmailpour — Thu, 25 Mar 2021 13:02:11 +0000

The 4th part of this series was about Error Message Reusability and Localization and in this article, I'm going to show you how to add resilience and transient fault handling to HttpClient with Polly.

When you create an ASP.NET Core Web API project, there is a WeatherForecastController that returns a list of weather forecasts. I'm going to use external weather API to get real weather forecasts instead of returning static data.

Step 1 - External API

Go to the weatherapi.com
Create an account
After login, you find the API key at the dashboard

Step 2 - Typed HTTP Client

There are several ways to use HttpClient class in a project:

Basic usage
Named clients
Typed clients
Generated clients

Most of the time I Prefer to use typed client because of:

Provide the same capabilities as named clients without the need to use strings as keys
Provides IntelliSense and compiler help when consuming clients
Provide a single location to configure and interact with a particular HttpClient. For example, a single typed client might be used:
- For a single backend endpoint.
- To encapsulate all logic dealing with the endpoint
Work with DI and can be injected where required in the app

Let's create a typed client for weather external API:

Create a new folder HttpClients inside Infrastructure folder
Add new class WeatherHttpClient.cs to HttpClients folder
Open appsettings.json file and add the following key/values:

"WeatherSettings": {
  "ApiKey": "YOURKEY"
  "BaseUrl": "https://api.weatherapi.com",
  "NoDaysForecast": 5
}

Open WeatherHttpClient.cs file and create a class for API settings:

public class WeatherSettings
{
    public string ApiKey { get; set; }

    public string BaseUrl { get; set; }

    public int NoDaysForecast { get; set; }
}

Open Startup.cs class and inside ConfigureServices method bind and register weather settings:

var weatherSettings = new WeatherSettings();
Configuration.GetSection("WeatherSettings").Bind(weatherSettings);
services.AddSingleton(weatherSettings);

Inside WeatherHttpClient.cs file create an interface:

public interface IWeatherHttpClient
{
    Task<IEnumerable<WeatherForecast>> GetForecastAsync(string cityName);
}

Let's implement the above interface:

public class WeatherHttpClient : IWeatherHttpClient
{
    private readonly HttpClient _client;
    private readonly WeatherSettings _settings;

    public WeatherHttpClient(HttpClient client, WeatherSettings settings)
    {
        _client = client;
        _settings = settings;
        _client.BaseAddress = new Uri(_settings.BaseUrl);
        _client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    }

    public async Task<IEnumerable<WeatherForecast>> GetForecastAsync(string cityName)
    {
        var url = $"v1/forecast.json?key={_settings.ApiKey}&q={cityName}&days={_settings.NoDaysForecast}";
        var response = await _client.GetAsync(url);
        response.EnsureSuccessStatusCode();
        var content = await response.Content.ReadAsStringAsync();
        ...
    }
}

The response of API is really verbose and I'm not going to create an object for deserializing JSON instead I use an anonymous object to extract only data that I need:

public async Task<IEnumerable<WeatherForecast>> GetForecastsAsync(string cityName)
{
    ...
    var days = JsonSerializerExtensions.DeserializeAnonymousType(content, new
    {
        forecast = new
        {
            forecastday = new[]
            {
               new
               {
                   date = DateTime.Now,
                   day = new { avgtemp_c = 0.0, condition = new { text = "" } }
               }
            }
        }
    }).forecast.forecastday;

    return days.Select(d => new WeatherForecast
    {
        Date = d.date,
        Summary = d.day.condition.text,
        TemperatureC = (int)d.day.avgtemp_c
    });

    // Other way to deserialize json without creating anonymous object
    // To get more information see https://docs.microsoft.com/en-us/dotnet/api/system.text.json.jsonelement?view=net-5.0
    //dynamic result = JsonSerializer.Deserialize<ExpandoObject>(content);
    //var days = result.forecast.GetProperty("forecastday").EnumerateArray();
    //foreach (var day in days)
    //{
    //    var date = day.GetProperty("date").GetDateTime();
    //    var temp = day.GetProperty("day").GetProperty("avgtemp_c").GetDouble();
    //    var condition = day.GetProperty("day").GetProperty("condition").GetProperty("text").GetString();
    //}
}

Open Startup.cs class and register weather HTTP client service:

services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>();

Open WeatherForecastController and inject IWeatherHttpClient:

namespace CoolWebApi.Apis.V1.Controllers
{
    [ApiController]
    [ApiVersion("1.0")]
    [Route("api/v{version:apiVersion}/[controller]")]
    public class WeatherForecastController : ControllerBase
    {
        private readonly IWeatherHttpClient _weatherClient;

        public WeatherForecastController(IWeatherHttpClient weatherClient)
        {
            _weatherClient = weatherClient;
        }

        [HttpGet]
        [ProducesResponseType(StatusCodes.Status200OK)]
        [ProducesResponseType(StatusCodes.Status400BadRequest)]
        public async Task<IEnumerable<WeatherForecast>> Get(string city = "London")
        {
            return await _weatherClient.GetForecastsAsync(city);
        }

It's time to simulate an error when we call weather API. Hence we have no control over external API, I use Fiddler Everywhere to capture and fiddle HTTP request and response.

Step 3 - Using Fiddler

Download and install Fiddler Everywhere
Run Fiddler and call the api/v1/weatherforecast API As you can see Fiddler captured HTTP traffics but we cannot see the request/response of HTTPS calls because they are encrypted, however, Fiddler has a mechanism to decrypt HTTPS traffics.
On the menu click View->Preferences or hit ctrl+, to open Settings
In the Settings window click on the HTTPS tab and click on Trust root certificate (You can remove the certificate anytime by clicking on Remove root certificate)
Call the API again and now you can see decrypted request/response Let's filter captured traffic to get rid of other requests.
On the URL column click on vertical dots to open the filter window and enter localhost:5001 and api.weatherapi.com URLs and change And to Or: Now it's time to change the response of api.weatherapi.com API.
On the Live Traffic tab right-click on api.weatherapi.com row and from the menu click on Add new rule(1). In the Auto Responder tab click on the switch button to enable it(2) then click on the edit icon(3)
On the Rule Editor window clear Raw input and the following text then click on the Save button:

HTTP/1.1 503

Service is unavailable

Let's test the rule. Again right-click on api.weatherapi.com row and from the menu click on Reply->Reissue requests and now API returns 503 instead of 200 status code

Try to call API with the Swagger and you will receive an Internal Server Error

Step 4 - Installing Polly

Install Microsoft.Extensions.Http.Polly nuget package
Open Startup.cs class and in ConfigureServices method modify WeatherHttpClient registration to this:

services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.WaitAndRetryAsync(3, _ => TimeSpan.FromSeconds(4)));

Now WeatherHttpClient retries 3 times and waits 4 seconds between each retry. Let's try again and see how many API calls will be made by WeatherHttpClient:

Let's try one more time and during API call I will turn off Fiddler Auto Responder:

This time after turning off Auto Responder API returns the status code 200 and forecast result.
If you hover your mouse on the AddTransientHttpErrorPolicy method, you get noticed that PolicyBuilder handles the following error categories:

Network failures (as System.Net.Http.HttpRequestException)
HTTP 5XX status codes (server errors)
HTTP 408 status code (request timeout) For example, no retry will be made on HTTP error code 400.

Polly offers multiple resilience policies:

Retry
Circuit-breaker
Timeout
Bulkhead Isolation
Cache
Fallback
PolicyWrap

Failing fast is better than making users/callers wait. If the external service is down or seriously struggling, it's better to give that system a break. In this case we can chain multiple policies in Polly to give a break. Let's chain retry policy with circuit breaker policy.

services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => 
        policy.WaitAndRetryAsync(2, _ => TimeSpan.FromSeconds(2)))
    .AddTransientHttpErrorPolicy(policy => 
        policy.CircuitBreakerAsync(2, TimeSpan.FromSeconds(5)));

We want to retry 2 times and wait 2 seconds between each retry and after 4 failed retries, stop for 5 seconds.

More examples:

Different retry times:

services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.WaitAndRetryAsync(new[]
        {
            TimeSpan.FromSeconds(2),
            TimeSpan.FromSeconds(6),
            TimeSpan.FromSeconds(10)
        }));

This policy will delay 2 seconds before the first retry, 6 seconds before the second retry, and 10 seconds before the third retry.

Handle other status codes

var policy = HttpPolicyExtensions
  .HandleTransientHttpError()
  .OrResult(response => (int)response.StatusCode == 417) // RetryAfter
  .WaitAndRetryAsync(...);

As I mentioned earlier by default HandleTransientHttpError handles HttpRequestException, 5XX and 408 errors. The above policy can handle the 429 status code too.

Custom error handling logic:

services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.RetryAsync(3, onRetry: (exception, retryCount) =>
    {
        //Add logic to be executed before each retry
    }));

Retry forever until it succeeds

services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.RetryForeverAsync());

Advanced Circuit Breaker

services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => policy.AdvancedCircuitBreakerAsync(
        failureThreshold: 0.5,
        samplingDuration: TimeSpan.FromSeconds(10),
        minimumThroughput: 8,
        durationOfBreak: TimeSpan.FromSeconds(30)
    ));

Breaks the circuit for 30 seconds if 50% of above of incoming requests fails or a minimum of 8 faults occur within a 10 second period. The circuit is reset/closed after 30 seconds.

Fallback

services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddTransientHttpErrorPolicy(policy => 
        policy.FallbackAsync(new HttpResponseMessage(HttpStatusCode.RequestTimeout)));

The fallback technique helps to return a fallback value instead of an exception re-throw when faults keep occurring. This helps the system to ensure that it gracefully tries to keep the system stable when it detects a fallback value coming by.

Timeout

var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(10);
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddPolicyHandler(timeoutPolicy);

Timeout policy allows you to specify how long a request should take to respond and if it doesn’t respond in the time period you specify, the request will be canceled.

Dynamic selection strategy

var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(10);
var noOpPolicy = Policy.NoOpAsync().AsAsyncPolicy<HttpResponseMessage>();
services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddPolicyHandler(request => request.Method == HttpMethod.Get 
        ? retryPolicy 
        : noOpPolicy
);

You may want to define a strategy that will only apply to GET requests but not other HTTP verbs. In the above example, the timeout policy will only be used for GET requests.

Policy Registry

var registry = services.AddPolicyRegistry();
registry.Add("DefaultRetryStrategy", HttpPolicyExtensions.HandleTransientHttpError().WaitAndRetryAsync(...));
registry.Add("DefaultCircuitBreaker", HttpPolicyExtensions.HandleTransientHttpError().CircuitBreakerAsync(...));

// Or

var registry = new PolicyRegistry()
{
    { "DefaultRetryStrategy", HttpPolicyExtensions.HandleTransientHttpError().WaitAndRetryAsync(...) },
    { "DefaultCircuitBreaker", HttpPolicyExtensions.HandleTransientHttpError().CircuitBreakerAsync(...) }
};
services.AddSingleton<IReadOnlyPolicyRegistry<string>>(registry);

services.AddHttpClient<IWeatherHttpClient, WeatherHttpClient>()
    .AddPolicyHandlerFromRegistry("DefaultCircuitBreaker")
    .AddPolicyHandlerFromRegistry("DefaultCircuitBreaker");

Polly provides a policy registry, it is equivalent to the strategy storage center, the registered strategy allows you to reuse it in multiple locations in the application.

You can find the source code for this walkthrough on Github.

DEV Community: Mohsen Esmailpour

Transaction middleware in ASP.NET Core

Extending ProblemDetails - Add error code to ValidationProblemDetails

Leaky abstraction and clean architecture template

How to extend identity UserManager class

Some Docker commands make your life easier

-f lets you specify the path to Dockerfile.

Tag or rename image

You Can use image id rather than image name:

Save image:

If the image name contains /:

Load image:

Remove images that name starting with myimage

Delete images whose names starting with myimage.

Delete all untagged images

Delete all images (warning)

Start all stopped containers

Stop all running containers

Delete all containers

Delete any resources — images, containers, volumes, and networks — that are dangling (warning)

Delete any stopped containers and all unused images (warning)

Copy file to container

Build an image from running container

Mount file to a container

Get into a Docker container's shell

Display volumes with mounted path

Delete volumes whose names starting with myvloume.

Delete dangling volumes

Build docker-compose from several files

Scale a service

How to scale Hangfire with docker

Producer Service

Worker Service

EF Core multiple Database providers

Step 1 - Create a project

Step 2 - Create Default DbContext

Step 3 - Create SQL Server DbContext

Step 4 - Create PosgreSQL DbContext

Step 5 - Registring DbContexts

Step 6 - Migrations

Step 4 - Entity Configuration

Create a dashboard to visualize application logs in Kibana

Step 1- Setup Elasticsearch and Kibana

Step 2- Setup ASP.NET Core Web API project

Step 3- Create index in Kibana

Step 4- Create Visualizer

Total logs metric visualizer

Log level metric visualizer

Log level pie visualizer

Vertical bar visualizer

Table visualizer

Error log level search

Pomodoro - A technique to rescue your eyes as a software developer

20 useful Visual Studio extensions

1- CodeMaid

2- Open Command Line

3- Solution Error Visualizer

4- Viasfora

5- Add New File

6- VSColorOutput | Output Enhancer

7- Visual Studio Spell Checker

8- Learn the Shortcut

9- Open in Visual Studio Code

10- Better Comments

11- GhostDoc Community

12- Trailing Whitespace Visualizer

13- File Icons

14- File Nesting

15- Bundler and Minifier

16- Image Optimizer

17- Color Themes for Visual Studio

18- Indent Guides

19- Match Margin

20- WakaTime

Paid

What every ASP.NET Core Web API project needs - Part 6 - IServiceCollection Extension

What every ASP.NET Core Web API project needs - Part 5 - Polly

Step 1 - External API

Step 2 - Typed HTTP Client

Step 3 - Using Fiddler

`-f` lets you specify the path to `Dockerfile`.

If the image name contains `/`:

Remove images that name starting with `myimage`

Delete images whose names starting with `myimage`.

Delete volumes whose names starting with `myvloume`.