DEV Community: Mahesh More

Hangfire Jobs -Background processing with DotNet core

Mahesh More — Sat, 28 Jan 2023 15:04:32 +0000

What's Hangfire?

Hangfire is an open-source library that allows you to kick-start the process in the background so you don't need a separate process or windows service for processing. It's also backed with persistent storage so you can pause, resume, stop and restart failed jobs at any point and it will keep track of activity in the persistent database, and everything is stored in storage which helps background jobs to survive application restart, server reboot, etc.

This library consists of three main components: Client, Storage, and Server. Below diagram will give you a high-level view of how these three components work together to complete the particular background process.

Overview diagram:

Types of background jobs

Hangfire supports several kinds of background tasks, below is the list of jobs that you can create using Hangfire. For more information about each type click here.

Fire-And-Forget
Delayed
Recurring
Continuation
Batches
Batch Continuation
Background process

In this article, we will be implementing the Hangfire server in the DotNet core application and SQL as persistent storage. So let's start with actual implementations. Here we are implementing recurring jobs, but you can follow the same project for other types of jobs too.

Recurring Jobs:

Create a new DotNet core application using Visual Studio:

Either you can create a new web application using VS or using a command prompt.

Install Hangfire:

Hangfire is available as a NuGet package so you can install the below packages from the NuGet feed.

Hangfire.Core
Hangfire.SqlServer
Hangfire.AspNetCore

dotnet add package Hangfire.Core
dotnet add package Hangfire.SqlServer
dotnet add package Hangfire.AspNetCore

Now we have all the required Hangfire packages so the next step is to prepare persistent storage, here I'm assuming that you have SQL Server installed and configured properly for use.

Create a new Database:

As I mentioned earlier we will be using SQL Server as job storage, so connect to the local instance of the SQL server using the SSMS tool and create the database using the below command.

CREATE DATABASE [Hangfire]
GO

Configure Hangfire Database in WebApp:

Open appsettings.json file and add a connection string.

{
  "ConnectionStrings": {
      "Hangfire": "Server=.;Database=Hangfire;Integrated Security=SSPI;"
    }
  .
  .
}

I have created the app using DotNet Core 6 so I don't have Startup.cs file so I will be configuring hangfire in Program.cs file, if you have DotNet 3.1 application you can use a similar configuration in Startup.cs. First, we need to import the required namespaces.

//...
using Hangfire;
using Hangfire.SqlServer;

By default DotNet core supports dependency injection and Hangfire.AspNetCore the NuGet package contains all the extension methods to register/configure Hangfire Server.

builder.Services.AddHangfire(config => config
                                .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
                                .UseSimpleAssemblyNameTypeSerializer()
                                .UseRecommendedSerializerSettings()
                                .UseSqlServerStorage(builder.Configuration.GetConnectionString("Hangfire"), new SqlServerStorageOptions
                                {
                                    CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
                                    SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
                                    QueuePollInterval = TimeSpan.Zero,
                                    UseRecommendedIsolationLevel = true,
                                    DisableGlobalLocks = true
                                })
                            );

builder.Services.AddHangfireServer();

Adding Dashboard UI:

Hangfire does have a super cool feature where it provides you a dashboard and you can enqueue, dequeue, re-queue, and many more operations you can perform from this UI. The following code snippet shows you how to add a dashboard UI in the web app, so you can use all features from the user-friendly UI itself and no need to remember complex commands.

app.UseEndpoints(endpoints =>
{
    endpoints.MapHangfireDashboard();
});

So finally your Program.cs file will look like the one below.

using Hangfire;
using Hangfire.SqlServer;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddControllers();

//Configure Hangfire

builder.Services.AddHangfire(config => config
                                .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
                                .UseSimpleAssemblyNameTypeSerializer()
                                .UseRecommendedSerializerSettings()
                                .UseSqlServerStorage(builder.Configuration.GetConnectionString("Hangfire"), new SqlServerStorageOptions
                                {
                                    CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
                                    SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
                                    QueuePollInterval = TimeSpan.Zero,
                                    UseRecommendedIsolationLevel = true,
                                    DisableGlobalLocks = true
                                })
                            );

builder.Services.AddHangfireServer();

var app = builder.Build();

app.UseHttpsRedirection();

app.UseRouting();

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllers();
    endpoints.MapHangfireDashboard();
});

app.Run();

Run the app:

Now we have everything configured to run the application so either you can press F5or run the app using the command dotnet run.

SQL Database Schema:

In step 3 we have created separate DB for Hangfire jobs and in step 4 configured the connection in appsettings.json file, so as soon as we run the application it will initialize the SQL table schema. You can find the list of tables in the below image.

Hangfire Dashboard:

Now our application is started, open the following URL to access the Hangfire Dashboard interface.

https://localhost:8057/hangfire

By default, the hangfire server runs on port 8057and dashboard can be served on /hangfire endpoint path but you can configure this endpoint path while registering the dashboard in Program/Startup.cs file. The following snippet will configure a custom path for accessing Dashboard.

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllers();
    endpoints.MapHangfireDashboard("/dashboard");
});

Now we have everything configured to run Hangfire Server but still, we don't have any recurring job configured that's why on Recurring Jobs tab we are seeing 0 jobs on the dashboard, so let's configure one and see how we can queue and re-queue jobs from UI.

Create a new project:

We can create new recurring jobs in the same project as well but just to keep everything isolated I have created new class library project within the same solution.

Add UserMigration job:

Now create IUsersMigration interface and UsersMigration class which will have an actual implementation of the job.

IUsersMigration.cs

  public interface IUsersMigration
    {
        Task ExecuteAsync();
    }

UsersMigration.cs I have created this job for demo purposes so it contains basic implementation but you can implement your own logic, but I believe you got an idea about how we can create a new job.

  public class UsersMigration : IUsersMigration
    {
        public Task ExecuteAsync()
        {
            Console.WriteLine("Successfully completed!");

            return Task.CompletedTask;
        }
    }

So our entire solution structure will look like the below.

In Program.cs file you have to register all the jobs with cronschedule. The following snippet will register recurring jobs as Never but you can change that schedule as per your requirement.

RecurringJob.AddOrUpdate<IUsersMigration>(x => x.ExecuteAsync(), Cron.Never);

Now your final Program.cs file will look as below.

using Hangfire;
using Hangfire.SqlServer;
using Workbench.Hangfire.Jobs.Migrations.Interfaces;
using Workbench.Hangfire.Jobs.Migrations.Jobs;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddControllers();

//Configure Hangfire

builder.Services.AddHangfire(config => config
                                .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
                                .UseSimpleAssemblyNameTypeSerializer()
                                .UseRecommendedSerializerSettings()
                                .UseSqlServerStorage(builder.Configuration.GetConnectionString("Hangfire"), new SqlServerStorageOptions
                                {
                                    CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
                                    SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
                                    QueuePollInterval = TimeSpan.Zero,
                                    UseRecommendedIsolationLevel = true,
                                    DisableGlobalLocks = true
                                })
                            );

builder.Services.AddHangfireServer();

// Configure job's dependencies
builder.Services.AddScoped<IUsersMigration, UsersMigration>();

var app = builder.Build();

app.UseHttpsRedirection();

app.UseRouting();

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllers();
    endpoints.MapHangfireDashboard("/dashboard");
});

// Register job
RecurringJob.AddOrUpdate<IUsersMigration>(x => x.ExecuteAsync(), Cron.Never);

app.Run();

Enqueue job:

You can select as many jobs as you wanted to enqueue and click on Trigger now button, as soon as you trigger the jobs it will maintain the information in persistent storage.

To check the status you can visit the Jobs tab.

Here you can filter the list of jobs by their status so in the above image we are in Succeeded tab so all successful jobs will be listed here.

You can also check the status from SQL Server as well.

SELECT * FROM [HangFire].[Job];

GitHub Repo link.

Conclusion

Hangfire is an open-source library and is free to use for commercial use. It also provides you with a dashboard so non-technical users can use the dashboard to queue, re-queue, and delete scheduled jobs from the server. It also supports authentication to view the dashboard so no need to worry about anonymous user access. Hangfire supports SQL, Redis, and In-Memory as persistent storage so jobs can survive application restarts and even machine reboots, etc.

Different ways to implement IHttpClientFactory in .NET core apps

Mahesh More — Mon, 28 Feb 2022 10:32:34 +0000

Why not HttpClient?

We have been using the HttpClient class for REST API calls and I am not saying this is bad practice but most of the times developer use it wrongly and it has some limitations which cause serious issues, I have noted a few of them:

Even we dispose of instance underlying socket doesn't release immediately, which leads to socket exhaustion.

using(var client = new HttpClient())
{
    //do something with HTTP client
}

Do you think creating an instance using the using block will work? The answer is it won't work. If you are curious about what is the current state or wanted to confirm that the underline socket is closed or not then run the below command and it will show a list of connections that are in a wait or established state.

> netstat -na | find 'Port_Number'

To avoid socket exhaustion if you create a singleton or static instance throughout the application lifetime then it's not the ultimate solution because when you reuse the instance which means you are reusing the connection until the socket is closed. So the connection won't get updates from the server, effectively it will fail to handle DNS changes when we switch from staging to production environment. You can find more details here.
Creating an instance inside the using directive is not good practice as it tries to dispose of instances of HttpClient along with that it will dispose of the HttpClientHandler as well. Under the hood, HttpClient uses the HttpClientHandler for actual socket connection, but underhood implementation is out of scope for this article but I will create a separate detailed article on it.

To address the above issue .Net core introduced the IHttpClientFactory interface which we can use for HttpClient instance creation using DI.

Different ways to use IHttpClientFactory:

1. Basic usage:

You can register IHttpClientFactory using AddHttpClient extension method.
Program.cs or Startup.cs

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddHttpClient();

TestController.cs: DI will inject the dependency in the constructor.

class TestController : Controller
{
   private readonly IHttpClientFactory  _factory;

   public TestController(IHttpClientFactory factory)
   {
     _factory = factory;
   }

   [HttpGet]
   public async Task<IActionResult> Get()
   {
     var httpClient = _factory.CreateClient();
   }
}

2. Named Clients:

If you need a different configuration for each HttpClient then you can use named client DI and get an instance using the same name. Let's say you need to interact with 2 third-party APIs using HTTP REST calls and both need to be configured differently using the named client as below.

Startup.cs:

builder.Services.AddHttpClient("ClientA", httpClient =>
{
    httpClient.BaseAddress = new Uri("https://apiA.com/");
    //ToDo more configuration
});

builder.Services.AddHttpClient("ClientB", httpClient =>
{
    httpClient.BaseAddress = new Uri("https://apiB.com/");
    //ToDo more configuration
});

TestController.cs:

var httpClientA = _factory.CreateClient("ClientA");

var httpClientB = _factory.CreateClient("ClientB");

3. Typed Clients

It provides the same capabilities as the named client but instead of identifying the client based on the key it actually intelligently resolve HttpClient in a specific HttpHandler class. Another advantage of this approach is you don't need to create HttpClient manually using IHttpClientFactory.CreateClient() method.

Let's say you decided to write one custom HttpHandler which would be the single source for every REST call, so your implementation would look something like below.

HttpCommands.cs:

public class HttpCommands
{
  private readonly HttpClient _httpClient;

  public HttpCommands(HttpClient httpClient)
  {
    _httpClient = httpClient;
  }
}

Look at the constructor we are injecting HttpClient itself and not IHttpClientFactory, which means _factory.CreateClient() is not required anymore because .Net core is smart enough to resolve the dependency.

Startup.cs: To register typed HttpClient using generic version of AddHttpClient() extension method.

services.AddHttpClient<HttpCommands>();

//Or if you need custom configuration you can configure that too as we did for the named client

services.AddHttpClient<HttpCommands>(client =>
{
  client.BaseAddress = new Uri("https://apiA.com/");
});

Conclusion

What are the issues in traditional HttpClient usage?
What are the issues with the static or singleton HttpClient instance.
What are the different ways to use the IHttpClientFactory.

Happy Coding!

Looking for a way to initialize IOptions manually?

Mahesh More — Tue, 22 Feb 2022 11:41:55 +0000

Background

Ok ... I know what you are thinking you must have thought that DotNet Core's dependency injection handles it for you and we just need to register configuration class then what is the need for manual initialization? And I completely agree with you but sometimes you may need to create an instance of a class that is dependent on IOptions dependency (take an example of Unit test cases) and then real struggle starts.

Let's say we have ValueController.cs and it's accepting one parameter of type IOptions<AppSettings>

public class ValueController : Controller
{
   public ValueController(IOptions<AppSettings> options)
   {
   }
}

Also, we have some app configuration in appsettings.json file.

{
   "AppSettings": {
       "SettingOne": "ValueOne"
   }
}

Issue

If you try to pass an instance of AppSettings class to ValueController then DotNet core will not allow it.

var settings = new AppSettings { SettingOne = "ValueOne" };
var obj = new ValueController(settings);

Even it will not allow you to explicitly cast the instance to IOptions<AppSettings> type. Take a look at the below code which will not work. The below code will throw a runtime exception.

var settings = (IOptions<AppSettings>) new AppSettings { SettingOne = "ValueOne" };
var obj = new ValueController(settings);

Then how to get it fixed?

Solution

Luckily DotNet core provides a static method to create an object of IOptions<T> type.

var settings = Options.Create(new AppSettings { SettingOne = "ValueOne" });
var obj = new ValueController(settings);

Done! You are all set to initialize IOptions<T> manually.

Happy Coding!

Identity Server 4 with .NET core app

Mahesh More — Wed, 15 Jul 2020 09:19:07 +0000

What is Identity Server?

Identity Server 4 (IdS4) is an OpenID Connect and OAuth 2.0 framework for .NET core application. It's an authentication service that provides you centralized authentication logic for different types of applications(Web, Mobile, or Services).

Implement IdS4 in ASP.NET Core Web app

First, you need to create an empty ASP.NET Core web app using the below command.

dotnet new web

Alternatively, you can achieve the same from Visual Studio by choosing the ASP.NET Core Web Application project using an empty template.

Now, let's add IdS4 by installing the NuGet package.

dotnet add package IdentityServer4

You have successfully installed the IdS4 package, now open your project's Startup.cs file and add the below code in ConfigureServices() function. Please keep in mind that the below sample code is just referring to an empty list, so you need to make sure that you have a valid list of In-Memory resources and clients in your app.

services.AddIdentityServer()
                .AddInMemoryClients(new List<Client>())
                .AddInMemoryApiResources(new List<ApiResource>())
                .AddInMemoryIdentityResources(new List<IdentityResource>())
                .AddInMemoryPersistedGrants()
                .AddTestUsers(new List<TestUser>())
                .AddDeveloperSigningCredential();

The above code will include IdS4 dependency and now you need to update Configure method with below code snippet.

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {            
            app.UseHttpsRedirection();            
            app.UseIdentityServer();
        }

Let's understand the configuration, we have added IdS4 dependency using .AddIdentityServer() with default signing certificate using .AddDeveloperSigningCredential() extension method. Then we have updated Configure() method with .UseIdentityServer() method here we are actually enabling OpenIdConnect endpoints. Please see below list of endpoints provided by IdS4:

connect/token
connect/authorize
connect/userinfo
connect/endsession
connect/revocation
connect/introspect
connect/deviceauthorization

You can get a list of available endpoints using /.well-known/openid-configuration endpoint.

The above list of endpoints are provided by IdS4/OpenIdConnect/OAuth2 framework. In case you need your own endpoint for your business need, yes you can definitely create your custom endpoint!

Follow below steps to add custom endpoint:

services.AddIdentityServer(options =>
{
 //Adding custom endpoint in the discovery document          
 options.Discovery.ExpandRelativePathsInCustomEntries = true;
 options.Discovery.CustomEntries = new Dictionary<string, object>
             {
                { "myCustomEndpoint", "connect/myCustomEndpoint"}
              };
})
.AddEndpoint<MyCustomEndpoint>("myCustomEndpoint", "connect/myCustomEndpoint");

Implement Handler:

The above code adds a custom endpoint in IdS4's endpoints, now you need to write a handler class for actual implementation.

using IdentityServer4.Hosting;

public class MyCustomEndpoint : IEndpointHandler
    {
       public async Task<IEndpointResult> ProcessAsync(HttpContext 
                                                        context)
        {
         // ToDo: Here you can add your custom business-specific 
                  logic
        }
    }

How to use these endpoints?

You can use these endpoints to get access/refresh/identity token from the IdS4 token provider, each endpoint serves a different purpose. e.g. connect/authorize endpoint used in a public-facing application where you can use IdS4 login screen for authentication (using implicit grant type). Similarly connect/token endpoint provides you access token programmatically (using password grant type).

Request for Password/ResourceOwner grant type:

POST /connect/token

Headers:
Content-Type: application/x-www-form-urlencoded

Body:
grant_type=password&scope=api1&client_id=testClient&client_secret=testSecret&username=test.user&password=testPassword

The above request will provide you accessToken, so now you can use this access token to pass along with your REST API request.

How to protect your API using IdS4?

You can protect your existing API or create a new one using dotnet new webapi command.

To protect your API you need to install below NuGet package.

dotnet add package IdentityServer4.AccessTokenValidation

This NuGet provides JWT and Reference token validation middleware, for reference token validation it provides caching as well. To validate your access token you will need to add below code in ConfigureService method:

services.AddAuthentication("Bearer")
    .AddIdentityServerAuthentication("Bearer", options =>
    {
        options.ApiName = "api1";
        options.Authority = "https://localhost:44385";
    });

Here Authority is the IdentityServer4's URL and ApiName is the Audience from the access token and API resource name from IdS4 configuration.

To add IdS4 authentication middleware you need to update your Configure method with the below code.

public void Configure(IApplicationBuilder app)
{
    app.UseAuthentication();
    app.UseAuthorization();
}

If you look at the business need Authenticated user isn't always authorized to access all resources. So to allow access to the authorized users you can implement policy-based authorization. e.g. In case you have decided to authorize a user if the user's accessToken contains specific scope then you can create a simple policy using .AddPolicy() method. You need to update the ConfigureServices method with the below sample code.

Startup.cs

services.AddAuthorization(option =>
            {
                option.AddPolicy("MyPolicy", p =>
               {
                   p.RequireAuthenticatedUser();
                   p.RequireClaim(JwtClaimTypes.Scope, 
                      new List<string>
                        {
                         "api1.read",
                         "api1.write:
                        }
                      );
               });
            });

Now your API is protected by IdS4 authentication provider, so any endpoint decorated with [Authorize] attribute is protected. To call these REST endpoints you need to pass an accessToken using the Authorization header.

GET /api/v1.0/getuser

Headers:
Authorization: Bearer <AccessToken>

Note: We have implemented/integrated IdS4 in .Net project but we haven't introduced the user interface yet, so to add UI for the IdS4 project you can copy-paste code from QuickStart folder.

In this blog, I am using In-Memory clients and users, which is definitely not acceptable for a production app so you can use entity framework to update/get clients/users/resources from SQL DB or you can create your own storage library.

Happy coding!

Picture Source: Identity Server 4 docs