Background Processing with Hangfire

Introduction

In developing applications, it is essential to ensure a smooth and responsive user experience. This is particularly important in web development where client-server communication can cause enough delay. However, some tasks, such as sending emails, processing data, media conversion or performing maintenance operations, can be time-consuming, impacting the user's interaction with your application. Background processing is important to maintain a smooth user experience by handling such time-consuming tasks in the background.

Hangfire is a powerful tool in the ASP.NET Core ecosystem for handling background processes.

Why Background Processing?

Background processing is needed for the following scenarios:

• Handling long-running processes so that the main app execution is not blocked
• To maintain smooth user interaction
• To keep execution load small: executing only as many tasks as possible per time
• For batch processing
• To handle scheduled jobs
• For notification systems
• For queue processing
• Maintenance and cleanup tasks
• Retry and error handling

Examples of tasks where background processing is needed include: payment notification, report generation, temporary file cleanup, file processing e.g. media conversion, IoT sensor data processing, search indexing, resume parsing for job boards, task scheduling, thumbnail generation, etc.

What is Hangfire?

Hangfire is an open source library for .NET that enables tasks be processed in the background in a flexible and reliable way. This allows time-consuming or repetitive tasks be offloaded from the main thread of the application, maintaining a seamless user experience.

It works with any .NET application and allows background processes be run with no substantial changes. It works with several database systems, supporting RDBMS and NoSQL. It comes with a dashboard that allows monitoring of jobs and queues. It is also scalable, allowing Hangfire servers be added or removed easily as required. Hangfire, by default, retries tasks: when a task fails, it is automatically enqueued again. This can also be manually done via the dashboard.

A major advantage of Hangfire is that it works within a .NET application, removing the need for a separate service.

Features of Hangfire

• Easy Integration: Hangfire seamlessly integrates with ASP.NET Core applications, making it easy to adopt in your existing projects.
• Persistent Storage: Jobs and tasks are stored in a persistent storage (such as a database), ensuring that scheduled tasks are not lost during application restarts.
• Dashboard for Monitoring: Hangfire provides a web-based dashboard that allows you to monitor background jobs, track their progress, and manage recurring tasks.
• Recurring Jobs: Schedule jobs to run at specific intervals or on a recurring basis, automating repetitive tasks.
• Support for Different Job Types: Hangfire supports various types of jobs, including fire-and-forget, delayed, recurring, batch, and continuations.

Hangfire Job Types

There are different types of jobs supported by Hangfire.

• Fire-and-forget
• Delayed
• Recurring
• Continuations
• Batches
• Batch continuations
• Background processes

Fire-and-Forget

This is a job that happens once. It is added to the queue and executed once.

var jobId = BackgroundJob.Enqueue( () => Console.WriteLine(“One Time!")); 

Delayed

A delayed job is scheduled for a specific time. Helpers are available to make scheduling easy.

var jobId = BackgroundJob.Schedule( () => Console.WriteLine(“Wait for it!"), TimeSpan.FromDays(7)); 

Recurring

A recurring job happens repeatedly according to a determined schedule. Hangfire uses CRON to schedule recurring jobs.

RecurringJob.AddOrUpdate( "myrecurringjob", () => Console.WriteLine("Recurring!"), Cron.Daily); 

Demo Project: Media Watermarking Application

This application applies a watermark to a media file. It allows the user to upload a media file (image, pdf or video) and a watermark image and creates a watermarked version of the file.

Because this could be a resource-intensive process as the size of the media file may vary, the processing is handled in the background. The user is able to view the status of the task and, when processing is complete, the user is given a link to download the result file.

Here is a sketch of how it works:

The code is available on Github at https://github.com/shegzee/Watermarker.

The media processing job is enqueued once the user uploads the media files. This is a fire-and-forget job.

In addition to this, a job is scheduled to clean out all files and jobs older than 30 minutes (or as configured in appsettings). This is a recurring job.

I will include the relevant code in this post.

Program.cs

using HangfireWatermarker.Data;
using HangfireWatermarker.Jobs;
using HangfireWatermarker.Repositories;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
using Hangfire;
using Hangfire.SqlServer;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddRazorPages();

builder.Services.AddControllers();

// Add services to the container.
builder.Services.AddDbContext<AppDbContext>(options =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("DefaultConnection")).EnableSensitiveDataLogging(),
    ServiceLifetime.Transient
);
builder.Services.AddScoped<IJobStatusRepository, SqlJobStatusRepository>();
builder.Services.AddScoped<WatermarkJob>();

builder.Services.AddHangfire(config => config.UseSqlServerStorage(builder.Configuration.GetConnectionString("HangfireConnection")));
builder.Services.AddHangfireServer();



var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseHangfireDashboard();

RecurringJob.AddOrUpdate<CleanupJob>(Guid.NewGuid().ToString(), j => j.CleanExpiredJobs(), Cron.MinuteInterval(app.Configuration.GetValue<int>("Settings:CleanupJobInterval", 30)));

app.UseRouting();

app.UseAuthorization();

app.MapControllers();

app.MapRazorPages();

app.Run();

We can see that the CleanupJob is added just before the program starts. This is the code for the job.

CleanupJob.cs

using HangfireWatermarker.Repositories;
using Microsoft.Extensions.Configuration;

namespace HangfireWatermarker.Jobs
{
    public class CleanupJob
    {
        private readonly IJobStatusRepository _jobStatusRepository;
        private readonly IConfiguration _configuration;

        public CleanupJob(IJobStatusRepository jobStatusRepository, IConfiguration configuration)
        {
            _jobStatusRepository = jobStatusRepository;
            _configuration = configuration;
        }

        public void CleanExpiredJobs()
        {
            DateTime expiryTime = DateTime.Now;
            var jobExpiryMinutes = _configuration.GetValue<int>("Settings:JobExpiryMinutes", 30);
            var outdatedJobItems = _jobStatusRepository.GetOutdatedJobItems(jobExpiryMinutes, expiryTime);

            var inputFilesDirectory = _configuration.GetValue<string>("FileDirectories:Input");
            var outputFilesDirectory = _configuration.GetValue<string>("FileDirectories:Output");

            foreach (var jobItem in outdatedJobItems)
            {
                var inputFile = Path.Combine(inputFilesDirectory, jobItem.InputFileName);
                var watermarkFile = Path.Combine(inputFilesDirectory, jobItem.WatermarkFileName);
                var outputFile = Path.Combine(outputFilesDirectory, jobItem.ResultFileName);
                File.Delete(inputFile);
                File.Delete(watermarkFile);
                File.Delete(outputFile);
            }
            _jobStatusRepository.DeleteOutdatedJobItems(jobExpiryMinutes, expiryTime);
        }
    }
}

The main job to watermark the media files is added when the files are uploaded.

Index.cshtml

@page
@using HangfireWatermarker.Models;
@model IndexModel
@{
    ViewData["Title"] = "Home page";
}

<h2>File Processing</h2>

<form method="post" enctype="multipart/form-data">
    <div class="form-group">
        <label for="inputfile">Choose a File:</label>
        <input type="file" id="inputfile" name="inputfile" class="form-control" required />
        <label for="watermarkfile">Choose a Watermark Image:</label>
        <input type="file" id="watermarkfile" name="watermarkfile" class="form-control" required />
    </div>
    <button type="submit" class="btn btn-primary">Submit</button>
</form>

<h3>Job Status</h3>
<div class="table-striped">
@foreach (var job in Model.JobStatusList)
{
    <div class="d-sm-table-row">
        Job: @job.InputFileName - 
        <span class="">Status: @job.Status</span> 
        @if (job.Status == JobStatus.Completed)
        {
            <a href="Download?jobId=@job.JobId">Download Result</a>
        }
    </div>
}
</div>

Index.cshtml.cs

public IActionResult OnPost()
{
    if (InputFile != null && WatermarkFile != null)
    {
        try
        {
            if (!IsValidFileType(InputFile) || !IsValidFileSize(InputFile))
            {
                TempData["ErrorMessage"] = "Invalid file type or size.";
                return RedirectToPage("/Index");
            }

            if (!IsValidWatermarkFileType(WatermarkFile) || !IsValidWatermarkFileSize(WatermarkFile))
            {
                TempData["ErrorMessage"] = "Invalid watermark file type or size.";
                return RedirectToPage("/Index");
            }

            var inputFilesDirectory = _configuration.GetValue<string>("FileDirectories:Input");
            if (!Directory.Exists(inputFilesDirectory))
            {
                Directory.CreateDirectory(inputFilesDirectory);
            }

            var originalFileName = InputFile.FileName;
            var inputFilePath = Path.Combine(inputFilesDirectory, UniqueFileName(InputFile.FileName));
            using (var stream = new FileStream(inputFilePath, FileMode.Create))
            {
                InputFile.CopyTo(stream);
            }

            // copy uploaded watermark image file (?)
            //var watermarkFilePath = Path.GetTempFileName();
            var watermarkFilePath = Path.Combine(inputFilesDirectory, UniqueFileName(WatermarkFile.FileName));
            using (var stream = new FileStream(watermarkFilePath, FileMode.Create))
            {
                WatermarkFile.CopyTo(stream);
            }

            var jobId = System.Guid.NewGuid();

            var sanitizedFileName = Path.GetFileName(inputFilePath);
            var sanitizedWatermarkFileName = Path.GetFileName(watermarkFilePath);

            // Update job status to enqueued
            _jobStatusRepository.AddJobItem(new JobItem
            {
                JobId = jobId,
                InputFileName = sanitizedFileName,
                WatermarkFileName = sanitizedWatermarkFileName,
                Status = JobStatus.Enqueued,
                CreatedOn = DateTime.Now,
            });

            // output file path
            var outputFilesDirectory = _configuration.GetValue<string>("FileDirectories:Output");
            if (!Directory.Exists(outputFilesDirectory))
            {
                Directory.CreateDirectory(outputFilesDirectory);
            }

            var outputFileNameWithoutExtension = "watermarked_" + originalFileName;
            var fileExtension = GetFileExtension(sanitizedFileName);
            var timestamp = DateTime.Now.ToFileTime();
            var outputFilePath = Path.Combine(outputFilesDirectory, $"{UniqueFileName(outputFileNameWithoutExtension)}");

            BackgroundJob.Enqueue(() => _watermarkJob.Process(jobId, fileExtension, inputFilePath, watermarkFilePath, outputFilePath));

            return RedirectToPage("/Index");
        }
        catch (System.Exception ex)
        {
            Console.WriteLine(ex.ToString());
            TempData["ErrorMessage"] = "An error occurred while processing the file.";
            return RedirectToPage("/Index");
        }
    }

    TempData["ErrorMessage"] = "No file selected.";
    return RedirectToPage("/Index");
}

The function, BackgroundJob.Enqueue(), adds the job to Hangfire.

The class, WatermarkJob.cs, contains methods to add the watermark for the different types of input media files. These are called by the Process method based on the input file type.

...
    public class WatermarkJob
    {
        private readonly IJobStatusRepository _jobStatusRepository;

        public WatermarkJob(IJobStatusRepository jobStatusRepository)
        {
            _jobStatusRepository = jobStatusRepository;
        }

        public void Process(Guid jobId, string fileExtension, string filePath, string watermarkPath, string outputPath)
        {
            try
            {
                var jobItem = _jobStatusRepository.GetJobItem(jobId);
                jobItem.Status = JobStatus.Processing;
                // Update job status to processing
                _jobStatusRepository.UpdateJobItem(jobItem);
                var originalFileName = jobItem.InputFileName;
                var fileType = GetFileType(fileExtension);

                // Apply watermark based on file type
                if (fileType.Equals("other", StringComparison.OrdinalIgnoreCase))
                {
                    throw new FileLoadException("Unsupported File Type");
                }

                CreateOutputFile(outputPath);

                if (fileType.Equals("pdf", StringComparison.OrdinalIgnoreCase))
                {
                    ApplyWatermarkToPdf(filePath, outputPath, watermarkPath);
                }
                else if (fileType.StartsWith("image", StringComparison.OrdinalIgnoreCase))
                {
                    ApplyWatermarkToImage(filePath, outputPath, watermarkPath);
                }
                else if (fileType.StartsWith("video", StringComparison.OrdinalIgnoreCase))
                {
                    ApplyWatermarkToVideo(filePath, outputPath, watermarkPath);
                }


                jobItem = _jobStatusRepository.GetJobItem(jobId);
                jobItem.ResultFileName = Path.GetFileName(outputPath);
                jobItem.Status = JobStatus.Completed;

                // Update job status to completed
                _jobStatusRepository.UpdateJobItem(jobItem);
            }
            catch (Exception ex)
            {
                var jobItem = _jobStatusRepository.GetJobItem(jobId);

                // Update job status to failed
                jobItem.Status = JobStatus.Failed;
                _jobStatusRepository.UpdateJobItem(jobItem);

                // let Hangfire know it failed
                throw new InvalidOperationException("Job Failed/n" + ex.Message);
            }
        }

        // other methods are implemented
        ...

    }

There are other files to handle download, the user interface, etc. The rest of the source code can be seen in Github

Conclusion

Hangfire is a reliable tool for background task processing. I personally like the fact that it requires minimal setup and requirements as you can see in the "Watermarker" example, as it plugs into my .NET application and uses any database system. This removes the need for a separate queue or a separate service as it works asynchronously within the .NET application.

With Hangfire, handling background processes is a breeze, as this example shows. If you haven't explored Hangfire yet, now is the time to use it to take your ASP.NET applications to the next level.

(This post was originally posted on LinkedIn)