A first look at load balancing the Umbraco backoffice

When Umbraco 17 released there was a lot of focus on it being the first LTS version of Umbraco using the new web component based backoffice, and for a lot of people that was important enough in itself.

I must admit the only other feature I paid attention to is that dates now consistently have a timezone on them making it much easier to work with them.

However, one thing I somehow managed to miss completely was that as of Umbraco 17 you can now load balance the backoffice of Umbraco.

Isn't load balancing Umbraco an old feature?

Load balancing Umbraco sites has been around for longer than I have used Umbraco - so at least since v7!

It used to work a certain way as shown in this diagram based on the official documentation:

Source: Umbraco documentation

Which basically meant that what most people doing load balancing would do is spin up 2 Azure Web Apps, deploy the same code to both of them but have some settings that would designate one web app as the frontend app and one as the backend app.

Both apps would use the same database, and with some specific settings added you could allow the frontend app to scale out - for example by setting some rules that determines what amount of load the frontend should be under before it starts additional instances to automatically scale out.

This works pretty well, but also means that the minimal amount of infrastructure was 2 web apps - and if you had a lot of editors then there was no way to scale out the backend to accommodate extra load there.

Load balancing the backoffice

So the new thing in Umbraco 17 is that you can now load balance everything. One immediate effect that could have is that you now just need a single web app for a basic scale out site, something like this:

Shout out MermaidJS charts. AI agents are very good at translating described flows into charts, and it makes it so much easier to create visuals!

This setup could theoretically make it cheaper, and it's also easier to have a regular site running just 1 instance with the option of scaling out for big peaks in load without a bunch of extra setup!

So first you have to decide between the "old" way of load balancing with separate backend and frontend webapps and the "new" way of having a single scaleable app for both.

Secondly if choosing the "new" way of scaling a single app there is also a choice of whether the backoffice should use sticky sessions or be stateless.

There are a lot of articles about pros and cons for both:

• https://www.linode.com/docs/guides/configuring-load-balancer-sticky-session/
• https://www.imperva.com/learn/availability/sticky-session-persistence-and-cookies/
• https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-best-practices#check-your-distribution-mode

But in short the difference is how traffic is distributed:

Sticky sessions

• Once a request hits the load balancer it is assigned to an instance and a cookie is returned on the response that has a reference to that instance.
• Future requests pass the cookie along to the load balancer, and it ensures that the new request hits the same instance.
• If x amount of users cause the server resources to spike to the point that more instances are added based on the load balancing rules, you can end up having a lot of users "stick" to the initial instance due to their cookies.
• If the instance a user is "stuck" to gets removed during a scale-down, the load balancer assigns them to a new instance - which can cause problems (more on that below).

Stateless

• Once a request hits the load balancer it is assigned to the instance with the least load
• This ensures a much more equal distribution between instances
• This also makes things like sessions or SignalR connections a lot more complicated to manage

Setting up a backoffice load balanced site

The following setup applies to both sticky sessions and stateless, with the exception of SignalR which differs between the two and is covered separately at the end.

The assumption here is that the load balanced site will be hosted on Azure as a web app.
The current documentation will always be the most relevant resource: https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/load-balancing-backoffice

As of writing this for Umbraco 17, there are some settings that should be set no matter if it is a sticky session or stateless setup:

1. Set Umbraco:Cms:Examine:LuceneDirectoryFactory to TempFileSystemDirectoryFactory
2. Set Umbraco:Cms:Hosting:LocalTempStorageLocation to EnvironmentTemp
3. Set the Server role accessor & IsolatedCaches:

public class LoadBalancingComposer : IComposer
{    
    public void Compose(IUmbracoBuilder builder)
    {        
        builder.LoadBalanceIsolatedCaches();
        builder.SetServerRegistrar(new StaticServerAccessor());
    }
}

public class StaticServerAccessor : IServerRoleAccessor
{
    public ServerRole CurrentServerRole => ServerRole.SchedulingPublisher;
}

Note: In the traditional setup you'd have one SchedulingPublisher and multiple Subscribers. With the new backoffice load balancing, check the current documentation for how server roles should be assigned, as this may differ from what you're used to.

Session management

If you use sticky sessions then it may seem like nothing is needed for managing sessions as you "stick" to one instance and that instance will handle your session just like on a non-loadbalanced site.

However, if you consider this scenario:

1. You run a webshop with automatic scaling set up & sticky sessions
2. You have a big sale, and the site scales up from 1 to 4 instances
3. A user is assigned to the 4th instance and starts adding things to their basket (which is stored as a session)
4. While that user is browsing traffic lowers, and the automatic scaling scales down to 3 instances
5. The user tries to add a new item to their basket - but the API controller request now hits the load balancer with a sticky session cookie to an instance that has been removed.
6. The load balancer assigns them to a new instance, which has no knowledge of their previous session. Their basket now only contains the item they just added - everything else is gone.

The same thing would happen for auth cookies as they use Data Protection - by default each instance generates its own encryption keys, meaning cookies encrypted on one instance can't be decrypted by another. So similarly to the example above, users may need to log in again each time they are moved to a new instance.

Dependent on your site this may not be a big problem, but if you use anything with sessions / auth cookies then it is very bad user experience.

Setting up out of process sessions and data protection

The solution to both the session and auth cookie problem is to move them out of the individual instances and into a shared store.

This is needed regardless of whether you use sticky sessions or stateless load balancing:

• With sticky sessions, users are normally pinned to one instance - but as shown in the scenario above, scaling events can force them onto a different instance. Without shared sessions and data protection, that means lost session data and forced re-authentication.
• With stateless load balancing, every single request can land on a different instance, making shared sessions and data protection essential for even basic functionality.

In this example we'll use Redis, but any IDistributedCache implementation would work.

You'll need the following NuGet packages:

• Microsoft.Extensions.Caching.StackExchangeRedis
• Microsoft.AspNetCore.DataProtection.StackExchangeRedis

There are three things we need to set up:

1. A shared distributed cache

builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = redisConnectionString;
    options.InstanceName = "my-site-";
});

This registers Redis as the IDistributedCache implementation. By default, ASP.NET Core uses an in-memory implementation which is exactly what causes the problem - each instance has its own cache. By pointing this at a shared Redis instance, all app instances share the same cache.

The InstanceName is used as a key prefix in Redis, which is useful if you share a Redis instance across multiple applications.

As a bonus, Umbraco's content and media cache is built on Microsoft's HybridCache - which automatically uses any registered IDistributedCache as a second-level cache. So by registering Redis here, Umbraco will also use it to share its content and media cache across instances.

2. Session configuration

builder.Services.AddSession(options =>
{
    options.IdleTimeout = TimeSpan.FromMinutes(30);
    options.Cookie.HttpOnly = true;
    options.Cookie.IsEssential = true;
    options.Cookie.SecurePolicy = CookieSecurePolicy.Always;
});

ASP.NET Core sessions use whatever IDistributedCache is registered - so because we registered Redis above, sessions are now automatically stored in Redis instead of in-memory.

The cookie settings are worth noting:

• HttpOnly prevents JavaScript from accessing the session cookie
• IsEssential ensures the cookie is set even if the user hasn't consented to non-essential cookies
• SecurePolicy = Always ensures the cookie is only sent over HTTPS

3. Shared Data Protection keys

var redisConnection = ConnectionMultiplexer.Connect(redisConnectionString);
builder.Services.AddDataProtection()
    .SetApplicationName("my-umbraco-site")
    .PersistKeysToStackExchangeRedis(redisConnection, "DataProtection-Keys");

This is what solves the auth cookie problem. By default, each instance generates its own Data Protection keys - meaning a cookie encrypted on instance 1 can't be decrypted by instance 2.

By persisting the keys to Redis and setting a shared ApplicationName, all instances use the same encryption keys. The ApplicationName is important - instances must share the same application name to be able to decrypt each other's cookies.

SignalR

This is the final piece of the puzzle, and the only one where the setup actually differs depending on whether you're using sticky sessions or stateless.

Umbraco uses SignalR in the backoffice for real-time client-to-server communication outside of standard HTTP requests - for example for preview functionality. In a load balanced setup, SignalR needs a way to send messages across all instances, not just the one the user is connected to. This is called a backplane.

Sticky sessions

With sticky sessions, client-side SignalR connections work as normal - the user has a persistent connection to one instance, just like on a single-server site. The only thing needed is a backplane so that server-side messages (like "this content was just published") can reach all instances.

Redis works well for this:

// Requires: Microsoft.AspNetCore.SignalR.StackExchangeRedis
builder.Services.AddSignalR().AddStackExchangeRedis(redisConnectionString);

Stateless

With stateless load balancing, each request can land on a different instance. SignalR relies on a persistent connection between client and server - and that connection can't just "jump" to a new instance on every request. So a Redis backplane alone isn't enough here.

The solution is to use a managed SignalR service like Azure SignalR Service, which handles both the backplane and the client-side connections externally:

// Requires: Microsoft.Azure.SignalR
builder.Services.AddSignalR().AddAzureSignalR(azureSignalRConnectionString);

This replaces the need for the Redis backplane entirely for SignalR - Azure SignalR Service handles everything. Note that you still need Redis for the distributed cache, sessions, and data protection as set up above.

Wrapping up

The stateless approach does give a smoother distribution of load across instances, but Azure SignalR Service adds a significant cost on top of the Redis you already need for everything else.

In most cases I would go with the sticky session setup - the lower cost and the simplicity of having everything managed through a single Redis instance makes it easy to set up and maintain. And once sessions and data protection are out of process, sticky sessions handle instance swaps gracefully anyway.

That said, I haven't done any load testing to compare the two approaches. If they scale differently under real-world traffic, that could easily be the biggest factor in choosing one over the other. This post is really just meant to highlight the differences and walk through the setup - not to declare a winner. Try both if you can, and pick what fits your situation.