DEV Community: Jesper Mayntzhusen

A first look at load balancing the Umbraco backoffice

Jesper Mayntzhusen — Thu, 12 Mar 2026 21:06:17 +0000

When Umbraco 17 released there was a lot of focus on it being the first LTS version of Umbraco using the new web component based backoffice, and for a lot of people that was important enough in itself.

I must admit the only other feature I paid attention to is that dates now consistently have a timezone on them making it much easier to work with them.

However, one thing I somehow managed to miss completely was that as of Umbraco 17 you can now load balance the backoffice of Umbraco.

Isn't load balancing Umbraco an old feature?

Load balancing Umbraco sites has been around for longer than I have used Umbraco - so at least since v7!

It used to work a certain way as shown in this diagram based on the official documentation:

Source: Umbraco documentation

Which basically meant that what most people doing load balancing would do is spin up 2 Azure Web Apps, deploy the same code to both of them but have some settings that would designate one web app as the frontend app and one as the backend app.

Both apps would use the same database, and with some specific settings added you could allow the frontend app to scale out - for example by setting some rules that determines what amount of load the frontend should be under before it starts additional instances to automatically scale out.

This works pretty well, but also means that the minimal amount of infrastructure was 2 web apps - and if you had a lot of editors then there was no way to scale out the backend to accommodate extra load there.

Load balancing the backoffice

So the new thing in Umbraco 17 is that you can now load balance everything. One immediate effect that could have is that you now just need a single web app for a basic scale out site, something like this:

Shout out MermaidJS charts. AI agents are very good at translating described flows into charts, and it makes it so much easier to create visuals!

This setup could theoretically make it cheaper, and it's also easier to have a regular site running just 1 instance with the option of scaling out for big peaks in load without a bunch of extra setup!

So first you have to decide between the "old" way of load balancing with separate backend and frontend webapps and the "new" way of having a single scaleable app for both.

Secondly if choosing the "new" way of scaling a single app there is also a choice of whether the backoffice should use sticky sessions or be stateless.

There are a lot of articles about pros and cons for both:

But in short the difference is how traffic is distributed:

Sticky sessions

Once a request hits the load balancer it is assigned to an instance and a cookie is returned on the response that has a reference to that instance.
Future requests pass the cookie along to the load balancer, and it ensures that the new request hits the same instance.
If x amount of users cause the server resources to spike to the point that more instances are added based on the load balancing rules, you can end up having a lot of users "stick" to the initial instance due to their cookies.
If the instance a user is "stuck" to gets removed during a scale-down, the load balancer assigns them to a new instance - which can cause problems (more on that below).

Stateless

Once a request hits the load balancer it is assigned to the instance with the least load
This ensures a much more equal distribution between instances
This also makes things like sessions or SignalR connections a lot more complicated to manage

Setting up a backoffice load balanced site

The following setup applies to both sticky sessions and stateless, with the exception of SignalR which differs between the two and is covered separately at the end.

The assumption here is that the load balanced site will be hosted on Azure as a web app.
The current documentation will always be the most relevant resource: https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/load-balancing-backoffice

As of writing this for Umbraco 17, there are some settings that should be set no matter if it is a sticky session or stateless setup:

Set Umbraco:Cms:Examine:LuceneDirectoryFactory to TempFileSystemDirectoryFactory
Set Umbraco:Cms:Hosting:LocalTempStorageLocation to EnvironmentTemp
Set the Server role accessor & IsolatedCaches:

public class LoadBalancingComposer : IComposer
{    
    public void Compose(IUmbracoBuilder builder)
    {        
        builder.LoadBalanceIsolatedCaches();
        builder.SetServerRegistrar(new StaticServerAccessor());
    }
}

public class StaticServerAccessor : IServerRoleAccessor
{
    public ServerRole CurrentServerRole => ServerRole.SchedulingPublisher;
}

Note: In the traditional setup you'd have one SchedulingPublisher and multiple Subscribers. With the new backoffice load balancing, check the current documentation for how server roles should be assigned, as this may differ from what you're used to.

Session management

If you use sticky sessions then it may seem like nothing is needed for managing sessions as you "stick" to one instance and that instance will handle your session just like on a non-loadbalanced site.

However, if you consider this scenario:

You run a webshop with automatic scaling set up & sticky sessions
You have a big sale, and the site scales up from 1 to 4 instances
A user is assigned to the 4th instance and starts adding things to their basket (which is stored as a session)
While that user is browsing traffic lowers, and the automatic scaling scales down to 3 instances
The user tries to add a new item to their basket - but the API controller request now hits the load balancer with a sticky session cookie to an instance that has been removed.
The load balancer assigns them to a new instance, which has no knowledge of their previous session. Their basket now only contains the item they just added - everything else is gone.

The same thing would happen for auth cookies as they use Data Protection - by default each instance generates its own encryption keys, meaning cookies encrypted on one instance can't be decrypted by another. So similarly to the example above, users may need to log in again each time they are moved to a new instance.

Dependent on your site this may not be a big problem, but if you use anything with sessions / auth cookies then it is very bad user experience.

Setting up out of process sessions and data protection

The solution to both the session and auth cookie problem is to move them out of the individual instances and into a shared store.

This is needed regardless of whether you use sticky sessions or stateless load balancing:

With sticky sessions, users are normally pinned to one instance - but as shown in the scenario above, scaling events can force them onto a different instance. Without shared sessions and data protection, that means lost session data and forced re-authentication.
With stateless load balancing, every single request can land on a different instance, making shared sessions and data protection essential for even basic functionality.

In this example we'll use Redis, but any IDistributedCache implementation would work.

You'll need the following NuGet packages:

There are three things we need to set up:

1. A shared distributed cache

builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = redisConnectionString;
    options.InstanceName = "my-site-";
});

This registers Redis as the IDistributedCache implementation. By default, ASP.NET Core uses an in-memory implementation which is exactly what causes the problem - each instance has its own cache. By pointing this at a shared Redis instance, all app instances share the same cache.

The InstanceName is used as a key prefix in Redis, which is useful if you share a Redis instance across multiple applications.

As a bonus, Umbraco's content and media cache is built on Microsoft's HybridCache - which automatically uses any registered IDistributedCache as a second-level cache. So by registering Redis here, Umbraco will also use it to share its content and media cache across instances.

2. Session configuration

builder.Services.AddSession(options =>
{
    options.IdleTimeout = TimeSpan.FromMinutes(30);
    options.Cookie.HttpOnly = true;
    options.Cookie.IsEssential = true;
    options.Cookie.SecurePolicy = CookieSecurePolicy.Always;
});

ASP.NET Core sessions use whatever IDistributedCache is registered - so because we registered Redis above, sessions are now automatically stored in Redis instead of in-memory.

The cookie settings are worth noting:

HttpOnly prevents JavaScript from accessing the session cookie
IsEssential ensures the cookie is set even if the user hasn't consented to non-essential cookies
SecurePolicy = Always ensures the cookie is only sent over HTTPS

3. Shared Data Protection keys

var redisConnection = ConnectionMultiplexer.Connect(redisConnectionString);
builder.Services.AddDataProtection()
    .SetApplicationName("my-umbraco-site")
    .PersistKeysToStackExchangeRedis(redisConnection, "DataProtection-Keys");

This is what solves the auth cookie problem. By default, each instance generates its own Data Protection keys - meaning a cookie encrypted on instance 1 can't be decrypted by instance 2.

By persisting the keys to Redis and setting a shared ApplicationName, all instances use the same encryption keys. The ApplicationName is important - instances must share the same application name to be able to decrypt each other's cookies.

SignalR

This is the final piece of the puzzle, and the only one where the setup actually differs depending on whether you're using sticky sessions or stateless.

Umbraco uses SignalR in the backoffice for real-time client-to-server communication outside of standard HTTP requests - for example for preview functionality. In a load balanced setup, SignalR needs a way to send messages across all instances, not just the one the user is connected to. This is called a backplane.

Sticky sessions

With sticky sessions, client-side SignalR connections work as normal - the user has a persistent connection to one instance, just like on a single-server site. The only thing needed is a backplane so that server-side messages (like "this content was just published") can reach all instances.

Redis works well for this:

// Requires: Microsoft.AspNetCore.SignalR.StackExchangeRedis
builder.Services.AddSignalR().AddStackExchangeRedis(redisConnectionString);

Stateless

With stateless load balancing, each request can land on a different instance. SignalR relies on a persistent connection between client and server - and that connection can't just "jump" to a new instance on every request. So a Redis backplane alone isn't enough here.

The solution is to use a managed SignalR service like Azure SignalR Service, which handles both the backplane and the client-side connections externally:

// Requires: Microsoft.Azure.SignalR
builder.Services.AddSignalR().AddAzureSignalR(azureSignalRConnectionString);

This replaces the need for the Redis backplane entirely for SignalR - Azure SignalR Service handles everything. Note that you still need Redis for the distributed cache, sessions, and data protection as set up above.

Wrapping up

The stateless approach does give a smoother distribution of load across instances, but Azure SignalR Service adds a significant cost on top of the Redis you already need for everything else.

In most cases I would go with the sticky session setup - the lower cost and the simplicity of having everything managed through a single Redis instance makes it easy to set up and maintain. And once sessions and data protection are out of process, sticky sessions handle instance swaps gracefully anyway.

That said, I haven't done any load testing to compare the two approaches. If they scale differently under real-world traffic, that could easily be the biggest factor in choosing one over the other. This post is really just meant to highlight the differences and walk through the setup - not to declare a winner. Try both if you can, and pick what fits your situation.

A deep dive into an award-winning Umbraco editor experience

Jesper Mayntzhusen — Tue, 24 Jun 2025 08:14:53 +0000

This week I attended Codegarden, the annual Umbraco tech conference. My company, Ecreo, had submitted a site that I had taken part in building for the Umbraco Award in the category "Best Editor Experience" - and wednesday evening at the awards ceremony we were announced as winners!

I've always found it a bit frustrating seeing all these amazing sites and features being showcased at the Umbraco Awards and then not seeing anything more about the tech behind them - so here is a post explaining one major feature built on the Dansani website to improve editor experience!

NOTE: This site is running Umbraco 13, any examples will be in AngularJS and are not useable on Umbraco 14+.

The problem to solve

When we were building a new site for Dansani they already had a PIM for product data and a DAM for media. In their DAM (Digital Asset Management) they had tagged a lot of images with product SKUs (Stock Keeping Units) from the PIM (Product Information Management) in order to keep track of which products appeared on the images.

Since they already made the connection and we had to do an integration with both the PIM and DAM they were hoping for the tagged SKUs to be displayed as products in the frontend of the site:

At first we thought we would have to make a custom media picker to pick images from their DAM. However, the DAM - Kontainer - already had a closed-source Umbraco package that the client was used to using. The smart part about this package is that it came with a media picker that sends you to the DAM portal where you can use the full capabilities of the system to manage your media. Once you are done and select an image it has a callback to Umbraco where the image is picked:

The package would then save a JSON blob to the database that looks something like this:

[
    {
        "url": "https://dansani.kontainer.com/cdn/5FPG1rz/GaqfKzSvrtHW/vcali-10d-shape120-25.webp",
        "urlBaseName": "https://dansani.kontainer.com/cdn/5FPG1rz/GaqfKzSvrtHW/vcali-10d-shape120-25",
        "thumbnailUrl": "https://dansani.kontainer.com/cdn/5FPG1rz/GaqfKzSvrtHW/vcali-10d-shape120-25.webp",
        "type": "image",
        "extension": "webp",
        "description": null,
        "alt": null,
        "fileId": 15245169,
        "fileName": "vcali_10d_shape120_25.tif",
        "folderId": 527324,
        "token": null,
        "external": null,
        "cf": {
            "Article number": "631112403\nC09-1099\nBI1\nBI2\nS939-099\nPO2SK\nC75-0099\n\nBreezy Blue\nNCS S 3010-B10G",
            "Variants": "631112403\nBI1\nBI2\nC09-1099\nC75-0099\nPO2SK\nS939-099",
            "File category": "Images room sets",
            "Series": "Dansani Calidris",
            "Alt tags DK": "Stort og lækkert badeværelse med badeværelsesmøbler i lyse, blå farver og holdt i nordisk stil",
            "Alt tags NO": "Stort og lekkert bad med baderomsmøbler i lyse, blå farger og holdt i nordisk stil",
            "Alt tags SE": "Stort och läckert badrum med badrumsmöbler i ljusa, blåa färger i nordisk stil",
            "Alt tags FI": "Suuri ja herkullinen kylpyhuone, jossa kylpyhuonekalusteet vaaleissa, sinisissä väreissä ja pohjoismaiseen tyyliin",
            "Alt tags UK": "Lovely large bathroom with furniture in light, blue colors oozing with Nordic style",
            "Alt tags DE": "Großes und stilvolles Badezimmer mit Badezimmermöbeln in hellblauen Farben und nordischem Stil",
            "Alt tags NL": "Grote moderne badkamer met badkamermeubel in lichte, blauwe kleuren in een Scandinavische stijl"
        },
        "downloadTemplateId": 7675,
        "originalWidth": 3000,
        "originalHeight": 3900,
        "originalSize": 70227052
    }
]

Which means when we saved an image we actually saved all the custom fields like alt texts in different languages and the tagged product SKUs.

However, the custom ValueConverter (the code that takes the saved value and converts into a typed model for the frontend) didn't use any of the custom fields so they were stripped away by the time the frontend is hit.

Additionally, the client didn't want the product tagging to always occur, and may not want every tagged product to occur even when some should.

Extending the closed-source media picker package

The backoffice in v13 is written in AngularJS which has some very powerful extension capabilities - for example you can create an interceptor that listens to a request for a view file and replaces that view file with your own.

Shout out to Dave Woestenborghs for writing about intercepting views with AngularJS many years ago: https://skrift.io/issues/changing-backoffice-functionality-without-changing-core-code/

We quickly found that the media picker used a view found in the App_Plugins folder on the path: /App_Plugins/Kontainer/editors/mediapicker.html, so we could register our own plugin to intercept that.

/App_Plugins/KontainerExtension/package.manifest:

{
    "javascript": [
        "~/App_Plugins/KontainerExtension/interceptor.js",
    ]
}

/App_Plugins/KontainerExtension/interceptor.js:

angular.module('umbraco.services').config([
    '$httpProvider',
    function ($httpProvider) {
        $httpProvider.interceptors.push(function () {
            return {
                'request': function (request) {
                    // Redirect any requests to the kontainer media picker to our own view
                    if (request.url.indexOf("/App_Plugins/Kontainer/editors/mediapicker.html") === 0) {
                        request.url = '/App_Plugins/KontainerExtension/mediapicker.html';
                    }
                    return request;
                }
            };
        });
    }]);

So basically we say any requests going to the packages mediapicker.html view, send it to our own mediapicker.html view instead - so far pretty easy!

Our own view was a direct copy of the package view, however, they had the little remove and edit buttons when you hover an image. We wanted to add a third button:

Which you can see I've added here at the top of the list of buttons:

<div class="umb-sortable-thumbnails__actions" data-element="sortable-thumbnail-actions">
    <!-- @@CUSTOM BUTTON EXTENSION-->
    <button type="button" aria-label="Edit products" class="umb-sortable-thumbnails__action btn-reset"
            data-element="action-products" ng-click="vm.products(item, $index)">
        <i class="icon icon-box-alt" aria-hidden="true"></i>
    </button>
    <!-- @@CUSTOM BUTTON EXTENSION-->
    <button type="button" aria-label="Edit media" class="umb-sortable-thumbnails__action btn-reset"
            data-element="action-edit" ng-click="ctrl.edit(item)">
        <i class="icon icon-edit" aria-hidden="true"></i>
    </button>
    <button type="button" aria-label="Remove" class="umb-sortable-thumbnails__action -red btn-reset"
            data-element="action-remove" ng-click="ctrl.remove(item)">
        <i class="icon icon-delete" aria-hidden="true"></i>
    </button>
</div>

Some of you may have noticed the click event on the button:

ng-click="vm.products(item, $index)"

Which is supposed to then open a separate modal for handling the tagged products. However, to be able to call this vm.products function we had to add to the controller the view is hooked up to - and we didn't want to also entirely replace the package's controller if we didn't have to.

Luckily AngularJS also has the capability of extending controllers (that are explicitly registered).

Shout out to Lennard Fonteijn who wrote about extending AngularJs controllers here: https://umbraco.com/blog/mvp-blog-beyond-umbraco-2-hooking-angularjs-again/

We could see in the view file that the controller was registered as mediapickerController, so we could add our own extending upon that:

/App_Plugins/KontainerExtension/package.manifest:

{
    "javascript": [
        "~/App_Plugins/KontainerExtension/interceptor.js",
        "~/App_Plugins/KontainerExtension/kontainerExtension.controller.js",
        "~/App_Plugins/KontainerExtension/infiniteEditor.controller.js",
    ]
}

/App_Plugins/KontainerExtension/kontainerExtension.controller.js:

(function () {
    "use strict";
    function KontainerController($scope, $controller, angularHelper, editorService) {
        var ctrl = $controller('mediapickerController as ctrl', {$scope: $scope, angularHelper: angularHelper})
        angular.extend(this, ctrl);

        var vm = this;        
        vm.products = products;

        function products(item, $index){
            var options = {
                title: "Select products",
                view: "/App_Plugins/KontainerExtension/infiniteEditor.html",
                size: "medium",
                item: item,
                itemIndex: $index,
                selection: $scope.model.value[$index].selection,
                submit: function(model, index) {                    
                    $scope.model.value[index].selection = model;
                    editorService.close();
                    vm.buttonState = "init";
                },
                close: function() {
                    editorService.close();
                    vm.buttonState = "init";
                }
            };
            editorService.open(options);
        }
    }    

    angular.module('umbraco').controller('KontainerController', [
        '$scope',
        '$controller',
        'angularHelper',
        'editorService',
        KontainerController]);
})();

As you can see the code to extend from an existing controller only takes 2 lines of code:

var ctrl = $controller('mediapickerController as ctrl', {$scope: $scope, angularHelper: angularHelper})
angular.extend(this, ctrl);

And then we can add a method for the products that will open an editor and pass along the selected items, as well as updating the list of selected items when the editor is "submitted".

You may notice that these are saved in $scope.model.value[index].selection which means we can stick with the existing JSON data and just add another object to it called selection.

Here is how the modal then ends up looking:

And the saved json to the database is the same but with this additional data attached:

"selection": {
    "showProducts": true,
    "productChoices": [
        {
            "name": "631112403",
            "checked": false
        },
        {
            "name": "BI1",
            "checked": true
        },
        {
            "name": "BI2",
            "checked": true
        },
        {
            "name": "C09-1099",
            "checked": true
        },
        {
            "name": "C75-0099",
            "checked": false
        },
        {
            "name": "PO2SK",
            "checked": false
        },
        {
            "name": "S939-099",
            "checked": false
        }
    ]
}

Now the modal itself is just a custom modal so I wont go into too much depth with that, but if curious here is the code for the view and the controller:

/App_Plugins/KontainerExtension/infiniteEditor.html

<div ng-controller="KE.InfiniteEditorController as vm">
    <umb-editor-view>
        <umb-editor-header
                name="model.title"
                name-locked="true"
                hide-alias="true"
                hide-icon="true"
                hide-description="true">
        </umb-editor-header>
        <umb-editor-container>
            <umb-box>
                <umb-box-content>
                    <umb-load-indicator ng-if="vm.loading"></umb-load-indicator>
                    <umb-toggle
                            checked="vm.selection.showProducts"
                            on-click="vm.toggle()"
                            show-labels="true"
                            label-on="Showing related products"
                            label-off="Hiding related products"
                            label-position="right">
                    </umb-toggle>
                    <br>
                    <br>                    
                    <div ng-show="vm.selection.showProducts">
                        <p>Select the products that should be shown for the image:</p>
                        <umb-toggle
                                checked="vm.toggleAllChecked"
                                on-click="vm.toggleAll()"
                                show-labels="true"
                                label-on="Toggle all off"
                                label-off="Toggle all on"
                                label-position="right">
                        </umb-toggle>                        
                        <br>
                        <div class="umb-toggle-group">
                            <div class="umb-toggle-group-item" ng-repeat="item in vm.selection.productChoices" ng-if="!item.disabled" ng-class="{'umb-toggle-group-item--disabled': item.disabled}">
                                <umb-toggle class="umb-toggle-group-item__toggle"
                                            checked="item.checked"
                                            disabled="item.disabled"
                                            input-id="{{item.inputId}}"
                                            aria-labelledby="{{item.labelId}}"
                                            on-click="vm.toggleChoice(item)">
                                </umb-toggle>
                                <img src="{{item.image}}" style="max-height:100px; padding-right:20px; max-width:100px;"/>
                                <div class="umb-toggle-group-item__content" ng-click="change(item)">
                                    <div><label id="{{item.labelId}}" for="{{item.inputId}}">{{ item.name }}</label> </div>
                                    <div class="umb-toggle-group-item__description">{{ item.description }}</div>
                                </div>
                            </div>
                        </div>

                        <br><br>
                        <details>
                            <summary>Image skus with no matching products</summary>
                            <div class="umb-toggle-group">
                                <div class="umb-toggle-group-item" ng-repeat="item in vm.selection.productChoices" ng-if="item.disabled" ng-class="{'umb-toggle-group-item--disabled': item.disabled}">
                                    <umb-toggle class="umb-toggle-group-item__toggle"
                                                checked="item.checked"
                                                disabled="item.disabled"
                                                input-id="{{item.inputId}}"
                                                aria-labelledby="{{item.labelId}}"
                                                on-click="vm.toggleChoice(item)">
                                    </umb-toggle>
                                    <div class="umb-toggle-group-item__content" ng-click="change(item)">
                                        <div><label id="{{item.labelId}}" for="{{item.inputId}}">{{ item.name }}</label> </div>
                                        <div class="umb-toggle-group-item__description">{{ item.description }}</div>
                                    </div>
                                </div>
                            </div>
                        </details>                        
                    </div>                    
                </umb-box-content>
            </umb-box>
        </umb-editor-container>
        <umb-editor-footer>
            <umb-editor-footer-content-right>
                <umb-button
                        type="button"
                        button-style="link"
                        label-key="general_close"
                        shortcut="esc"
                        action="vm.close()">
                </umb-button>
                <umb-button
                        type="button"
                        button-style="action"
                        label-key="general_submit"
                        action="vm.submit(model)">
                </umb-button>
            </umb-editor-footer-content-right>
        </umb-editor-footer>
    </umb-editor-view>
</div>

/App_Plugins/KontainerExtension/infiniteEditor.controller.js

(function () {
    "use strict";

    function KeInfiniteEditorController($scope, $routeParams, $http, editorState) {

        var vm = this;

        vm.submit = submit;
        vm.close = close;
        vm.toggle = toggle;
        vm.toggleChoice = toggleChoice;
        vm.toggleAll = toggleAll;

        vm.loading = false;
        vm.item = $scope.model.item;
        vm.articleNumbers = vm.item.cf['Article number'] === undefined ? [] : vm.item.cf['Article number'].split(/\r\n|\r|\n/);
        vm.selection = getSelection($scope.model.selection);
        vm.toggleAllChecked = getToggleState();

        function getToggleState(){
            let obj = vm.selection.productChoices.find(f => f.checked === false);
            if(obj === undefined) return true;
            return false;
        }

        function toggleChoice(item){
            let obj = vm.selection.productChoices.find(f => f.description === item.description);
            obj.checked = !obj.selected;
            obj.selected = !obj.selected;
        }

        function toggleAll(){
            vm.toggleAllChecked = !vm.toggleAllChecked;
            vm.selection.productChoices.forEach(function (item){
                if(!item.disabled){
                    item.checked = vm.toggleAllChecked;
                }
            })
        }

        function getSelection(selection){

            if(selection === undefined){
                var initSelection = {
                    "showProducts": true,
                    "productChoices": []
                }

                var options = [];

                vm.articleNumbers.forEach(function(item){
                    options.push({"name": item.trim(), "checked": false});
                })

                var products = GetProducts(options);
                initSelection.productChoices = products;

                return initSelection;
            }

            var formattedChoices = GetProducts(selection.productChoices);            
            selection.productChoices = formattedChoices;

            return selection;
        }

        function GetProducts(options){
            vm.loading = true;
            var formattedProducts = [];

            if (options === undefined || options.length == 0) {
                vm.loading = false;
                return formattedProducts;
            }

            if(Object.hasOwn(options[0], 'description')){
                var strippedOptions = [];

                options.forEach(function(item){
                    strippedOptions.push({"name": item.description, "checked": item.checked});
                })

                options = strippedOptions;
            }

            $http({
                method: 'POST',
                url: '/umbraco/backoffice/api/KontainerExtension/GetProducts',
                headers: {'Content-Type': 'application/json'},
                data: JSON.stringify({
                    options: options,
                    culture: $routeParams.mculture,
                    nodeId: editorState.current.id
                })
            })
            .then(function (response) {

                response.data.forEach(function(item){
                    formattedProducts.push({"name": item.Name, "checked": item.Checked, "disabled": !item.Exists, "description": item.Sku, "image": item.Image});
                })

                vm.loading = false;
            });

            return formattedProducts;
        }

        function toggle(){
            vm.selection.showProducts = !vm.selection.showProducts
        }

        function submit() {
            if($scope.model.submit) {
                var res = [];
                vm.selection.productChoices.forEach(function (item){
                    res.push({"name": item.description, "checked": item.checked});
                })
                vm.selection.productChoices = res;
                $scope.model.submit(vm.selection, $scope.model.itemIndex);
            }
        }

        function close() {
            if($scope.model.close) {
                $scope.model.close();
            }
        }
    }

    angular.module("umbraco").controller("KE.InfiniteEditorController", KeInfiniteEditorController);
})();

This controller would take the skus and pass along to a secure controller endpoint that would return the product name, image, sku and toggle state to use in the preview.

Finally, we needed to replace the custom property value converter that came with the package. This is quite easy to do, we can just unregister it and register our own:

Composer.cs:

public class Composer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
    {
        builder.PropertyValueConverters().Remove<Kontainer.ValueConverters.MediaPickerValueConverter>();
    }
}

KontainerOverwriteValueConverter.cs:

public class KontainerOverwriteValueConverter : IPropertyValueConverter
{
    private readonly IVariationContextAccessor _variationContextAccessor;
    private readonly KontainerMapper _kontainerMapper;

    public KontainerOverwriteValueConverter(IVariationContextAccessor variationContextAccessor, KontainerMapper kontainerMapper)
    {
        _variationContextAccessor = variationContextAccessor;
        _kontainerMapper = kontainerMapper;
    }

    public bool IsConverter(IPublishedPropertyType propertyType)
    {
        return propertyType.EditorAlias.Equals("Kontainer.MediaPicker");
    }

    // Taken directly from Umbraco core - https://github.com/umbraco/Umbraco-CMS/blob/cb544f15eee862361c7830640ac0d0c4252bf488/src/Umbraco.Core/PropertyEditors/PropertyValueConverterBase.cs#L9
    public bool? IsValue(object? value, PropertyValueLevel level)
    {
        switch (level)
        {
            case PropertyValueLevel.Source:
                // the default implementation uses the old magic null & string comparisons,
                // other implementations may be more clever, and/or test the final converted object values
                return value != null && (value is not string stringValue || !string.IsNullOrWhiteSpace(stringValue));
            case PropertyValueLevel.Inter:
                return null;
            case PropertyValueLevel.Object:
                return null;
            default:
                throw new NotSupportedException($"Invalid level: {level}.");
        }
    }

    public Type GetPropertyValueType(IPublishedPropertyType propertyType)
    {
        var isMultiple = IsMultipleDataType(propertyType.DataType);
        return isMultiple
            ? typeof(List<KontainerData>)
            : typeof(KontainerData);
    }

    private bool IsMultipleDataType(PublishedDataType dataType)
    {
        if (dataType.Configuration is not Dictionary<string, object> config)
        {
            return true;
        }

        if(!config.TryGetValue("multiSelect", out var val))
        {
            return true;
        }

        return val.ToString() != "0";
    }

    public PropertyCacheLevel GetPropertyCacheLevel(IPublishedPropertyType propertyType)
    {
        return PropertyCacheLevel.Elements;
    }

    public object? ConvertSourceToIntermediate(IPublishedElement owner, IPublishedPropertyType propertyType, object? source,
        bool preview)
    {
        var culture = string.IsNullOrWhiteSpace(_variationContextAccessor.VariationContext?.Culture) ? "da-DK" : _variationContextAccessor.VariationContext.Culture;
        var isMultiple = IsMultipleDataType(propertyType.DataType);

        var files = new List<KontainerData>();

        if (source == null)
        {
            return isMultiple ? files : null;
        }

        var sourceString = source.ToString();
        if (sourceString is null)
        {
            return isMultiple ? files : null;
        }

        if (!string.IsNullOrWhiteSpace(sourceString))
        {
            var kontainerMedia = JsonSerializer.Deserialize<List<KontainerDataJsonModel>>(sourceString);
            if (kontainerMedia is not null)
            {
                files.AddRange(_kontainerMapper.MapKontainerData(kontainerMedia, culture).GetAwaiter().GetResult());
            }
        }

        return isMultiple ? files : files.FirstOrDefault();
    }

    public object? ConvertIntermediateToObject(IPublishedElement owner, IPublishedPropertyType propertyType,
        PropertyCacheLevel referenceCacheLevel, object? inter, bool preview)
    {
        return inter;
    }

    public object? ConvertIntermediateToXPath(IPublishedElement owner, IPublishedPropertyType propertyType,
        PropertyCacheLevel referenceCacheLevel, object? inter, bool preview)
    {
        return null;
    }
}

Where our KontainerMapper takes the skus and fetches the products from our product cache which then allows us to return our own custom model that contains all the information we want, and we can also access things like the culture of the request so we can build the url for our product nodes, which means when you open the modal you can navigate directly to the product page:

As for rendering the media, we made a custom taghelper that could differentiate between the different media types and would render dependant on that so in the FE you could always just pass along the model like this:

<ec-kontainer media="@article.Image" container-class="overflow-hidden" img-class="w-full object-cover sm:h-120 xl:h-155 max-sm:w-full"></ec-kontainer>

Additionally, my colleague Søren Kottal created https://marketplace.umbraco.com/package/umbraco.community.imagesharpremoteimages to allow us to continue using imagesharp for cropping the DAM images on our end, caching the crops, etc.

That meant we didn't have to change anything about how we normally work with media, and could do things like a src-set and lazysizes.js for generating multiple media crops to fit multiple browser formats!

I hope you liked this little preview into one of the custom extensions we made for this site!

You can reach me on these socials:

GitHub: https://github.com/jemayn
Mastodon: https://umbracocommunity.social/@Jmayn
LinkedIn: https://www.linkedin.com/in/jemayn

Facetted search with Examine - Umbraco 13

Jesper Mayntzhusen — Mon, 26 Aug 2024 20:10:37 +0000

Intro

I've been doing a few projects where we used the new facet engine in Examine 4, have seen several questions about it so figured I throw together a basic example.

This example is using the Umbraco starter kit: https://github.com/umbraco/The-Starter-Kit/tree/v13/dev

All the products have prices between 2 and 1899, the goal now is to set up facets for the price where it shows facet price range values.

To get facets running we first need to do a few things:

Enabling facets

First thing is to update the Examine NuGet package version to one that has facets (as of writing this it is 4.0.0-beta.1).

So after installing it I have this in my csproj:

<PackageReference Include="Examine" Version="4.0.0-beta.1" />

Next I need to ensure that the price field is indexed as a facet field. As this is in the external index it can be changed by adding new config:

In a composer:

public class SearchComposer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
    {
        builder.Services.ConfigureOptions<ConfigureExternalIndexOptions>();

    }
}

ConfigureExternalIndexOptions:

using Examine;
using Examine.Lucene;
using Microsoft.Extensions.Options;
using Umbraco.Cms.Core;

namespace FacetBlog.Search;

public class ConfigureExternalIndexOptions : IConfigureNamedOptions<LuceneDirectoryIndexOptions>
{
    private readonly IServiceProvider _serviceProvider;

    public ConfigureExternalIndexOptions(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
    }

    public void Configure(string name, LuceneDirectoryIndexOptions options)
    {
        if (name.Equals(Constants.UmbracoIndexes.ExternalIndexName))
        {
            // Index the price field as a facet of the type long (int64)
            options.FieldDefinitions.AddOrUpdate(new FieldDefinition("price", FieldDefinitionTypes.FacetTaxonomyLong));
            options.UseTaxonomyIndex = true;

            // The standard directory factory does not work with the taxonomi index.
            // If running on azure it should use the syncedTemp factory
            options.DirectoryFactory = _serviceProvider.GetRequiredService<global::Examine.Lucene.Directories.TempEnvFileSystemDirectoryFactory>();
        }
    }

    public void Configure(LuceneDirectoryIndexOptions options)
    {
        throw new System.NotImplementedException();
    }
}

At this point if the index is rebuilt it will add the facet fields for the price.

Adding a searchservice

Can now add a bit of search logic so show the facet values on the frontend when searching on products. First we add a SearchService:

using Examine;
using Examine.Lucene;
using Examine.Search;
using Umbraco.Cms.Core.Models.PublishedContent;
using Umbraco.Cms.Web.Common;

namespace FacetBlog.Search;

public interface ISearchService
{
    SearchResult SearchProducts(string query);
}

public class SearchResult
{
    public IEnumerable<IPublishedContent> NodeResults { get; set; }
    public IEnumerable<IFacetValue> Facets { get; set; }
}

public class SearchService : ISearchService
{
    private readonly IExamineManager _examineManager;
    private readonly UmbracoHelper _umbracoHelper;

    public SearchService(IExamineManager examineManager, UmbracoHelper umbracoHelper)
    {
        _examineManager = examineManager;
        _umbracoHelper = umbracoHelper;
    }

    public SearchResult SearchProducts(string query)
    {
        var res = new SearchResult();
        var nodeResult = new List<IPublishedContent>();
        var facetResults = new List<IFacetValue>();

        if (_examineManager.TryGetIndex("ExternalIndex", out IIndex? index))
        {
            // Start searching product pages
            var queryBuilder = index
                .Searcher
                .CreateQuery("content")
                .NodeTypeAlias("product");

            // If a query string is added, search the nodenames for that query string
            if (!string.IsNullOrEmpty(query))
            {
                queryBuilder.And()
                    .Field("nodeName", query.MultipleCharacterWildcard());
            }

            // Add facets for the price field split up into several ranges
            var results = queryBuilder
                .WithFacets(f => f.FacetLongRange("price", new[]
                    {
                        new Int64Range("0-100", 0, true, 100, false),
                        new Int64Range("100-500", 100, true, 500, false),
                        new Int64Range("500-1500", 500, true, 1500, false),
                        new Int64Range("1500+", 1500, true, long.MaxValue, true)
                    }))
                .Execute();

            // Loop through results and add to a list of IPublishedContent
            foreach (var result in results)
            {
                nodeResult.Add(_umbracoHelper.Content(int.Parse(result.Id)));
            }

            var priceFacet = results.GetFacet("price");

            // Loop through facet results and add to a list
            foreach (var facetValue in priceFacet)
            {
                facetResults.Add(facetValue);
            }
        }

        res.NodeResults = nodeResult;
        res.Facets = facetResults;

        return res;
    }
}

The main difference to a "normal" search is that before we execute the search we can add a list of facet fields we want to get facets for based on the result set. Because our facet in this example is a number we may not want to get every single number but instead get the ones in certain ranges - that is configured like this:

var results = queryBuilder
    .WithFacets(f => f.FacetLongRange("price", new[]
        {
            new Int64Range("0-100", 0, true, 100, false),
            new Int64Range("100-500", 100, true, 500, false),
            new Int64Range("500-1500", 500, true, 1500, false),
            new Int64Range("1500+", 1500, true, long.MaxValue, true)
        }))
    .Execute();

Then once we retrieve the result the facets can be gotten based on the facet fieldname:

var priceFacet = results.GetFacet("price");

Now in the view we can add a bit of markup with an input field and outputting a list of results and a list of facets:

<form action="@Model.Url()" method="get">
    <input type="text" placeholder="Search" name="query" value="@Context.Request.Query["query"].FirstOrDefault()"/>
    <button>Search</button>
</form>
<div>
    @if (Model.SearchResults.NodeResults.Any())
    {
        <p>Content results:</p>
        <ul>
            @foreach (var content in Model.SearchResults.NodeResults)
            {
                <li>
                    <a href="@content.Url()">@content.Name</a>
                </li>
            }
        </ul>

        <p>Facet results:</p>
        <ul>
            @foreach (var facet in Model.SearchResults.Facets)
            {
                <li>
                    @facet.Label - amount: @facet.Value
                </li>
            }
        </ul>
    }
    else if(Model.HasSearched)
    {
        <p>No results found</p>
    }
</div>

And now we get something like this:

And with a query:

Outro

This is a simple example so there are likely several use cases that are not covered - if you have any specific requests for what I can cover next let me know in a comment 🙂.

Feel free to reach out to me on Mastodon and let me know if you liked the blogpost: https://umbracocommunity.social/@Jmayn

Searching with Umbraco Examine: Avoid these common filtering mistakes

Jesper Mayntzhusen — Sun, 09 Jun 2024 16:22:55 +0000

With Umbracos Codegarden conference mere days away I am starting to feel the Codegarden spirit! And that has helped motivate me to blog a bit again, at the top of my blogpost ideas list is something interesting I found a few months back about how you can filter in Examine - Umbracos API layer on top of Lucene.

Setup

The starting point of this blogpost is an Umbraco 13 site with The Starter Kit.
And a simple search setup inspired by this docs article.

The starter kit has a People section, where people have tags assigned in a property called "department".
I've added my own "Person" with a nice AI generated image:

Here is a quick overview of the starting code:

PeopleController.cs

using ExamineTesting.Models;
using ExamineTesting.Services;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.ViewEngines;
using Umbraco.Cms.Core.Models.PublishedContent;
using Umbraco.Cms.Core.PublishedCache;
using Umbraco.Cms.Core.Web;
using Umbraco.Cms.Web.Common.Controllers;

namespace ExamineTesting.Controllers;

public class PeopleController : RenderController
{
    private readonly IPublishedValueFallback _publishedValueFallback;
    private readonly ISearchService _searchService;
    private readonly ITagQuery _tagQuery;

    public PeopleController(ILogger<RenderController> logger,
        ICompositeViewEngine compositeViewEngine,
        IUmbracoContextAccessor umbracoContextAccessor,
        IPublishedValueFallback publishedValueFallback,
        ISearchService searchService,
        ITagQuery tagQuery) : base(logger,
        compositeViewEngine,
        umbracoContextAccessor)
    {
        _publishedValueFallback = publishedValueFallback;
        _searchService = searchService;
        _tagQuery = tagQuery;
    }

    public override IActionResult Index()
    {
        var tags = HttpContext.Request.Query["tags"];

        var allTags = _tagQuery.GetAllContentTags();

        var searchResults = _searchService.SearchContentByTag(tags);

        // Create the view model and pass it to the view
        SearchViewModel viewModel = new(CurrentPage!, _publishedValueFallback)
        {
            SearchResults = searchResults.results,
            Tags = allTags,
            Query = searchResults.query
        };

        return CurrentTemplate(viewModel);
    }
}

SearchService.cs

using Examine;
using Microsoft.Extensions.Primitives;
using Umbraco.Cms.Core.Models.PublishedContent;
using Umbraco.Cms.Web.Common;

namespace ExamineTesting.Services;

public class SearchService : ISearchService
{
    private readonly IExamineManager _examineManager;
    private readonly UmbracoHelper _umbracoHelper;

    public SearchService(IExamineManager examineManager,
        UmbracoHelper umbracoHelper)
    {
        _examineManager = examineManager;
        _umbracoHelper = umbracoHelper;
    }

    public (IEnumerable<IPublishedContent> results, string query) SearchContentByTag(StringValues tags)
    {
        IEnumerable<string> ids = Array.Empty<string>();
        var queryString = string.Empty;
        if (_examineManager.TryGetIndex("ExternalIndex",
                out IIndex? index))
        {
            var q = index
                .Searcher
                .CreateQuery("content")
                .NodeTypeAlias("person");

            if (tags.Any())
            {
                q.And().Field("department", tags.FirstOrDefault());
            }

            ids = q
                .Execute()
                .Select(x => x.Id);

            queryString = q.ToString();
        }

        var results = new List<IPublishedContent>();

        foreach (var id in ids)
        {
            results.Add(_umbracoHelper.Content(id));
        }

        return (results, queryString);
    }
}

people.cshtml

@using Microsoft.AspNetCore.Mvc.TagHelpers
@inherits Umbraco.Cms.Web.Common.Views.UmbracoViewPage<ExamineTesting.Models.SearchViewModel>
@{
    Layout = "master.cshtml";
}
@{
    void SocialLink(string content, string service)
    {
        if (!string.IsNullOrEmpty(content))
        {
            ; //semicolon needed otherwise <a> cannot be resolved
            <a class="employee-grid__item__contact-item" href="http://@(service).com/@content">@service</a>
        }
    }
}

@Html.Partial("~/Views/Partials/SectionHeader.cshtml")

<section class="section">

    <div class="container">
        <div>
            <span>@Model.Query</span>
            <form action="@Model.Url()" method="get">
                <label for="tags">Choose a tag:</label>

                <select name="tags" id="tags">
                    @foreach (var tag in Model.Tags)
                    {
                        <option value="@tag?.Text">@tag?.Text</option>
                    }
                </select>
                <button>Search</button>
            </form>
        </div>
        <div class="employee-grid">
            @foreach (Person person in Model.SearchResults)
            {

                <div class="employee-grid__item">
                    <div class="employee-grid__item__image" style="background-image: url('@person.Photo?.Url()')"></div>
                    <div class="employee-grid__item__details">
                        <h3 class="employee-grid__item__name">@person.Name</h3>
                        @if (!string.IsNullOrEmpty(person.Email))
                        {
                            <a href="mailto:@person.Email" class="employee-grid__item__email">@person.Email</a>
                        }
                        <div class="employee-grid__item__contact">
                            @{ SocialLink(person.FacebookUsername, "Facebook"); }
                            @{ SocialLink(person.TwitterUsername, "Twitter"); }
                            @{ SocialLink(person.LinkedInUsername, "LinkedIn"); }
                            @{ SocialLink(person.InstagramUsername, "Instagram"); }
                        </div>
                    </div>
                </div>
            }
        </div>
    </div>

</section>

So at this point we have a dropdown with all available tags, and can choose one, click search and it filters into that specific tag. (To help debugging I've also added the Lucene query string output):

The problem

Now at this point is where I would have often stopped in the past. As I showed in the image just above it works, the query adds +department:denmark, the + in Lucene queries means AND, and it returns the only person in our set of people that has the tag "denmark".

However, what if I try to find the test person - King Arthur - who I added earlier based on his tag "Fairytale kingdom":

Suddenly a bunch of unexpected results are added, and while these are all great people - none of them have the "Fairytale kingdom" tag.

And we can also see that the apparent way that examine treats a query string with a space is to split it in two and do an OR search: +(department:fairytale department:kingdom) (the space in Lucene means OR)

After a bit of digging in the data, it occurs that all of these extra people have the tag "United Kingdom", thus they are treated as hit based on the "kingdom" part.

Handling multiword phrases in filters

The fix for this problem is actually quite easy - in our code we have this which is what adds the tag filter:

q.And().Field("department", tags.FirstOrDefault());

Now without an IDE it may be a bit tough to understand what tags are - but in this case the var tags is of the type StringValues which is what we get back from the HTTP Query.

It allows you to fx have a query like this:

domain.com?q=searchterm&color=red&color=blue

Where if you then tried to get the color query value you would get StringValues with 2 values - red and blue.

In our case we only ever pass along one tag, so adding a .FirstOrDefault() to it will return it as a string.

The Examine .Field() method has two versions, one that takes the query as a string, and one that takes it as an IExamineValue:

An IExamineValue is basically a search term with additional logic applied. So if we fx wanted to boost this specific part of our queries importance we can add the boost extension which then turns out string into an ExamineValue: tags.FirstOrDefault().Boost(10).

But there is another string extension that takes a search string and turns it into a "phrase match" string, where the string will basically need to be an exact match otherwise it wont work.

We can achieve this by changing it to tags.FirstOrDefault().Escape()

A new search at this point shows the query now has it in quotes: +department:"Fairytale Kingdom"

However, it also doesn't return King Arthur as it should:

It does in the Examine dashboard in the backoffice though:

The problem is reported and discussed here: https://github.com/Shazwazza/Examine/issues/329

So rather than going into it I will just ensure it works, but making sure it is indexed and searched for in lowercase:

We add a quick TransformingIndexValues event where we can take the tags values from the "department" property, and lowercase them and save into a new "departmentLower" field:

using Examine;
using Umbraco.Cms.Core.Events;
using Umbraco.Cms.Core.Notifications;
using Umbraco.Cms.Web.Common.PublishedModels;

namespace ExamineTesting.Notifications;

public class ExternalIndexTransformations : INotificationHandler<UmbracoApplicationStartedNotification>
{
    private readonly IExamineManager _examineManager;

    public ExternalIndexTransformations(IExamineManager examineManager)
    {
        _examineManager = examineManager;
    }

    public void Handle(UmbracoApplicationStartedNotification notification)
    {
        if (!_examineManager.TryGetIndex(Umbraco.Cms.Core.Constants.UmbracoIndexes.ExternalIndexName, out var index))
        {
            throw new InvalidOperationException(
                $"No index found by name {Umbraco.Cms.Core.Constants.UmbracoIndexes.ExternalIndexName}");
        }

        index.TransformingIndexValues += IndexOnTransformingIndexValues;
    }

    private void IndexOnTransformingIndexValues(object? sender, IndexingItemEventArgs e)
    {
        if(e.ValueSet.ItemType is not Person.ModelTypeAlias) return;

        var hasDepartment = e.ValueSet.Values.TryGetValue("department", out var values);
        if (!hasDepartment) return;

        var tagsLowerCase = values!.Select(x => x.ToString()?.ToLowerInvariant());

        var valuesDictionary = e.ValueSet.Values.ToDictionary(x => x.Key, x => x.Value.ToList());
        var newValues = new List<object>();

        foreach (var tag in tagsLowerCase)
        {
            if (!string.IsNullOrWhiteSpace(tag))
            {
                newValues.Add(tag);
            }
        }
        valuesDictionary.Add("departmentLower", newValues );

        e.SetValues(valuesDictionary.ToDictionary(x => x.Key, x => (IEnumerable<object>)x.Value));
    }
}

After a quick reindex we can see the new field:

Now after changing our tags filter in the searchservice to look into that field and also lowercase the searchterm:

q.And().Field("departmentLower", tags.FirstOrDefault()?.ToLowerInvariant().Escape());

Now we get the search result that we expected:

Wohoo now the filtering works just as expected!! 🎉

Or does it... 🤔

Partial word matches

What if you were to add a dragon, and that dragon is tagged with "Fairytale":

The exact phrase "fairytale" matches the value "fairytale kingdom", so now we get extra results again:

In some cases that may be exactly what we want - however in others it will be considered a wrong match.

So how can we handle this case?

In my search for a solution I stumbled upon this old blogpost about lucene phrase matching from 2012, so shout out to Mark Leighton Fisher who describes an easy workaround in this blogpost!

Basically what he suggests is to wrap your indexed value and searchterm in a delimeter word - in his example he uses the word lucenematch.

The reason this works is that we go from having the indexed values:

fairytale
fairytale kingdom

lucenematch fairytale lucenematch
lucenematch fairytale kingdom lucenematch

And while fairytale will be a match to fairytale kingdom.
lucenematch fairytale lucenematch will not match lucenematch fairytale kingdom lucenematch

So I'll add a quick string extension to add the delimiter word:

namespace ExamineTesting.Extensions;

public static class SearchExtensions
{
    private const string DelimiterWord = "lucenematch";

    public static string AddLuceneDelimiterWord(this string value)
    {
        return $"{DelimiterWord} {value} {DelimiterWord}";
    }
}

And in the TransformingIndexValues event I will ensure that we add it to the values before indexing:

foreach (var tag in tagsLowerCase)
{
    if (!string.IsNullOrWhiteSpace(tag))
    {
        newValues.Add(tag.AddLuceneDelimiterWord());
    }
}

If I then go and reindex the externalIndex I can see it is added to our departmentLower field:

Then we can add it to our searchService as well:

if (tags.Any())
{
    q.And().Field("departmentLower", tags.FirstOrDefault()?.AddLuceneDelimiterWord().ToLowerInvariant().Escape());
}

And when trying the search again we can see the query now adds the delimiter word, and the results are back to only the specific one we wanted:

Outro

Note: Joe has written a similar blogpost that solves the same problem with a different approach and goes a bit more in-depth with explaining the underlying Examine/Lucene parts.

Please check it out here: https://joe.gl/ombek/blog/tag-style-exact-matching-with-examine/

Thanks for following along! Please let me know if this was helpful to you 🙂

Also feel free to reach out to me on Mastodon: https://umbracocommunity.social/@Jmayn

Index PDFs on their pages in Umbraco

Jesper Mayntzhusen — Sun, 29 Oct 2023 17:07:05 +0000

Intro

The Umbraco documentation has an example on how to set up the PDF indexing package and adding a multisearcher. However that means it will index all of the PDFs in the media library in a seperate PDF index, and then it extends the searcher to search in both Umbraco nodes and PDF documents.
This means that when the user searches they can get PDF results which you can link directly to from the search results.

This blogpost will take a different approach where we don't necessarily search in a separate document index, but instead it will go through how to index PDFs in Umbraco and to add the pdf index to the content nodes that it's included on.

Setting up

To begin with I've started a new site on Umbraco 12.2.0 with the Clean starter kit to have some start content.

I've added a block for the blocklist where you can upload pdf files, and I've downloaded an Umbraco whitepaper as PDF from Umbraco.com to test with.

You can see the block with files being added to one of the blogposts in the starter kit, and the whitepaper is selected:

To help with typed models we will also enable Umbraco Modelsbuilder in the appsettings.json file and generate models:

"Umbraco": {
  "CMS": {
    "ModelsBuilder": {
      "ModelsMode": "SourceCodeManual"
    }
  }
},

Installing the PDF indexing package

To index the PDFs we can use the official Umbraco package, it can be installed from NuGet where it is named Umbraco.ExaminePDF.

After installing and restarting the site you can find the new index in the settings section for Examine indexes - if you rebuild you will see your PDF's from your media section. In my example I only have the Umbraco whitepaper:

However this data is not indexed on any content nodes - let's fix that!

Extending the External Index with PDF data

First of all we will hook into an event on the ExternalIndex in order to add a new pdf field we can put the data into:

We add a new file with the notification handler where we hook into the TransformingIndexValues event on the external index:

using Examine;
using Umbraco.Cms.Core.Events;
using Umbraco.Cms.Core.Notifications;

namespace PdfIndexing.Searching;

public class ExternalIndexTransformer : INotificationHandler<UmbracoApplicationStartedNotification>
{
    private readonly IExamineManager _examineManager;

    public ExternalIndexTransformer(IExamineManager examineManager)
    {
        _examineManager = examineManager;
    }

    public void Handle(UmbracoApplicationStartedNotification notification)
    {
        if (!_examineManager.TryGetIndex(Umbraco.Cms.Core.Constants.UmbracoIndexes.ExternalIndexName, out var index))
        {
            throw new InvalidOperationException(
                $"No index found by name {Umbraco.Cms.Core.Constants.UmbracoIndexes.ExternalIndexName}");
        }

        index.TransformingIndexValues += IndexOnTransformingIndexValues;
    }

    private void IndexOnTransformingIndexValues(object? sender, IndexingItemEventArgs e)
    {
        throw new NotImplementedException();
    }
}

And a composer to ensure it is registered in dependency injection:

using Umbraco.Cms.Core.Composing;
using Umbraco.Cms.Core.Notifications;

namespace PdfIndexing.Searching;

public class Composer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
    {
        builder.AddNotificationHandler<UmbracoApplicationStartedNotification, ExternalIndexTransformer>();
    }
}

In the indexing event we want to add code that does:

Only extend the Article doctype
Get the node content from the cache and cast it to the Article type
Get it's blocklist property (called ContentRows in this site)
Loop through the blocks and only do something if it's the files row type which is the new PDF block.

private void IndexOnTransformingIndexValues(object? sender, IndexingItemEventArgs e)
{
    if(e.ValueSet.ItemType is not Article.ModelTypeAlias) return;

    if (!int.TryParse(e.ValueSet.Id, out int id)) return;

    using var context = _umbracoContextFactory.EnsureUmbracoContext();
    var content = context.UmbracoContext.Content?.GetById(id);

    if (content is not Article typedContent) return;

    var blockList = typedContent.ContentRows;

    if(blockList is null) return;

    foreach (var block in blockList)
    {
        if (block.Content is not FilesRow filesRow) continue;

    }
}

Next we want to do the following:

Loop through the selected files and get the umbracoFile property of the files - which in Umbraco correspond to the files path.
If it finds it we can use the PdfTextService from the UmbracoExamine.PDF package to extract the file content.
We add all of the files' content to a string which we can later index on the node.

foreach (var block in blockList)
{
    if (block.Content is not FilesRow filesRow) continue;
    if (filesRow.Files is null) continue;

    foreach (var file in filesRow.Files)
    {
        var filePath = file.Value<string>("umbracoFile");
        if(string.IsNullOrWhiteSpace(filePath)) continue;

        try
        {
            var pdfString = _pdfTextService.ExtractText(filePath);
            if (pdfString is not null)
            {
                pdfContent += pdfString + " ";
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, $"Could not index the file content from path: {filePath}");
        }
    }
}

Finally, after the blocklist foreach loop we can take the new PDF string content and add as a new field in the index:

if(string.IsNullOrWhiteSpace(pdfContent)) return;

var indexFields = e.ValueSet.Values.ToDictionary(x => x.Key, x => x.Value.ToList());
indexFields.Add("pdfTextContent", new List<object>{pdfContent});
e.SetValues(indexFields.ToDictionary(x => x.Key, x => (IEnumerable<object>)x.Value));

See full event code here:

using Examine;
using Umbraco.Cms.Core.Events;
using Umbraco.Cms.Core.Notifications;
using Umbraco.Cms.Core.Web;
using Umbraco.Cms.Web.Common.PublishedModels;
using UmbracoExamine.PDF;

namespace PdfIndexing.Searching;

public class ExternalIndexTransformer : INotificationHandler<UmbracoApplicationStartedNotification>
{
    private readonly IExamineManager _examineManager;
    private readonly IUmbracoContextFactory _umbracoContextFactory;
    private readonly PdfTextService _pdfTextService;
    private readonly ILogger<ExternalIndexTransformer> _logger;

    public ExternalIndexTransformer(IExamineManager examineManager,
        IUmbracoContextFactory umbracoContextFactory,
        PdfTextService pdfTextService,
        ILogger<ExternalIndexTransformer> logger)
    {
        _examineManager = examineManager;
        _umbracoContextFactory = umbracoContextFactory;
        _pdfTextService = pdfTextService;
        _logger = logger;
    }

    public void Handle(UmbracoApplicationStartedNotification notification)
    {
        if (!_examineManager.TryGetIndex(Umbraco.Cms.Core.Constants.UmbracoIndexes.ExternalIndexName, out var index))
        {
            throw new InvalidOperationException(
                $"No index found by name {Umbraco.Cms.Core.Constants.UmbracoIndexes.ExternalIndexName}");
        }

        index.TransformingIndexValues += IndexOnTransformingIndexValues;
    }

    private void IndexOnTransformingIndexValues(object? sender, IndexingItemEventArgs e)
    {
        if(e.ValueSet.ItemType is not Article.ModelTypeAlias) return;

        if (!int.TryParse(e.ValueSet.Id, out int id)) return;

        using var context = _umbracoContextFactory.EnsureUmbracoContext();
        var content = context.UmbracoContext.Content?.GetById(id);

        if (content is not Article typedContent) return;

        var blockList = typedContent.ContentRows;

        if(blockList is null) return;

        var pdfContent = string.Empty;

        foreach (var block in blockList)
        {
            if (block.Content is not FilesRow filesRow) continue;
            if (filesRow.Files is null) continue;

            foreach (var file in filesRow.Files)
            {
                var filePath = file.Value<string>("umbracoFile");
                if(string.IsNullOrWhiteSpace(filePath)) continue;

                try
                {
                    var pdfString = _pdfTextService.ExtractText(filePath);
                    if (pdfString is not null)
                    {
                        pdfContent += pdfString + " ";
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, $"Could not index the file content from path: {filePath}");
                }
            }
        }

        if(string.IsNullOrWhiteSpace(pdfContent)) return;

        var indexFields = e.ValueSet.Values.ToDictionary(x => x.Key, x => x.Value.ToList());
        indexFields.Add("pdfTextContent", new List<object>{pdfContent});
        e.SetValues(indexFields.ToDictionary(x => x.Key, x => (IEnumerable<object>)x.Value));
    }
}

Finally we can reapply our new code, and go and save and publish an article with a PDF on it. After it has published the new event will have added the pdf content to the index for the content node in the new field "pdfTextContent":

Which means whatever search implementation used can be extended to also search on this new field to include the node into search results if the node contains matches in its pdf content.

NOTE: If an editor goes on the media node of the PDF and replaces the file the index will end up out of date - you can read more carefully why and what you can possibly do to get around the problem of referenced content being indexed in my other blogpost.

The problem of referenced content in Examine indexing (Umbraco 11)

Jesper Mayntzhusen — Wed, 08 Mar 2023 19:59:26 +0000

Intro

When you use a picker property editor in Umbraco, it will reference some other content - the built-in Umbraco index which automatically indexes all content properties will also index the field, but only with a reference to the other content node.

I will run through the details of how this works, and some potential solutions as well as their shortcommings in this blogpost.

The examples below are using a fresh Umbraco 11.2.0 site with the Clean starter kit installed for some base content.

NOTE: While this blog is using code and examples for v11, the core concepts are valid for all Umbraco versions!

The problem

Imagine you have a blogging website where you have authors saved as nodes on their own and then referenced on blog posts like this:

This works great, and you can fetch the data from the referenced node to output on the frontend quite easily.

However, when it comes to the search index, if you want the blogpost to be searchable by author then you need to do some extra work. Currently the blogpost will have the author node referenced like this:

Indexing External node data

So the first step towards solving the problem is that we can hook into the TransformingIndexValues event. This event happens whenever you index content - so either on a full index rebuild or when a specific node is indexed on a publish.
In the event we can add one or more new fields to the index where we save the external node data.

We add this on the UmbracoApplicationStartedNotification, which means it gets added on startup:

using Examine;
using Umbraco.Cms.Core;
using Umbraco.Cms.Core.Events;
using Umbraco.Cms.Core.Notifications;

namespace IndexingBlog.Indexing;

public class CustomIndexer : INotificationHandler<UmbracoApplicationStartedNotification>
{
    private readonly IExamineManager _examineManager;

    public CustomIndexer(IExamineManager examineManager)
    {
        _examineManager = examineManager;
    }

    public void Handle(UmbracoApplicationStartedNotification notification)
    {
        if (!_examineManager.TryGetIndex(Constants.UmbracoIndexes.ExternalIndexName, out IIndex index))
            throw new InvalidOperationException($"No index found by name {Constants.UmbracoIndexes.ExternalIndexName}");

        if (index is not BaseIndexProvider indexProvider)
            throw new InvalidOperationException("Could not cast");

        indexProvider.TransformingIndexValues += IndexProvider_TransformingIndexValues;
    }

    private void IndexProvider_TransformingIndexValues(object? sender, IndexingItemEventArgs e)
    {
        // Do stuff
    }
}

And we can then register it in the startup like this:

public void ConfigureServices(IServiceCollection services)
{
    services.AddUmbraco(_env, _config)
        .AddBackOffice()
        .AddWebsite()
        .AddComposers()
        .AddNotificationHandler<UmbracoApplicationStartedNotification, CustomIndexer>()
        .Build();
}

In the TransformingIndexValues event we can get the node UDI, then get the node from the cache and pull the values we want - then add them to the index for the node:

private void IndexProvider_TransformingIndexValues(object? sender, IndexingItemEventArgs e)
{
    // We don't do anything if the indextype isn't content (so fx media)
    if (e.ValueSet.Category != IndexTypes.Content) return;

    // We get the author field, if it's not there we return
    var hasAuthorProp = e.ValueSet.Values.TryGetValue("author", out var values);
    if (!hasAuthorProp) return;

    // The author field contains the content UDI as a string, we get it and parse it as a UDI
    var contentUdiString = values?.FirstOrDefault()?.ToString();
    var contentUdi = UdiParser.Parse(contentUdiString);

    using var context = _umbracoContextFactory.EnsureUmbracoContext();
    // Get the content node from the UDI
    var node = context?.UmbracoContext?.Content?.GetById(contentUdi);

    // Get the author name and description from the node
    var authorName = node?.Value<string>("authorName");
    var authorDescription = node?.Value<string>("description");

    // We can't add to the values collection, so we get it and add our fields - then overwrite it with our new collection
    var valuesDictionary = e.ValueSet.Values.ToDictionary(x => x.Key, x => x.Value.ToList());
    if(authorName is not null) 
        valuesDictionary.Add("authorName", new List<object>() { authorName });
    if(authorDescription is not null) 
        valuesDictionary.Add("authorDescription", new List<object>() {authorDescription});

    e.SetValues(valuesDictionary.ToDictionary(x => x.Key, x => (IEnumerable<object>)x.Value));
}

At this point whenever the blog node is saved it will also index the author name and description from the referenced node.

This works nicely and is searchable - however there is one major problem:
Imagine you have one author that is referenced on several blogposts - then you update the description of that author.
At that point the description would not be updated in the searchindex for all of the blog posts referencing that author - as their index values are only set when the node itself is published or the entire index is rebuilt.

So how do we keep the index values up to date?

Since Umbraco 8.6 there is a default relation that gets added when you pick a content item from a different content item.

If you are unfamiliar with Umbracos relations, then it is a built-in functionality that tracks entities that are connected somehow.

For example there is a media relation that shows where media items are used:

The relations can be found in the settings section - and there is a new relation type called Related document - and you can see that it already tracks the items that are connected by being picked:

So the flow we want to implement is something like this:

When a node is published we check if it is the author type
If it is the author type we use the relation to look up the nodes it is used on
We create a list of all the related nodes and then reindex them

First thing we want to do is to create a Notification event handler:

using Examine;
using Umbraco.Cms.Core;
using Umbraco.Cms.Core.Events;
using Umbraco.Cms.Core.Notifications;
using Umbraco.Cms.Core.Services;
using Umbraco.Cms.Infrastructure.Examine;

namespace IndexingBlog.Notifications;

public class ContentNotifications : INotificationHandler<ContentPublishedNotification>
{
    private readonly IRelationService _relationService;
    private readonly IExamineManager _examineManager;
    private readonly IPublishedContentValueSetBuilder _publishedContentValueSetBuilder;
    private readonly IContentService _contentService;

    public ContentNotifications(IRelationService relationService, IExamineManager examineManager, IPublishedContentValueSetBuilder publishedContentValueSetBuilder, IContentService contentService)
    {
        _relationService = relationService;
        _examineManager = examineManager;
        _publishedContentValueSetBuilder = publishedContentValueSetBuilder;
        _contentService = contentService;
    }

    public void Handle(ContentPublishedNotification notification)
    {
        // Do stuff
    }
}

And register it in the startup:

services.AddUmbraco(_env, _config)
        .AddBackOffice()
        .AddWebsite()
        .AddComposers()
        .AddNotificationHandler<UmbracoApplicationStartedNotification, CustomIndexer>()
        .AddNotificationHandler<ContentPublishedNotification, ContentNotifications>()
        .Build();

Finally we add the code needed to find the relations and to reindex the related nodes:

public void Handle(ContentPublishedNotification notification)
{
    // We start a list of content ids we need to reindex
    var nodesToReindex = new List<int>();

    foreach (var node in notification.PublishedEntities)
    {
        // We only want to do this for the author type, so if it's not that type we continue the loop
        if(node.ContentType.Alias != "author") continue;

        // When we get a author node that is published we use the relations service and check the related documents
        var relations = _relationService.GetByChildId(node.Id, "umbDocument");
        // We add all the related node ids to our list for reindexing
        nodesToReindex.AddRange(relations.Select(relation => relation.ParentId));
    }

    if (!nodesToReindex.Any()) return;

    // Get the content from the id list
    var contentToReindex = _contentService.GetByIds(nodesToReindex);

    // Get the external index
    if (!_examineManager.TryGetIndex(Constants.UmbracoIndexes.ExternalIndexName, out IIndex index))
        throw new InvalidOperationException($"No index found by name {Constants.UmbracoIndexes.ExternalIndexName}");

    // Get the valuesets and index them
    var valueSets = _publishedContentValueSetBuilder.GetValueSets(contentToReindex.ToArray());
    index.IndexItems(valueSets);
}

Things to keep in mind

This is one way to keep the indexed content up to date. And it works great for a local small site. However you should be wary of how many nodes this may affect - it also hits the db several times through the service usages.

So if you have a large site with many editors this may not be the right way to solve the problem!

Indexing blocklist data in Umbraco 9

Jesper Mayntzhusen — Mon, 14 Mar 2022 20:16:46 +0000

Here is a short example on how to create a new index field and then index specific data to that field from each blocktype.

Setup

As a test site I've started a fresh v9 site with the Portfolio starter kit package.

The site is not that important, it is just to have some starter content.

Next we create a new indexer and hook it up in the startup.cs class:

BlocklistIndexer.cs:

using Examine;
using System;
using Umbraco.Cms.Core.Events;
using Umbraco.Cms.Core.Notifications;
using static Umbraco.Cms.Core.Constants;

namespace BlocklistIndexing.Indexing
{
    public class BlocklistIndexer : INotificationHandler<UmbracoApplicationStartingNotification>
    {
        private readonly IExamineManager _examineManager;

        public BlocklistIndexer(IExamineManager examineManager)
        {
            _examineManager = examineManager ?? throw new ArgumentNullException(nameof(examineManager));
        }

        public void Handle(UmbracoApplicationStartingNotification notification)
        {
            if (!_examineManager.TryGetIndex(UmbracoIndexes.ExternalIndexName, out IIndex index))
                throw new InvalidOperationException($"No index found by name {UmbracoIndexes.ExternalIndexName}");

            if (index is not BaseIndexProvider indexProvider)
                throw new InvalidOperationException("Could not cast");

            indexProvider.TransformingIndexValues += IndexProvider_TransformingIndexValues;

        }

        private void IndexProvider_TransformingIndexValues(object sender, IndexingItemEventArgs e)
        {
            // TODO
        }
    }
}

startup.cs:

services.AddUmbraco(_env, _config)
    .AddBackOffice()
    .AddWebsite()
    .AddComposers()
    .AddNotificationHandler<UmbracoApplicationStartingNotification, BlocklistIndexer>()
    .Build();

This adds a new BlockListIndexer which runs whenever one or more fields in the index is updated. We also hook it up to the configure services in startup.cs on the ApplicationStarting notification so it's added on the site when it starts up.

Add blocklist indexing logic

Next step is to add some steps where we check if the node being indexed has a blocklist property, if it does we get the blocklist and pass it on to a helper function to extract all searchable text then save that into a new search field.

private void IndexProvider_TransformingIndexValues(object sender, IndexingItemEventArgs e)
{
    if (!int.TryParse(e.ValueSet.Id, out int id)) return;

    using var context = _umbracoContextFactory.EnsureUmbracoContext();
    var content = context.UmbracoContext.Content.GetById(id);

    // Exit if content can't be found or doesn't have the blocklist property
    if (content == null || !content.HasProperty("mainContent")) return;

    var blockList = content.Value<BlockListModel>("mainContent");

    var searchableText = SearchableText(blockList);
    if (string.IsNullOrWhiteSpace(searchableText)) return;

    e.ValueSet.Add("blockListSearch", searchableText);
}

Finally we add the helper function, in this we loop through all the blocks in the blocklist, check their types (based on the generated modelsbuilder models), then we add different properties or even nested blocks to one text string that we in the end add to the field:

private string SearchableText(BlockListModel blocklist)
{
    var searchableText = new List<string>();

    foreach (var block in blocklist)
    {
        if(block.Content is CallToAction)
        {
            var typedBlock = block.Content as CallToAction;
            searchableText.Add(typedBlock.Text);
        }

        if(block.Content is CardsRow)
        {
            var typedBlock = block.Content as CardsRow;
            searchableText.Add(typedBlock.Title);

            // This block has a blocklist with blocks of the type IconCard below, we recurse through it
            var blocks = typedBlock.Cards;
            searchableText.Add(SearchableText(blocks));
        }

        if(block.Content is ProjectPreview)
        {
            var typedBlock = block.Content as ProjectPreview;
            searchableText.Add(typedBlock.Title);
            searchableText.Add(typedBlock.Description);
            searchableText.Add(typedBlock.Image.Name);
        }

        if(block.Content is IconCard)
        {
            var typedBlock = block.Content as IconCard;
            searchableText.Add(typedBlock.Title);
            searchableText.Add(typedBlock.Description);
        }
    }

    return string.Join(" ", searchableText.ToArray());
}

Comparing search fields

That's all it takes. At this point we can rebuild the index, and now we can compare the auto generated search field with our new one:

Auto generated one:

{
    "layout": {
        "Umbraco.BlockList": [
            {
                "contentUdi": "umb://element/649184f94d1340ce959c0cefcd005acd",
                "settingsUdi": "umb://element/1dc5f7680cb84520a2e7145a1c349be4"
            },
            {
                "contentUdi": "umb://element/7a2243381cb74d56a53a370c22e18731",
                "settingsUdi": "umb://element/915b74c12c564a228ff31e716119181e"
            },
            {
                "contentUdi": "umb://element/8ee0bd6662d24961b0882f1da940b50e",
                "settingsUdi": "umb://element/f31f872400324fb4bb266a4356c395a6"
            }
        ]
    },
    "contentData": [
        {
            "contentTypeKey": "0ff0a31a-3727-4cba-b45b-4ba8519499bd",
            "udi": "umb://element/7a2243381cb74d56a53a370c22e18731",
            "title": "Special Skills",
            "cards": {
                "layout": {
                    "Umbraco.BlockList": [
                        {
                            "contentUdi": "umb://element/ac777ec30f5446acb4c2cf3ba7955818",
                            "settingsUdi": "umb://element/4e7e112c781549c2b89dda81c51334d6"
                        },
                        {
                            "contentUdi": "umb://element/81b42c2e8f39415b8b1c6aa656c45e0f",
                            "settingsUdi": "umb://element/4ccd9e7eb32d46dab99670c6bce28b5d"
                        },
                        {
                            "contentUdi": "umb://element/0d402062e033403788f1498d13f1d625",
                            "settingsUdi": "umb://element/002ad580c3f940a29890f17204727d57"
                        }
                    ]
                },
                "contentData": [
                    {
                        "contentTypeKey": "3f6a6761-efa8-426e-9b4b-344c596f8aae",
                        "udi": "umb://element/ac777ec30f5446acb4c2cf3ba7955818",
                        "iconClass": [
                            "Ios Star Outline"
                        ],
                        "title": "Umbraco Master",
                        "description": "I am an Umbraco Certified Master. I love everything about Umbraco, especially v9"
                    },
                    {
                        "contentTypeKey": "3f6a6761-efa8-426e-9b4b-344c596f8aae",
                        "udi": "umb://element/81b42c2e8f39415b8b1c6aa656c45e0f",
                        "iconClass": [
                            "Social Github Outline"
                        ],
                        "title": "Git Expert",
                        "description": "I am an expert at Git. Talk to me if you need some help with rebasing or merging with confilicts."
                    },
                    {
                        "contentTypeKey": "3f6a6761-efa8-426e-9b4b-344c596f8aae",
                        "udi": "umb://element/0d402062e033403788f1498d13f1d625",
                        "iconClass": [
                            "Trophy"
                        ],
                        "title": "Umbraco MVP",
                        "description": "I have been awarded an Umbraco MVP award for my contributions to the Umbraco Community"
                    }
                ],
                "settingsData": [
                    {
                        "contentTypeKey": "0c9bd42b-0bd4-4b98-a899-95d27494e348",
                        "udi": "umb://element/4e7e112c781549c2b89dda81c51334d6",
                        "hide": ""
                    },
                    {
                        "contentTypeKey": "0c9bd42b-0bd4-4b98-a899-95d27494e348",
                        "udi": "umb://element/4ccd9e7eb32d46dab99670c6bce28b5d",
                        "hide": ""
                    },
                    {
                        "contentTypeKey": "0c9bd42b-0bd4-4b98-a899-95d27494e348",
                        "udi": "umb://element/002ad580c3f940a29890f17204727d57",
                        "hide": ""
                    }
                ]
            }
        },
        {
            "contentTypeKey": "8728e9e8-749f-4719-a3f3-6ed780b42141",
            "udi": "umb://element/649184f94d1340ce959c0cefcd005acd",
            "text": "Like what you see?",
            "link": [
                {
                    "name": "Hire me",
                    "url": "/contact/"
                }
            ]
        },
        {
            "contentTypeKey": "96dd9b3c-04ae-47fb-b610-d546b05740bb",
            "udi": "umb://element/8ee0bd6662d24961b0882f1da940b50e",
            "title": "Online Blogging Site",
            "description": "I was fortunate enough to be involved in building the popular blog called codeshare.co.uk. I even migrated it from Umbraco 7 to Umbraco 8.",
            "image": [
                {
                    "key": "e5d880bc-55f4-4201-9373-843372bb39ec",
                    "mediaKey": "121e4357-98a2-4d70-ac44-d87e1732f2e4",
                    "crops": [],
                    "focalPoint": {
                        "left": 0.5,
                        "top": 0.5
                    }
                }
            ]
        }
    ],
    "settingsData": [
        {
            "contentTypeKey": "e2ec53a3-602d-4134-a56e-7ff7fe92a587",
            "udi": "umb://element/915b74c12c564a228ff31e716119181e",
            "itemName": "",
            "hide": "0"
        },
        {
            "contentTypeKey": "e1d2ae73-a9e2-42f5-85ac-c2a1d4c27c3d",
            "udi": "umb://element/1dc5f7680cb84520a2e7145a1c349be4",
            "itemName": "",
            "hide": "0"
        },
        {
            "contentTypeKey": "40c5160c-386f-42cf-9101-b4c816fc8a2d",
            "udi": "umb://element/f31f872400324fb4bb266a4356c395a6",
            "itemName": "",
            "hide": "0"
        }
    ]
}

Our new field:

Like what you see? Special Skills Umbraco Master I am an Umbraco Certified Master. 
I love everything about Umbraco, especially v9 Git Expert I am an expert at Git. 
Talk to me if you need some help with rebasing or merging with confilicts. 
Umbraco MVP I have been awarded an Umbraco MVP award for my contributions to the Umbraco Community Online Blogging Site 
I was fortunate enough to be involved in building the popular blog called codeshare.co.uk. 
I even migrated it from Umbraco 7 to Umbraco 8. 
Codeshare

That's all it takes!

Umbraco backoffice listview + infinite editing - part 3

Jesper Mayntzhusen — Sat, 26 Feb 2022 19:47:42 +0000

Hi my name is Jesper 👋

More than a year ago I started a series of blogposts I never finished, now I've been promised cake if I do - so here is the third and final part to the guide about creating a custom listview with infinite editing in Umbraco 8!

In part 2 we left off with a fully functioning with pagination - if you forgot where we left off, don't worry.. I pretty much had to start from scratch as well 😊

Here is the starting point:

In ~/App_Plugins/ListViews:

package.manifest

{
  "dashboards": [
    {
      "alias": "myCustomDashboard",
      "view": "/App_Plugins/ListViews/dashboard.html",
      "sections": [ "content" ],
      "weight": 15
    }
  ],
  "javascript": [
    "~/App_Plugins/ListViews/dashboard.controller.js"
  ]
}

dashboard.html

<div ng-controller="MyDashboardController as vm">
    <h1>Upcoming publish/unpublish events!</h1>

    <div class="umb-table" ng-show="!vm.loading && vm.weHaveRecords">
        <div class="umb-table-head">
            <div class="umb-table-row">
                <!-- We leave this first on empty in the header & show icons in the body -->
                <div class="umb-table-cell"></div>
                <div class="umb-table-cell umb-table__name">Name</div>
                <div class="umb-table-cell">Publish time</div>
                <div class="umb-table-cell">Culture</div>
            </div>
        </div>

        <div class="umb-table-body">
            <div class="umb-table-row -selectable" ng-repeat="item in vm.records">
                <div class="umb-table-cell">
                    <i class="umb-table-body__icon umb-table-body__fileicon icon-document" ng-if="item.ScheduleInfo.FullSchedule[0].Action == 0"></i>
                    <i class="umb-table-body__icon umb-table-body__fileicon icon-document-dashed-line color-grey" ng-if="item.ScheduleInfo.FullSchedule[0].Action == 1"></i>
                </div>
                <div class="umb-table-cell umb-table__name">{{item.Content.Name}}</div>
                <div class="umb-table-cell">{{item.ScheduleInfo.FullSchedule[0].Date | date : medium}}</div>
                <div class="umb-table-cell">{{item.ScheduleInfo.FullSchedule[0].Culture}}</div>
            </div>
        </div>
    </div>

    <div class="flex justify-center" ng-show="!vm.loading && vm.weHaveRecords">
        <umb-pagination page-number="vm.pagination.pageNumber"
                        total-pages="vm.pagination.totalPages"
                        on-next="vm.nextPage"
                        on-prev="vm.prevPage"
                        on-change="vm.changePage"
                        on-go-to-page="vm.goToPage">
        </umb-pagination>
    </div>

    <umb-box ng-if="!vm.weHaveRecords && !vm.loading">
        <umb-box-content>
            No records at this point!
        </umb-box-content>
    </umb-box>

    <umb-load-indicator ng-if="vm.loading">
    </umb-load-indicator>
</div>

dashboard.controller.js

(function () {
    "use strict";

    function DashboardController($http) {

        var vm = this;

        function init() {
            getContentForRelease(0, vm.recordsPerPage);
        }

        vm.nextPage = nextPage;
        vm.prevPage = prevPage;
        vm.changePage = changePage;
        vm.goToPage = goToPage;
        vm.recordsPerPage = 2;

        function nextPage(pageNumber) {
            vm.loading = true;
            var offset = (pageNumber - 1) * vm.recordsPerPage;
            getContentForRelease(offset, vm.recordsPerPage)
        }

        function prevPage(pageNumber) {
            vm.loading = true;
            var offset = (pageNumber - 1) * vm.recordsPerPage;
            getContentForRelease(offset, vm.recordsPerPage)
        }

        function changePage(pageNumber) {
            vm.loading = true;
            var offset = (pageNumber - 1) * vm.recordsPerPage;
            getContentForRelease(offset, vm.recordsPerPage)
        }

        function goToPage(pageNumber) {
            vm.loading = true;
            var offset = (pageNumber - 1) * vm.recordsPerPage;
            getContentForRelease(offset, vm.recordsPerPage)
        }

        function getContentForRelease(offset, limit) {
            vm.loading = true;
            $http({
                method: 'GET',
                url: '/Umbraco/backoffice/api/Dashboard/GetContentForReleaseNextWeek?offset=' + offset + '&limit=' + limit,
                headers: {
                    'Content-Type': 'application/json'
                }
            }).then(function (response) {
                console.log(response); // logging the response so we know what to do next!

                if (response.data.ScheduledNodes.length > 0) {
                    vm.weHaveRecords = true;
                    vm.records = response.data.ScheduledNodes;
                } else {
                    vm.weHaveRecords = false;
                }

                var totalPages = Math.ceil(response.data.TotalRecords / limit);

                vm.pagination = {
                    pageNumber: (offset / vm.recordsPerPage) + 1,
                    totalPages: totalPages
                };

                vm.loading = false;
            });
        }

        init();
    }

    angular.module("umbraco").controller("MyDashboardController", DashboardController);

})();

lang/en-US.xml

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<language>
    <area alias="dashboardTabs">
        <key alias="myCustomDashboard">Custom Dashboard</key>
    </area>
</language>

In your class library:

~/Controllers/DashboardController.cs

using ListViews.Models;
using System;
using System.Collections.Generic;
using System.Linq;
using Umbraco.Core.Models;
using Umbraco.Core.Services;
using Umbraco.Web.WebApi;

namespace ListViews.Controllers
{
    public class DashboardController : UmbracoAuthorizedApiController
    {
        private readonly IContentService _contentService;

        public DashboardController(IContentService contentService)
        {
            _contentService = contentService;
        }

        public PagedContent GetContentForReleaseNextWeek(int offset = 0, int limit = 10)
        {
            var endDate = DateTime.Now.AddDays(7);

            var response = _contentService.GetContentForRelease(endDate);

            var contentWithScheduledInfo = new List<ContentWithScheduledInfo>();

            foreach (IContent item in response)
            {
                contentWithScheduledInfo.Add(new ContentWithScheduledInfo
                {
                    Content = item,
                    ScheduleInfo = item.ContentSchedule
                });
            }

            var pagedContent = new PagedContent
            {
                // filter the result set with offset and limit
                ScheduledNodes = contentWithScheduledInfo.Skip(offset).Take(limit).ToList(),
                TotalRecords = contentWithScheduledInfo.Count
            };

            return pagedContent;
        }
    }

}

~/Models/ContentWithScheduledInfo.cs

using Umbraco.Core.Models;

namespace ListViews.Models
{
    public class ContentWithScheduledInfo
    {
        public IContent Content { get; set; }
        public ContentScheduleCollection ScheduleInfo { get; set; }
    }
}

~/Models/PagedContent.cs

using System.Collections.Generic;

namespace ListViews.Models
{
    public class PagedContent
    {
        public List<ContentWithScheduledInfo> ScheduledNodes { get; set; }
        public int TotalRecords { get; set; }
    }
}

Step 6 - Hooking up infinite editing

It's actually surprisingly simple to set up infinite editing - here is what we need to do!

Firstly, we add a click event on each row in the listview, so in the dashboard html file we change out the first line within the umb-table-body, from:

<div class="umb-table-row -selectable" ng-repeat="item in vm.records">

to:

<div class="umb-table-row -selectable" ng-repeat="item in vm.records" ng-click="vm.open(item)">

in other words - we add the ng-click event listener. Note how we pass in the item, which is the model we get from our API controller.

Next we go to the dashboard controller, where we need to set up this new vm.open function:

function DashboardController($http, editorService) {

        var vm = this;

        function init() {
            getContentForRelease(0, vm.recordsPerPage);
        }

        vm.nextPage = nextPage;
        vm.prevPage = prevPage;
        vm.changePage = changePage;
        vm.goToPage = goToPage;
        vm.recordsPerPage = 2;
        vm.open = open;

        function open(item) {
            // do stuff
        }

So we've added the following:

New open function
Added the editorService to be injected in the constructor (we will need this in a second)
Assigned vm.open to the new open function

Next let's see what we need the open function to do:

function open(item) {
    var options = {
        id: [node id here],
        submit: function (model) {
            editorService.close();
        },
        close: function () {
            editorService.close();
        }
    };
    editorService.contentEditor(options);
}

That's how easy it is! We set up some config for the editor in an options object, we need to give it the id of the content node we want to open in the infinite editor, then we need to tell it to close the editor again when we submit or close the window.

Note: Curious about other options for the contentEditor? Check them out here!

Final bit is to get the ID to pass in - luckily our item model has the IContent model wrapped in it:

So we can just set it like this:

id: item.Content.Id,

Final result - and ensuring that it actually saves the values:

Note: While testing and writing this post I found an issue with nodes having a listview, so if that isn't really working as expected. See the issue here.

Outro

Thanks for reading so far!

If you like this post, have any feedback, suggestions, improvements to my hacky code or anything else please let me know on Twitter - @jespermayn

Creating a boilerplate Umbraco 9 site using project templates - part 1

Jesper Mayntzhusen — Thu, 30 Sep 2021 18:46:19 +0000

With Umbraco 9 freshly releases and running on .NET 5 lots of people need to change the way they work. One of those changes is probably to replace your current boilerplate solutions with a fresh one.

This blogpost will show you how to leverage the built in dotnet templates to do that in a very nice and easy way - let's get started!

Setting up a basic template based on an Umbraco site

Set up the site

Before we turn it into a template - is to create the basic site you want to reuse. For the purpose of this tutorial I will use the wonderful demo site made by Dennis Adolfi as a starting point: https://github.com/Adolfi/UmbracoNineDemoSite.git

Create the template files

In the root of the repository create a folder called .template.json. Inside it create a template.json file, with the following basic content:

{
    "$schema": "http://json.schemastore.org/template",
    "author": "Jesper Mayntzhusen",
    "classifications": ["Umbraco", "WebApp"],
    "identity": "Testing.Boilerplate",
    "name": "Testing boilerplate solution",
    "shortName": "test-bp",
    "tags": {
        "type": "project",
        "language": "C#"
    },
    "sourceName": "UmbracoNineDemoSite"
}

Testing out the basic template

At this point we have what we need to pack the site up into a dotnet template - of course we will do lots more later. But first let's check that it works!

First of all, let's run a super helpful PowerShell script created by Microsoft Program Manager Sayed Ibrahim Hashimi:

function Reset-Templates{
    [cmdletbinding()]
    param(
        [string]$templateEngineUserDir = (join-path -Path $env:USERPROFILE -ChildPath .templateengine)
    )
    process{
        'resetting dotnet new templates. folder: "{0}"' -f $templateEngineUserDir | Write-host
        get-childitem -path $templateEngineUserDir -directory | Select-Object -ExpandProperty FullName | remove-item -recurse
        &dotnet new --debug:reinit
    }
}

If you run this in a PowerShell terminal you can then run the function later on to clean up - more on that later.

In the terminal navigate to a folder you want to install your new site based on the template.

Install your new template with a path to your template repository - for me it in a sibling folder:

dotnet new --install ..\BoilerPlate\

You should see something like this:

Note the shorthand for creating the new template is test-bp - which is what I set in the template.json file in the beginning.

Now that the template is installed we can use it to create our test site:

dotnet new test-bp -o MyTestSite

-o specifies an output folder, and if no name is specified it will use the foldername so don't have to set -n as well

If you look at the new project it created you will see it is a copy of the one we cloned from Github - except it renamed all instances of UmbracoNineDemoSite to MyTestSite. This is because that was what we set as the sourceName in the template.json file.

Pretty easy so far!

Adding additional parameters

The sourceName parameter is a standard required parameter for dotnet templates - but if your sites are set up like ours you may need to run replacements more than for just 1 name. For that we can add additional custom parameters - these will all be available as command line params or form fields in VS.

There is another special parameter for setting the framework - for that we can add this in the template.json (remember to do this in the boilerplate solution - not the new one you created based on it):

Add it right at the end below sourceName:

    "sourceName": "UmbracoNineDemoSite",
    "symbols": {
        "Framework": {
            "type": "parameter",
            "description": "The target framework for the project.",
            "datatype": "choice",
            "choices": [
                {
                    "choice": "netcoreapp3.1",
                    "description": "Target netcoreapp3.1"
                }
            ],
            "replaces": "netcoreapp3.1",
            "defaultValue": "netcoreapp3.1"
        }
    }
}

We can also create a second symbol similar to the sourceName param that replaces a value in the template files. The UmbracoNineDemoSite has Products that has controllers, a composer, view models - etc. Let's say we use this as a baseline for different shop sites and this new site is going to be about selling Sofas - we may want to replace Product with Sofa then:

Right below the Framework symbol we add the new ProductName param:

"symbols": {
    "Framework": {
        "type": "parameter",
        "description": "The target framework for the project.",
        "datatype": "choice",
        "choices": [
            {
                "choice": "netcoreapp3.1",
                "description": "Target netcoreapp3.1"
            }
        ],
        "replaces": "netcoreapp3.1",
        "defaultValue": "netcoreapp3.1"
    },
    "ProductName": {
        "type": "parameter",
        "defaultValue": "",
        "replaces": "Product",
        "datatype":"text",
        "fileRename": "Product",
        "description": "Product to be replaced"
    }
}

WARNING: This could mess up your site if you use something too generic, Product is a terrible example - please use something that is way more specific!

Note that the replaces value is for within files, whereas the fileRename value is for actual filenames being changed, so you have a bit of granular control if you only want one type.

At this point we can reset and reinstall the template:

Reset-Templates; dotnet new --install ..\BoilerPlate\

Now we can see the new param in the help description by running:

dotnet new test-bp --help

Let's install it while using the new param:

dotnet new test-bp -o MyTestSiteTwo --ProductName Sofa

If you open and compare your solution you can now see that filenames, namespace, etc have all been changed:

Closing note

There are lots more you can do, but will leave it at this for part 1. I hope this gave a good first view of setting up your own dotnet templates!

Thanks for reading! If you like this post, have any feedback, suggestions, improvements to my hacky code or anything else please let me know on Twitter - @jespermayn

Indexing data from another node and keeping it up to date with changes in Umbraco

Jesper Mayntzhusen — Mon, 20 Sep 2021 19:41:05 +0000

Usecase and setup

This will be a short tutorial that shows you how to index data from another node on specific content nodes in the external index.

Imagine for example you are creating a library site with books nested under author nodes. You may want to index some of the author information on the books so if you search for fx authorname, author time period, etc the books show up.

So we start with a structure like this:

We have 2 parts - getting the author data indexed on the book documents and ensuring the index data is up to date when changes occur.

Indexing the author data on the book documents

To index the data we will create a Component and register it in a Composer:

using Umbraco.Core;
using Umbraco.Core.Composing;

namespace AutoImages.Composers
{
    public class SiteComposer : IUserComposer
    {
        public void Compose(Composition composition)
        {
            composition.Components().Append<BookIndexerComponent>();
        }
    }
}

In the Component we will use TransformingIndexValues event which runs when the index is being built and allows you to extend or edit the fields in the index.

In our case we will check on the currently indexed document to see if it is of the type book - if it is we continue and find it's parent and the values from it. Finally we add those values to the books index:

using Examine;
using Examine.Providers;
using System;
using System.Collections.Generic;
using System.Linq;
using Umbraco.Core.Composing;
using Umbraco.Core.Logging;
using Umbraco.Web;

namespace AutoImages.Composers
{
    public class BookIndexerComponent : IComponent
    {
        // The doc type id of the "Book" document type
        const string BookDoctypeId = "1154";

        private readonly IExamineManager _examineManager;
        private readonly IUmbracoContextFactory _umbracoContextFactory;
        private readonly IProfilingLogger _logger;

        public BookIndexerComponent(IExamineManager examineManager, IUmbracoContextFactory umbracoContextFactory, IProfilingLogger logger)
        {
            _examineManager = examineManager;
            _umbracoContextFactory = umbracoContextFactory;
            _logger = logger;
        }

        public void Initialize()
        {
            if (!_examineManager.TryGetIndex(Umbraco.Core.Constants.UmbracoIndexes.ExternalIndexName, out IIndex index))
                throw new InvalidOperationException($"No index found by name {Umbraco.Core.Constants.UmbracoIndexes.ExternalIndexName}");
            if (!(index is BaseIndexProvider indexProvider))
                throw new InvalidOperationException("Could not cast");
            indexProvider.TransformingIndexValues += IndexProviderTransformingIndexValues;
        }

        private void IndexProviderTransformingIndexValues(object sender, IndexingItemEventArgs e)
        {
            // Get the nodetypealias from the index
            if (e.ValueSet.Values.TryGetValue("nodeType", out List<object> nodeType) == false) return;

            // if it is not type book we exit
            if (nodeType.FirstOrDefault()?.ToString() != BookDoctypeId) return;
            if (e.ValueSet.Values.TryGetValue("id", out List<object> id) == false) return;
            if (id.FirstOrDefault()?.ToString() == "") return;

            using (var context = _umbracoContextFactory.EnsureUmbracoContext())
            {
                _logger.Debug<BookIndexerComponent>($"Indexing Book node with id {id}");

                try
                {
                    var node = context.UmbracoContext.Content.GetById(int.Parse(id.FirstOrDefault().ToString()));

                    if (node == null) return;

                    var author = node.Parent;

                    // get values from the parent node and index them on this document
                    e.ValueSet.Add("authorName", author.Name);
                    e.ValueSet.Add("authorStyle", author.Value<string>("style")?.ToLowerInvariant());
                    e.ValueSet.Add("authorCentury", author.Value<string>("century")?.ToLowerInvariant());               
                }
                catch (Exception ex)
                {
                    _logger.Error<BookIndexerComponent>(ex, $"Error occured while indexing fabric node with id: {id.FirstOrDefault()}");
                }
            }
        }

        public void Terminate()
        {
        }
    }
}

Finally we can check the index and see that the new author fields are added to the documents:

So if we search for some data that is only saved on the author - like their style, we will get the books as results as well since they now index that value:

Keeping the index up to date

The problem with this approach though is that we only update the book index fields whenever the index is being rebuilt for the book pages - so if we were to change some data on the author the books wouldn't automatically stay up to date.

I've gone and changed J.R.R. Tolkien to have the style high-fantasy instead, and now the books don't show up:

So let's write some code that ensures the books are re-indexed whenever the author is updated:

using Examine;
using System;
using System.Linq;
using Umbraco.Core.Composing;
using Umbraco.Core.Models;
using Umbraco.Core.Services;
using Umbraco.Core.Services.Changes;
using Umbraco.Examine;
using Umbraco.Web.Cache;

namespace AutoImages.Events
{
    public class ContentCacheRefresherEvent : IComponent
    {
        private readonly IContentService _contentService;
        private readonly IPublishedContentValueSetBuilder _valueSetBuilder;
        private readonly IExamineManager _examineManager;
        private IIndex _index;

        public ContentCacheRefresherEvent(IContentService contentService, IPublishedContentValueSetBuilder valueSetBuilder, IExamineManager examineManager)
        {
            _contentService = contentService;
            _valueSetBuilder = valueSetBuilder;
            _examineManager = examineManager;
        }

        public void Initialize()
        {
            if (!_examineManager.TryGetIndex(Umbraco.Core.Constants.UmbracoIndexes.ExternalIndexName, out IIndex index))
                throw new InvalidOperationException($"No index found by name {Umbraco.Core.Constants.UmbracoIndexes.ExternalIndexName}");
            _index = index;
            ContentCacheRefresher.CacheUpdated += ContentCacheRefresher_CacheUpdated;
        }

        private void ContentCacheRefresher_CacheUpdated(ContentCacheRefresher sender, Umbraco.Core.Cache.CacheRefresherEventArgs e)
        {
            foreach (var payload in (ContentCacheRefresher.JsonPayload[])e.MessageObject)
            {
                // only continue if the cache change is to the node content, not it being deleted, etc.
                if (payload.ChangeTypes.HasType(TreeChangeTypes.RefreshNode))
                {
                    var content = _contentService.GetById(payload.Id);
                    if (content.ContentType.Alias == "author")
                    {
                        // Get all children of the author
                        var children = _contentService.GetPagedChildren(payload.Id, 0, 999, out long totalRecords);

                        if (children.Count() == 0) return;

                        // Cast to array to fit valuesetbuild signature
                        IContent[] childArray = children.Cast<IContent>().ToArray();

                        // Rebuild the index for the authors children
                        _index.IndexItems(_valueSetBuilder.GetValueSets(childArray));
                    }
                }
            }
        }

        public void Terminate()
        {
            ContentCacheRefresher.CacheUpdated -= ContentCacheRefresher_CacheUpdated;
        }
    }
}

And as always - make sure to register the component in the composer:

using AutoImages.Events;
using Umbraco.Core;
using Umbraco.Core.Composing;

namespace AutoImages.Composers
{
    public class SiteComposer : IUserComposer
    {
        public void Compose(Composition composition)
        {
            composition.Components().Append<BookIndexerComponent>();
            composition.Components().Append<ContentCacheRefresherEvent>();
        }
    }
}

And that'll do it! At this point we can change the author info and the related books will automatically have their fields updated:

Thanks for reading! If you like this post, have any feedback, suggestions, improvements to my hacky code or anything else please let me know on Twitter - @jespermayn

Assign images to content on upload in Umbraco

Jesper Mayntzhusen — Mon, 13 Sep 2021 20:04:46 +0000

Let's say you are creating an Umbraco site that has fx a product catalogue that is imported. You may have a lot of different products that need images as well, more than you are interested in going through one by one.

Here is an example on how you can automate it, this example will use the Umbraco Starter kit as it already has a product catalogue with SKU's:

First thing first - let's limit the automation to a "Product Uploads" folder in the media library:

We will save the ID of this folder in a constants file:

namespace AutoImages.Constants
{
    public static class Content
    {
        public const int MediaUploadFolder = 1145;
    }
}

Then we will create a media event to listen for uploads to that folder. For this we can use the cacherefresher event - it is the last event after media saving, plus this way we can trigger it for everything by rebuilding the cache:

using Umbraco.Core.Composing;
using Umbraco.Core.Services;
using Umbraco.Web.Cache;

namespace AutoImages.Events
{
    public class MediaEvents : IComponent
    {
        public void Initialize()
        {
            MediaCacheRefresher.CacheUpdated += MediaCacheRefresher_CacheUpdated;
        }

        public void Terminate()
        {
            MediaCacheRefresher.CacheUpdated -= MediaCacheRefresher_CacheUpdated;
        }

        private void MediaCacheRefresher_CacheUpdated(MediaCacheRefresher sender, Umbraco.Core.Cache.CacheRefresherEventArgs args)
        {
            // do stuff
        }        
    }
}

Before diving into the event code we need to make sure it runs on the site, for that we need to register it in a composer:

using AutoImages.Events;
using Umbraco.Core;
using Umbraco.Core.Composing;

namespace AutoImages.Composers
{
    public class SiteComposer : IUserComposer
    {
        public void Compose(Composition composition)
        {
            composition.Components().Append<MediaEvents>();
        }
    }
}

Alright so now that the basics are laid out - what we actually want to happen is something like this:

Image is uploaded to our uploads folder
That image has some sort of naming convention we can use to match it with the product nodes
We search the product nodes to find a match
If one is found we "select" the image on that node

To make it easy we will just say we will overwrite the product image if the name matches the sku:

using Examine;
using System;
using System.Collections.Generic;
using System.Linq;
using Umbraco.Core;
using Umbraco.Core.Composing;
using Umbraco.Core.Services;
using Umbraco.Core.Services.Changes;
using Umbraco.Core.Sync;
using Umbraco.Web.Cache;

namespace AutoImages.Events
{
    public class MediaEvents : IComponent
    {
        private readonly IContentService _contentService;
        private readonly IMediaService _mediaService;
        private readonly IExamineManager _examineManager;

        public MediaEvents(IContentService contentService, IMediaService mediaService, IExamineManager examineManager)
        {
            _contentService = contentService;
            _mediaService = mediaService;
            _examineManager = examineManager;
        }

        private void MediaCacheRefresher_CacheUpdated(MediaCacheRefresher sender, Umbraco.Core.Cache.CacheRefresherEventArgs args)
        {
            if (args.MessageType != MessageType.RefreshByPayload)
                throw new NotSupportedException();

            var products = new Dictionary<string, string>();

            // for each media item the cache has refreshed
            foreach (var payload in (MediaCacheRefresher.JsonPayload[])args.MessageObject)
            {
                // only when the node is refreshed - so not on delete, move, etc
                if (payload.ChangeTypes.HasType(TreeChangeTypes.RefreshNode))
                {
                    var media = _mediaService.GetById(payload.Id);
                    if (media == null || media.Trashed) continue;

                    // only continue if the parent is the uploads folder
                    if (media.ParentId == Constants.Content.MediaUploadFolder)
                    {
                        // example name: UMB-BIKER-JACKET.png
                        var splitName = media.Name.Split('.');

                        // we save the sku and the media udi for later
                        products.Add(splitName[0], media.GetUdi().ToString());
                    }
                }
            }

            _examineManager.TryGetIndex(Umbraco.Core.Constants.UmbracoIndexes.ExternalIndexName, out IIndex index);
            var searcher = index.GetSearcher();

            foreach (var product in products)
            {
                var searchResults = searcher.CreateQuery().NativeQuery($"__NodeTypeAlias: product +sku: {product.Key}").Execute();

                if (searchResults.Any())
                {
                    foreach (var searchResult in searchResults)
                    {
                        if (!int.TryParse(searchResult.Id, out int contentId))
                            continue;

                        var node = _contentService.GetById(contentId);

                        // MediaPicker v2 saves the udi of the image, which was saved in the value of the product dictionary already
                        node.SetValue("photos", product.Value);
                        _contentService.SaveAndPublish(node);
                    }
                }
            }
        }

        public void Initialize()
        {
            MediaCacheRefresher.CacheUpdated += MediaCacheRefresher_CacheUpdated;
        }

        public void Terminate()
        {
            MediaCacheRefresher.CacheUpdated -= MediaCacheRefresher_CacheUpdated;
        }             
    }
}

And here is a gif of it all working 😉

That's it for now! I am not entirely happy with having to use services as much as I do, so if anyone know of a more performant way to do this please let me know!

It is however still way faster than having to do it all manually!

Set page color based on an img in Umbraco

Jesper Mayntzhusen — Fri, 03 Sep 2021 19:55:59 +0000

The problem

Currently working on a new website that shows a product catalogue. Part of the design is that each page has a background color that is based on the main product image. Here is how I've solved it.

The code

We will calculate the avg color of an image when it is added to one of the product page nodes - so in the content saving event:

~/Events/ContentServiceEvents.cs

using System;
using System.Drawing;
using Umbraco.Core;
using Umbraco.Core.Composing;
using Umbraco.Core.Models;
using Umbraco.Core.Models.PublishedContent;
using Umbraco.Core.Services.Implement;
using Umbraco.Web;

namespace Test.Events {
    public class ContentServiceEvents : IComponent {
        private readonly IUmbracoContextFactory _umbracoContextFactory;

        public ContentServiceEvents(IUmbracoContextFactory umbracoContextFactory) {
            _umbracoContextFactory = umbracoContextFactory;
        }

        public void Initialize() {
            ContentService.Saving += ContentService_Saving;
        }

        public void Terminate() {
            ContentService.Saving -= ContentService_Saving;
        }

        private void ContentService_Saving(Umbraco.Core.Services.IContentService sender, Umbraco.Core.Events.ContentSavingEventArgs e) {
            // When content is saved, we loop through each saved node
            foreach (IContent node in e.SavedEntities) {

                // We only continue if it's a product page
                if (node.ContentType.Alias == "productPage") {

                    // We only continue if the saved node has the product image set
                    if (node.GetValue("productImage") != null) {

                        // We only continue if the color isn't already set
                        if (string.IsNullOrWhiteSpace(node.GetValue<string>("imageBgColor"))) {
                            using (var context = _umbracoContextFactory.EnsureUmbracoContext()) {

                                // MediaPicker v2 saves as a media udi, so we get it to grab the media object
                                var udi = node.GetValue<Udi>("productImage");
                                var img = context.UmbracoContext.Media.GetById(udi);

                                // From https://stackoverflow.com/a/20757431/16493772
                                // This is for external media storage like Azure blob storage or AWS S3 buckets
                                // We download the image based on the media url to create a bitmap we use to calculate the avg color
                                System.Net.WebRequest request = System.Net.WebRequest.Create(img.Url(mode: UrlMode.Absolute));
                                System.Net.WebResponse response = request.GetResponse();
                                System.IO.Stream responseStream = response.GetResponseStream();
                                Bitmap bm = new Bitmap(responseStream);
                                var color = AverageColor(bm);

                                // Finally we set the avg color in a property - since this is a content saving event
                                // we don't need to save exeplicitly, it'll do that at the end
                                node.SetValue("imageBgColor", color);
                            }
                        }
                    }                                
                }
            }
        }

        // From https://stackoverflow.com/a/2546080/16493772 - Edited to get around a bug where the values would overflow the max Int32 value and become negative,
        // changed ints to longs then converting back to int in the end.
        // In short - it goes through every pixel of the img, finds its RGBA value, add them all together and divide by the number of pixels to find the average color of the image.
        public string AverageColor(Bitmap bmp) {            
            long width = bmp.Width;
            long height = bmp.Height;
            long red = 0;
            long green = 0;
            long blue = 0;
            long alpha = 0;
            for (int x = 0; x < width; x++)
                for (int y = 0; y < height; y++) {
                    var pixel = bmp.GetPixel(x, y);
                    red += pixel.R;
                    green += pixel.G;
                    blue += pixel.B;
                    alpha += pixel.A;
                }

            Func<long, long> avg = c => c / (width * height);

            red = avg(red);
            green = avg(green);
            blue = avg(blue);
            alpha = avg(alpha);

            var color = Color.FromArgb(Convert.ToInt32(alpha), Convert.ToInt32(red), Convert.ToInt32(green), Convert.ToInt32(blue));

            var hexColor = "#" + color.R.ToString("X2") + color.G.ToString("X2") + color.B.ToString("X2");

            return hexColor;
        }
    }
}

~/Composers/SiteComposer.cs

using Test.Events;
using Umbraco.Core;
using Umbraco.Core.Composing;

namespace Test.Composers {

    [RuntimeLevel(MinLevel = RuntimeLevel.Run)]
    public class SiteComposer : IUserComposer {

        public void Compose(Composition composition) {            
            composition.Components().Append<ContentServiceEvents>();
        }
    }
}

Let me know if you found this interesting, or if you know of any improvements I could make!

Checking each pixel of the image is not super performant, so there may be better ways to get the color. For 8mb images this can take ~10 seconds on the first save on my current site.