DEV Community: Anna Bastron

Learnings from using the Sitecore ADM module

Anna Bastron — Tue, 26 Nov 2024 11:00:51 +0000

If you worked with a Sitecore XP website that has been live for a long time, I am sure you are familiar with the xDB cleanup requirements. I usually recommend to agree xDB data retention and set up the clean up process from day 1, but it is not always possible. Some websites were developed on an older version of Sitecore that did not have the built-in functionality for data removal, or the volume of xDB data was underestimated, or maybe tracking was enabled later down the line without considering data retention.

This is a common issue and the exact reason why Sitecore Analytics Database Manager (ADM) module exists. It allows viewing xDB records statistics and removing collection data (raw analytical data collected by Sitecore XP) via Sitecore UI while keeping the data integrity.

However, when I tried to use ADM for the first time on Sitecore 9 and Sitecore 10 websites, I faced a few challenges, particularly when dealing with large volumes of old data. In this article, we’ll explore these challenges and look at optimisation strategies for efficient data cleanup.

How Sitecore ADM works

One thing that is critical to understand is that any ADM cleanup process consists of two key phases: task generation and task processing.

Task generation

During this phase, the module retrieves all contacts from the xDB regardless of the date range specified by the user. It begins processing the most recent contacts first and systematically works backward through the database. The module reads additional data associated with each contact, such as interactions and facets, to determine whether a record meets the deletion criteria. If a contact qualifies for deletion, its ID is stored in the [Tasks] table of the [ADM.Tasks] database for later task processing.

Task processing

In the second phase, the module processes the tasks generated in the previous step, deleting all records marked for removal.

While this approach seems robust at first, it comes with a few significant limitations, particularly around the performance and scalability of the task generation phase.

Challenges

1. Inefficient SQL queries

One of the main challenges is the way how the ADM module works with xDB Shard databases. Instead of relying on SQL indexes to efficiently filter contacts by date, the module retrieves all contacts and iterates through them, starting with the most recent. This results in slower data retrieval times, especially when working with large datasets spanning several years.

For instance, even if you specify a short date range in 2022, the module will still need to process and check all contacts, including those from 2023-2024. This significantly increases the load on the shard databases, leading to higher data input/output operations and longer processing times.

2. Threading and timeout issues

By default, the ADM module operates with 4 threads and processes 1,000 contacts per batch. While these settings may be sufficient for smaller databases, you may want to tweak these settings if you have a larger database.

The module’s batching system divides work between threads at the start of the process, meaning that if there is an SQL timeout in one of the threads, the entire thread will be aborted and the cleanup of this batch will not be completed. Therefore, the same process will have to be restarted from scratch to pick up remaining DB records. This can be frustrating, particularly when processing large datasets that take several hours to be processed.

3. Single task limit

Another challenge is that only one cleanup process can run at a time. Attempting to create a new task while another is running will terminate the existing process, which makes it critical to monitor and manage the process carefully.

4. Limited pause functionality

The task generation phase cannot be paused, it is supported only for task processing. There was an idea to run the cleanup in batches outside of business hours for one of the websites with a lot of xDB data to remove, but it was impossible to do run task generation in batches because it cannot be paused and it had to be done in one go.

Optimisation tactics

Here are some tricks that helped me to reduce processing times and achieve the best performance for the cleanup in the past:

1. Read documentation

I know this sounds obvious, but if you plan to use ADM and have not looked at the documentation that comes with it, do it now! It is quite technical and can answer many questions you already have.

2. Upscale to Premium tier

If your server or database resources are under strain, especially Shards and ProcessingPools databases, you can try temporarily upscaling them for the cleanup. Premium Azure SKUs are optimised for high data I/O, which is critical when working with the ADM intensive data processing.

3. Adjust performance settings

There is some flexibility in tweaking the performance of the ADM module. You can increase multiple NumberOfThreads, RetrieveDataBatchSize and NumberOfConnectionRetries settings in the ADM configuration files (see section "6. Performance Tuning" in the ADM module documentation). This helps speed up task generation and processing but should be balanced against server and database available resources.

4. Measure and plan accordingly

Although tempting, splitting the cleanup process into smaller date ranges may not necessarily improve performance. The ADM module will still read all contacts every time, so this strategy may not be the most efficient for some cases. Instead, allow the process to finish at least one time and monitor its completion, note the time taken to better estimate and plan additional runs.

Also, if you plan to perform the cleanup in multiple runs, start with the recent date ranges because they will be accessed first during the task generation phase.

5. Consider direct SQL queries

For older versions of Sitecore, or if you struggle to run ADM process on your xDB Shards even after tweaking performance settings, an alternative approach could be used - running direct SQL queries to clean up data (you can find a few useful scripts here). However, this method requires understanding of SQL scripts and xDB tables structure, so do this only if you have backed up your Shard DBs and you are confident in your skills.

Sitecore xDB cleanup is not always simple and the ADM module can be useful tool for managing and cleaning up contact and interaction data. I hope this article helped you to understand how the module works, its limitations and tactics for optimising its performance.

How personalisation works in Sitecore XM Cloud

Anna Bastron — Thu, 11 Jul 2024 11:07:43 +0000

In my previous article, I shared a comprehensive troubleshooting guide for Sitecore XM Cloud tracking and personalisation. The guide addresses common issues, explains investigation steps and provides solutions for these issues.

However, understanding how the personalisation engine works in depth can further help in diagnosing persistent issues and developing personalised websites. This article visualises what happens behind the scenes when you enable personalisation and tracking in your Sitecore XM Cloud applications.

Overview of the personalisation workflow

Before we start, let's familiarise ourselves with key elements of the diagram we are going to look at.

On the left hand site we can see the Browser, it is responsible for sending requests to our application and displaying the result to end users.
The JSS app sits at the top centre of the diagram and it represents the Rendering Host role in Sitecore Headless topology. It is the application that processes incoming requests and handles the presentation layer. In this case it is a Next.js application based on the XM Cloud foundation template.
Edge / XM API is shown on the right hand side, it is a GraphQL endpoint that returns layout definition and content for requested pages, including personalised variants. It can be Experience Edge API for cloud-based setups or a local CM container API endpoint for development purposes.
Finally, at the bottom we can see Tracking & Interactive API powered by the embedded instance of Sitecore CDP & Personalize. This is where audiences are stored, conditions are executed and analytics is collected.

When we talk about Next.js applications, there are two main rendering methods: Server-Side Rendering (SSR) and Static Site Generation (SSG). Sitecore XM Cloud personalisation engine works slightly differently with these two approaches so we will cover both of them to understand their specifics.

Server-Side Rendering

This is what happens when a website visitor opens a page that has personalised variants.

Step 1. When a user loads the website or navigates to a new page, the browser sends an HTTP request to the Next.js application, including cookies and HTTP headers that can be used in personalisation conditions.

Step 2. The Next.js application runs all registered middleware modules, with the Personalize middleware being of particular interest. This middleware sends an API request to the GraphQL endpoint to fetch all personalised variants for the current page configured in the CMS.

If there are no personalised variants configured for this page or the page is not found, this middleware will exit and page generation will continue as usual.

Step 3. If a page has more than one variant, the middleware sends another API request to the Personalize API to detect if the current visitor matches any of the audiences configured for this page. This is where cookies and HTTP headers received from the browser will help as they will be passed to the Personalize API to identify the visitor.

Step 4. By combining responses from these two API requests, the middleware determines which personalised page variant is suitable for the current visitor (cat-themed page in the diagram below 😺).

If the visitor matches an audience configured for the page, the middleware will rewrite the page path to a special personalised variant path (for example, from /_site_Test/Pets to /_variantId_0dd7b00680be49c6815ca4d0793a36da/_site_Test/Pets) and this will instruct the Next.js application to use the specific page variant when rendering the page. So the personalised version of the page will be rendered on the server and returned to the browser.

If the visitor does not match any audiences, then the default page variant will be rendered and returned to the user.

Step 5. Once the page is rendered in the browser, a special React component responsible for tracking will send an API request to the CDP Stream API to register the page view, including which personalised variant was shown. This data is later will be aggregated and shown in analytics reports.

Static Site Generation (SSG)

The SSG process flow is similar to SSR but has some specifics related to this rendering method. Now, let's see what are these differences.

Step 1. This step is exactly the same as for SSR - the browser sends an HTTP request to the Next.js application with cookies and HTTP headers.

Step 2. This is where things get different from the SSR process. When the Personalize middleware kicks in, it checks if there are any pre-rendered page variants for this page (the default, cat-themed 😺 and dog-themed 🐶 variants in the diagram). If yes, it skips the API request to the Edge / XM API, otherwise it will fall back to the standard SSR process and fetch personalised variants for the current page.

Step 3. This step is the same as in the SSR flow - if a page has personalised variants, the middleware sends an API request to the Personalize API to identify visitor's audience.

Step 4. If there is a match and personalised page variants are pre-generated, the middleware will rewrite the page path and then the appropriate personalised page variant will be chosen and returned to the browser (looks like it's the cat-themed variant again! 😺).

If there are no pre-generated personalised variants, but they exist in the CMS and the visitor matches one of the audiences, then the middleware will rewrite the page path, the Next.js app will generate the page variant and save the static output for future requests. This is the default process, see notes at the end of the article to learn more about static generation of personalised page variants.

If the visitor does not match any audiences, then the default page variant will be returned to the user using the statically generated HTML if it exists.

Step 5. As with SSR, once the page is returned to the browser the CdpPageView React component will send an API request to track the page view event for reporting.

As you can see, the flow is very similar for SSR and SSG. The SSG method with static HTML generation and skipping some API requests can give us a performance boost, especially for websites with high traffic and personalisation enabled on frequently visited pages.

Notes

Personalize middleware

The middleware is provided by Sitecore as a part of the JSS XM Cloud add-on for Next.js. Please note that this add-on is compatible with JSS version 21.6 and later. For earlier versions the Next.js Personalize add-on is used that is now obsolete.

This add-on is only compatible with Sitecore XM Cloud due to specific naming conventions and pre-configured settings required for the embedded CDP and Personalize instance.

Build-time static generation of personalised page variants

To expand on the step 4 of SSG process, let's see when exactly personalised page variants are generated. As you may know, hosting providers often limit the time available for SSG builds. By default, pre-generation of personalised page variants during build is disabled in to avoid long build times.

However, if sufficient build time is available (for example, your website does not have too many pages) or you have critical personalisation rules on key pages (for instance, you only have a small number of personalised variants on the homepage or an important campaign page), then SSG for personalised variants can be explicitly enabled.

This can be done by modifying the file src/lib/sitemap-fetcher/plugins/graphql-sitemap-service.ts and setting the includePersonalizedRoutes parameter to true in the sitemap service constructor:

this._graphqlSitemapService = new MultisiteGraphQLSitemapService({
    clientFactory,
    sites: [...new Set(siteResolver.sites.map((site: SiteInfo) => site.name))],
    includePersonalizedRoutes: true,
});

Just make sure to watch your build time after enabling this setting to avoid build failing or incurring unnecessary hosting costs.

Sitecore XM Cloud provides robust support for personalisation out-of-the-box for both SSR and SSG rendering methods. The add-on with Personalize middleware and CDP tracking component streamlines the process of fetching, matching, and delivering personalised content to website visitors while tracking interactions for reporting.

Hope this article helps to understand the entire process of personalisation and tracking in XM Cloud and allows you to build well-performing and personalised applications. Feel free to share your thoughts and questions in the comments!

Sitecore Personalize: tips & tricks for decision models and programmable nodes

Anna Bastron — Fri, 24 Nov 2023 12:42:23 +0000

While working with Sitecore Personalize, I've collected various findings around decision models and programmable nodes. Some of them were the result of a human error, while others were just specific rules and conventions I discovered while using the platform. So I'm sharing these tips and tricks in the hope that they prove as valuable to you as they did to me.

1. Remember to return a value from programmable nodes

Starting with a fundamental tip: when creating custom programmable nodes, ensure you include a return statement at the end of the JavaScript function. It's easy to overlook, but without it, your decision model might return unexpected results.

When I added my very first custom programmable node to a decision model, it took me a while to find why the model was working incorrectly. It turned out my programmable node was calculating everything correctly, but there was no return statement after calling an internal function. I don't know how I expected Sitecore Personalize to know which value to use! 😅

2. Set the correct return type in programmable nodes

Specify the correct return type in programmable nodes, aligning it with the decision table's comparison rules. Whether it's integer, long, double, string, boolean, date, or map, choosing the right type is necessary for seamless integration with the decision table.

3. Avoid leaving empty lines in decision tables

They are not as empty as I thought! Beware of seemingly harmless empty lines in decision tables. A forgotten line can lead to hit policy errors or produce incorrect output, because hyphens in input columns act as wildcards. For instance, in the example below, the first row applies to all guests of "customer" type, irrespective of their page views count.

4. Name decision table output columns carefully

When naming output columns in decision tables, remember that output of the decision models can be used in Freemarker templates for web or full stack experiences. Opt for output reference names without spaces or special characters for easier integration later.

5. Pay attention to workflow in decision model

Before using a decision model, move at least one variant to the production state. Once in production, the variant becomes non-editable so make sure to test it before moving it to production. But don't worry, if any changes are required further down the line, you can always duplicate the live variant and update the newly created draft.

6. Save decision model versions with comments

When making significant changes to your decision model variant, choose the "Save with comment" button. Comments will be visible in the revision log, making it easier to track changes. They will also help finding the right version to revert to if that's required.

7. Use the right case in input columns

Be cautious with the values in decision model input columns as they are case-sensitive. Inconsistent cases may lead to unexpected behaviour, so ensure you are using the right values, especially when copying them from other sources.

8. Use decision templates

If there is a programmable node that you or your marketing team will likely reuse, then consider saving it as a decision template. Decision templates allow adding configurable parameters that will be displayed as a user-friendly form and you won't need copying the same JavaScript code from one decision model to another.

9. Watch `executionTime` in test canvas

This is probably the most important recommendation from me: optimise performance of your decision models. Regardless of the type of experience and integration approach you use, long execution times will negatively affect user experience on the website and all other channels connected to Sitecore Personalize.

To ensure the most efficient performance, Sitecore Personalize imposes data limits on the amount of data that is available in decisioning and conditions. In addition to this, you can apply the following techniques to improve execution time of programmable nodes and decision models:

Look only at the data that you need by limiting the number of sessions and events you check
Break the loops and return values from functions as soon as you have found what you need to avoid accessing and iterating through unnecessary data
Leverage batch segments and user data extensions where possible to make decisions based on already calculated and saved data
Reduce the number of elements and connections on the decision canvas
Limit the number of programmable decisions on the canvas to three or four as recommended by Sitecore. If multiple programmable decisions use the same data, consider combining them into one element.
Keep the number of external data systems and AI models minimal in a decision model. Sitecore recommends to include one or two data systems and/or AI elements.
Reduce the number of rows and columns in decision tables. Where possible, combine conditions by using hyphen (-) to eliminate unnecessary checks.
Ensure that the model returns only one decision

I hope these insights ease your journey with Sitecore Personalize. Feel free to share your own tips or ask questions in the comments!

Custom guest profile attributes in Sitecore CDP

Anna Bastron — Fri, 06 Oct 2023 14:23:03 +0000

When my colleagues and I started working on a proof-of-concept for Sitecore CDP & Personalize, one of the important aspects of our plan was validation of how custom guest profile properties can be effectively stored and used in the CDP.
Many organisations need to save custom pieces of data associated with website visitors and their interactions on the website. These data properties may include marketing consent status, preferred language, or specific topics of interest just to name a few. During our research we found that Sitecore CDP offers a built-in feature for storing and reading these custom fields known as data extensions.

Types of data extensions

Sitecore CDP supports data extension attributes for the following data models:

Guests
Events
Orders

In this article I will primarily focus on event and guest data extensions, but it is worth noting that similar principles apply to orders as well.

How to create data extensions

Depending on your application specifics and requirements, you can choose one or multiple available APIs: Stream API, REST API and Batch API. Regardless of the integration method you select, Sitecore CDP will provide an option for sending custom data and saving it in the system for future use.

Stream API

For seamless integration, Engage SDK provides an easy way of passing event data extensions to the CDP. The optional object extensionData can be included as the last function argument when sending events, page views or identifying guests.
Client-side tracking methods supporting the extensionData parameters are:

Engage.pageView(eventData[, extensionData])
Engage.identity(eventData[, extensionData])
Engage.event(type, eventData[, extensionData])
Engage.addToEventQueue(type, eventData[, extensionData])

Server-side SDK tracking methods are similar, but they also require passing the HTTP request parameter req:

EngageServer.pageView(eventData, req[, extensionData])
EngageServer.identity(eventData, req[, extensionData])
EngageServer.event(type, eventData, req[, extensionData])

If you opt not to use Engage SDK and prefer to integrate through direct HTTP requests, you can still supply additional data with an event by utilising the ext data extension object. Please note that the name of the event data extension object must be ext, otherwise Sitecore CDP will not recognise it as a data extension.

Here's an example of a data extension object for HTTP requests:

{
  "type": "VIEW",
  "channel": "WEB",
  "pos": "POS name",
  "browser_id": "…",
  "language": "EN",
  "currency": "EUR",
  "page": "home",
  "ext": {
    "pageCategory": "value",
    "bonusPoints":10
  }
}

Please note that after you send the event, the data in this object becomes available in the event, but not in the guest profile.

Some other limitations that are worth noting are:

An event data extension object can have a maximum of 50 attributes
The attribute name must be alphanumeric, i.e. it should only contain characters A-Z, a-z, or 0-9
The attribute name must be unique within the entire event type

REST API

If you integration requires a synchronous communication method and you can’t use Stream API, the REST API is a good alternative. It provides a special API endpoint for sending extension data:

POST /v2/guests/<guestRef>/extext

It’s important to note that extext consists of two parts: the first ext is constant and means that we want to send extension data. The second ext represents the name of the extension property in Sitecore CDP. The system supports multiple extension group names allowing you to send data extensions from various source systems and store them separately. Failing to do so will result in overwriting guest extension data from other sources. You can create a maximum of six guest data extension groups and use only predefined names: ext, ext1, ext2, ext3, ext4, ext5.

For instance, if you want to send extension data from a second system, you can modify the request to use the extension group ext1 as follows:

POST /v2/guests/<guestRef>/extext1

Request body example:

{
    "key": "default",
    "loyaltyTier": "level2",
    "loyaltyNumber": 12345,
    "marketingConsent": true,
}

Please note that the key attribute must be set to default and guest data extension can have a maximum of 100 attributes. Naming conventions are the same as for the event extensions:

Unique attribute name
The attribute name must be alphanumeric and written in camel case (see examples above)

REST API also supports GET, POST and DELETE endpoints for retrieving, updating and deleting specific data extensions properties. To retrieve a data extension property from the guest profile, you can send a request like this:

GET /v2/guests/f7aabbca-1c1b-4fc2-be72-3e16294a4f03/extext1/09bfc8cc-3815-5b77-b75a-d3cc6a867c1a

where f7aabbca-1c1b-4fc2-be72-3e16294a4f03 is the guest reference, ext1 is the data extension group and 09bfc8cc-3815-5b77-b75a-d3cc6a867c1a is the data extension property reference.

Batch API

Additionally, you can send send additional data via Batch API https://doc.sitecore.com/cdp/en/developers/api/send-additional-guest-data.html. The principle is similar to the REST API, where you send data extensions using the ext property. You can specify one of the six extensions group names to differentiate data from multiple systems. Batch API is particularly suitable for asynchronous integration scenarios, such as daily imports from a CRM system to synchronise customer records updated within the last 24 hours.

How to use data extensions

Now, why would you need to send some guest information to CDP as data extensions?

One of the most common use cases is batch segmentation which enables creation of guest segments of guests based on their profile, event, and order extensions. This can be extremely helpful for implementing marketing automation campaigns such as sending newsletters or push notifications to specific groups of users. You can read more about batch segmentation in Sitecore CDP in my colleague’s blog post here.

Batch Segment configuration screen showing examples of customer profile fields and data extensions.

However, usage of data extensions is not limited to batch segmentation. Custom properties can be leveraged to build real-time audiences for live personalisation and A/B testing. They can be accessed within programmable nodes in decision models and returned as a part of API response in web and full stack experiences. Furthermore, this data can be extracted from Sitecore CDP and pushed to external systems, such as Tableau or Power BI for custom reporting.

In conclusion, Sitecore CDP's data extensions feature is essential for storing additional data that will help building marketing automation and personalisation, as well as getting insights into visitor segments and interactions.

By understanding how to create, manage, and read data extensions, you can harness the full potential of your customer data within Sitecore CDP & Personalize.

Server-side integration with Sitecore Personalize API

Anna Bastron — Mon, 25 Sep 2023 08:26:51 +0000

As promised, in this post I will walk you through an example of API integration with Sitecore Personalize full stack experiences. While I am using C# for demonstration purposes, keep in mind that the underlying principles are the same for applications written in JavaScript, Kotlin, Swift or any other programming language.

In my previous article I explained how to configure API response in Sitecore Personalize. Now, let's take the next step to read this response in our application.

Imagine you have a website that requires personalisation functionality, for example displaying content tailored to the website visitors’ behaviour. Some scenarios can be achieved with web experiences and JavaScript injected on the page. However, in this example I will show you how it can be done with interactive experiences for scenarios that require a deeper level of integration between the application and personalisation engine.

Preparation Steps

Before we start, I would recommend creating a Postman request for testing purposes. This allows you experiment with different request types and parameters to make sure API behaves as expected and your custom API response is correctly formatted.

You should already have an interactive experience configured in Sitecore Personalize. In case you need a quick refresher, refer to the useful video guide in this article. Go to your experience details and grab the friendly ID from the Setup Details section. You will also need a browser ID or an identifier of a guest that exists in CDP.

With these prerequisites in place, open Postman or a similar tool and set up a POST request to /v2/callFlows. The full specification for this API request is available in Sitecore documentation here.

You must include one of the following attributes: browserId or email or identifiers to identify the guest you are sending this request for. Optionally, if your experience supports custom fields, you can include them using the params object:

{
    "clientKey": "…",
    "channel": "WEB",
    "language": "en",
    "currencyCode": "EUR",
    "pointOfSale": "…",
    "browserId": "…",
    "params": {
      "someKey":"someValue"
    },
    "friendlyId": "…"
}

This is how my test request looks in Postman:

Writing Code

Once you have verified that the API responds as expected via Postman, it is time to start developing the integration. I have set up a vanilla Sitecore website with basic Sitecore CDP tracking, generating page view events and guest profiles.

API integration code consists of the following elements:

1. FlowExecutionService.cs

This class is responsible for sending API requests to /v2/callFlows. I am using the standard System.Net.Http.HttpClient for simplicity but you are free to choose your preferred REST API library. Here is a code snippet:

public class FlowExecutionService
{
        private readonly HttpClient _httpClient;

        public FlowExecutionService(HttpClient httpClient)
        {
            _httpClient = httpClient;
            _httpClient.BaseAddress = new Uri(ConfigSettings.APIEndpoint);

        }

        public async Task<T> ExecuteFlow<T>(FlowExecutionRequest request)
        {
            using (var httpRequest = new HttpRequestMessage(HttpMethod.Post, "callFlows"))
            {
                var json = JsonConvert.SerializeObject(request);
                httpRequest.Content = new StringContent(json, System.Text.Encoding.UTF8, "application/json");

                using (var response = await _httpClient.SendAsync(httpRequest).ConfigureAwait(false))
                {
                    var content = await response.Content.ReadAsStringAsync();
                    var result = JsonConvert.DeserializeObject<T>(content);
                    return result;
                }
            }
        }
}

2. Then there are two model classes: FlowExecutionRequest.cs and FlowExecutionResult.cs. FlowExecutionRequest is quite generic and can be reused for calling different experiences.

public class FlowExecutionRequest
{
        [JsonProperty("clientKey")]
        public string ClientKey { get; set; }

        [JsonProperty("channel")]
        public string Channel { get; set; }

        [JsonProperty("language")]
        public string Language { get; set; }

        [JsonProperty("currencyCode")]
        public string CurrencyCode { get; set; }

        [JsonProperty("pointOfSale")]
        public string PointOfSale { get; set; }

        [JsonProperty("email")]
        public string Email { get; set; }

        [JsonProperty("browserId")]
        public string BrowserId { get; set; }

        [JsonProperty("friendlyId")]
        public string FriendlyId { get; set; }
}

FlowExecutionResult should be specific to your experience and match the API response your configured with FreeMarker. That’s why I defined ExecuteFlow<T> as a generic method so that it can return different response models. This is just one example of how this model can look like:

public class FlowExecutionResult
{
        [JsonProperty("guestEmail")]
        public string GuestEmail { get; set; }

        [JsonProperty("guestName")]
        public string GuestName { get; set; }

        [JsonProperty("contentItemId")]
        public string ContentItemId { get; set; }
}

3. Finally, models and service calls are combined in a controller. Take a look at the method Banner() below, it prepares the FlowExecutionRequest model, populates it with API configuration settings and passes email address of the logged-in user. If you want to use browser Id instead of email, it can be passed from the front-end code or read from the bid_ cookie like this:

BrowserId = HttpContext.Request.Cookies["bid_" + ConfigSettings.ClientKey];

The controller calls the FlowExecutionService and retrieves FlowExecutionResult which in my case contains guest’s name and email address as well as personalised component data source ID. This data source is taken from the website database and passed to the view for rendering the output:

public class BannerController : Controller
{
        private readonly FlowExecutionService _flowExecutionService;

        public BannerController(FlowExecutionService flowExecutionService)
        {
            _flowExecutionService = flowExecutionService;
        }

        public ActionResult Banner()
        {

            var request = new FlowExecutionRequest()
            {
                Channel = ConfigSettings.Channel,
                ClientKey = ConfigSettings.ClientKey,
                CurrencyCode = ConfigSettings.Currency,
                Email = Sitecore.Context.User.Profile?.Email,
                FriendlyId = ConfigSettings.ExperienceId,
                Language = Sitecore.Context.Item.Language.Name,
                PointOfSale = ConfigSettings.POS
            };

            var result = _flowExecutionService.ExecuteFlow<FlowExecutionResult>(request).GetAwaiter().GetResult();
            var model = new BannerModel()
            {
                GuestEmail = result?.GuestEmail,
                GuestName = result?.GuestName
            };

            if (result != null && ID.TryParse(result.ContentItemId, out var id))
            {
                model.ContentItem = Sitecore.Context.Database.GetItem(id);
            }

            return View("~/Views/FullStackExperience/Banner.cshtml", model);
        }
}

Results

By creating a simple proof-of-concept like this, you will be able to integrate with much more complex interactive experiments and experiences in Sitecore Personalize. Depending on your business requirements, you can build a framework that allows specifying experience’s friendly ID and other parameters in the calling application. For instance, it can be a data source in Sitecore content tree or a configurational content item in a headless CMS. Building a foundation layer can streamline the process of adding new experiences for your marketing teams.

For a complete working example, feel free to explore the code repository on GitHub.

Using FreeMarker template engine in Sitecore Personalize

Anna Bastron — Mon, 28 Aug 2023 07:30:00 +0000

If you have tried creating an experiment or experience within Sitecore Personalize or completed a training course, the phrase "FreeMarker syntax" might have caught your attention. In this article I will explain what is FreeMarker and how it helps to unlock the full potential of personalisation and experimentation capabilities in Sitecore Personalize.

What is FreeMarker?

FreeMarker is a template engine, it allows to generate text output based on templates and dynamic data. It is similar to Mustache, Handlebars, Thymeleaf and other template engines. Templates are written in the FreeMarker Template Language (FTL) that supports conditional blocks, iterations, formatting, and many other capabilities.

How is FreeMarker used in Sitecore Personalize?

As a template engine, FreeMarker helps to produce output of the predefined format and make it dynamic based on the provided data input.

There are two main areas where FreeMarker is used in Sitecore Personalize:

Web experiences and experiments
Interactive experiences and experiments

For both of them Sitecore Personalize allows to construct API response using FreeMarker language that will be executed on the server side. Do not confuse this with Handlebars template language that is used for HTML, CSS and JavaScript sections. The key difference is that HTML, CSS and JavaScript templates will be executed on the client-side by Handlebars.js library:

What data is available in API response?

API response can reference and transform any server-side data that is available for the current guest: including guest profile data, his sessions, orders, segment memberships, and the decision model that is associated with this experience.

API response data can then subsequently be used in the HTML, CSS and JavaScript sections of a web experience. Imagine wanting to access the guest’s name or email address in your JavaScript code. To achieve this, you must configure the API response to include the dynamic data so that when the experience is executed, the data is requested from the server and then passed to the front-end code.

Interactive experiences work in a similar manner, but they don’t contain any HTML, CSS and JavaScript. Irrespective of the programming language you choose, your application can interact with the REST API to get dynamically produced API response and then use this data as required. I will show a simple example of such application in my next article.

Examples and tips

Example 1: Basic API Response

Please see a basic example of API response mark-up below. This API response returns guest’s email address and full name so that it can be accessed by the application:

{
    "guestEmail": "${guest.email}", 
    "guestName": "${guest.firstName} ${guest.lastName}" 
}

Example 2: Conditional Sections and Segments

Let’s add a conditional section with segments that are applicable to the current guest:

{
    "guestEmail": "${guest.email}", 
    "guestName": "${guest.firstName} ${guest.lastName}" 
<#if (guest.segmentMemberships)??> 
    , 
    "segmentMemberships": [ 
    <#list guest.segmentMemberships as segment> 
        ${toJSON(segment)}<#if (segment?has_next)>,</#if> 
    </#list> 
    ] 
</#if> 
}

You may have noticed multiple <#if> statements in the example above, they are required to ensure that the final JSON result is valid. If there are any extra commas or non-closed brackets, Sitecore Personalize will return an error saying that the syntax is not correct.

Example 3: Decision Model Results

Another common use case for API response is returning results of the linked decision model:

{
    "guestEmail": "${guest.email}", 
    "guestName": "${guest.firstName} ${guest.lastName}" 
<#if (decisionModelResults.decisionModelResultNodes)??> 
    , 
    "contentItemId": 
    <#list decisionModelResults.decisionModelResultNodes as resultNode> 
        <#list resultNode.outputs as output> 
            <#if (output.ContentItemID)??> 
                "${output.ContentItemID}" 
            </#if> 
        </#list> 
    </#list> 
</#if> 
}

Depending on how the decision model is configured, it can return one or multiple output nodes so you should consider this when designing the API response.

As you can see, FreeMarker templates in Sitecore Personalize are quite flexible and allow to tailor API response depending on your application needs and data available within Sitecore CDP. In my next blog post I will show an example of how an application can integrate with the REST API to get dynamic API responses from Sitecore Personalize. If you found this article helpful, feel free to share it with fellow developers and personalisation enthusiasts.

Building decision models with Sitecore Personalize: usage of programmable nodes

Anna Bastron — Fri, 28 Jul 2023 08:32:47 +0000

In our recent research of Sitecore CDP and Personalize, my colleagues and I looked into decision models, aiming to understand how to dynamically recommend the most relevant content variants based on specific rules. We soon discovered that decision models offer much more than a simple drag-and-drop canvas! In this article I will focus on programmable nodes, an important element of the toolset available to us in Sitecore Personalize.

Understanding decision models

Decision models are the foundational element of personalised content delivery in Sitecore Personalize. They incorporate several essential components:

Programmable nodes: these nodes are where the magic happens. Technical users familiar with JavaScript can utilise programmable nodes to define custom business logic for dynamic decision making.
Offers: content repositories that can be used for generating recommendations. Offers are what users eventually see on your platform.
External systems: connections to external analytical models or data systems that help drive decisions within the models. A typical reason for using an external system in a decision model is to retrieve data or perform a calculation and then pass it into another decision component.
Decision tables: a powerful tool for crafting decision logic that allows you to map conditions and outcomes efficiently.

The importance of programmable nodes

Programmable nodes are where developers and marketers collaborate to fine-tune the decision models. These nodes allow you to incorporate server-side JavaScript to process and analyse user behaviour data captured from Sitecore CDP across multiple sessions.

Why are programmable nodes so important?

Flexibility: programmable nodes offer unparalleled flexibility, enabling you to implement custom business logic tailored to your specific requirements.
Access to user data: you can access guest objects and their properties from programmable nodes, including data extensions, sessions, orders, and more. This rich data allows you to make decisions and identify which content to display for this guest.
Endless possibilities: with JavaScript expertise at your disposal, there are plenty of possibilities. You can easily write custom code to meet your business expectations and create unique, engaging user experiences.

Creating custom programmable nodes in Sitecore Personalize

To illustrate the process, let's look at an example of a custom programmable node. Imagine you want to calculate the total number of "VIEW" events across all sessions for a given user and output it as an integer.

In this code snippet, we define a function called getNumberOfEvents that calculates the number of "VIEW" events within each session and returns the total count.

Custom programmable nodes for unique use cases

Besides event calculations, you can build programmable nodes to cater to various other use cases. For instance, you could check the session state to determine if a user's session is still active:

Alternatively, you might want to trigger specific actions based on the last event type, such as completion of checkout:

Empowering personalised experiences

Programmable nodes in Sitecore Personalize provide a powerful tool to create highly personalised user experiences. By leveraging server-side JavaScript and collaborating with technical experts, marketers can configure decision models to cater for various scenarios and user preferences. So, unlock the potential of programmable nodes and deliver outstanding content tailored precisely to your users' needs! 🚀

For a detailed guide on creating decision models, check out my colleague's article here. Once you have that foundation, you're all set to start your journey of customisation with programmable nodes. In my next articles I will talk more about other technical features of Sitecore Personalize. Stay tuned! 😊

Sitecore Federated Authentication and that annoying error IDX21323: RequireNonce is '[PII is hidden]'

Anna Bastron — Tue, 02 May 2023 18:17:55 +0000

Have you ever experienced an error that keeps appearing under different circumstances and seems to be caused by random reasons? You can fix it once or twice but it comes back in a couple of days or in another environment or for another user. This was the error IDX21323 for me, it was so annoying that after a few cases I started saying "No, not you again!" 😅

I worked with Federated Authentication in different versions of Sitecore and integrated it with multiple identity platforms such as Okta, Auth0 and Azure AD B2C. It is a good foundational layer that provides a lot of core functionality out-of-the-box, but there is still plenty of room for mistakes and learnings. In this article I want to share some of these learnings, specifically about the error IDX21323: RequireNonce is '[PII is hidden]'.

The main symptom of it is the fact that Sitecore does not authenticate the user correctly after successful redirect back from the identity provider login page. Instead of the configured redirectURL, it sends user to the /error page with the following message in the URL query string:

IDX21323: RequireNonce is '[PII is hidden]'. OpenIdConnectProtocolValidationContext.Nonce was null, OpenIdConnectProtocol.ValidatedIdToken.Payload.Nonce was not null. The nonce cannot be validated. If you don't need to check the nonce, set OpenIdConnectProtocolValidator.RequireNonce to 'false'. Note if 'nonce' is found it will be evaluated.

The message itself is not very clear but we can see more details in the Owin.log file:

26672 08:37:29 WARN  Microsoft.Owin.Security.OpenIdConnect.OpenIdConnectAuthenticationMiddleware - The nonce cookie was not found.
26672 08:37:29 ERROR Microsoft.Owin.Security.OpenIdConnect.OpenIdConnectAuthenticationMiddleware - Exception occurred while processing message: 
Exception: Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectProtocolInvalidNonceException
Message: IDX21323: RequireNonce is '[PII is hidden]'. OpenIdConnectProtocolValidationContext.Nonce was null, OpenIdConnectProtocol.ValidatedIdToken.Payload.Nonce was not null. The nonce cannot be validated. If you don't need to check the nonce, set OpenIdConnectProtocolValidator.RequireNonce to 'false'. Note if a 'nonce' is found it will be evaluated.
Source: Microsoft.IdentityModel.Protocols.OpenIdConnect
   at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectProtocolValidator.ValidateNonce(OpenIdConnectProtocolValidationContext validationContext) in C:\agent2\_work\56\s\src\Microsoft.IdentityModel.Protocols.OpenIdConnect\OpenIdConnectProtocolValidator.cs:line 639
   at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectProtocolValidator.ValidateAuthenticationResponse(OpenIdConnectProtocolValidationContext validationContext) in C:\agent2\_work\56\s\src\Microsoft.IdentityModel.Protocols.OpenIdConnect\OpenIdConnectProtocolValidator.cs:line 264
   at Microsoft.Owin.Security.OpenIdConnect.OpenIdConnectAuthenticationHandler.<AuthenticateCoreAsync>d__11.MoveNext()

Ultimately this error means that the Owin middleware was not able to read the nonce cookie. In fact, OpenIdConnectProtocolValidator can return many other error codes but I'll focus just on one of them today. So why can this error appear?

Federated Authentication process

To answer this question, we need to understand how Sitecore Federated Authentication works and what it does behind the scenes.
Here is how the happy path should look like:

Step 1. Browser requests a page that contains a login link.
Step 2. Sitecore finds the IdentityProvider registered for the specified website name and generates login URL using the pipeline GetSignInUrlInfo.
Step 3. User clicks the login link and browser sends a POST request to the generated login URL.
The URL should start from /identity/externallogin?...
Step 4. The processor HandleLoginLink performs the following logic:

ensures that the request method is POST
includes RedirectUri parameter in the Authentication properties
calls OWIN AuthenticationManager.Challenge() method passing the correct authentication type Then OWIN generates nonce, saves it in a cookie and redirects to the identity provider passing the nonce in the message.

Step 5. Browser redirects user to the Identity Provider sign in page.
Step 6. IDP authenticates the user and returns a JWT token.
Step 7. Browser redirects back to Sitecore application with JWT token.
Step 8. OWIN validates that nonce returned in the JWT token is valid and matches the original nonce cookie.
Sitecore IdentityProviderProcessor validates the token, authenticates the user and redirects to the specified RedirectUri.

If something goes wrong at any of these steps, the error IDX21323 can appear. There are some common reasons that can cause it:

Sitecore pipeline GetSignInUrlInfo was not used for generating the sign in URL.
Original domain of the website and the configured redirect URL domain are different, therefore Sitecore does not have access to the nonce cookie after redirect back from the identity provider.
Cookies or session was cleared between the sign in URL generation and redirect back from the external provider.
Nonce cookie is returned by the server but the browser blocks it.
ASP.NET Session ID cookie is not created.

For example, the last time I saw this error it was intermittent and happened only on one machine and we could not reproduce it anywhere else. It turned out that the problem was caused by the disabled xDB.Enabled and xDB.Tracking.Enabled settings. If a request to the login page was made in a new incognito browser window, there was no ASP.NET Session ID cookie created before the redirect was made to the identity provider, therefore Sitecore could not validate that the response from the identity provider was made from the same user session.

The root cause of this behaviour is a bug in Microsoft's Owin implementation for System.Web - sometimes cookies are not saved correctly due to a conflict between Owin and System.Web libraries, as a result ASP.NET_SessionId and Owin-related cookies are not saved and it breaks authentication process. You can read more about this bug and possible workarounds on this page.

Troubleshooting steps

If you faced this error, the following steps can help to troubleshoot it:

Look at requests on the browser Network tab. It can be an IIS redirect that breaks the POST request and the problem is not even in the nonce cookie!
Check that the cookies are generated and saved correctly - both ASP.NET Session ID and nonce cookie.
If the website redirects back to the /error page, check the error message in query string.
Review Sitecore log and Owin.log for any relevant error messages.

Bonus tip:
If you have just started integration and don't have any components that display the logged in user details yet, create a simple test rendering that will output all information required for debugging.

Troubleshooting Sitecore content tests

Anna Bastron — Thu, 05 Jan 2023 12:37:58 +0000

Sitecore Experience Optimization is a powerful tool that allows running A/B and multi-variant tests to help content and marketing teams improve their website content and overall customer experience. However, a fully-scaled Sitecore XP has many parts that are involved in test data processing and aggregation. I met teams who had set up their A/B tests in Sitecore but struggled to see them working or couldn't get any reports.

In this post I will go through the test lifetime from creation to reporting to help you identify which part of the system is not working as expected.

Is my A/B test working?

The first step is to check that the test works and visitors can see different experiences. This can be done by reloading the test page a few times in an incognito browser window or by clearing cookies between attempts. Eventually you should see an alternative version of the page or component.

If it does not happen and you always see the default variant, it means that Sitecore XP features are not configured properly or the Content Delivery server does not see the test definition in web database.

The first and obvious thing to check is that these three settings are enabled in Sitecore config on the Content Delivery server: xDB.Enabled, xDB.Tracking.Enabled and ContentTesting.Enabled.
ContentTesting.Enabled setting alone is not enough because Sitecore checks xDB.Enabled and xDB.Tracking.Enabled behind the scenes to determine whether content tests should run or not.
Check that you have SC_ANALYTICS_GLOBAL cookie in the browser. Cookie consent modules and custom code can block this cookie if there is no consent given.
Another possible reason for missing analytics cookie can be lack of @Html.Sitecore().VisitorIdentification() tag in the website layout file.
Make sure that your page with the test and all relevant data sources are published to the Web database. If you are not sure, try republishing the page with its related items (child items are not required).
Another important element is the test definition item, it should be located in the folder /sitecore/system/Marketing Control Panel/Test Lab, have the workflow state Deployed and published to the Web database too.
When in doubt, it is always good to check Content Delivery server logs, never underestimate this step!

Is test data being saved to xDB?

If you can see test experiences, the next step is making sure that your visits are recorded correctly in the xDB on session end. You will need access to the Shard DBs via SQL Management Studio or Azure Query Editor.

Use the script below to check recent entries in the Interaction table of Shard DBs (if you have multiple shards, remember to check each of them!).

USE [database_name] -- replace with your Shard DB name, you may need to check all your Shard DBs
GO

SELECT TOP(100) * --always limit number of results, especially if you Shard DB is large
FROM [xdb_collection].[Interactions]
WHERE ContactId = '0E3CC254-C440-0000-0000-062C9FE15922' -- replace with your contact ID if it is known
AND StartDateTime > '2022-01-01 12:00' -- replace with your date and time, keep the time range as small as possible

-- be careful when adding more fields to the WHERE condition because Interactions table has only 2 indexes and searching for large number of interactions can be slow

Your interaction should contain the special event called MVTestTriggered in the Events column:

You can also try checking that the contact has the facet TestCombinations:

USE [database_name] -- replace with your Shard DB name, you may need to check all your Shard DB
GO

SELECT *
FROM [xdb_collection].[ContactFacets]
WHERE ContactId = '0E3CC254-C440-0000-0000-062C9FE15922' -- replace with your contact ID
AND FacetKey = 'TestCombinations'

There should be a row with data like this in the FacetData column:

If you don’t see this data in any of the Shard databases, the problem is between the CD server, xConnect collection service and xDB.

Make sure there are no files in the folder /App_Data/SubmitQueue on CD server. If there are any, it means that CD collected tracking information but had problems pushing sessions to the xConnect collection.
Have you already checked your CD server logs? 😊

If xConnect collection service returns an error or unavailable, it will be written to the CD log. I will not go into every single scenario here because most of these errors have already been discovered and solved by the community. Here are some of the articles that helped me to resolve similar issues in the past:

Is xDB data being aggregated?

Moving on, if you see raw data in the Interactions table correctly, the next step is finding this data in the Reporting database. The following tables will be of interest to us:

Fact_MvTesting
Fact_TestConversions
Fact_TestPageClicks

USE [database_name] -- replace with your Reporting DB nam
GO

-- [Fact_MvTesting] contains aggregated number of visits, engagement value, bounces, etc. for each test variant
-- TestSetId is ID of test definition item and TestValues represent each of the test variants
SELECT *
FROM [dbo].[Fact_MvTesting]
WHERE TestSetId = 'DB071F98-C2CE-4DBB-8873-2E378F88CEB4' -- replace with your test ID

-- [Fact_TestConversions] stores aggregated number of triggered goals, associated visits and engagrement value
-- counted for each test variant and goal ID
SELECT *
FROM [dbo].[Fact_TestConversions]
WHERE TestSetId = 'DB071F98-C2CE-4DBB-8873-2E378F88CEB4' -- replace with your test ID
AND GoalId = '3430F877-4767-4B2E-A121-FDA73122FEFB' -- optionally you can filter by a specific goal ID

-- [Fact_TestPageClicks] represents aggregated number of page views
-- counted for each test variant and page ID
SELECT *
FROM [dbo].[Fact_TestPageClicks]
WHERE TestSetId = 'DB071F98-C2CE-4DBB-8873-2E378F88CEB4' -- replace with your test ID
AND ItemId = 'D8771EF4-47D5-4EDA-9F96-0785207AA052' -- optionally you can filter by a page ID

If there is no data in these tables or data is too old, it’s time to look at the Processing role.

Check log files on your Processing server.
Ensure that xDB.Enabled, xDB.Tracking.Enabled and ContentTesting.Enabled are set to true on the Processing server. If they are not, the Processing server will not aggregate test data correctly.
Review server resource consumption on the Processing server and databases ReferenceData, Shards and Reporting. If the server or databases are maxing out, you will see delays in aggregated data appearing in the Reporting DB.

Is aggregated data being returned correctly?

Starting from Sitecore 10.1 the xDB Reporting role has been combined with the Content Management role to reduce hosting costs. So if you have version 10 or lower, check your Content Management and xDB Reporting roles, especially connectivity between them.

For Sitecore 10.1 and higher, it must be an issue with the Content Management role or its connection to the Reporting DB.

Hopefully this article shed some light on how the A/B test data transforms and flows between different parts of Sitecore platform and which things to check when troubleshooting A/B tests.

All scripts from this article are saved in the GitHub repository https://github.com/geann/troubleshooting-sitecore-content-tests so you can save them for future use.

geann / troubleshooting-sitecore-content-tests

Scripts for troubleshooting Sitecore content tests

This repository contains SQL scripts used in the article Troubleshooting Sitecore content tests:

Interactions.sql - this script checks raw interactions data in Shard databases with ability to filter by a date and/or contact ID.
Contact facets.sql - checks if a contact contains a special facet with A/B test combinations, i.e. validates the fact that this contact participated in an A/B test. Results can be filtered by contact ID.
Reporting.sql - checks test results in the Reporting database for specific test ID:
- overall test statistics
- test conversions for specific goal
- test page clicks

Screeenshots with example data are saved for reference to the folder /screenshots.

View on GitHub

DEV Community: Anna Bastron

Learnings from using the Sitecore ADM module

How Sitecore ADM works

Task generation

Task processing

Challenges

1. Inefficient SQL queries

2. Threading and timeout issues

3. Single task limit

4. Limited pause functionality

Optimisation tactics

1. Read documentation

2. Upscale to Premium tier

3. Adjust performance settings

4. Measure and plan accordingly

5. Consider direct SQL queries

How personalisation works in Sitecore XM Cloud

Overview of the personalisation workflow

Server-Side Rendering

Static Site Generation (SSG)

Notes

Personalize middleware

Build-time static generation of personalised page variants

Sitecore Personalize: tips & tricks for decision models and programmable nodes

1. Remember to return a value from programmable nodes

2. Set the correct return type in programmable nodes

3. Avoid leaving empty lines in decision tables

4. Name decision table output columns carefully

5. Pay attention to workflow in decision model

6. Save decision model versions with comments

7. Use the right case in input columns

8. Use decision templates

9. Watch executionTime in test canvas

Custom guest profile attributes in Sitecore CDP

Types of data extensions

How to create data extensions

Stream API

REST API

Batch API

How to use data extensions

Server-side integration with Sitecore Personalize API

Preparation Steps

Writing Code

Results

Using FreeMarker template engine in Sitecore Personalize

What is FreeMarker?

How is FreeMarker used in Sitecore Personalize?

What data is available in API response?

Examples and tips

Example 1: Basic API Response

Example 2: Conditional Sections and Segments

Example 3: Decision Model Results

Building decision models with Sitecore Personalize: usage of programmable nodes

Understanding decision models

The importance of programmable nodes

Why are programmable nodes so important?

Creating custom programmable nodes in Sitecore Personalize

Custom programmable nodes for unique use cases

Empowering personalised experiences

Sitecore Federated Authentication and that annoying error IDX21323: RequireNonce is '[PII is hidden]'

Federated Authentication process

Troubleshooting steps

Troubleshooting Sitecore content tests

Is my A/B test working?

Is test data being saved to xDB?

Is xDB data being aggregated?

Is aggregated data being returned correctly?

geann / troubleshooting-sitecore-content-tests

Scripts for troubleshooting Sitecore content tests

Scripts for troubleshooting Sitecore content tests

9. Watch `executionTime` in test canvas