DEV Community: Sam Kirkpatrick

Azure DevOps Pipeline Caching

Sam Kirkpatrick — Tue, 01 Mar 2022 22:32:23 +0000

If you're an Azure-DevOps-using organisation, pipelines are obviously key to an effective CI/CD strategy. And there's a skill to getting the steps right for your context. Some tasks you still need to do every single time, even though the contents might not change (much) between builds.

We build .NET solutions which often have framework-based front-ends (or at least some form of front-end packaging process). So we've always got NuGet references and node_modules folders. It's a rare thing that these change between runs of a pipeline, but it can happen, so we need to fetch updates or run npm install each time.

Within pipelines, these steps can occasionally take quite some time - possibly several minutes depending on the agent count and capability, size of the payloads or the number of packages/references you have. And it really starts to be come a factor if you're running multiple pipelines for multiple projects simultaneously throughout the day.

Alas, "Cache" tasks to the rescue...

Cache tasks allow you to store blobs based on a key. Adding a cache step to your pipeline will check against the key and set a variable indicating whether there has been a hit or not. If there is a cache hit, the blob will be downloaded to the pipeline working directory and the value of the hit variable can be used a condition on running later steps in the pipeline.

Worked Example - `node_modules`

Let's work through how exactly we use Cache steps for a NodeJS build (which would ordinarily create a node_modules folder). Since, during the development of our application, we may add new dependencies, we do need to run the npm install command on each execution of the pipeline.

Take a pipeline with a traditional "npm install" step.
Add a new "Cache" task above that step, and fill it in something like this: This initialises a compound cache key containing the static string "npm", the Agent OS name and the unique hash of the "package-lock.json" file and tells Azure to store the contents of the specified "node_modules" folder as the blob attached to the cache key.
Modify the "Control Options" of the "npm install" task to use the cache hit variable value defined above. This tells Azure to only run this step if the "NodeJsApp" cache hit variable is not equal to 'true' (i.e. a cache miss).

And, um, that's it.

On first run, the pipeline logs will get a cache miss and run the "npm install". At the end of the pipeline, the referenced cache blobs will be automatically uploaded to storage. On subsequent runs of the pipeline (provided the scope criteria match), the matching blob will be downloaded and the "npm install" step will be skipped entirely.

Some Points of Note

Azure uses generalised and self-managed blob storage for cache entries. You have no control nor ability to configure where exactly your cache goes, nor to easily access it manually.
Cache blobs are stored in a scope - essentially mapping to a branch. So if you're working on a development branch which triggers the pipeline with each commit, your first run will create the cache blob and all subsequent runs (provided they don't alter the key) will re-use that blob.
The size of your blob can be a factor in whether or not caching is actually useful. If it takes as long (or potentially longer) to download the cached blob to your build agent as it does to run the command, you've a dilemma. Well, not really - you choose whichever option is faster in a given content.

Cache Effectiveness

The original driver for looking into this was an attempt to reduce build times for pipelines with NuGet or Node steps which took significant chunks of time and were run often. On one project we've applied this to, a pipeline with two distinct "npm install" steps has gone from those steps taking an average of over 7mins per run to approximately 2mins per run.

Given the following legend:

...can you spot when we might have enabled the caching of these 2 steps?

A thoroughly useful technique for speeding up the execution of pipelines with required but potentially time consuming steps. Just take note to get the cache key correct and don't fall foul of creating a blob larger than the payload required to run the command afresh on each execution.

Adding React to an Optimizely UI - Part 2

Sam Kirkpatrick — Fri, 19 Nov 2021 13:52:04 +0000

In Part 1 of this series, I gave a start-to-finish overview of how we at UNRVLD integrated a React application into an Optimizely ASP.NET MVC website.

I'd like to delve into several aspects of the process in more detail over the next few parts, starting with the choice of React itself and the specifics of how we actually implemented it, given there were (as there always is with anything front-end...) several options.

Why a Front-End Framework?

JavaScript is easily the most flexible way of implementing a performant, user-centric experience in a browser. Your HTML gives you structure, CSS makes it pretty and JavaScript manipulates it as necessary.

At some point, the amount of JavaScript code increases and the amount of manipulation required - even with the help of some still very popular libraries... cough...jQuery...cough - reaches the point of being unmanageable and unsustainable. FE frameworks have, in recent years, really risen to address these limitations.

Taking on a project to do exactly what these frameworks are designed to do really meant it was a no-brainer to go with one.

But which one?

The decision of which front-end framework to build on given a reasonably blank slate is always a challenging one. The stereotype of the front-end development landscape changing every 15 minutes is not entirely undeserved - the choices are, indeed, ever more numerous.

But the big name players are still React, Angular and Vue, in my opinion (and I fully confess there was a lot of my opinion in the decision phase here!). Other noteworthy candidates were Next.JS and Svelte.

However, for reasons including (but not limited to):

In-house developer experience
Learning curve
Community support and documentation
Training availability
Package library diversity

...we chose React.

And then we added TypeScript.

As I mentioned in Part 1, I've been someone who was - despite my years of experience and interest in new development tools - actively resistant to adopting TypeScript. But I'm willing to admit I was mistaken. It has definitely grown on me.

The supporting tools in IDEs like VS Code are really excellent and the assistance you get when developing actually does help significantly. I can't see us starting any significant piece of JS development in the future without using TypeScript.

I have no doubt that:

perfectly valid arguments could have been made for using a number of other frameworks
we would have successfully produced a solution using any one of those number of other frameworks

But React it would be for us for now.

Integrating with .NET

Since we remain somewhat constrained by a traditional MVC architecture and a UI rendered using Razor views, we needed a way to integrate the React app into the site.

One potential option would have been to use ReactJS.NET - a platform almost specifically built to do what we were trying to do. But the coupling of Razor and React required by this tool just felt "wrong", and would have left us inseparably tied to React (a view also expressed by the Optimizely team themselves). Given our "proof of concept" stage, maintaining the ability to swap out one framework for another one in the (unlikely-but-you-just-never-know) event of reaching an insurmountable road block was important.

`create-react-app`

The de-facto standard when starting a new React application is to open a terminal and run npx create-react-app. Again, we decided against this. Whilst this app does indeed give you absolutely everything you need to create, build and deploy a React application, it includes a bulk load of features that we didn't expect to need and we weren't creating a Single Page Application (SPA) - something create-react-app is very much angled towards.

Our app was going to be a fully built-by-hand endeavour. Starting with a brand new working folder and npm init and a src/index.tsx, off we went.

Adding React to an Optimizely UI - Part 1

Sam Kirkpatrick — Fri, 05 Nov 2021 14:19:37 +0000

At UNRVLD, we build solutions on a number of DXP platforms, including the Optimizely Content and Commerce platforms.

We have a significant number of clients using Optimizely Content up to v11 and we've been considering how we might be able to deliver some of the more complex and heavily interactive user journeys our clients need by leveraging modern front-end frameworks. In this post, I want to walk you through how we've recently architected the introduction of React to part of the site for one of our largest e-commerce partners.

This is the first part of a series looking at how we approached this challenge as an organization in which I'll summarize the technologies and solutions. In subsequent parts, I'll break down some of the key aspects in more technical detail - keep an eye out for these over the coming weeks.

What's the motivation?

Front-end frameworks are obviously becoming huge in terms of developing web solutions and user interfaces. With Optimizely Cloud being built on a server-based ASP.NET MVC architecture, there's a challenge to leveraging some JavaScript tools.

But user expectations are growing ever higher. Performance, personalization and quality experiences are essential for any site and significant improvements in these areas can have huge impact on conversions in an e-commerce context. As an agency, we have the design and UX expertise to know what works in these areas and this exercise was about how to implement those ideas technically.

What would we need?

So how would we inject a React app with all its dependencies on CMS content and configuration into the existing site and have it render on-screen?

We would need:

The React application itself
A placeholder within the Razor views of the MVC site in which to inject the React code
A mechanism to provide the required data to the React app
Support for server-side rendering (SSR)

I'll go into the thought processes (e.g., why React?) and structure of these individual items in further posts throughout this series.

The React App

Items 1 and 2 were intrinsically tied. We had to have a React app and then confirm that we could get it to render in the context of using Razor views. On the surface, this seems pretty trivial - you just need some JS files declared as a bundle and included in your Razor! But this was key to deciding if the approach would work long-term, so we had to prove it as a concept.

We also had to decide whether or not to use TypeScript. With 20+ years of writing JavaScript under my belt, I've been a slightly hesitant adopter of TypeScript, but it honestly feels like starting any significant JS development project nowadays without it is churlish. TypeScript it was - more on that in a later post.

We created a very simple app initially to ensure it could be embedded into the site as we wanted. Since "create-react-app" generally focuses on SPAs and includes a lot of features we didn't expect to need, we decided against it but instead hand-crafted the packages and files we needed.

For context, our first index.tsx app looked something like this:

import React from 'react';
import ReactDOM from 'react-dom';

import App from './app';

ReactDOM.render(
  <App />,
  document.getElementById('react-app');
)

Using Webpack, we built this to an existing folder within the .NET web application to make it easily referenceable using the following partial config:

module.exports = (env, argv) => {
  return {
    entry: {
      'react-app': './src/index.tsx'
    },
    output: {
      filename: '[name].js',
      path: path.resolve(__dirname, '../static')
    },
    resolve: {
      extensions: ['.ts', '.tsx', '.js']
    }
  }
}

We quickly reached a point of having a compiled JavaScript file that we could reference in our Optimizely Content site.

Razor Placeholder

The next task was to add something to the Razor views to get the app to appear on screen, leading us to this:

<div id="react-app"></div>
@Html.VersionedJs("~/static/react-app.js")

With a quick npm run build of our React app and a spin-up of the Optimizely MVC site, we browsed to the necessary address. Et voilà! A React app rendered inside an Optimizely view.

Injecting Data (Populating State)

The most important element of making this work was ensuring that the React app had the required data. This would obviously be built and prepared by the back-end C#, but how best to get it into the app? Really, it comes down to one of two options:

Render the data in some way on the page (e.g. hidden field) and consume it within your React start-up.
Mount the React app and trigger an immediate callback to fetch the data.

The trade offs here are about performance and user perception of speed. We went for the former initially, purely because a lot of the work to build the data was done on page load behind the scenes, so it didn't really make sense to get a blank, placeholder UI on screen before then re-requesting much of the already built data. We may review that over time as our app grows.

Using Redux for state management (which will be important in the SSR section below), our app instantiation code expanded to look like this:

import React from 'react';
import ReactDOM from 'react-dom';
import { Provider } from 'react-redux';

import App from './app';
import { initStore } from './redux/store';

const dataBlob = (document.getElementById('initial-state') as HTMLElement).value;
const storeData = JSON.parse(dataBlob);

const store = initStore(storeData);
ReactDOM.render(
  <Provider store={store}>
    <App />
  </Provider>
);

Server Side Rendering

Given this React app was being included in a major e-commerce solution with significant SEO foundations already in place, it became clear we would need to implement server-side rendering (SSR) of our React app to maintain those high scores and visibility. This was definitely one of the more interesting aspects of the overall project and will be covered in detail in a later post - it turned into a very expansive aspect, too!

Summary

So, that's the overview - a proof of concept idea to include a significant, standalone React element in a long-established ASP.NET MVC application, with state management and SSR to support continued SEO efforts, taken through to a production-quality implementation.

In the rest of the series, I'll go into more detail about the different parts of the solution and hopefully be able to share something of how the changes have been received by our client and their customers once released.