DEV Community: Erik Lieben

Using playwright during SPA/Angular development to mock your environment

Erik Lieben — Sun, 12 Mar 2023 14:59:19 +0000

Developing Single Page Applications (SPAs) can be complex, particularly when combining a SPA that communicates with one or more API's with complex data sets. It's often time-consuming to create a full range of testing projects that cover all the required scenarios in your testing environment due to complex dependencies or additional data structures necessary to make it work.

With the emergence of modern testing frameworks and libraries like Playwright, it is now possible to build SPAs without being dependent upon the entire environment and mock out parts to fit a given need. This allows developers to speed up the development process, be more fluent and agile, and ultimately deliver high-quality applications faster.

My experience

I've experienced this as a great way to develop modules that are part of a more extensive application (with many dependencies), and in cases where we applied event sourcing to the way we store 'data', the business flow. In the last case, we mock the API to truly focus on how the data would be best represented for consumption in the required scenario. This gave us the option to build our actual projections after we've finalized the code that consumes them and is less impacted by incorrect assumptions.

Next to an improved flow of work, it's also a great way to start earlier with preparing your e2e tests. You build API interactions to cover specific scenarios, what happens when there is nothing to show, how pagination works with a lot of data, what happens when we fail to get a response from the API, etc. All in a reusable way, you're not adjusting your code (adding hacks) to your data loading process to throw exceptions during development, which are later removed when you're done with development (otherwise, the application would always break).

Now, this does not mean that there will be no work left when your focus shifts to e2e or integration testing, but it does help you to be able to quickly get into a state to reduce risk when deploying to production in continuous flow.

Let's get started

In this article, I will build a small test application to demonstrate some of the benefits gained. This application is not meant to be perfect, but just here to illustrate the benefits of mocking your development environment with Playwright.

Building the Angular test application

For this demo, we will create a small testing application using Angular to illustrate the purpose. An Angular application is not required for this setup; it will work with any SPA framework or anything that works inside of a browser.

To illustrate some of the benefits, we will not load our data immediately. Still, the application requires a couple of clicks to reach the imaginary position we're working on/ developing. In this case, that is:

a loading button
a button to show the actual data

You can compare this to a wizard, for which you're working on the 3rd step of a 5-step wizard. We want to avoid clicking through the first two steps each time the browser refreshes due to changes detected in your code by the HMR process.

To create the sample application, generate a new default Angular app, and replace the content of the files with the content of the next two code blocks.

app.component.html

<h1>Posts</h1>
<button #loadbtn>load posts</button>
<ng-container *ngIf="posts$ | async as data">
  <div *ngIf="data.isLoading">loading</div>
  <div *ngIf="data.error">error loading</div>
  <div *ngIf="data.value as posts">
    <button (click)="togglePosts()" [disabled]="data.isLoading ? 'disabled': null">click me to show posts</button>
    <ul *ngIf="showPosts">
      <li *ngFor="let post of posts">{{post.title}}</li>
    </ul>
  </div>
</ng-container>

Next, we will modify the code behind of the component (app.component.ts). We will review some of it below to give you a quick view of what occurs.

On line 12 in the code block below, we get a reference to our load posts button (line 2 in the above code snippet). On line 22, we turn this button's click event into an observable stream. This means each time a new click is detected, a new click value will be emitted to the stream.

Once a new click event arrives in the observable stream (loadButtonClick$), we transform this signal (switchMap) into an HTTP get request retrieving data from the URL https://jsonplaceholder.typicode.com/posts (this endpoint generates random json, but could be seen as our production API).

The HTTP get method in Angular returns another observable, which results in either the retrieved data mapped (line 28) to the object { isLoading: false, value } and emitted or if the call fails/ returns an error state it transforms the observable into a new one (line 29) and emits the object/signal { isLoading: false, error}.

And if your head isn't spinning yet, on line 30, we set the first value/object to be emitted on the stream to be { isLoading: true }.

All in all, this means we will have a stream of objects/signals/events occurring that we are listening to; the first event will have the state of the boolean isLoading set to true and once it completed the HTTP call either successfully or in error state the boolean isLoading will be set to false. This isLoading boolean will be used to display a loading screen.

app.component.ts

import { HttpClient, HttpErrorResponse } from '@angular/common/http';
import { AfterViewInit, Component, ElementRef, ViewChild } from '@angular/core';
import { catchError, debounceTime, fromEvent, map, Observable, of, startWith, switchMap } from 'rxjs';

@Component({
  selector: 'app-root',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.css']
})
export class AppComponent implements AfterViewInit  {

  @ViewChild('loadbtn', { static: true }) loadButton: ElementRef|undefined;

  public loadButtonClick$: Observable<unknown>|undefined;
  public posts$: Observable<HttpRequestState<Post[]>>|undefined;
  public showPosts = false;

  constructor(private http: HttpClient) {}

  ngAfterViewInit(): void {

    this.loadButtonClick$ = fromEvent(this.loadButton?.nativeElement, 'click').pipe(debounceTime(300));

    this.posts$ = this.loadButtonClick$.pipe(
      switchMap(
        () => this.http.get<Post[]>('https://jsonplaceholder.typicode.com/posts')
          .pipe(
            map((value) => ({ isLoading: false, value })),
            catchError(error => of({ isLoading: false, error})),
            startWith({ isLoading: true })
          )
      ));
  }

  togglePosts() {
    this.showPosts = !this.showPosts;
  }
}

interface Post {
  userId: number;
  id: number;
  title: string;
}

export interface HttpRequestState<T> {
  isLoading: boolean;
  value?: T;
  error?: HttpErrorResponse | Error;
}

Installing the required npm packages

We need to install a couple of developer dependencies into our project to use this setup. Of course, we will need Playwright. Next, we will need ts-node to run our custom typescript script using node. The last three dependencies are just here to control execution.

Nodemon allows you to watch for changes to running scripts and reboot the ts-node run, which makes it easier to make adjustments to mock data/steps. Concurrently and wait-on help with starting up the Angular application and running it in parallel with our playwright script, wait-on specifically allows us to wait for the HTTP server started by ng serve to respond to HTTP traffic.

npm i playwright ts-node concurrently nodemon wait-on -D

Create a custom task to launch our browser

Let's start by creating our npm scripts to run our Playwright browser. Open up your package.json file, replace the default start script, and add a script called launch-browser to your scripts.

To go over the scripts quickly, it all starts with the concurrently npm package that can run multiple scripts/commands simultaneously. In the start script, you can see that it will try to run two scripts. One of them is the one we know ng serve, which is the default Angular option to start our Angular application. In parallel, it will use wait-on and wait for content on the default port used by Angular (4200); once content is detected, it will start the script/task npm run launch-browser.

The launch-browser script will start our typescript script tasks/launch.ts and use nodemon to restart it if you change it while running. The --mock=%npm_config_mock% part of this script will allow us to start our script and supply a name for the mocks to use. For example, npm start --mock=largeDataSet.

{
  "scripts": {
    "launch-browser": "nodemon --exec ts-node tasks/launch.ts --mock=%npm_config_mock% -e ts",
    "start": "concurrently \"ng serve\" \"wait-on tcp:4200 && npm run launch-browser\"\""
  }
}

Next, we're going to create our script. First, we need to add a custom tsconfig file in our folder to run our script in node. Create a folder named tasks and add a file called tsconfig.json and add the content below:

tasks/tsconfig.json

{
  "compileOnSave": false,
  "compilerOptions": {
    "sourceMap": true,
    "declaration": false,
    "downlevelIteration": true,
    "experimentalDecorators": true,
    "module": "CommonJS",
    "moduleResolution": "node",
    "resolveJsonModule": true,
    "importHelpers": true,
    "target": "ES2015",
    "typeRoots": [
      "./../node_modules/@types"
    ]
  }
}

Finally, we will create the script launch.ts, containing the core functionality.

Let's go roughly over the script to give you an idea of what it does:

Lines 2 to 12 allow us to retrieve the mock argument if it is provided. It will use the default.ts mock when nothing is provided.
Line 14 loads the specified mock you've provided using the argument.
On line 21, we create a new browser (see lines 23 to 28) set to headless is false (this means it will show us the browser and start the visible part of the browser as well).
On line 20, we create a new browser context, use a default desktop profile supplied by Playwright (see line 15), and use the method defaultContextaccording to our defaults.
The method defaultContext on lines 32 to 43 creates a new page, calls the mockHttp method to initialize the mocking (covered later), and navigates to our default Uri.
On line 35 in the method defaultContext, we check if the entire dom is loaded and then execute our steps if the URI equals our default starting URI.
Lines 46 and 47 contain the steps to navigate our app to the position in which we would like to have it. In this example, we wait for the button with the text 'load posts' to be visible/clickable and click it. Next we wait for the button with the text 'click me to show posts' to be visible/clickable and click it.

The steps function is most likely the one you will modify often; this function allows you to specify to which part of your application you want Playwright to navigate after each page refresh.

tasks/launch.ts

import {chromium, Page} from 'playwright';
import { argv } from 'process';

const args = (key: string) => {
  let value = argv.find( element => element.startsWith( `--${ key }=` ) );
  if (!value) {
    return undefined;
  }
  value = value.replace( `--${ key }=` , '' );
  if (value == '%npm_config_mock%') value = undefined;
  return value;
}

const mockConfiguration = args('mock') ?? 'default';
const mockHttp = require(`./../mocks/${mockConfiguration}`).default;

const desktopEdge = devices['Desktop Edge'];
const uri = 'http://localhost:4200/';

(async () => {
  const browser = await defaultBrowser();
  defaultContext(await browser.newContext({ ...desktopEdge }));
})();

async function defaultBrowser(args: string[] = []) {
  return await chromium.launch({
    headless: false,
    args
  });
}

async function defaultContext(context: BrowserContext) {
  const page = await context.newPage();
  await mockHttp(page);
  page.on('domcontentloaded', async (data) => {
    if (page.url() == uri)
    {
      await steps(page);
    }
  });

  await page.goto(uri, { timeout: 60000 });
}

async function steps(page: Page) {
  await page.getByRole('button', { name: 'load posts'}).click();
  await page.getByRole('button', { name: 'click me to show posts' }).click();
}

Mocking HTTP calls

Create a folder named mocks in the root of your app folder. In this folder, we will create a couple of files containing mocking scenarios.

To demonstrate more clearly what is happening, I've created a default script that mocks the API and returns one post with the title 'this is the default mock'. I hope this clarifies the process because our endpoint also produces random fake data.

Our default mocking scenario

When mocking API endpoints, we want to ensure we don't accidentally call an endpoint we did not cover with mocks. We can do this by instructing Playwright to mock all routes to our endpoint (lines 5 to 12) and return a 404. Not mocking these routes could, for example, result in deleting a given object because the id occurred in our mocked case.

Next, we instruct Playwright to mock our call to the posts API endpoint and return a default object titled 'this is the default mock' (lines 14 to 27). This route will override the catch-all route resulting in the 404 for this specific path.

default.ts

import { Page } from "playwright";

export default  async function (page: Page) {

  await page.route('https://jsonplaceholder.typicode.com/**', (route) => {
    route.fulfill({
      status: 404,
      contentType: 'text/html',
      headers: { 'access-control-allow-origin': '*'},
      body: `This call was intercepted and blocked, we don't want to go to production`
    })
  });

  await page.route('https://jsonplaceholder.typicode.com/posts', (route) => {
    route.fulfill({
      status: 200,
      contentType: 'text/html',
      headers: { 'access-control-allow-origin': '*'},
      body: JSON.stringify([
        {
          userId: 1,
          id: 2,
          title: 'this is the default mock'
        }
      ])
    })
  });

};

We're now ready to start up our application to see the result.

To give you an idea of how the application looks when you run it against the actual environment, start by running ng serve.

Close it, and now start our application using our created script; run npm start, which runs the application with our default (the above) mocking instructions.

Our application isn't aware of any of this, as can be seen when going to the inspector and looking in the network tab; the URL used to access the API is still the same. It's retrieving the mocked data set, but from the perspective of our application, this comes from the actual/real environment.

Adjusting the default scenario to be a bit more realistic

By now, I assume you got a rough view of what is occuring when we start the application using our custom startup script.

Let's now slightly adjust our default mocking script and, by default, return more data and add a delay to emulate the processing time of our API. To do this, we will install an additional package as a development dependency:

npm i @faker-js/faker -D

Generating the data with a fake data generator isn't required; this could also be a set of data you've created beforehand. To reduce the size of the code samples, I've decided to use fake data generation in this case, as you can see on line 26.

On line 17, we've added logic to wait for 1 second to emulate the time required for the API to return the request. You can remove or adjust this timeout to your needs. Next, we create an array with 20 items and map it to our data object (line 23).

default.ts

import { Page } from "playwright";
import { faker } from '@faker-js/faker';

export default async function (page: Page) {

  await page.route('https://jsonplaceholder.typicode.com/**', (route) => {
    route.fulfill({
      status: 404,
      contentType: 'text/html',
      headers: { 'access-control-allow-origin': '*'},
      body: `This call was intercepted and blocked, we don't want to go to production`
    })
  });

  await page.route('https://jsonplaceholder.typicode.com/posts', async (route) => {

    await new Promise(f => setTimeout(f, 1000));

    route.fulfill({
      status: 200,
      contentType: 'text/html',
      headers: { 'access-control-allow-origin': '*'},
      body : JSON.stringify([...Array(20)].map((v,i) => ({
        userId: i,
        id: i,
        title: faker.lorem.paragraph()
      })))
    })
  });
};

The above gives you a default state, which you can use for most development needs. We will add a couple more mock scripts to show you that we can also make additional use cases.

Create a file named largeDataset.ts in our mocks folder and add the content from the code sample below. As seen on line 23, this creates a more extensive data set of 1000 items and will increase the call to 2 seconds on line 17.

largeDataset.ts

import { Page } from "playwright";
import { faker } from '@faker-js/faker';

export default async function (page: Page) {

  await page.route('https://jsonplaceholder.typicode.com/**', (route) => {
    route.fulfill({
      status: 404,
      contentType: 'text/html',
      headers: { 'access-control-allow-origin': '*'},
      body: `This call was intercepted and blocked, we don't want to go to production`
    })
  });

  await page.route('https://jsonplaceholder.typicode.com/posts', async (route) => {

    await new Promise(f => setTimeout(f, 2000));

    route.fulfill({
      status: 200,
      contentType: 'text/html',
      headers: { 'access-control-allow-origin': '*'},
      body : JSON.stringify([...Array(1000)].map((v,i) => ({
        userId: i,
        id: i,
        title: faker.lorem.paragraph()
      })))
    })
  });
};

This could be a great case to test what happens to your application when there is a lot of data to show, for example, to try if your pagination logic is activated and if it works and looks like expected. (This code is not included in the test application, so you will get a list of 1000 items without pagination).

To see it in action, start up our application using the following:

npm start --mock=largeDataset

By now, we've covered two scenarios concerning the size of the result, but we also want to handle cases for which no data could be loaded. This logic can be seen in the code snippet below on lines 14 to 17; we define the route, wait for 2 seconds, and then abort the route.

failRequest.ts

import { Page } from "playwright";

export default async function(page: Page) {

  await page.route('https://jsonplaceholder.typicode.com/**', (route) => {
    route.fulfill({
      status: 404,
      contentType: 'text/html',
      headers: { 'access-control-allow-origin': '*'},
      body: `This call was intercepted and blocked, we don't want to go to production`
    })
  });

  await page.route('https://jsonplaceholder.typicode.com/posts', async (route) => {
    await new Promise(f => setTimeout(f, 2000));
    route.abort();
  });
};

To see this in action, create another file in the mocks folder, named failRequest.ts and add the content from the code snippet above and run it by using:

npm start --mock=failRequest

Other scenarios

Next to mocking APIs, many other options can be adjusted in our environment. Let's go over some of the possibilities to inspire you to make optimal use of Playwright in your development flow.

Develop dark and light mode

Creating a dark and light mode and previewing its effect on your application can be time-consuming if you need to switch between the two modes all the time.

What we can do is adjust our tasks/launch.ts (see below) to open up two browsers, one in dark mode and one in light mode, and then position them next to each other, running our steps function to bring the application to the spot we're interested in. When using the chromium browser, we can specify the location it opens up on by starting it using the option ---window-position.

(async () => {

  const browser1 = await defaultBrowser(['--window-position=0,0']);
  const browser2 = await defaultBrowser(['--window-position=0,800']);
  defaultContext(await browser1.newContext({ ...desktopEdge, colorScheme: 'dark' }));
  defaultContext(await browser2.newContext({ ...desktopEdge, colorScheme: 'light' }));

})();

To see this in action, we also need to make a minor adjustment to the head section of the index.html file, add the following to display a dark background when the browser is set to prefer the dark color scheme:

<style>
  @media (prefers-color-scheme: dark) {
    body {
      color: #e3e3e3;
      background-color: #3e3e3e;
    }
  }
</style>

Next, we start our development environment again to see the changes in action: npm start.

Develop different locales

When building applications with different locales, the length of text in one locale can be different from the other locale, which can impact how your application looks. Next, you might also have the requirement to support LTR (left to right text direction) and RTL (right to left text direction) locales and want to preview that during development.

Running an additional browser window in a different locale could help with this. Adjust our tasks/launch.ts to the snippet below to start two browsers in different locales.

(async () => {

  const browser1 = await defaultBrowser(['--window-position=0,0']);
  const browser2 = await defaultBrowser(['--window-position=0,800']);
  defaultContext(await browser1.newContext({ ...desktopEdge, locale: 'en-US' }));
  defaultContext(await browser2.newContext({ ...desktopEdge, locale: 'nl-NL' }));

})();

Another option would be to mock the call to the localization files and replace the localized text with the key of the localization resource item. This allows you to see where a specific localization key is used and use Playwright to screenshot it so that it can be shared with stakeholders in the development process.

Develop desktop and browser resolutions

A common practice during development is to preview and test the application in different browsers and resolutions. We can adjust our tasks/launch.ts to align with this need.

We can get the default device configuration from Playwright; this configures the defaults and actual browser to use. In the snippet below, we use the default chromium-based browser and start the WebKit browser to emulate an iPhone, and it's resolution.

import {chromium, webkit, Page, devices, BrowserContext} from 'playwright';

const desktopEdge = devices['Desktop Edge'];
const mobileBrowser = devices['iPhone 13'];
const uri = 'http://localhost:4200/';

(async () => {

  await defaultContext(await (await defaultBrowser()).newContext({ ...desktopEdge }));
  await defaultContext(await (await webkitBrowser()).newContext({ ...mobileBrowser }));

  //repositionWindows();

})();

async function webkitBrowser() {
  return await webkit.launch({
    headless: false,
  });
}

One downside of using the Webkit browser is that there is no option to specify the starting position of the window, which isn't the most helpful when refreshing or restarting our instances. We can resolve this issue with another npm package called node-window-manager.

Install the package using the following command:

npm i node-window-manager -D

Import the windowManager to our tasks/launch.ts, and add the function repositionWindows from the code snippet below to correct the positions of the browser instances/ windows that opened up.

import { windowManager } from 'node-window-manager';

function repositionWindows() {
  const windows = windowManager
    .getWindows()
    .filter(window => window.path.indexOf('AppData\\Local\\ms-playwright') > -1 && window.isVisible());

  windows[0].setBounds({ ...windows[0].getBounds(), x: 1026, y: 0 });
  windows[1].setBounds({ ...windows[1].getBounds(), x: 0, y: 0 });
}

We're currently filtering out the windows we need based upon the path of the executable containing AppData\Local\ms-playwright this is the path to the browser executables used by Playwright on Windows and might differ on Mac/Linux.

Ultimately, this solution is not the most generic and requires you to play around a bit to get the right windows, but it works okay.

Finally

In conclusion, I hope that the above offered you some ideas on improving your process or at least inspired you to look into Playwright and what it can do for you because there is much more possible than what we covered in this article.

Customized pull request status policy using Azure Functions for semantic release PR titles (part 1)

Erik Lieben — Sun, 05 Mar 2023 17:50:03 +0000

In my previous blog post, I showed you how to automate releases using pull request titles for your NuGet packages.

When you use this process less often, or if you have new people onboarding your team, it is beneficial to give some additional help/ insights when they go through the process. This blog post will look at how to build an Azure Function to provide more information about the process during your regular pull request workflow.

What will we build

To extend your Azure DevOps Pull Request workflow, you can build a customized workflow by creating an HTTP endpoint. This endpoint will be notified whenever a pull request is created or updated. To set up Azure DevOps to notify the HTTP endpoint, we can use Service Hooks.

The logic behind the HTTP endpoint will perform the check and post back the status to Azure DevOps using a PAT token for authentication.

Let's start with creating the PAT token.

Creating the PAT (Personal Access Token) token

A personal access token contains your security credentials for Azure DevOps. A PAT identifies you, your accessible organizations, and scopes of access. As such, they're as critical as passwords, so you should treat them the same way. That's why we will store this token in Azure KeyVault later on.

Let's create the token with the required scope. Go to your Azure DevOps environment and click the user settings icon next to your profile image in the top right of the window. Pick the option Personal Access Token.

This will open up the User settings page on the section Security with Personal access tokens selected. By default, the PAT tokens screen will show you active tokens and allow you to create a new token. Click the + New Token button to generate the required token.

This will open up the window to generate a personal access token window. Set the title to a new one you will remember; in this case, we will use semantic-release title validation. Select the expiration date for the token. This is the period the token will be active; after this date, the token will stop working, and you will need to extend or re-create it. Next, we will specify the scopes that can be accessed using the token. Select Custom defined to specify a subset of scopes. In the Section Code check the item Status. This will give the token access to read and write commit and pull request statuses.

Click the Create button on the bottom of the screen to create the token. A new token will be generated; copy this token, because it won't be shown again. Keep this safe for now, we will later on store this in Azure KeyVault.

Creating the Azure Function

Open Visual Studio and create an Azure Function; in the project creation wizard, select .NET 7.0 Isolated as the Function worker option. This will generate a project for us with a default Function1 class - rename this class to SemanticReleaseConventionFunctions.

Ingestion of the PAT token secret

To send back status information to Azure DevOps we require access to the PAT token from our code. To supply this token, we will create a record called ApplicationSettings and add one property to called PatToken to the record.

public record ApplicationSettings
{
    public string? PatToken { get; set; }
}

Next, we set up the startup logic to load the settings from local.settings.json file. Open up the Program.cs file and add the ConfigureServices method and the code to setup the application settings configuration (line 8 till 14 below).

using AzureDevopsSemanticRelease;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var host = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults()
    .ConfigureServices(s =>
    {
        s.AddOptions<ApplicationSettings>().Configure<IConfiguration>((settings, configuration) =>
        {
            configuration.GetSection(nameof(ApplicationSettings)).Bind(settings);
        });
    })
    .Build();

host.Run();

Add the setting for the PAT token to your local.settings.json and paste the token you've generated above. Your local.settings.json file should not be checked into source control, because it contains a secret. We will temporarily keep the token in this file to simplify debugging our code.

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
    "ApplicationSettings:PatToken": "REPLACE_WITH_YOUR_PAT_TOKEN"
  }
}

We can now retrieve the application settings in our Function class by letting the dependency injection process inject it into our constructor. Create a constructor for your class and add the parameter IOptions<ApplicationSettings> settings and store it's value in a private field.

public class SemanticReleaseConventionFunctions
{
    private readonly string pat;

    public SemanticReleaseConventionFunctions(IOptions<ApplicationSettings> settings)
    {
        pat = settings.Value.PatToken ?? throw new ConfigurationException("Unable to find setting 'ApplicationSettings:PatToken'");
    }
}

public class ConfigurationException : Exception
{
    public ConfigurationException(string message) : base(message) { }
}

Create the HTTP trigger #

We're now ready to add the HTTP endpoint and the logic to validate the pull request title. Create an Azure Function for the route validate-pr-title that accepts POST requests in the SemanticReleaseConventionFunctions class.

On line 5 and 6 we retrieve the body from the request and parse the json to our object structure to understand the message.
Line 8 takes this object and generates the status message and submits it to the Azure DevOps rest API endpoint.
On line 10 till 12 we create a valid response for the webhook.

[Function(nameof(ValidatePRTitle))]
public async Task<HttpResponseData> ValidatePRTitle(
    [HttpTrigger(AuthorizationLevel.Function, "post", Route = "validate-pr-title")] HttpRequestData req)
{
    var body = await req.ReadAsStringAsync() ?? throw new MessageProcessingException("Unable to read body");
    var pr = JsonSerializer.Deserialize<Message>(body) ?? throw new MessageProcessingException("Unable to parse body");

    await PostStatusOnPullRequest(GenerateStatusMessage(pr.resource.title, pr.resource.repository.webUrl), pr.resource.url);

    var response = req.CreateResponse(HttpStatusCode.OK);
    response.Headers.Add("Content-Type", "text/plain; charset=utf-8");
    return response;
}

public class MessageProcessingException : Exception
{
    public MessageProcessingException(string message) : base(message) { }
}

In addition, we need a few objects to store the data structure in which Azure Devops sends the event.

public record Message
{
    public PullRequestResource resource { get; set; } = new();
}

public record PullRequestResource
{
    public string title { get; set; }

    public int pullRequestId { get; set; }

    public string url { get; set; }

    public PullRequestRepository repository { get; set; } = new();

}

public record PullRequestRepository
{
    public string webUrl { get; set; }
}

Create the GenerateStatusMessage method #

In the generate status method, we will generate the message to post to Azure DevOps based on the title text. We return a succeeded state by default, and if it's not in the right format the pending state. In both cases, we return the generated description, either providing the error description or the type of release it will create.

private static string GenerateStatusMessage(string pullRequestTitle, string repositoryUri)
{
    string state = "succeeded";
    var (valid, description) = ParseTitle(pullRequestTitle);
    if (!valid)
    {
        state = "pending";
        description = "PR title is not following PR title conventions (<type>(<scope>): or <type>:); click me for more information.";
    }

    return JsonSerializer.Serialize(
        new
        {
            State = state,
            Description = description,
            TargetUrl = $"{repositoryUri}?path=/docs/release-instructions.md&version=GBmain&_a=preview",

            Context = new
            {
                Name = "PullRequest-SemanticRelease-Commit-Title",
                Genre = "pr-azure-function-ci"
            }
        });
}

In the ParseTitle method we parse the title and process it to define the type of release. On line 13 to 24 you can define your rules around releases.

private static (bool, string) ParseTitle(string title)
{
    Match m = SemanticReleaseRegex().Match(title);

    string message = "No release will be created.";

    if (m.Success)
    {
        var tag = m.Groups["tag"].Value;
        var scope = m.Groups["scope"].Value;

        string? releaseType = null;
        switch (tag)
        {
            case "feat":
                releaseType = "minor";
                break;
            case "docs" when scope == "README":
            case "perf":
            case "fix":
            case "deps":
                releaseType = "patch";
                break;
        }

        message = string.IsNullOrEmpty(releaseType) ?
              $"PR title is according to PR title conventions for semantic release and will _not_ perform a release."
            : $"PR title is according to PR title conventions for semantic release and will perform a {releaseType} release";
    }

    return (m.Success, message);
}

[GeneratedRegex("^(?<tag>docs|perf|fix|deps|feat|style|test|refactor)\\(?(?<scope>.*?)\\)?:(?<description>.*)", RegexOptions.IgnoreCase)]
private static partial Regex SemanticReleaseRegex();

We defined the following rules below: If the tag is feat we will expect a minor release. When the tag is docs and the scope is README we will expect a patch release and when it is in any other scope but uses the tag docs no release will be created. The tags perf, fix, and deps will expect a patch release. All the other tags or scopes will not expect a release.

Line 26 to 28 will generate the correct description based upon the release type.

Line 34 and 35 contain a regular expression parsing the title text. The GeneratedRegex was introduced in .NET 7 that uses Roslyn enabled source generators to provide the regex implementation; for more information see .NET regular expression source generators.

The regex will roughly perform the following:

We expect the title string to start with a tag:

docs
perf
fix
deps
feat
style
test
refactor

Then optionally, we would detect a scope and the rest as the description (in our sample, we don't use the description).

Next, we look at the last bit, sending the generated response to Azure DevOps in the PostStatusOnPullRequest method.

We use the pat token previously retrieved from the settings, convert it to a base64 string, and add it as basic authorization header. Finally, we send the message generated in the GenerateStatusMessage method to the Azure DevOps REST API and validate the response (The EnsureSuccessStatusCode method throws an Exception if the status code is not a successful one).

private async Task PostStatusOnPullRequest(string message, string prUri)
{
    using HttpClient client = new();
    client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(Encoding.ASCII.GetBytes($":{pat}")));

    var request = new HttpRequestMessage(new HttpMethod("POST"), $"{prUri}/statuses?api-version=4.1")
    {
        Content = new StringContent(message, Encoding.UTF8, "application/json")
    };

    using HttpResponseMessage response = await client.SendAsync(request);
    response.EnsureSuccessStatusCode();
}

Testing/ running your code locally using the public preview of dev tunnels #

It's always more fun and easier to understand first to see your code working on your machine and be able to debug it and step through your code. This is a bit difficult for most because our development machine is most likely not directly exposed to the internet/ publicly accessible. To still perform this action, we need a temporary tunnel. This can either be done using ngrok or a new tool in Visual Studio called dev tunnels. We will use the latter.

For this to work, you must ensure you have version 17.4 of Visual Studio. The first step is to enable the preview version; go to: Tools, Options in the Visual Studio toolbar. The options window will show; on the left side, open up the Environment section and select the subsection Preview Features. Ensure the checkbox before 'Enable dev tunnels for Web Applications' is checked.

Next, we need to configure the Azure Function to use dev tunnels. In your solution explorer, open the folder Properties and open the file launchSettings.json. See the code below, and make sure the launchBrowser option is set to true; by default, this is set to false and add the two devTunnel properties on like 7 and 8.

{
  "profiles": {
    "AzureDevopsSemanticRelease": {
      "commandName": "Project",
      "commandLineArgs": "--port 7068",
      "launchBrowser": true,
      "devTunnelEnabled": true,
      "devTunnelAccess": "public"
    }
  }
}

Start your Azure Function app by clicking play or F5 to run it. A new browser window will open, which contains the link to your dev tunnel. In the screenshot, the actual URL is masked with *'s; yours will have a random set of characters and is personal to you/ made at random for testing.

Copy the URL you get, because we will use it in the next step to setup the webhook.

Setting up the webhook in Azure DevOps

We need to set up a service hook to be made aware of new pull requests or changes to a pull request. A service hook is comparable to a webhook; when something occurs, it calls your HTTP endpoint to inform you of the change.

Next, go to the section Service hooks and click the + Create subscription button.

If you already have service hooks configured, the screen looks a bit different, clicking the + button will bring you to the same spot.

A new window will open up, scroll down in the list on the left and select Web Hooks and click Next.

On the next screen, we must set the type of events we would like to receive. In the dropdown below Trigger on this type of event select Pull request created and click Next.

Optionally, you can filter on repository, target branch, or members. For example, if you use a lot of different branches, then set this to your main branch (the branch that triggers the release).

The last part of the wizard steps allows you to specify the target URL. For testing purpose and to see the code in action, we will add the URL we received above when creating the dev tunnel appended with /api/validate-pr-title.

Note: once the system is deployed to production, we must replace this URL with the production URL.

Our first service hook/webhook is now configured.

Repeat the process above once more for pull request updates. In the event to trigger step, select Pull request updated for the field Trigger on this type of event. Once this is completed, you will have two service hooks, one that triggers the creation of pull requests and one that triggers changes/updates to pull requests.

We can now create a testing pull request and try to adjust the title when our Function App is running on our local machine. If you set a breakpoint, it will be hit, and you can step through the code.

Be aware that your setup is now configured to temporarly work together with the local instance running on your machine. We don't want to keep this on all the time, so after confirming that everything works it might be good to remove the webhook from Azure DevOps so we can, once you completed the later on reconfigure it.

Finally

With the above step completed, we've seen that the code does what we expect it to do and also discovered how to debug and step through the code if we ever require that. Now, running the function locally on your machine isn't an option. In the second part of the blog post, we will look at how we can bring our code to production/ host it in Azure.

Up next

In part 2 of this blog post, we will cover how to create our IaC (Infrastructure as Code) a bicep template to deploy our infrastructure, setup our Azure build pipeline to release/deploy everything, and finally hook it all up to our production environment, so that it is ready for use.

Automate your release flow of NuGet packages using Azure DevOps and Node's semantic-release

Erik Lieben — Sun, 17 Apr 2022 19:03:18 +0000

In the NodeJS ecosystem, a great solution is available for automating the workflow of releasing packages, explicitly concerning the versioning of packages, named semantic-release.

All it takes for you to use this in your .NET project is a willingness to accept a little bit of JavaScript in your .NET deployment pipeline. A well worth exception you should be willing to take to improve your overall development experience.

In this blog post, I will take you through my setup for a project that builds & publishes a NuGet package using Azure DevOps to an Azure DevOps artifact feed.

Semantic-release you say? #

According to its tagline, semantic-release is a fully automated version management and package publishing system. You can configure it to manage your entire workflow around the versioning of your NuGet packages. All based upon the git commit message (pull request title in our case) you write to save your changes.

The versioning structure is based upon semantic versioning 2.0, which, as stated on their site, follows the following pattern:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,

MINOR version when you add functionality in a backward-compatible manner, and

PATCH version when you make backward-compatible bug fixes.

If you are using NuGet 4.3.0+ and Visual Studio 2017 version 15.3+, NuGet has support for the same versioning structure, as can be read in the Package versioning - Version basics section of the documentation site.

How do I use it? #

After the configuration described in this blog post, the pipeline will use your PR (pull request) title to determine if a new package needs to be released and the version number of that package.

Your PR titles will need to be written confirming a convention. There are many preset conventions to pick from; the default is the Angular Commit Message Conventions format. In this blog post, we will use the preset conventionalcommits, which is the barest one.

Our configuration for the convention will mean that your titles need to confirm to:

<type>(<scope>): <short summary>

Where both the <type> and <short summary> fields are mandatory, and the (<scope>) field is an optional field. The scope field can be the area or scope of the package you are working on, or in the case of a repository that shares the source multiple packages, this might be the package name.

Create a release, bump the version #

If you want to create a release, the <type> field of the PR title needs to be one of the following:

BREAKING CHANGE: A breaking change (causes a release with a MAJOR version bump)
feat: A new feature (causes a release with a MINOR version bump)
fix: A bug fix (causes a release with a PATCH version bump)
docs(README): An update to the README documentation (causes a release with a PATCH version bump)
perf: A performance fix (causes a release with a PATCH version bump)
deps: A dependency update (causes a release with a PATCH version bump)

These are the ones we will create as defaults in this blog post, but you can adjust these to your liking.

Commit work, that should not cause a version bump #

Next to the above PR titles, which create a new release for you, you might also have work that won't require an immediate release—for example, code style or some non-crucial documentation changes.

We will add the following types for those use cases:

refactor:: A refactor of a section of your code
docs: A non-crucial documentation update
style: A adjustment to the style of your code
test: A added or adjusted (unit)test case

What if I forget the format/ convention? #

If you want to follow the convention by the letter, that's up to you because semantic-release will ignore anything other than the setup types.

So when you start to use the convention as a team, and from time to time you forget about it, it won't break anything. It won't release your packages or leave the added PRs out of the documentation, but the build won't break due to this.

What if my PR breaks the pipeline and I need to fix it? #

In the unfortunate case where the modifications you've made in your PR bring the state of the main branch to a non-releasable condition and the Azure DevOps pipeline that runs your PR failed to block this.

No additional steps are required to fix the above issue other than to fix the actual problems that brought your pipeline into a breaking condition with a new PR of which the title starts with fix:.

Semantic-release will, in this case, retry its actions and perform the bump that includes the previous change.

So far, we have covered the information needed for you to get started, so we will move on to how to configure this. If you want to see the information covered so far in action, see the section You're now all set.

How do I configure it? #

To configure semantic release in your .NET project, a package.json file is required in the root folder of your repository. By default, there is no requirement to perform npm install (which will retrieve these packages and install them in the node_modules folder on your machine). However, if you want to debug the workflow or run it locally, this is required.

{
  "name": "nuget-package",
  "private": true,
  "scripts": {
    "ci:release": "semantic-release"
  },
  "devDependencies": {
    "@semantic-release/changelog": "^6.0.1",
    "@semantic-release/commit-analyzer": "^9.0.2",
    "@semantic-release/exec": "^6.0.3",
    "@semantic-release/git": "^10.0.1",
    "@semantic-release/release-notes-generator": "^10.0.3",
    "semantic-release": "^19.0.2",
  }
}

This file contains a couple of properties; the name field can be anything you like that confirms to the standard naming format of an npm package. It's not used by the release workflow or will appear anywhere in your final release.

The private property set to truemakes sure that if you accidentally try to release this as an npm package, it will refuse to publish the package.

The package file contains one script that is defined, ci:release, which executes the semantic-release package/code. The Azure DevOps pipeline only calls this script; thus, there is no need to run this script locally (only needed if you want to debug your semantic release setup locally).

The devDependencies section contains the packages required to run our workflow.

Package name	Description
semantic-release	This is the main semantic release executable
@semantic-release/commit-analyzer	plugin to analyse the commit message format
@semantic-release/release-notes-generator	plugin to generate the release notes history
@semantic-release/changelog	plugin to generate a CHANGELOG.md file/ document with version history
@semantic-release/git	plugin to perform a commit with the CHANGELOG.md & new version info
@semantic-release/exec	plugin to hook into stages of the version upgrade workflow and execute scripts

Next up, we need to set up semantic-release and add some adjustments to make it work together with Azure DevOps. Create a file named release.config.js in your root folder next to the package.json file created above.

module.exports = {
  branches: [
    'main'
  ],
  plugins: [
    [
      '@semantic-release/commit-analyzer',
      {
        preset: 'conventionalcommits',
        releaseRules: [
          {breaking: true, release: 'major'},
          {type: 'docs', scope:'README', release: 'patch'},
          {type: 'perf', release: 'patch'},
          {type: 'fix', release: 'patch'},
          {type: 'deps', release: 'patch'},
          {type: 'feat', release: 'minor'},
        ],
        parserOpts: {
          mergePattern: '^Merged PR (\\d+): (\\w*)(?:\\([\\w\\$\\.\\-\\*]*)\\))?\\: (.*)$',
          mergeCorrespondence: [
            'id',
            'type',
            'scope',
            'subject'
          ],
          noteKeywords: [
            'BREAKING CHANGE',
            'BREAKING CHANGES'
          ]
       }
      }
    ],
    ['@semantic-release/release-notes-generator', {
      preset: 'conventionalcommits',
      presetConfig: {
        types: [
          {
            type: 'docs',
            section: 'Documentation',
            hidden: false
          },
         {
            type: 'fix',
            section: 'Bug fixes',
            hidden: false
          },
          {
            type: 'feat',
            section: 'New features',
            hidden: false
          },
          {
            type: 'perf',
            section: 'Performance improvement',
            hidden: false
          },
          {
            type: 'style',
            section: 'Code style adjustments',
            hidden: false
          },
          {
            type: 'test',
            section: '(Unit)test cases adjusted',
            hidden: false
          },
          {
            type: 'refactor',
            section: 'Refactor',
            hidden: false
          },
          {
            type: 'deps',
            section: ':arrow_up: Dependency updates',
            hidden: false
          }
        ],
        issueUrlFormat: '//' + process.env.SYSTEM_TEAMPROJECT + '/_workitems/edit/'
      },
      writerOpts: {
        finalizeContext: function (context, options, filteredCommits, keyCommit, commits) {
          const parts = /(.*)\/_git\/(.*)/gm.exec(context.repository);
          const repoUrl = `${context.host}/${context.owner}/${parts[1]}`;
          return {
            ...context,
            repository: null,
            repoUrl,
            commit: `_git/${parts[2]}/commit`,
            issue: '_workitems/edit',
            linkCompare: false
          };
        }
      },
      parserOpts: {
        mergePattern: '^Merged PR (\\d+): (\\w*)(?:\\(([\\w\\$\\.\\-\\*]*)\\))?\\: (.*)$',
        mergeCorrespondence: [
          'id',
          'type',
          'scope',
          'subject'
        ],
        noteKeywords: [
          'BREAKING CHANGE',
          'BREAKING CHANGES'
        ]
      }
    }],
    [
      '@semantic-release/changelog',
      {
        changelogFile: 'docs/CHANGELOG.md'
      }
    ],
    ['@semantic-release/exec', {
      prepareCmd: "pwsh -NoLogo -NoProfile -NonInteractive -Command ./prepare.ps1 '${ nextRelease.version }' '${ nextRelease.gitHead }' '${ options.repositoryUrl }' '${process.env.CSPROJ}'",
    }],
    [
      '@semantic-release/git',
      {
        message: 'chore(release): ${nextRelease.version} [skip ci]\n\n${nextRelease.notes}',
        assets: [
          'docs/CHANGELOG.md',
          '${process.env.CSPROJ}'
        ]
      }
    ]
  ]
}

To reduce the size of this post, here is a rough overview of the above file content:

Line 3 : The name of the release branch to work against, in our case main.
Line 7 to 23 : custom configuration of the commit-analyzer plugin to understand commits made AzureDevOps and the release rules (when to perform, what release). The mergePattern is a custom regex that helps the commit analyzer understand the Azure DevOps format.
Line 33 to 109 : custom configuration of the release-note-generator plugin to generate release notes to a format of your choice. The presetConfig.types section allows you to define the sections each PR/ change falls under. The writerOpts and parserOpts are custom configurations for the release note generator to understand/ generate notes in an Azure DevOps format. Line 110 to 115 : Set up the changelog package and specify where you want your changelog to be generated.
Line 116 to 118 : setup of the exec plugin to execute a custom PowerShell script to update version information in our .csproj file as preparation for the release of our NuGet package.
Line 119 to 128 : setup of the git plugin defines which files to commit and what message to use for the commit. The [skip ci] label in this message makes sure that there isn't a additional build triggered due to a commit in the Azure DevOps repository.

As shown on lines 116 to 118 in the above file, we use a custom PowerShell script to perform the required updates to the .csproj file. So in the root, we also require a file named prepare.ps1 with the following content:


param($version, $gitHead, $repoUri, $filename)

$repoUriWithoutKey = $repoUri -replace 'https:\/\/(.*)@','https://'
$xml = [xml](Get-Content $fileName)
$versionPrefixNode = $xml.SelectSingleNode("//Project/PropertyGroup/VersionPrefix");
$versionPrefixNode.InnerText = $version
$repositoryCommit = $xml.SelectSingleNode("//Project/PropertyGroup/RepositoryCommit");
$repositoryCommit.InnerText = $gitHead
$repositoryUrl = $xml.SelectSingleNode("//Project/PropertyGroup/RepositoryUrl");
$repositoryUrl.InnerText = $repoUriWithoutKey
$packageProjectUrl = $xml.SelectSingleNode("//Project/PropertyGroup/PackageProjectUrl");
$packageProjectUrl.InnerText = $repoUriWithoutKey
[System.Xml.Linq.XDocument]::Parse($Xml.OuterXml).ToString() | Out-File $filename
Write-Host "##vso[task.setvariable variable=version;]$version"

This file roughly performs the following actions:

Line 1 : this allows us to call the script with four parameters, the new version, the git hash of the current head, the URI to the repository, and the filename of the .csproj to adjust.
Line 3 : by default, the system provides us with the URI to the git repository that includes the PAT token to access it. Of course, we don't want to publish this token, so we replace/ remove that value at this line.
Line 5 to 14 : we open up the .csproj file and add the data provided (version, hash, repo), and store it again on disk.
Line 16 : we write the variable version to the Azure DevOps pipeline for later usage.

Next up, we need to open up our .csproj file and make some adjustments:

MyProject.csproj


<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <PackageId>MyProjectPackageId</PackageId>
    <PackageIcon>icon.png</PackageIcon>
    <PackageTags>tag A;tag B</PackageTags>
    <VersionPrefix>0.0.1</VersionPrefix>
    <Authors>Your name</Authors>
    <Company>Your company</Company>
    <Copyright>Copyright © $(Company) $([System.DateTime]::Now.Year)</Copyright>
    <Description>package description</Description>
    <PackageReleaseNotes>docs/CHANGELOG.md</PackageReleaseNotes>
    <SymbolPackageFormat>snupkg</SymbolPackageFormat>
    <RepositoryUrl></RepositoryUrl>
    <PackageProjectUrl></PackageProjectUrl>
    <RepositoryCommit></RepositoryCommit>
    <RepositoryBranch>main</RepositoryBranch>
    <RepositoryType>git</RepositoryType>
    <PublishRepositoryUrl>true</PublishRepositoryUrl>
    <EmbedUntrackedSources>true</EmbedUntrackedSources>
    <DebugType>embedded</DebugType>
  </PropertyGroup>
  <PropertyGroup Condition="'$(TF_BUILD)' == 'true'">
    <ContinuousIntegrationBuild>true</ContinuousIntegrationBuild>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.SourceLink.AzureRepos.Git" Version="1.1.1" PrivateAssets="All" />
  </ItemGroup>
  <ItemGroup>
      <None Include="..\icon.png" Visible="false">
        <Pack>True</Pack>
        <PackagePath>\</PackagePath>
      </None>
  </ItemGroup>
    <ItemGroup>
      <None Include="..\docs\CHANGELOG.md" Visible="false">
        <Pack>True</Pack>
        <PackagePath>\docs</PackagePath>
      </None>
  </ItemGroup>
</Project>

The modifications we make are largely in relation to information shared in the NuGet package:

Line 7 to 14 : general information about your NuGet package
Line 15 : link to the generated version history changelog
Line 16 to 24 : links to your Azure DevOps git repository & embedding debug symbols.
Line 28 to 33 : include a package for source linking
Line 35 to 46 : include the icon in the package, but don't show it in the Visual Studio solution explorer.

That leaves us with one last step, the build pipeline that brings this all together. Create a azure-devops.yml file and add the following content:

trigger:
- main

pool:
  vmImage: ubuntu-latest

variables:
- name: csproj
  value: ./MyProject/MyProject.csproj

steps:
- task: NodeTool@0
  displayName: Use node v16
  inputs:
    versionSpec: '16.x'
- task: UseDotNet@2
  displayName: Use .NET v6.0
  inputs:
    packageType: 'sdk'
    version: '6.0.x'
- task: NuGetAuthenticate@0
  displayName: Authenticate to NuGet
- task: Npm@1
  displayName: Install NPM packages
  inputs:
     command: 'install'
- task: Npm@1
  displayName: Bump version & Create release doc (semantic-release)
  env:
    GIT_CREDENTIALS: $(System.AccessToken)
    CSPROJ: $(csproj)
    HUSKY: 0
  inputs:
    command: custom
    verbose: false
    customCommand: run ci:release
- task: DotNetCoreCLI@2
  displayName: Build & pack package
  condition: ne(variables['version'], '')
  inputs:
    command: pack
    projects: $(csproj)
    arguments: '-c $(buildConfiguration) /p:ContinuousIntegrationBuild=true'
- task: NuGetCommand@2
  displayName: Publish package
  condition: ne(variables['version'], '')
  inputs:
    command: push
    publishVstsFeed: 'AzureDevopsProjectName/FeedName'
    allowPackageConflicts: true

This is a simplified version of a build pipeline to demonstrate the functionality discussed in the post. You most likely want to use a more advanced pipeline in your production deployment with additional steps.

Let's go over the lines to get an overview of what occurs in the pipeline:

Line 1 to 5 : define the basic setup of the branch that triggers a build and the type of build server to use.
Line 7 to 9 : defines the variables that can be used in this file, replace the value of the csproj varaible with the path to your .csproj file.
Line 12 to 26 : These steps prepare the build process. Ensure that node js is installed, use .NET v6 SDK (this is optional today), Authenticate NuGet with the PAT token from the build agent, and install the required npm packages.
Line 28 to 37 : This is the step that performs the complete versioning workflow. Any tests or things to validate if the package can be released need to be performed before this step.
Line 39 to 45 : build & pack the NuGet package using the new version generated in the previous step.
Line 47 to 53 : publish the NuGet package to the Azure DevOps Artifacts feed.

As you can see on line 31,we use the $(System.AccessToken) access token or PAT (Personal Access Token) that is temporarily generated to perform this build by the build agent. Since the versioning workflow adds git tags, commits code, and more to the git repository, you also need to assign the build services client.

Assign rights to build services #

In the above process, during the 'Bump version & Create release doc (semantic-release)' step, the build server performs the following tasks to your git repository:

create a versioning tag
adds additional metadata to git commits (notes)
pushes a commit directly to your main branch

Your build services need to have rights to perform these actions, which can be set up per repository or for all repositories in your project.

Open your project, and on the left bottom, click Project Settings and go to Repositories in the Repos section. Select either a repository and select the tab Security or select the tab Security in the All repositories view. Select the user (AzureDevOpsProjectName Build Service (OrganizationName)) and assign the rights on the left side.

You're now all set #

With the above configuration, you're all set to release your NuGet packages with ease. Let's go over some of the scenarios and see what occurs.

Releasing a new feature #

The release of a new feature will cause a minor version (0.x.0) release.

Your pull request:

How the docs/CHANGELOG.md will look after the new release:

As you can see in the above screenshot, the release was created with:

A link to the git diff compare (1.1.0) between the new version tag and the previous version tag
A link to the git commit (652864d) with the code adjustments that caused this update
A link to the story (#143) that was attached to this PR

Releasing a bug fix #

The release of a bug fix will cause a patch version (0.0.x) release.

Your pull request:

How the docs/CHANGELOG.md will look after the new release:

As you can see in the above screenshot, the release was created with:

A link to the git diff compare (1.1.1) between the new version tag and the previous version tag (1.1.0)
A link to the git commit (38bffff) with the code adjustments that caused this update
A link to the story (#145) that is attached to this PR

A non essential (documentation) update #

A pull request for non-essential documentation changes will not cause a release.

Thus, it won't cause any changes to the docs\CHANGELOG.md file. However, in the next release, the changes will be included:

The next pull request:

The docs/CHANGELOG.md after a the new release:

Finally #

You're now all set to release NuGet packages with ease and keep a nice changelog of changes that occurred. The semantic versioning structure will help the consumers of your package upgrade their dependencies with ease, and the changelog gives them easy access to the changes in your package.

For example, when using Renovate to keep dependencies of solutions up to date. More often than not, the person upgrading the dependency wants easy and quick insights into the changes to decide if they can apply the upgrade or if more work is required to perform the upgrade.

With the above setup, you allow them to access that with ease, while at the same time, you can release and generate that information with ease.

Up next #

In a follow-up blog post, I will show you how to add a customized pull request status policy using Azure Functions to help your team members format PR titles according to the new convention required. The policy can either enforce the pull request title convention or be a helpful optional reminder to create titles according to the pattern.

And after that, I have another blog post for you in which we will look at how you could use the same workflow to streamline your release flow for Node/npm packages published to your Azure Artifacts npm feed.

Keep an eye out on the blog or follow me on Twitter to get an update once these additional blog posts are published.

Have fun releasing packages!

Building a serverless blog site on Azure

Erik Lieben — Fri, 20 Sep 2019 05:58:18 +0000

This article is part of #ServerlessSeptember. You'll find other helpful articles, detailed tutorials, and videos in this all-things-Serverless content collection. New articles are published every day — that's right, every day — from community members and cloud advocates in the month of September.

Find out more about how Microsoft Azure enables your Serverless functions at https://docs.microsoft.com/azure/azure-functions/.

Introduction

In this blog post, I want to take you through the story of a serverless application and teach you how to build a serverless application that runs at minimal costs while maintaining scalability. I hope, to inspire you, to try, play, and get experience with serverless ideas and implementations to gain knowledge of serverless scenario's.

We will build an application that allows us to post articles in markdown and render them out to static HTML pages for easy consumption even if you don't have JavaScript enabled (search engine) and will, later on, look at ways to enhance the site if you do have JavaScript enabled.

This article takes you through the story and gives a global overview of the application with some code samples, but is in no way meant as a copy and paste example for a full application. I will go more in-depth into the specific topics in follow-up blog posts looking at each of the parts separately.

Architecture / Helicopter view

The application can be divided in to a few sections:

the hosting of the static files (below green bar)
the API for performing modifications to content (below red bar)
processing/ generation part (below purple bar)

The goal of serverless in our case is to remove as much of the idle CPU processing parts as possible, while still allowing us to be able to scale out to handle traffic or processes.

The hosting of the static files (below the green bar)

In the first section, we host the files/content of the blog on Azure Storage and serve files to clients using Azure CDN. This allows us to pay only for the storage of files and transfer of files from Azure Blob Storage to the Azure CDN. We won't require anything that is potentially wasting CPU cycles (idle VM or App Services). The CDN allows us to scale and deliver content rapidly to our clients, and we again only pay for the usage of the CDN (no idle machine if there is no traffic).

The API for performing modifications to content (below the red bar)

The second part consists of Azure Functions that we can run as part of the consumption plan. This allows us to remove the need for a machine that is spinning (adding to our costs) and waiting for requests from clients. With Azure Functions in the consumption plan, we only pay for the startup of a function and the amount of CPU/memory it uses during execution. So when no one is writing blog posts (retrieving and storing), the system is, in a sense, turned off and not generating costs. One of the downsides of running your code in this manner is that it takes a bit of time for functions to wake up or cold start. For now, we accept that we sometimes need to wait a few seconds to save or retrieve our content when editor.

Processing/ generation part (below purple bar)

The last part of the application is a set of Azure Functions that handle generating static content that can be consumed by clients. This allows us to serve our content quickly and to all clients (also clients that don't have JavaScript enabled, like search engines) without the need to render static content on each request.

Infrastructure

The central part of our application visited by most of the consumers of our application are the static files (either the JavaScript app/bundles or generated static blog articles). To serve those to the consumers, we require only a small portion of the services Azure offers: Azure Blob Storage and the Azure CDN service.

Static file hosting using Azure Blob static website hosting

Azure Blob Storage supports static website hosting. A feature that allows us to only pay for traffic/transfer and the storage of our files, a feature that fits perfectly into the Serverless story. It also allows us to define an index and error document path, which is very useful for single-page applications using push state.

You can set up a custom domain name for blob storage, but it won't allow you to use a custom SSL certificate for your domain name. So if you want to serve files over HTTPS, it will give you a warning about an incorrect SSL certificate, because it serves the certificate for blob.core.windows.net instead of the one you need for your custom domain. This can be resolved by using the Azure CDN service, which has the option to generate or use a custom certificate for your domain.

Azure Content Delivery Network

Azure CDN is a distributed network of servers managed by Azure that allows us to cache our content close to the end-users to minimize latency. The CDN has worldwide POP (point of presence) locations to provide content as quickly as possible to anyone, anywhere in the world, at any load.

As mentioned above, it also resolves our issue with the SSL certificate, because we can either upload or own SSL certificate or get one for free for our domain.

The CDN on top of Azure Blob storage gives us the perfect scalability and performance targets because the Azure CDN service supports much higher egress limits than a single storage account.

Costs

Calculating costs is difficult if we don't know the exact usage patterns of a site, but we can come up with some quick estimates that give us an idea of the bill that we could get at the end of the month.

Azure Storage

Local redundant storage, which is sufficient for our use case, will cost us €0.0166 per GB per month for the storage we need. The process for transactions are a bit more specific, but if we generalize them, they cost €0.0456 per 10.000 transactions. We get 5GB/month for free on outbound data transfer. After that, we pay €0.074 per GB.

The static files we store aren't GB's of data, it's most likely below a GB of data, which means €0.0166 and let's say we do 50.000 operations (which is a lot, but let's say our authors save their work often) that's €0.228 and a GB of data transfer for €0.074 per GB. That gives us an overall amount of 32 euro cents to host all the content for a month, which is nearly free and we will probably have a lower usage pattern because the Azure CDN does most of the data transfer.

Azure CDN

The costs for Azure CDN are the costs we will start to pay for transfer to clients because they will most likely hit one of the CDN Edge points. We will use Azure Premium from Verizon which is a bit more expensive than the standard one (but supports HTTP to HTTPS redirect rules).

Each zone has a different price, but if we take the most expensive one, which is €0.3930 per GB and estimate 5 GB of transfer, we will end up with a total cost of around 2 euro.

zone	area	per GB/ month
Zone 1	North America, Europe, Middle East and Africa	€0.1333
Zone 2	Asia Pacific (including Japan)	€0.1965
Zone 3	South America	€0.3930
Zone 4	Australia	€0.2202
Zone 5	India	€0.2674

Setup Azure Blob Storage hosting

Azure blob storage can be set up for hosting static content quite easily. Once your storage account is created, go to the 'Static website' section under Settings and enable it using the toggle.

There are two options to configure, the 'Index document name' and the 'Error document name'. If you want to host a SPA application with 'pushState' enabled, set both of these options to the 'index.html' or the root document of your SPA application to make it possible for the SPA application to activate on deeper routes than the base route (deep link into your SPA application/ pushState enabled).

Setup Azure CDN

We can now create a new Azure CDN Profile and point the endpoint to our newly created Azure Storage static site URL. You can find the URL for your static site in the same screen as were you enabled static site hosting. It's the 'Primary endpoint'. When creating the Azure CDN Profile, check the box before 'Create a new CDN endpoint now' and supply the name you want to use. Select 'Custom origin' from the dropdown box 'Origin type' and paste the 'Primary endpoint' URL into the textbox named 'Origin hostname'. Be sure to remove the leading 'https://' to make it valid.

Adding a custom domain name

If you own your own domain name you can set it up to point to the CDN endpoint.

Enable HTTPS

Once you've added your custom domain name you can click on it to setup HTTPS for the custom domain. You can either buy your own SSL certificate or get one for free from Microsoft Azure by using the option 'CDN managed'.

API

The editor will need a way to access the blog articles that are still unpublished and require a way to publish/ save a blog article in a secure way.

Secure API (Azure Function with HTTP trigger) with Azure AD

As we don't want anyone to be able to modify our blog post we need to limit the access to the Azure Functions with HTTP endpoints.

The Azure Functions team created a very easy to use option to accomplish this. We can simply add a provider that takes care of it in the 'Platform features' tab of the 'Functions App' in the section 'Networking' under 'Authentication/ Authorization' without making any modifications to our code.

There are a lot of different authentication providers. For now, I will use 'Azure Active Directory' as the authentication provider and create a user in AD with 2-factor authentication enabled. This will add an additional cost of around €1,- to our overall costs (for a user that has 2-factor authentication enabled).

Azure Functions C-sharp

Our REST API is used by the admin interface and takes care of serving and saving our blog articles. Using the input and output binding of Azure Functions allows us to build our REST API without a lot of code to maintain/ write.

Get blog post

        [FunctionName(nameof(Get))]
        public async Task<IActionResult> Get(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req,
            [Blob("posts", FileAccess.Read, Connection = "connection")] CloudBlobContainer container)
        {
            string slug = req.Query["slug"];
            var blobRef = container.GetBlockBlobReference(slug + ".md");
            string markdownText = await blobRef.DownloadTextAsync();
            return new OkObjectResult(markdownText);
        }

Save blog post

       [FunctionName(nameof(Save))]
        public async Task<IActionResult> Save(
            [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] HttpRequest req,
            [Blob("posts", FileAccess.ReadWrite, Connection = "connection")] CloudBlobContainer container,
            [Queue("get-markdown-metadata", Connection = "blogeriklieben")]CloudQueue outputQueue)
        {
            string slug = req.Query["slug"];
            if (string.IsNullOrWhiteSpace(slug))
            {
                return new BadRequestObjectResult("slug cannot be empty");
            }

            var blobRef = container.GetBlockBlobReference(slug + ".md");

            await blobRef.UploadFromStreamAsync(req.Body);
            blobRef.Properties.ContentType = "text/markdown";
            await blobRef.SetPropertiesAsync();

            // request update to the index file
            await outputQueue.AddMessageAsync(new CloudQueueMessage(slug));

            return new OkObjectResult(slug);
        }

List markdown files

        [FunctionName(nameof(List))]
        public IActionResult List(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req,
            [Blob("posts/index.json", FileAccess.ReadWrite, Connection = "connection")] string index)
        {
            return new JsonResult(index);
        }

Azure Functions TypeScript

The great thing about Azure Functions is that you can make small functions that handle a single responsibility and pass it on to the next function for processing further. That function doesn't even need to be written in the same programming language, you can use the language that best fits the use case.

In our case, we will use TypeScript/JavaScipt to render out markdown files out using markdown-it. This is the markdown to HTML transformer we will use in our client-side editor. Markdown-it is a JavaScript framework for generating HTML from markdown with a rich set of plugins/ extensions.

This way, we don't need to find a C# framework or a port of markdown-it that does precisely the same, we can rather use the same logic in a small function and pass it back to our C# functions.

So even if you don't feel like you have a lot of experience or knowledge of JavaScript, at least you can use a small part of JavaScript code and don't need to worry about gaining the knowledge to host it as a service along with other concerns one might have to keep it running during the lifespan of our application.

In this case, I will use two TypeScript functions; one for gathering metadata and one for generating out static content using Aurelia.

Read markdown metadata

In our editor, we can provide metadata of a blog post by adding the following in key/value sets to the top of our markdown text:

---
title: 'amazing blog post',
publishDate: 2019-09-09,
state: published,
tags: amazing, awesome, superb
---

The only way to get this metadata out of our blog post, is by processing the markdown file itself. What we will do is listen to modifications to markdown files stored in our blob storage account.

Once a markdown file is saved, we need to process the markdown metadata to check if the blog post is in the published state which means that we need to queue it for publication and we will need to update the blog post index file that we keep in blob storage, with the latest information.

The function code index.ts:

const MarkdownIt = require('markdown-it');

module.exports = async function (context, markdownFilePath, markdownFile) {

    context.log('Processing metadata for markdown file: ', markdownFilePath);  

    const md = new MarkdownIt();
    md.use(require('markdown-it-meta'));
    const html = md.render(markdownFile);

    const meta = md.meta;
    meta.fileName = markdownFilePath;
    return JSON.stringify(meta);
};

As you can see this isn't much code and it's still easy to understand and maintain.

The function imports the markdown library and creates an instance of it. The next line imports the markdown-it-meta plugin for parsing the metadata and tells markdown-it to use the plugin/ extension. It will render the markdown to HTML and save the metadata in a separate property on the markdown instance. This is the data we need for further processing; we extend it with the markdownFilePath fileName and return the object serialized as JSON.

Now, if you don't want to use a SPA for rendering out the static HTML, you could just as well use the HTML variable in the above code snippet and combine that with your template HTML, and write it out to blob storage as an .HTML file.

A part of the magic of the above code sample is in the bindings. The Azure Functions runtime is injected in to our function. To let the runtime inject these, we define the following functions.json file with binding definitions:

{
  "bindings": [
    {
      "name": "markdownFilePath",
      "type": "queueTrigger",
      "direction": "in",
      "queueName": "get-markdown-metadata",
      "connection": "ConnectionString_STORAGE"
    },
    {
      "name": "markdownFile",
      "type": "blob",
      "path": "{queueTrigger}",
      "connection": "ConnectionString_STORAGE",
      "direction": "in",
      "dataType": "string"
    },
    {
      "name": "$return",
      "type": "queue",
      "direction": "out",
      "queueName": "markdown-metadata",
      "connection": "ConnectionString_STORAGE"
    }
  ]
}

The first binding is a trigger that activates as soon as a new message arrives in the storage queue, named get-markdown-metadata. The message content is the filename of the modified markdown file.

The second binding provides us with the content of the markdown file. To get the path of the markdown file we use the dynamic variable {queueTrigger} to get the message content from the queue that activated the Azure Function.

The last binding is the binding on the return value of the function and writes out the return value in a different storage queue named markdown-metadata.

Generate static files

I want to enhance my blog, later on, to become more dynamic and use a SPA (single page application) framework to do this. For now, generating static files using a SPA framework might look a bit odd, but it will be instrumental, to be revealed soon (in a future blog post-:-)).

One of the downsides of a SPA is that it is Client-Side Rendered by default, which isn't optimal for visitors that depend upon the static content and it also requires a little bit of time to initialize the SPA framework on the first load of the page. An example of a visitor that isn't starting up your SPA application is a search engine and it will miss out on most of your content. Luckily, there are a few options to mitigate the downsides.

Enhancing

With the enhance technique, you take a static, (or a server-side rendered) part of the site (rendered using another framework such as ASP.NET) and progressively enhance it using client-side code. This technique works well if the page has static content and doesn't use any dynamic content on each page load to render/understand the page. Content doesn't need to be static forever; the number of reads/views of the content just needs to succeed the amount of writes/modifications to the content.

Examples of these might be one blog post, a product page, and the news section.

This technique works well in a serverless context because we only need CPU cycles to generate static content from time to time. You will need to think about the amount of content you have and the timeframe in which you require the static content to refresh. It does its job right if the number of views is higher than the number of times the content is regenerated.

Server-side rendering

With the SSR (Service Side Rendering) technique, you run the framework on the server-side on each request to dynamically generate the first view that the client will be presented with. Now, this doesn't feels like anything new Since we've been doing that for ages using ASP.NET.

The main difference with this technique is that you use the same SPA framework as on the client-side and run it using Node.JS on the server. This allows you to have one code base and let the framework handle the rehydration of the page from the static content.

An example of this might be a (very active) discussion in a discussion board. You want to present the latest discussions at the page load, and let the client-side rendering handle the new posts that arrive after the first page load. Alternatively, if you have a profile page that due to the content changes, changes every hour, but only receives a visitor once a week, SSR might also be a better fit.

You can use this technique in a serverless manner, but you will need to keep in mind that it will require CPU cycles for each request because you need to render on each request. This works great if you have a large amount of content and the change rate is higher than the read/ visitor rate or if you need to render out pages with a 1-to-1 rate for write/ modifications and reads/visits.

The implementation

The SPA framework I like to use is Aurelia, which has been around since late 2015. The framework consists of a set of different libraries that can be used together as a robust framework. Due to this separation and all the different use cases, the libraries can be used in; from the start of the development of the framework, it provided high extensibility. One of the examples of that is the PAL (platform abstraction library) that is used throughout the libraries to abstract away the dependency on an actual browser, which means we can use it with a 'virtual browser' implementation in NodeJS. The next version of Aurelia which I will use during this post contains a similar implementation that is built on top of JSDOM in the library @aurelia/runtime-html-jsdom, which runs perfectly inside of on Azure Function.

A small disclaimer: the next version of Aurelia (vNext or 2) is still under development, which means it might not be the best choice for production usage at the time of writing this blog, but for this blog post I accept that things might be different in the final release of the next version of Aurelia.

side note: I will create a separate blog post later on that will walk through the Aurelia application step-by-step, for this post I will focus on rendering the static file content using Azure Functions and starting up the SPA app.

At the first try to generate static pages, I created code to start Aurelia and used the @aurelia/runtime-html-jsdom, which worked smoothly for everything related to Aurelia. One of the things that didn't work as well was the webpack plugin style-loader because I could not find a way to provide or inject a custom implementation of the DOM; it seems to have a hard dependency on objects in the browser. The easiest way around this was to load it inside the 'virtual browser' (that is created by JSDOM) where all the objects it requires exist.

Let's first look at the code required to render out the static page:

import { AzureFunction, Context } from "@azure/functions";
import * as jsdom from 'jsdom';
import * as fetch from 'node-fetch';

const queueTrigger: AzureFunction = async function (context: Context, slug: string): Promise<void> {

    context.log('Slug to render', slug);

    // Retrieve the SPA application html and javascript bundle
    const mainjs = await getFile('main.js');
    const indexhtml = await getFile('index.html');

    // Create a new JSDOM instance and use the index.html as the open document
    const dom = new jsdom.JSDOM(indexhtml, {
        contentType: "text/html",
        includeNodeLocations: true,
        pretendToBeVisual: true,
        storageQuota: 10000000,
        runScripts: "dangerously",
        resources: "usable"
    });

    // JSDOM has no default support for fetch, let's add it because we use fetch for performing calls to our API in our SPA app
    dom.window.fetch = fetch["default"];

    // Once JSDOM is done loading all the content (our index file)
    dom.window.document.addEventListener("DOMContentLoaded", async function () {

        // Tell JSDOM to load our webpack bundle and execute it
        dom.window.eval(mainjs);

        // Wait for the Aurelia application to start
        await dom.window.au.wait();

        // Change the url to let the aurelia-router open the component blog-post with the specified slug (the component will load the file from our get-post API)
        dom.window.location.hash = `blog-post(${slug})`;

        // Wait a second for the routing to complete
        await new Promise(resolve => setTimeout(resolve, 1000));

        // Serialize the state of the DOM to a string 
        let result = dom.serialize();

        // Replace the bundle, so that the app doesn't directly startup when the page is loaded (we want to keep it static for now)
        result = result.replace('<script type="text/javascript" src="main.js"></script>', '');

        // Store the result and notify Azure Functions we are done
        context.done(await saveFile(slug, result));
    });
};

export default queueTrigger;

As you can see in this case, we don't use blob input or output bindings. This is because at the point of writing this blog post the option to access blobs from the $web container (which is used by Azure Blob Storage static site hosting as the root container) is still not supported or I could not find a way to escape the $ character.

What we can do for the time being is use the azure blob storage SDK to get and save the files ourselves. The functions getFile and saveFile in the code block below will do that for us. It's a bit less pleasant, but it also gives us the insights into how much code we can save/remove by using the Azure Functions bindings :-)

import {
  Aborter,
  BlockBlobURL,
  ContainerURL,
  ServiceURL,
  SharedKeyCredential,
  StorageURL} from '@azure/storage-blob';

// credentials should not be in code, but just here to make it easier to read
const storageAccount = 'storage-account-name';
const pipeline = StorageURL.newPipeline(new SharedKeyCredential(storageAccount, 'key'));
const serviceURL = new ServiceURL(`https://${storageAccount}.blob.core.windows.net`, pipeline);
const containerURL = ContainerURL.fromServiceURL(serviceURL, '$web');

async function getFile(file) {   
    const blockBlobURL = BlockBlobURL.fromContainerURL(containerURL, file);
    const aborter = Aborter.timeout(30 * 1000);
    const downloadResponse = await blockBlobURL.download(aborter, 0);
    return await streamToString(downloadResponse.readableStreamBody);
}

async function streamToString(readableStream) {
    return new Promise((resolve, reject) => {
      const chunks = [];
      readableStream.on("data", data => {
        chunks.push(data.toString());
      });
      readableStream.on("end", () => {
        resolve(chunks.join(""));
      });
      readableStream.on("error", reject);
  });
}

async function saveFile(slug: string, content: string) {

  const blockBlobURL = BlockBlobURL.fromContainerURL(containerURL, `${slug}\\index.html`);
  const uploadBlobResponse = await blockBlobURL.upload(Aborter.none, content, content.length, {
    blobHTTPHeaders: {
      blobContentType: "text/html",
      blobContentEncoding: "utf-8",
    }
  });

  return uploadBlobResponse.errorCode;
}

The only content left for the above function is the function.json file that contains our binding information.
As you can see we generate a new static page as soon as we get a new item in the render-static-page storage queue.
The slug we push into the queue is a short identifier for the blog post itself, mostly with dashes to create a readable URL.

{
  "bindings": [
    {
      "name": "slug",
      "type": "queueTrigger",
      "direction": "in",
      "queueName": "render-static-page",
      "connection": "connectionString_STORAGE"
    }
  ],
  "scriptFile": "../dist/RenderFile/index.js"
}

So what are our approximate monthly running costs?

€1,18 a month for a Active Directory user
~ €0.32 for hosting our content on Azure Storage
~ €2,- for proving our content using the Azure CDN

So for the price of a coffee or a beer a month at a café we are able to serve our application in optimal conditions around the world.

Where can we go next?

There are a lot of different services in Azure that you can attach to your system or external system you can talk to using web hooks.

A few examples are:

Generate audio transcript using Azure Cognitive services text to speech
Tweet new blog post created (Azure Function => twitter API)
Notify Microsoft Teams channel (Azure Function => Teams API)
Generate PDF/ EPUB (Azure Function)

I hope this article could inspire you to think differently about the things you need to build and that you don't always need a AppService or VM that is costing money while it's idle.