DEV Community: Kees C. Bakker

Async Replace All for JavaScript

Kees C. Bakker — Tue, 25 Apr 2023 14:16:36 +0000

I love the replaceAll string API in JavaScript, as it makes replacing a string far more intuitive than the "good old" global regular expression. This week I had to replace strings with the results of async calls. Yeah, that is not supported by any API in standard JavaScript.

The idea

So, let's create a replaceAllAsync that looks like this:

export async function replaceAllAsync(
  input: string,
  regex: RegExp,
  replacement: (match: RegExpMatchArray) => Promise<string>,
  mode: PromiseExecutionMode = PromiseExecutionMode.All
): Promise<string> {
  // implementation
}

export enum PromiseExecutionMode {
  // Uses Promise.All -- which is the fastest, but may overwhelm
  All,
  // Uses an await on each replacement, making it synchronous
  ForEach,
}

The input is the input string that we'll work on. The regex is the regular expression that will be used to find matches that need to be replaced. Every match is fed to the replacement function, which will be awaited. The mode indicates if we should process all the promises at once or one by one.

In the body of the function, we need to:

Capture all the matches and feed them to the replacement function.
Split the input on the regular expression.
Stitch all the components back together into a string.
Return that string.

Easy, right? Well... it turns out there are a view caveats when it comes to regular expressions.

Caveats of reusing a global regex

Consider the following code:

const str = "Numb3r!1"
const regex = /\d+/g

console.log("Test 1", regex.test(str)) // true
console.log("Test 2", regex.test(str)) // true
console.log("Test 3", regex.test(str)) // false
console.log("Test 4", regex.test(str)) // true

Why? It turns out the RegExp oject is very stateful when the global flag is set:

JavaScript RegExp objects are stateful when they have the global or sticky flags set (e.g., /foo/g or /foo/y). They store a lastIndex from the previous match. Using this internally, test() can be used to iterate over multiple matches in a string of text (with capture groups).

Source: MDN Web Docs - RegExp.prototype.test()

The lastIndex is changed per test! So when you reuse a global regex, you might not get what you think. We'll create a new regular expression based on the given one.

Caveats of split with capture groups

Consider the following code:

const str = "I have 12 bananas and 3 apples!"

// no groups:
const regex1 = /\d+/g
console.log("Test 1", str.split(regex1))
// [ 'I have ', ' bananas and ', ' apples!' ]

// capture groups:
const regex2 = /(\d+)/g
console.log("Test 2", str.split(regex2))
// [ 'I have ', '12', ' bananas and ', '3', ' apples!' ]

// non-capture groups:
const regex3 = /(?:\d+)/g
console.log("Test 3", str.split(regex3))
// [ 'I have ', ' bananas and ', ' apples!' ]

Conclusion: if you use capture groups in your regular expression, they will end up in the splitted result.

Caveats of matchAll

So, when you want to capture all the matches of a string you can do a str.matchAll(regex). But this only work for global regular expressions.

Implications

Based on the caveats, we have to do a view things:

It is better not to reuse the given regular expression, but to create a new one from it. This prevents problems with lastIndex.
Convert the regular expression to a global one. The end user should already know, but this might make it a bit easier, and it prevents us from having to debug all the time.
Use 2 regular expressions: one for matching and one for splitting. The regular expression used for splitting should have its capturing groups replaced by non-capturing groups.

The code

Now, let's implement the function using what we know:

export async function replaceAllAsync(
  input: string,
  regex: RegExp,
  replacement: (match: RegExpMatchArray) => Promise<string>,
  mode: PromiseExecutionMode = PromiseExecutionMode.All
): Promise<string> {
  // replace all implies global, so append if it is missing
  const addGlobal = !regex.flags.includes("g")
  let flags = regex.flags
  if (addGlobal) flags += "g"

  // get matches
  let matcher = new RegExp(regex.source, flags)
  const matches = Array.from(input.matchAll(matcher))

  if (matches.length == 0) return input

  // construct all replacements
  let replacements: Array<string>
  if (mode == PromiseExecutionMode.All) {
    replacements = await Promise.all(matches.map(match => replacement(match)))
  } else if (mode == PromiseExecutionMode.ForEach) {
    replacements = new Array<string>()
    for (let m of matches) {
      let r = await replacement(m)
      replacements.push(r)
    }
  }

  // change capturing groups into non-capturing groups for split
  // (because capturing groups are added to the parts array
  let source = regex.source.replace(/(?<!\\)\((?!\?:)/g, "(?:")
  let splitter = new RegExp(source, flags)

  const parts = input.split(splitter)

  // stitch everything back together
  let result = parts[0]
  for (let i = 0; i < replacements.length; i++) {
    result += replacements[i] + parts[i + 1]
  }

  return result
}

export enum PromiseExecutionMode {
  // Uses Promise.All -- which is the fastest, but may overwhelm
  All,
  // Uses an await on each replacement, making it synchronous
  ForEach,
}

Conclusion

JavaScript is not always as intuitive as I would like it to be. But with regular expressions, you can build some pretty powerful async replacements.

.NET Console Application with injectable commands

Kees C. Bakker — Tue, 07 Mar 2023 04:11:57 +0000

Console applications are alive and kicking. Setting them up might be a bit hard. In this article I'll explore how to create a .NET console application that provides commands using the new System.CommandLine package. This will provide arguments to command mapping out of the box. I'll be showing how to combine it with dependency injection for even more power ⚡.

Goals
NuGet Packages
Project structure
Fake weather service
Commands
1. Current Temperature Command
2. Forecast Command
Dependency injection
Final thoughts

Goals

We want to create a CLI application with the following goals:

System.CommandLine — this is a fairly new project by .NET that helps to create better CLI applications. It offers the ability to add commands, arguments and options to your application. It comes with a --help feature and it will do the command line argument mapping for you.
Dependency Injection — why go anywhere without it? Dependency injection has made ASP.NET way more composable. I wrote an entire article on how to add it to console applications as well. We'll be reusing some of the code.
Environment variable injection support — some of the configuration should be overridable using environment variables.

We're making a CLI, so what's a better way to describe it than showing what the --help should look like?

Description:
  Weather information using a fake weather service.

Usage:
  MyCli [command] [options]

Options:
  --version       Show version information
  -?, -h, --help  Show help and usage information

Commands:
  current   Gets the current temperature.
  forecast  Get the forecast. Almost always wrong.

Note: if you want to use command line argument when executing a dotnet run, you can use -- to feed the arguments to the application instead of the .NET CLI (so dotnet run -- --help in this case).

NuGet Packages

If you say .NET, you say NuGet packages. We'll be using the following packages:

Install-Package System.CommandLine -Version 2.0.0-beta4.22272.1  
Install-Package Microsoft.Extensions.Configuration -Version 7.0.0  
Install-Package Microsoft.Extensions.Configuration.EnvironmentVariables -Version 7.0.0  
Install-Package Microsoft.Extensions.DependencyInjection -Version 7.0.0  
Install-Package Microsoft.Extensions.DependencyInjection.Abstractions -Version 7.0.0  
Install-Package Microsoft.Extensions.Options -Version 7.0.1  
Install-Package Microsoft.Extensions.Options.ConfigurationExtensions -Version 7.0.0

The System.CommandLine package is still in ~~beta~~ preview. I expect it to be released soon, but things might still change. It is used by the .NET dotnet CLI.

Project structure

I'm using the following project structure:

.
├── src/
│   └── MyCli/
│       ├── Commands/
│       │   ├── CurrentCommand.cs
│       │   └── ForcastCommand.cs
│       ├── Services/
│       │   ├── FakeWeatherService.cs
│       │   └── FakeWeatherServiceSettings.cs
│       └── Program.cs
└── MyCli.sln

Fake weather service

What is injection without a good service? Let's create a fake weather service that returns the temperature based on a randomizer:

namespace MyCli.Services;

public class FakeWeatherServiceSettings
{
    public string DefaultCity { get; set; } = "Zwolle, NLD";

    public int DefaultForecastDays { get; set; } = 5;
}

public class FakeWeatherService
{
    public FakeWeatherService(IOptions<FakeWeatherServiceSettings> settings)
    {
        Settings = settings?.Value ?? throw new ArgumentNullException(nameof(settings));
    }

    public FakeWeatherServiceSettings Settings { get; }

    public Task<string> GetTemperature(string? city = null)
    {
        if (city == null) city = Settings.DefaultCity;

        var report = $"In {city} it is now {Random.Shared.Next(-20, 40)} degrees celcius.";
        return Task.FromResult(report);
    }

    public Task<string[]> Forecast(int days, string? city = null)
    {
        if (city == null) city = Settings.DefaultCity;

        var reports = new List<string>
        {
            $"Report for {city} for the next {days} days:"
        };

        for (var i = 0; i<days; i++)
        {
            var date = DateTime.Now.AddDays(i + 1).ToString("yyyy-MM-dd");
            var report = $"- {date}: {Random.Shared.Next(-20, 40),3} degrees celcius.";
            reports.Add(report);
        }

        return Task.FromResult(reports.ToArray());
    }
}

Commands

Commands are implementations of the System.CommandLine.Command class. To make them injectable, we create classes that are derived from the Command class (see dependency injection section).

Current Temperature Command

To get our current temperature command, we'll need to do the following:

Call the base constructor with the name and description of the command. This will be used by the --help feature.
Inject the FakeWeatherService, as it does the actual work.
Use the FakeWeatherService.Settings to get the default value for the --city option.
Map it all together using a SetHandler. The option in automatically mapped to the city parameter of the Execute method.

Now the implementation is very easy:

using MyCli.Services;
using System.CommandLine;

namespace MyCli.Commands;

class CurrentCommand : Command
{
    private readonly FakeWeatherService _weather;

    public CurrentCommand(FakeWeatherService weather) : base("current", "Gets the current temperature.")
    {
        _weather = weather ?? throw new ArgumentNullException(nameof(weather));

        var cityOption = new Option<string>("--city", () => _weather.Settings.DefaultCity, "The city.");

        AddOption(cityOption);

        this.SetHandler(Execute, cityOption);
    }

    private async Task Execute(string city)
    {
        var report = await _weather.GetTemperature(city);
        Console.WriteLine(report);
    }
}

What I like about the setup is that we can add optional arguments with defaults. Here we get the default value from an object from our dependency injection. When we do a current --help, we can a nice description and the actual injected value:

Description:
  Gets the current temperature.

Usage:
  MyCli current [options]

Options:
  --city <city>   The city. [default: Amsterdam, NLD]
  -?, -h, --help  Show help and usage information

Forecast Command

The same goes for the forecast command, but now we have 2 options: --city and --days.

using Microsoft.Extensions.Options;
using MyCli.Services;
using System.CommandLine;

namespace MyCli.Commands;

class ForecastCommand : Command
{
    private readonly FakeWeatherService _weather;

    public ForecastCommand(FakeWeatherService weather) : base("forecast", "Get the forecast. Almost always wrong.")
    {
        _weather = weather ?? throw new ArgumentNullException(nameof(weather));

        var cityOption = new Option<string>("--city", ()=> _weather.Settings.DefaultCity, "The city.");
        var daysOption = new Option<int>("--days", () => _weather.Settings.DefaultForecastDays, "Number of days.");

        AddOption(cityOption);
        AddOption(daysOption);

        this.SetHandler(Execute, cityOption, daysOption);
    }

    private async Task Execute(string city, int days)
    {
        var report = await _weather.Forecast(days, city);
        foreach (var item in report)
        {
            Console.WriteLine(item);
        }
    }
}

Dependency injection

Now, let's tie it all together using dependency injection. We need to do the following:

Setup a ServiceCollection to store our dependencies.
Setup the configuration to use environment variables and read them into our WeatherServiceSettings object.
Add the commands CurrentCommand and ForecastCommand to the service collection.
Add the WeatherService to the service collection.
Create a System.CommandLine.RootCommand and tie it to the registered Command implementation.
Invoke the root command with the given command line arguments.

This leads to the followingProgram.cs code:

using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using MyCli.Commands;
using MyCli.Services;
using System.CommandLine;

static void ConfigureServices(IServiceCollection services)
{
    // build config
    var configuration = new ConfigurationBuilder()
        .AddEnvironmentVariables()
        .Build();

    // settings
    services.Configure<FakeWeatherServiceSettings>(configuration.GetSection("Weather"));

    // add commands:
    services.AddTransient<Command, CurrentCommand>();
    services.AddTransient<Command, ForecastCommand>();

    // add services:
    services.AddTransient<FakeWeatherService>();
}

// create service collection
var services = new ServiceCollection();
ConfigureServices(services);

// create service provider
using var serviceProvider = services.BuildServiceProvider();

// entry to run app
var commands = serviceProvider.GetServices<Command>();
var rootCommand = new RootCommand("Weather information using a fake weather service.");
commands.ToList().ForEach(command => rootCommand.AddCommand(command));

await rootCommand.InvokeAsync(args);

To make dependency injection work, we do a GetServices to retrieve all the commands and add them to the root command.

Final thoughts

And that's all: now you have a CLI that supports commands and a --help feature out of the box!

I've added the code to GitHub, so check it out: github.com/KeesCBakker/dotnet-cli-di-poc

Rethinking our ASP.NET Docker CI

Kees C. Bakker — Tue, 28 Feb 2023 15:10:33 +0000

At Wehkamp we've been using ASP.NET for a long time. Containers brought .NET with us to our microservices architecture. We used a shell script in a special build container to build our service. Today we'll discuss how we turned that shell script into a multi stage build Dockerfile.

Intro
Goals
Project structure
Step 1: Configuration with global arguments
Step 2: Build stage
1. (2.1) Restore in cacheable layers
2. (2.2) The build script
3. (2.3) Optional testing and publishing
Step 3: runtime stage
1. (3.1) Which runtime base image should we use?
2. (3.2) Go self contained or not?
3. (3.3) The final runtime image
All together
Reading list
Changelog
Comments

Goals

With the new setup we have some goals in mind:

Reproducibility — it should be on my machine as it is in the pipeline. It would be great if I can just do a docker build . and everything should start building as it does in Jenkins.
Cacheability — Docker uses layers, which greatly speeds up the system. When we do a restore as a set of layers, we don't need to generate network traffic repeatedly.
Upgradability — the current solution will focus on .NET 7.0, but we need things to be easily upgradable to a newer (or older) version of .NET.
Readability — the Dockerfile should be treaded as a DevOps manifest, showing how to build and run our application. Let's make sure people understand what's going on. But also the output that Docker shows in the terminal should be easy to read and understand.

If at some point we can add security to the Dockerfile, that would also be great.

Project structure

Most of what we do is based on convention. Our project structure reflects those conventions:

.
├── src/
│   └── Blaze.Prime.Service.Api/
│       ├── Blaze.Prime.Service.Api.csproj
│       └── ...
├── test/
│   └── Blaze.Prime.Service.Api.UnitTests/
│       ├── Blaze.Prime.Service.Api.UnitTests.csproj
│       └── ...
├── .dockerignore
├── .gitignore
├── Blaze.Prime.sln
├── Dockerfile
└── nuget.config

It is important that the name of the C# project directory matches the name of the C# project file.

Step 1: Configuration with global arguments

Our Dockerfile may grow. Let's make it easy to upgrade and changing global settings by using ARG instructions. We will use these arguments in the build and runtime images.

########################
# Configuration settings
########################

ARG API_NAME="Blaze.Prime.Service.Api"
ARG \
   API_PROJECT="./src/$API_NAME/$API_NAME.csproj" \
   API_DLL="$API_NAME.dll" \
   PORT=5000 \
   ASPNET_VERSION="7.0" \
   # options: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic]
   # minimal keeps your pipeline readable while inforing you what's going on
   VERBOSITY="minimal" \
   APP_DIR="/app" \
   TAG="" \
   EXECUTE_TESTS="true"

Let's construct the project and DLL names by convention. To make debugging easier, we set the VERBOSITY argument centrally. We find that the minimal settings is a great default, as it just gives enough output to understand what's going on.

Our CI fills the TAG with the tag of the Docker image that is being built.

Step 2: Build stage

Now that we have the arguments in place, let's start with the beginning of our build image. We use the standard SDK provided by Microsoft. We don't feel the need to share telemetry and we would like to have our times in local Amsterdam time.

################################################
# Build image, uses ASPNET_VERSION, API_PROJECT, 
# VERBOSITY, TAG, APP_DIR, EXECUTE_TESTS
################################################

FROM mcr.microsoft.com/dotnet/sdk:$ASPNET_VERSION AS build

ARG API_PROJECT VERBOSITY APP_DIR
ENV \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_NOLOGO=0

WORKDIR /build

Notice how we're not defining TAG and EXECUTE_TESTS here. If we would, we would lose our caching ability when a new tag is used or when testing is switched off.

2.1 Restore in cacheable layers

In our old script we just executed a dotnet restore as part of the build script, which is fine, but nothing is cached by Docker. Our build script will take longer than necessary. Andrew Lock did an excellent job of writing a solution that makes the restore cacheable, so let's use his solution:

# Let's restore the solution, nuget and project files and do a restore
# in cachable layers. This will speed up the build process greatly

# copy global files to restore
COPY *.sln *.*config ./

# copy src files to restore
COPY src/*/*.csproj .
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

# copy test files to restore
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done; \
   echo "" \
   && echo "-------------" \
   && echo "1/4 RESTORING" \
   && echo "-------------" \
   && echo "" \
   && dotnet restore --verbosity "$VERBOSITY" || exit 1

This method restores the project structure and makes the dotnet restore cacheable.

Notice how we are using echo to make our output more readable?

Output of the Dockerfile.

2.2 The build script

Now what should we copy? The base files like the solution and the NuGet config are already in there. Let's copy src and test only. This will help us cache builds that are triggered just because some documentation was changed (outside of the directory).

# copy dirs that are only needed for building and testing
COPY src ./src
COPY test ./test

# Note on build: don't use --no-restore, sometimes certain packages cannot be
# restored by the dotnet restore. The build will add them, as it has more context (!?)
# example: Package System.Text.Json, version 6.0.0 was not found

RUN echo "" \
   && echo "------------" \
   && echo "2/4 BUILDING" \
   && echo "------------" \
   && echo "" \
   && dotnet build --configuration Release --verbosity "$VERBOSITY" -nowarn:NETSDK1004 || exit 1

2.3 Optional testing and publishing

At the top of our Dockerfile, we've defined the ARG named EXECUTE_TESTS. Let's use it to test if we should do any testing. You might want to be able to skip testing to run faster locally.

# defining the argument here caches the previous layers when the value switches
ARG EXECUTE_TESTS
RUN echo "" \
   && echo "-----------" \
   && echo "3/4 TESTING" \
   && echo "-----------" \
   && echo ""; \
   if [ "$EXECUTE_TESTS" = "true" ]; then \
      dotnet test --configuration Release --logger "console;verbosity=$VERBOSITY" --no-build || exit 1; \
   else \
      echo "Skipping unit tests"; \
   fi; \
   echo "" \
   && echo "--------------" \
   && echo "4/4 PUBLISHING" \
   && echo "--------------" \
   && echo "" \
   && dotnet publish "$API_PROJECT" --configuration Release --output "$APP_DIR" --no-restore -nowarn:NETSDK1004 || exit 1

Step 3: runtime stage

The hard part is over: we have our application built. But there are still some choices to make!

3.1 Which runtime base image should we use?

So what runtime image are we going to use? We've built the entire project and checked the size of each resulting image:

Base image	Total output size
mcr.microsoft.com/dotnet/aspnet:7.0	218MB
mcr.microsoft.com/dotnet/nightly/aspnet:7.0-jammy-chiseled	114MB
mcr.microsoft.com/dotnet/aspnet:7.0-alpine	112MB

Looks like the chiseled project is very promising, as it promises a secure and minimal Debian image, but they have not released it for .NET 7. Most of our projects will run fine on Alpine.

3.2 Go self contained or not?

If you want to go even smaller, you can go to a self contained (and trimmed) application. It is a great way to limit the attack surface even further and a will result in an even smaller image (which is also great). But there are some caveats: your application may take longer to boot and you can't cache your ASP.NET dependencies, so there might be some more network traffic involved. For now we just stick with the old adage as we try to read more on what the best way forward is.

3.3 The final runtime image

We've decided on Alpine for now, so our runtime image looks like this:

#################################################
# Runtime image, uses PORT, TAG, API_DLL, APP_DIR
#################################################

FROM mcr.microsoft.com/dotnet/aspnet:$ASPNET_VERSION-alpine as runtime

RUN apk add --no-cache icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib tzdata

ARG PORT API_DLL APP_DIR

# create a new user and change directory ownership
RUN adduser --disabled-password \
   --home "$APP_DIR" \
   --gecos '' dotnetuser && chown -R dotnetuser "$APP_DIR"

# impersonate into the new user
USER dotnetuser

ENV \
   ASPNETCORE_URLS=http://*:$PORT \
   ASPNETCORE_ENVIRONMENT=Production \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

WORKDIR $APP_DIR
COPY --from=build $APP_DIR .

EXPOSE $PORT

ENV PROGRAM="$API_DLL"
ENTRYPOINT dotnet "$PROGRAM"

ARG TAG

For security reasons, we create a new user, so we don't run as root.

All together

When we put it all together, we get the following Dockerfile:

########################
# Configuration settings
########################

ARG API_NAME="Blaze.Prime.Service.Api"
ARG \
   API_PROJECT="./src/$API_NAME/$API_NAME.csproj" \
   API_DLL="$API_NAME.dll" \
   PORT=5000 \
   ASPNET_VERSION="7.0" \
   # options: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic]
   # minimal keeps your pipeline readable while inforing you what's going on
   VERBOSITY="minimal" \
   APP_DIR="/app" \
   TAG="" \
   EXECUTE_TESTS="true"

################################################
# Build image, uses ASPNET_VERSION, API_PROJECT, 
# VERBOSITY, TAG, APP_DIR, EXECUTE_TESTS
################################################

FROM mcr.microsoft.com/dotnet/sdk:$ASPNET_VERSION AS build

ARG API_PROJECT VERBOSITY APP_DIR
ENV \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_NOLOGO=0

WORKDIR /build

# Let's restore the solution, nuget and project files and do a restore
# in cachable layers. This will speed up the build process greatly

# copy global files to restore
COPY *.sln *.*config ./

# copy src files to restore
COPY src/*/*.csproj .
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done

# copy test files to restore
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done; \
   echo "" \
   && echo "-------------" \
   && echo "1/4 RESTORING" \
   && echo "-------------" \
   && echo "" \
   && dotnet restore --verbosity "$VERBOSITY" || exit 1

# copy dirs that are only needed for building and testing
COPY src ./src
COPY test ./test

# Note on build: don't use --no-restore, sometimes certain packages cannot be
# restored by the dotnet restore. The build will add them, as it has more context (!?)
# example: Package System.Text.Json, version 6.0.0 was not found

RUN echo "" \
   && echo "------------" \
   && echo "2/4 BUILDING" \
   && echo "------------" \
   && echo "" \
   && dotnet build --configuration Release --verbosity "$VERBOSITY" -nowarn:NETSDK1004 || exit 1

# defining the argument here caches the previous layers when the value switches
ARG EXECUTE_TESTS
RUN echo "" \
   && echo "-----------" \
   && echo "3/4 TESTING" \
   && echo "-----------" \
   && echo ""; \
   if [ "$EXECUTE_TESTS" = "true" ]; then \
      dotnet test --configuration Release --logger "console;verbosity=$VERBOSITY" --no-build || exit 1; \
   else \
      echo "Skipping unit tests"; \
   fi; \
   echo "" \
   && echo "--------------" \
   && echo "4/4 PUBLISHING" \
   && echo "--------------" \
   && echo "" \
   && dotnet publish "$API_PROJECT" --configuration Release --output "$APP_DIR" --no-restore -nowarn:NETSDK1004 || exit 1


#################################################
# Runtime image, uses PORT, TAG, API_DLL, APP_DIR
#################################################

FROM mcr.microsoft.com/dotnet/aspnet:$ASPNET_VERSION-alpine as runtime

RUN apk add --no-cache icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib tzdata

ARG PORT API_DLL APP_DIR

# create a new user and change directory ownership
RUN adduser --disabled-password \
   --home "$APP_DIR" \
   --gecos '' dotnetuser && chown -R dotnetuser "$APP_DIR"

# impersonate into the new user
USER dotnetuser

ENV \
   ASPNETCORE_URLS=http://*:$PORT \
   ASPNETCORE_ENVIRONMENT=Production \
   TZ=Europe/Amsterdam \
   DOTNET_CLI_TELEMETRY_OPTOUT=1 \
   DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

WORKDIR $APP_DIR
COPY --from=build $APP_DIR .

EXPOSE $PORT

ENV PROGRAM="$API_DLL"
ENTRYPOINT dotnet "$PROGRAM"

ARG TAG

Pretty cool, right?

Reading list

While studying the subject we found some interesting blogs to read:

On security: How to build smaller and secure Docker Images for .NET
On dotnet restore & Docker optimizations: Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files (Part 2)
On the new Chiselled project: Chiselled Ubuntu: the perfect present for your containerised and cloud applications
On Windows you might get the error "ERROR: failed to solve: error from sender: readdir: open src\*: The filename, directory name, or volume label syntax is incorrect." This is due to a bug in the Docker Moby BuildKit, which cannot handle the wildcards. On PowerShell you can execute $env:DOCKER_BUILDKIT=0 to disable BuildKit.

Changelog

2023-05-26: removed the section on compiling git info into the binary.
2023-05-26: installed the tzdata package to fix timezone support.
2023-05-26: notes on how to fix the Windows Docker error: "error from sender: readdir".
2023-05-26: nuget.config is now optional.
2023-05-26: make sure docker build exits on error.

Chrome Extension Development: Fetch non CORS URLs in v3

Kees C. Bakker — Sun, 15 Jan 2023 08:29:53 +0000

As browser security improves, many applications have implemented CORS headers to protect against unauthorized access. I'm working on a Chrome extension that needs to check if a certain URL is present on GitHub, before it opens the URL in a new tab. Let's explore what we can do.

The Manifest

Due to CORS headers, it is not possible to perform a fetch request for the HEAD method on a domain that is not included in the headers. However, by utilizing a worker script (in Chrome Extension version 3), it is possible to bypass this restriction. Let's configure the manifest:

{
  "manifest_version": 3,
  "name": "My Extension Name",
  "description": "My Extension Description",
  "version": "1.0",
  "content_scripts": [
    {
      "matches": [ "https://*/*" ],
      "js": ["content.js"]
    }
  ],
  "background": {
    "service_worker": "background.js"
  },
  "permissions": ["activeTab"]
  "host_permissions": ["https://github.com/*"]
}

It's important to note that the host_permissions array in the manifest file needs to include the URL that the extension will inspect, in this case "github.com" must be added in the array. If you fail to do so, you won't ever be able to fetch the data!

The Background Script

Interaction between the content and background scripts can be achieved through message passing. In our background.js script we'll add an event listener.

chrome.runtime.onMessage.addListener((request, sender, sendResponse) => {
  if (request.checkIfUrlExists) {
    fetch(request.checkIfUrlExists, { method: "HEAD" })
      .then(response => sendResponse({ status: response.ok }))
      .catch(error => sendResponse({ status: false, error: JSON.stringify(error) }))
  }
  return true
})

The Content Script

With the background script set up, we can use the chrome.runtime.sendMessage API to send messages to it. To make it easier to use async/await in our extension, we'll wrap the call in a Promise:

async function checkIfUrlExists(url) {
  const response = await new Promise(resolve => {
    chrome.runtime.sendMessage({ checkIfUrlExists: url }, resolve)
  })
  return response.status
}

Resolve URLs

My extension needs to check if a certain URL is present on GitHub. Because I don't know the exact name of the file, I'll check multiple possibilities and return the first that has a successful response status:

async function resolveUrl(name) {
  let repo = "https://github.com/keescbakker/my-workload-services"
  let commits = repo + "/commits/master/apps/applicationsets"

  let urls = [
    `${commits}/${name}.yml`,
    `${commits}/${name}.yaml`,
    `${commits}/${name}-service.yml`,
    `${commits}/${name}-service.yaml`,
    `${commits}/${name}-site.yml`,
    `${commits}/${name}-site.yaml`
  ]

  for (let url of urls) {
    if (await checkIfUrlExists(url)) {
      return url
    }
  }

  return null
}

Enjoy!

Dependency injection (with IOptions) in Console Apps in .NET

Kees C. Bakker — Mon, 12 Sep 2022 20:33:30 +0000

When you are used to building web applications, you kind of get hooked to the ease of Dependency Injection (DI) and the way settings can be specified in a JSON file and accessed through DI (IOptions<T>). It's only logical to want the same feature in your Console app.

After reading many - many! - outdated blogs, I decided to add my 50 cents specifically on a Console app. It all comes down to packages and separation of concerns.

The first version of this article was written in 2018. It has gotten lots of views since and it should stay green. That's why I've rewritten the article to include instructions for .NET 7.

👋 from 2023! Console Applications evolve and Microsoft is working on a System.CommandLine package that you might want to include in your CLI. Read more at .NET Console Application with injectable commands. I've changed this article to include the RootCommand.

Nuget me!

First, we need the right packages. And that can be quite a hassle - because the ASP.NET Core All package will include them for you when you are building an ASP.NET application - but there's no such thing for a Console app. They might be hard to ~~google~~ find:

dotnet add package Microsoft.Extensions.Configuration --version 7.0.0  
dotnet add package Microsoft.Extensions.Configuration.EnvironmentVariables --version 7.0.0  
dotnet add package Microsoft.Extensions.Configuration.FileExtensions --version 7.0.0  
dotnet add package Microsoft.Extensions.Configuration.Json --version 7.0.0  
dotnet add package Microsoft.Extensions.DependencyInjection --version 7.0.0  
dotnet add package Microsoft.Extensions.DependencyInjection.Abstractions --version 7.0.0  
dotnet add package Microsoft.Extensions.Logging --version 7.0.0  
dotnet add package Microsoft.Extensions.Logging.Console --version 7.0.0  
dotnet add package Microsoft.Extensions.Logging.Debug --version 7.0.0  
dotnet add package Microsoft.Extensions.Options --version 7.0.1  
dotnet add package System.CommandLine --version 2.0.0-beta4.22272.1

The project file

Edit your console project file and replace its contents with this.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Options" Version="7.0.1" />
    <PackageReference Include="System.CommandLine" Version="2.0.0-beta4.22272.1" />
  </ItemGroup>

</Project>

Note: this project uses implicit usings, a C# 10 feature, so we don't have to declare all usings.

Now you can upgrade the packages to the latest version in your Nuget manager. If you're not on .NET 5 or higher, you might need to add a <LangVersion>latest</LangVersion> to the first PropertyGroup. This will enable the latest version of C# including the async main (needed for .NET Core < 3).

AppSettings class + json

This blog has been around for a while now and people asked for an example of the actual settings. I use this class as AppSettings:

public class AppSettings
{
    public string Greeting { get; set; }
}

Next, you can add an appsettings.json to the root of your application:

{
  "App": {
    "Greeting": "Hello {0}! Welcome to the tool."
  }
}

Make sure the file has properties: Build Action must be None and Copy to Output Directory must be Copy always. It should like this in your project file:

  <ItemGroup>
    <None Update="appsettings.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
  </ItemGroup>

The JSON file works with sections, and we'll see later in the article how we bind the App section to an AppSettings instance. Want to know more about the setup with json files? Please read Setup multiple setting-files with a .NET Console Application.

Program vs. App

Normally the program class contains the main method and all the program-logic. If you want to do DI, it might be handy to separate the application from the setup that's required for your DI. So I propose to build an App class that does the actual running. For argument's sake, I've created such a class with a logger and some settings. The class is derived from RootCommand that will do the argument parsing.

using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using System.CommandLine;

public class App: RootCommand
{
    private readonly ILogger<App> _logger;
    private readonly AppSettings _appSettings;

    public App(IOptions<AppSettings> appSettings, ILogger<App> logger):
        base("Sends a nice greeting to the user.")
    {
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        _appSettings = appSettings?.Value ?? throw new ArgumentNullException(nameof(appSettings));

        var nameArgument = new Argument<string>("name", "The name of the person to greet.");
        AddArgument(nameArgument);

        this.SetHandler(Execute, nameArgument);
    }

    private async Task Execute(string name)
    {
        _logger.LogInformation("Starting...");
        var greeting = String.Format(_appSettings.Greeting, name);
        _logger.LogDebug($"Greeting: {greeting}");

        Console.WriteLine(greeting);

        _logger.LogInformation("Finished!");

        await Task.CompletedTask;
    }
}

This basically gives us an abstracted application class that is even testable. It also gives us the opportunity to focus on setting up the application in the program class (separating the actual application logic).

Setting up the program

In the program, we'll focus on DI. The App will also be added through DI. We're using the C# 10 top level statement, so we don't need a Program class:

using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using System.CommandLine;

static void ConfigureServices(IServiceCollection services)
{
    // configure logging
    services.AddLogging(builder =>
    {
        builder.AddConsole();
        builder.AddDebug();
    });

    // build config
    var configuration = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", optional: false)
        .AddEnvironmentVariables()
        .Build();

    services.Configure<AppSettings>(configuration.GetSection("App"));

    // add services:
    // services.AddTransient<IMyRespository, MyConcreteRepository>();

    // add app
    services.AddTransient<App>();
}

// create service collection
var services = new ServiceCollection();
ConfigureServices(services);

// create service provider
using var serviceProvider = services.BuildServiceProvider();

// entry to run app
await serviceProvider.GetService<App>().InvokeAsync(args);

Note: since C# 8, the using statement does not need braces anymore.

Invoke the program

When you do a dotnet run -- --help in your project dir, you'll see the following:

Description:
  Sends a nice greeting to the user.

Usage:
  ConsoleApp <name> [options]

Arguments:
  <name>  The name of the person to greet.

Options:
  --version       Show version information
  -?, -h, --help  Show help and usage information

When you do a dotnet run -- "peeps from KeesTalksTech" you'll see:

info: App[0]
      Starting...
Hello peeps from KeesTalksTech! Welcome to the tool.
info: App[0]
      Finished!

Final thoughts

We've injected a few things:

Logging − a lot of my applications run as stand-alone services in Docker containers, so logging is key. For convenience, I added a console logger. You can add any form of logging to the LoggerFactory.
Configuration − I use an appsettings.json file in my application. It will contain a section with the app variables. The DI will auto-map it to the setting object. To make things easier I added the AddEnvironmentVariables() so I'm able to override settings in production.
Services − this gives us nice abstractions.
RootCommand − this provides command line parsing. It also gives us a nice --help feature.

The program will resolve the App through DI and start your program. So that's it. Easy to use.

Need the code? Check GitHub.

Changelog

2023-03-11: Added .NET 7 and RootCommand support. Added the Invoke the program section.
2022-04-09: Added .NET 6 support with implicit usings and top level statements. Also updated the code on GitHub.
2021-07-23: Added .NET 5 support and a link to a GitHub repo.
2020-04-07: Added .NET 3.1 and support for args in the App.
2019-09-25: Added "Hello World" message and .NET Core 3.0 support.
2019-01-10: Added AppSettings class and JSON config examples.
2018-12-14: Added 2 packages missing in the list of Nuget packages.
2018-12-14: Logging is now done using the logging builder - the other version is obsolete.
2018-12-14: Added async main support by specifying the project example.

Don’t use Slack Incoming Webhooks; App Creation is Dead Simple!

Kees C. Bakker — Mon, 05 Sep 2022 17:31:53 +0000

At Wehkamp we ❤️ Slack! Seriously, in order to improve our efficiency, we've connected many of our applications, alerts and dashboards to Slack channels. But, as with all things, there is a right way and a wrong way of integrating a webhook at enterprise level.

As a Slack admin I see the old integration being requested every now and then, so let's see how easy it is to move away from Slack Incoming Webhooks. This guide was created with the help of Harry Kroes, one of our cloud engineers, as we needed to create a webhook to connect alerts from Nifi to Slack.

Why is it a problem?

Let me be clear: the Slack Incoming Webhooks integration will work, but is legacy and you should migrate as soon as possible.

Please note, this is a legacy custom integration - an outdated way for teams to integrate with Slack. These integrations lack newer features and they will be deprecated and possibly removed in the future. We do not recommend their use.

Source: Slack Incoming Webhooks

At Wehkamp, that's not even the main problem! Whenever the creator of a legacy webhook leaves the organization, the webhook is disabled. The code of the reactivated webhook changes; so vital integrations break.

Nooooo: some configurations of Incoming WebHooks have been deactivated.

Fast Guide To Create An App

It is super simple to create an application, especially now that we have Slack App Manifests. Follow this wizard and you'll have an app in 2 minutes:

Goto api.slack.com/apps?new_app=1
Choose From An App Manifest:

Slack wizard for app creation.
Select your workspace and click Next
Select workspace.

Add this manifest, change the contents to your liking and click the Next button.

display_information:
name: My Alert
description: Alerts from the My Monitoring System. Used for My App monitoring.
background_color: "#8B0000"
features:
bot_user:
display_name: My Alert
always_online: true
oauth_config:
scopes:
bot:
  - incoming-webhook
settings:
org_deploy_enabled: false
socket_mode_enabled: false
token_rotation_enabled: false

Review your app and hit the Create button. 🎉 Congratz, your app is now created 🎉
Next, we need to install the app. Navigate to Settings > Basic Information and click the Install your app section and hit the Install to Workspace button:
Install the app.
Select a default channel and hit the Allow button:
Select a default Slack channel.
🥳 Your app is now installed and you can copy the webhook! 🥳 Navigate to Features > Incoming Webhooks > Webhook URLs for Your Workspace and do your thing!
Configure one or more webhooks.

Add collaborators

It is super easy to add a new application. Now you can add collaborators to your app, so when you leave the organization, the application is still managed by your peers. Navigate to Settings > Collaborators to add more members.

Enjoy!

Task: Save Article to Markdown

Kees C. Bakker — Sat, 03 Sep 2022 03:03:17 +0000

WordPress rules, but I would like my content to be on other platforms as well. Some platforms like DEV, use Markdown, but I seem to struggle to import my articles. That's why I created a small snippet application to convert an article to markdown.

Final result

Just paste the URL of this blog into this small reppl.it program and watch how it converts the article into a big string of Markdown:

Packages

This solution uses Node.js. NPM has some great packages to work with:

node-fetch - to download the HTML. Depending on the version of Node.js, you might not need this to implement fetch. I use version 2, as I don't use ESM.
linkedom - to parse HTML into a workable DOM. I used to use jsdom, but I switch for performance reasons.
node-html-markdown - to parse HTML into markdown.

Install them like:

npm install node-fetch@2 linkedom node-html-markdown
npm install -D @types/node-fetch

Simple scraper

We're going to do the following:

Fetch the text of the URL. This is HTML, of course.
Parse it to DOM nodes.
Detect the article node.
Convert the article node to Markdown.

This results in the following lines of code:

async function scrape(url: string) {
  let f = await fetch(url)
  let txt = await f.text()

  const { document } = parseHTML(txt)

  // custom parsing:
  // parseCodeFields(document)
  // parseEmbeds(document)

  let article = (
    document.querySelector('article .entry-content') ||
    document.querySelector('article .crayons-article__main') ||
    document.querySelector('article') ||
    document.querySelector('body'))

  let html = article?.innerHTML || ""
  let content = NodeHtmlMarkdown.translate(html).trim()
  // let header = parseHeader(document)
  // content = header + content

  return content
}

Code Language Support

Now, my WordPress generates <pre class="lang-ts"><code></code></pre> blocks. Looks like node-html-markdown only takes <pre><code class="language-ts></code></pre>. Now, that's easily fixed by adding some extra processing before converting the document to markdown:

function parseCodeFields(document: Document) {
  document.querySelectorAll("pre code").forEach(code => {
    let lang = [...code.parentElement?.classList || []]
      .filter(x => x.startsWith("lang-"))
      .find(x => x)

    if(!lang) return

    lang = lang.replace("lang-", "language-")
    code.classList.add(lang)
  })
}

Embed rich content

Fortunately, dev.to supports liquid tags to embed rich content like repl.it and tweets. Let's parse our iframe elements into a liquid tag:

function parseEmbeds(document: Document) {
  document.querySelectorAll('iframe').forEach(iframe => {
    if (!iframe.src) return

    const url = new URL(iframe.src)
    const type = url.host
    const name = url.pathname

    const p = document.createElement("p")
    const n = document.createTextNode(`{% ${type} ${name} %}`)
    p.appendChild(n)

    iframe.parentNode?.insertBefore(p, iframe)
  })
}

This will not work for every embed, but it will get you started.

Header support

To be complete, we also need to add a YAML header with the title, tags and the canonical URL. It requires some parsing, but it'll make things easier:

function parseHeader(document: Document) {

  let header = '---\n'

  let title = (document.querySelector('h1')?.textContent || '').trim()
  if (title) {
    header += `title: ${title}\n`
  }

  let tags = [...document.querySelectorAll(".categories a, .tags a")]
    .map(a => (a.textContent || '').trim().toLowerCase())
    .filter(t => t)
  if (tags.length > 0) {
    tags.sort()
    let t = [... new Set(tags)].join(", ")
    header += `tags: [${t}]\n`
  }

  let canonical = document.querySelector('link[rel=canonical]')?.getAttribute("href")
  if (canonical) {
    header += `canonical_url: ${canonical}\n`
  }

  header += '---\n\n'

  return header;
}

Final thoughts

~~I still need to find a better way to detect the language of code snippets, so I don't have to add them by hand.~~ When I look at the result, I know one thing for sure: I'll keep using WordPress to write my blogs, as Markdown does not make it more readable!

Oh, and when you read this post on dev.to: it was created using this code (and yes, that's super meta 🤓).

Changelog

2022-08-31 Initial article.
2022-01-09 Fixed language support for WordPress code fields (see Code Language Support).
2022-03-09 Fixed embedding of repl.it through iframe parsing (see Embed rich content).
2022-03-09 Added title support.
2022-03-09 Added YAML header support (see Header support)

Dependency Injection based on request headers

Kees C. Bakker — Wed, 31 Aug 2022 04:31:22 +0000

Dependency Injection (DI) helps us to change the behavior of parts of our application on the fly. This is especially neat when you want to test your domain services against a mocked data-store. But what if you need to change the behavior in your API based on a request header?

Yesterday I had a discussion with my colleague Robert Kranenburg about this subject. He showed an example of a console application changing its behavior based on an argument. I took the idea and converted it into .NET Core 3.1 code for a Web API.

Let's change the message
(3) lifetime types
Accessing request headers
DI config
The result
Final thoughts
Further reading

Let's change the message

Our objective: change the response message based on the presence of a cookie named "hidden", using dependency injection.

First, we need to define the controller and the service interface:

public interface IMessageService
{
    string GetMessage();
}

[ApiController]
[Route("")]
public class MyController : ControllerBase
{
    private readonly IMessageService _messageService;

    public MyController(IMessageService messageService)
    {
        _messageService = messageService;
    }

    [HttpGet]
    public string Get()
    {
        return _messageService.GetMessage();
    }
}

.NET Core uses constructor injection and will provide an instance for each constructor parameter. It will inject an IMessageService instance into the controller.

Now, let's define 2 implementations of the IMessageService:

DefaultMessageService just returns a "Hello world!" message
HiddenMessageService returns a different message. To make things more interesting, it uses a dependency to construct that message.

Let's view the code:

public class DefaultMessageService : IMessageService
{
    public string GetMessage() => "Hello world!";
}

public class HiddenMessageService : IMessageService
{
    private readonly ISecretKey _key;

    public HiddenMessageService(ISecretKey key)
    {
            _key = key;
    }

    public string GetMessage() => 
        "The answer to life the universe and everything: " + 
        _key.GetKey();
}

public interface ISecretKey
{
    public string GetKey();
}

public class SecretKey : ISecretKey
{
    public string GetKey() => "42";
}

Before we set up the dependency injection, let's dive deeper into lifetimes.

3 lifetime types

A DI service is added with a lifetime:

AddTransient: Adding a transient service means that each time the service is requested, a new instance is created.
AddSingleton: A singleton is an instance that will last the entire lifetime of the application. In web terms, it means that after the initial request of the service, every subsequent request will use that same instance, across all requests.
AddScoped: A scoped service is instantiated per scope. For web this means the same instance per request, but you can actually create your own scopes.

Because we want to change the behavior based on request information, it makes sense to use a scoped service.

Accessing request headers

In the olden days, we could do anything we wanted with the static HttpContext.Current and be done with it. In .NET Core we use the IHttpContextAccessor and dependency injection to interact with the HttpContext. We can use the AddHttpContextAccessor to set this up.

DI config

Let's go to the Startup.cs and set up the dependency injection:

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddHttpContextAccessor();

    services.AddTransient<DefaultMessageService>();

    services.AddTransient<ISecretKey, SecretKey>();
    services.AddTransient<HiddenMessageService>();

    services.AddScoped<IMessageService>(provider =>
    {
        var context = provider.GetRequiredService<IHttpContextAccessor>();
        var isHidden = context.HttpContext?.Request.Cookies.ContainsKey("hidden") == true;

        if (isHidden)
        {
            return provider.GetRequiredService<HiddenMessageService>();
        }

        return provider.GetRequiredService<DefaultMessageService>();
    });
}

Why do we add DefaultMessageService and HiddenMessageService without interfaces to the service collection? Well, this helps us to use DI in the classes themselves. HiddenMessageService needs this, because this is how we get an ISecretKey instance. Adding these classes will make our lives way easier when we need to instantiate them. If we would "new them up" ourselves, we would need to resolve all the dependencies as well, leading to a lot of unnecessary code.

The IMessageService is resolved via a factory method. This method gets an IServiceProvider as input to locate any services that are required. We use this provider to locate the HTTP accessor and to check if a cookie named hidden is present. If so, we resolve the IMessageService as HiddenMessageService; otherwise as DefaultMessageService.

The result

When we add the code together we get this behavior:

The contents of the message is changed based on the presence of a cookie named hidden.

Final thoughts

It is easy to change de behavior of your program based on a request header or cookie. But things can get messy quite fast when you are doing dependency injection. Startup classes get large quickly when you need to inject many classes.

To make things easier, you can use an extension method like this:

public static class HiddenCookieExtensions
{
    public static IServiceCollection AddScopedByHiddenCookie<TService, THiddenImplementation, TDefaultImplementation>(this IServiceCollection services)
        where TService: class
        where THiddenImplementation: TService
        where TDefaultImplementation: TService
    {
        services.AddScoped<TService>(provider =>
        {
            var context = provider.GetRequiredService<IHttpContextAccessor>();
            var isHidden = context?.HttpContext?.Request.Cookies.ContainsKey("hidden") == true;

            if (isHidden)
            {
                return provider.GetRequiredService<THiddenImplementation>();
            }

            return provider.GetRequiredService<TDefaultImplementation>();
        });


        return services;
    }
}

Which can be used like this:

services.AddScopedByHiddenCookie<IMessageService, HiddenMessageService, DefaultMessageService>();

In this example, we used a cookie, but any header can be used to change the behavior of your application.

Named Arguments in a Bash Script

Kees C. Bakker — Wed, 31 Aug 2022 04:23:08 +0000

Let's face it, using positional arguments like $1 and $2 for your arguments is not very descriptive and not very flexible. There is a better way to supply arguments: with this simple trick you'll get --named arguments --in your-script, which is waaaay better 🤓.

Parsing script arguments

The following script parses all arguments and turns them into variables:

while [ $# -gt 0 ]; do
    if [[ $1 == "--"* ]]; then
        v="${1/--/}"
        declare "$v"="$2"
        shift
    fi
    shift
done

echo "\$this: '$this' \$a: '$a'"

When we execute the script with: ./test.sh --this is --a test, it returns:

$this: 'is' $a: 'test'

Note: this method is not very strict, it might skip parameters that are not adhering to the double dash format. If you want to support --single-parameters, check Support for --help?

Note 2: this method does not work with set -u 🤷

Argument validation

First, let's define a usage function to show how our script works:

programname=$0
function usage {
    echo ""
    echo "Deploys an ECR image to Atlas using GitOps and ArgoCD."
    echo ""
    echo "usage: $programname --service_name string --tag string --atlas string --env string "
    echo ""
    echo "  --service_name string   name of the service"
    echo "                          (example: blaze-search-term-redirect-service)"
    echo "  --tag string            tag of the image to deploy"
    echo "                          (example: 804-a325d6a)"
    echo "  --namespace string      namespace of the cluster"
    echo "                          (example: eos)"
    echo "  --env string            env to which to deploy the tag"
    echo "                          (example: dev)"
    echo ""
}

Next, let's define a die function:

function die {
    printf "Script failed: %s\n\n" "$1"
    exit 1
}

Now, let's check if the parameters are supplied. If one of the parameters is missing, display usage and terminate the script. We use the -z conditional expression to check if the variable is empty:

if [[ -z $service_name ]]; then
    usage
    die "Missing parameter --service_name"
elif [[ -z $tag ]]; then
    usage
    die "Missing parameter --tag"
elif [[ -z $namespace ]]; then
    usage
    die "Missing parameter --namespace"
elif [[ -z $env ]]; then
    usage
    die "Missing parameter --env"
fi

Support for --help?

The easiest way of adding --help to your script, is by adding it to the loop that turns the arguments into variables:

while [ $# -gt 0 ]; do
    if [[ $1 == "--help" ]]; then
        usage
        exit 0
    elif [[ $1 == "--"* ]]; then
        v="${1/--/}"
        declare "$v"="$2"
        shift
    fi
    shift
done

Any singular parameter can be added to this loop.

How about defaults?

Here you have 2 strategies:

Declare the variables at the top of your script and initialize them with defaults.
Use parameter expansion: my_parameter=${my_parameter:-default value}

Parameter expansion

If you use parameter expansion, you need to know that the empty string "" will also be replaced. Observe the following script:

my_parameter="${my_parameter:-default value}"
printf "%s\n\n" "$my_parameter"

It will give the following output:

$ ./test.sh
default value

$ ./test.sh --my_parameter "my value"
my value

$ ./test.sh --my_parameter ""
default value

So, settings the defaults at the top of the script has my preference.

Write better scripts: use ShellCheck

The article Please stop writing shell scripts makes some excellent points on why writing a bash script is hard for programmers: bash behavior is most likely not like the programming language you're familiar with. The ShellCheck extension for Visual Studio Code will help you catch some of the pitfalls. With every "offence", it links to a wiki article for education:

Show the explanation of SC2086.

Conclusion

Adding named arguments to a bash script is pretty easy. There are some caveats, as you might "override" arguments in your script from the outside. If you declare the arguments at the beginning of your script and the reset of the parameters after the while section, you mitigate that risk.

Changelog

2022-04-04: made the parameter validation script a bit shorter by using elif.

Jupyter Notebooks + VSCode Dev Container with Puppeteer support

Kees C. Bakker — Wed, 31 Aug 2022 04:11:51 +0000

I have a love and hate relationship with Python; it is super easy to develop in Python (abundance of examples and packages, cross platform), but the setup of your development environment (Python versions, package versions) is too cumbersome. This increases when you need to share that setup with another developer on another platform. I work on Windows, my colleagues on Mac and Linux.

That's why I now use Visual Studio Code and "dev containers":

The Visual Studio Code Remote - Containers extension lets you use a Docker container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of Visual Studio Code's full feature set.

Source: Developing inside a Container

I have some reasons for switching to this setup:

Visual Studio Code is already my go tool for many languages (Node.js, bash, Terraform). Many of my colleagues also use it for various tasks.
Running the whole dev setup in Docker helps keeping the host system clean. It would not be the first time that someone "bricked" their Python setup due to a new version requirement.
Docker gives you a nice, replicable, cross platform setup: on each dev machine it should work the same.

Now, the main question is: will this work for Jupyter Notebooks? Visual Studio Code already provides an excellent UI, but will it run Jupyter in a development container? Let's find out! My use case is a scraper, so I need support for Puppeteer as well.

Default Python Dev Container Setup
Install IPython & Pandas in the Dev Container
1. Do I still need to install packages in my notebook?
How about Puppeteer?
Don't commit Jupyter Notebook output
1. Auto save?
Improving performance
VSCode extensions
Final thoughts
Changelog

Default Python Dev Container Setup

Let's start with a Python 3 development container:

Open the command palette (F1 on Windows)
Search for: Remote-Containers: Add Development Container Configuration files... This will open up a small wizard.
Select Python 3 as the language.
Select 3 as the version (it will add the latest anyway).
We don't need Node.js, so let's select None (can be enabled later, but this option will make your container build faster).

The setup generates the configuration files in the .devcontainer folder of your project.

Install IPython & Pandas in the Dev Container

Now, open the Dockerfile from the .devcontainer folder and uncomment the following lines:

# [Optional] If your pip requirements rarely change, uncomment this section to add them to the image.
COPY requirements.txt /tmp/pip-tmp/
RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
   && rm -rf /tmp/pip-tmp

This makes it possible to start using a requirements.txt file, to install packages with PIP on a container level.

Next, add the requirements.txt to the root of your project and enter the following lines:

ipython
ipykernel
pandas

This makes sure the container installs the packages you need for your notebook.

Do I still need to install packages in my notebook?

Yo don't have too, but it might make things easier if you do and it caters to people that don't use your setup. With this simple line, you will install all the packages specified in your requirements.txt file:

%pip install --quiet --exists-action i --disable-pip-version-check -r ../requirements.txt --user

It will complete fast, as all things are already installed in your dev container. This also makes adding new packages easier, as you don't have to restart your dev container.

How about Puppeteer?

We're going to use the Python version of Puppeteer called Pyppeteer. Running Puppeteer from a container is not straightforward. Let's airlift the code from this article into our setup.

Add the following lines to the Dockerfile:

# Install Google Chrome Stable and fonts
# Note: this installs the necessary libs to make the browser work with Puppeteer.
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install gnupg wget -y && \
  wget --quiet --output-document=- https://dl-ssl.google.com/linux/linux_signing_key.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/google-archive.gpg && \
  sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' && \
  apt-get update && \
  apt-get install google-chrome-stable -y --no-install-recommends && \
  rm -rf /var/lib/apt/lists/*

This will download and install Chrome to the container. Now we just need to add the Puppeteer packages to our requirements.txt file:

ipython
ipykernel
pandas
pyppeteer

Add the following line to your notebook:

%pip install --quiet --exists-action i --disable-pip-version-check pyppeteer

Now, when we launch the browser in the notebook, we only have to test if we're in our development container:

browser_options = {
    'headless': True,
    'args': ["--no-sandbox"]
}

if os.getenv('PUPPETEER_SKIP_CHROMIUM_DOWNLOAD', '') != '':
    browser_options["executablePath"] = '/usr/bin/google-chrome'

browser = await launch(browser_options)

This makes sure that the pre-installed version of Chrome is used in your container.

Don't commit Jupyter Notebook output

We want to improve what we commit to Git: let's not commit the output of the notebook, by implementing the code of this article. First, let's add the nbconvert package to your requirements.txt:

ipython
ipykernel
nbconvert
pandas
pyppeteer
pyppeteer_stealth

Let's configure Git so it can use it; add a .gitconfig file to the root of your project:

[filter "strip-notebook-output"]
    clean = "jupyter nbconvert --ClearOutputPreprocessor.enabled=True --to=notebook --stdin --stdout --log-level=ERROR"

Now add a file called .gitattributes:

*.ipynb filter=strip-notebook-output

The last file you'll need to add is repo_init.sh:

#!/usr/bin/env bash

git config --local include.path ../.gitconfig
git add --renormalize .

Now, start your project in the dev container and hook things up with bash repo_init.sh. Every time a user clones your repository for the first time, this script needs to be executed.

Now Git is able to understand what has changed, and will no longer show a change when you just ran the script. Note: VSCode will still think the notebook has changed, but Git itself will not commit the change.

Auto save?

When you run the notebook, the file is changed. Notebooks store both the code and the output. I find it very annoying that VSCode shows an unsaved file in my IDE (and tries to restore it if I close the editor). To mitigate this, you can enable auto save to your dev container settings:

Open up .devcontainer/devcontainer.json
Navigate to customizations > vscode > settings
Add: "files.autoSave": "afterDelay",
Add "files.autoSaveDelay": 1000

Pet peeve fixed.

Improving performance

Our setup currently is naïve, as it does not fully leverage Docker caching. Let's look at our final Dockerfile:

# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.245.0/containers/python-3/.devcontainer/base.Dockerfile

# [Choice] Python version (use -bullseye variants on local arm64/Apple Silicon): 3, 3.10, 3.9, 3.8, 3.7, 3.6, 3-bullseye, 3.10-bullseye, 3.9-bullseye, 3.8-bullseye, 3.7-bullseye, 3.6-bullseye, 3-buster, 3.10-buster, 3.9-buster, 3.8-buster, 3.7-buster, 3.6-buster
ARG VARIANT="3.10-bullseye"
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}

# [Choice] Node.js version: none, lts/*, 16, 14, 12, 10
ARG NODE_VERSION="none"
RUN if [ "${NODE_VERSION}" != "none" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi

# Install Google Chrome Stable and fonts
# Note: this installs the necessary libs to make the browser work with Puppeteer.
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
RUN apt-get update && apt-get install gnupg wget -y && \
  wget --quiet --output-document=- https://dl-ssl.google.com/linux/linux_signing_key.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/google-archive.gpg && \
  sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' && \
  apt-get update && \
  apt-get install google-chrome-stable -y --no-install-recommends && \
  rm -rf /var/lib/apt/lists/*

# [Optional] If your pip requirements rarely change, uncomment this section to add them to the image.
COPY requirements.txt /tmp/pip-tmp/
RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
   && rm -rf /tmp/pip-tmp

By swapping the installation of the packages, you don't need to reinstall Chrome every time your Python packages changes.

VSCode extensions

The nice thing about this setup is the ability to share your Visual Studio Code Extensions. They are stored in the devcontainer.json, just like the settings. I'm using these extensions:

Jupyter
Jupyter Keymap
Jupyter Notebook Renderers
vscode-icons - to provide better icons in your project explorer
TODO Highlight - to provide a clearer highlighting of stuff you still need to do
ShellCheck - a linter for writing bash scripts - somehow I always end up with writing scripts in my projects.

I replaced the customizations > vscode > extensions node with this:

      "extensions": [
        "ms-python.python",
        "ms-python.vscode-pylance",
        "ms-toolsai.jupyter",
        "ms-toolsai.jupyter-keymap",
        "ms-toolsai.jupyter-renderers",
        "vscode-icons-team.vscode-icons",
        "wayou.vscode-todo-highlight",
        "timonwong.shellcheck"
      ]

When somebody opens the projects it is notified about the extensions.

Final thoughts

I like the fact that this setup works on any machine. Consider locking your versions in, as both Chrome and your packages are not versioned (so you might download breaking changes).

I don't like the fact that Visual Studio Code thinks a notebook has changed, while Git knows it isn't. According to issue #9514, this is something that should be fixed in the core of Visual Studio Code. So, I'm not really sure why issue #83232 (or #24883) is closed.

Changelog

2022-08-09 Changed the %pip install line to use the requirements.txt file.

DEV Community: Kees C. Bakker

Async Replace All for JavaScript

The idea

Caveats of reusing a global regex

Caveats of split with capture groups

Caveats of matchAll

Implications

The code

Conclusion

.NET Console Application with injectable commands

Goals

NuGet Packages

Project structure

Fake weather service

Commands

Current Temperature Command

Forecast Command

Dependency injection

Final thoughts

Rethinking our ASP.NET Docker CI

Goals

Project structure

Step 1: Configuration with global arguments

Step 2: Build stage

2.1 Restore in cacheable layers

2.2 The build script

2.3 Optional testing and publishing

Step 3: runtime stage

3.1 Which runtime base image should we use?

3.2 Go self contained or not?

3.3 The final runtime image

All together

Reading list

Changelog

Chrome Extension Development: Fetch non CORS URLs in v3

The Manifest

The Background Script

The Content Script

Resolve URLs

Dependency injection (with IOptions) in Console Apps in .NET

Nuget me!

The project file

AppSettings class + json

Program vs. App

Setting up the program

Invoke the program

Final thoughts

Changelog

Don’t use Slack Incoming Webhooks; App Creation is Dead Simple!

Why is it a problem?

Fast Guide To Create An App

Add collaborators

Task: Save Article to Markdown

Final result

Packages

Simple scraper

Code Language Support

Embed rich content

Header support

Final thoughts

Changelog

Dependency Injection based on request headers

Let's change the message

3 lifetime types

Accessing request headers

DI config

The result

Final thoughts

Further reading

Named Arguments in a Bash Script

Parsing script arguments

Argument validation

Support for --help?

How about defaults?

Parameter expansion

Write better scripts: use ShellCheck

Conclusion

Changelog

Jupyter Notebooks + VSCode Dev Container with Puppeteer support

Default Python Dev Container Setup