DEV Community: Nils Diekmann

Mediator and CQRS with MediatR

Nils Diekmann — Wed, 23 Oct 2024 22:11:43 +0000

ultimate beautiful code

Challenge

Today I am talking about a coding dojo that my younger and older self are carrying out against each other. A developer is active in the coding dojo all the time. The active developer is free to decide what to develop. But he must explain his thoughts to the other developer.

The company they work for has developed a donation system. People can donate some money and see the total amount of all donations. The challenge for the Coding Dojo is now to develop this system further. The sample application is available on GitHub.

KinNeko-De / sample-codingdojo-donation

Read my articles to find out how my [younger](https://medium.com/@kinneko-de/344fe6e8e4f6) and [older](https://medium.com/@kinneko-de/0de8a4351da2) selves solved this coding dojo.

Motivation

This is a sample exercise for a coding dojo in C#.

The application consists of a user interface part and a server part with a database. Everything is in one application to avoid unnecessary complexity. API calls are reduced to method calls.

Read my articles to find out how my younger and older selves solved this coding dojo.

Challenge

Develop the application further. You are completely free in your decisions

Architecture in a real world example

In a real world the UI would run on the client. The UI would access the server over an API.

Sources

The headliner image was generated using Gemini's image generation capabilities.

Gemini is a large language model developed by Google AI. It can generate realistic and diverse images based on text prompts. Learn more about Gemini.

View on GitHub

My younger self will take on the role of the active developer in this article. Read my other article to find out what my older self would do instead.

Coding Dojo

Younger Self [excited]: Oh my God, today we had a training session with an external consultant. He talked about what we should use in our project. He told us to use patterns to improve the code. There were a lot of patterns, I can’t remember them all, but there was one very special pattern called Mediator.

Older Self [relaxed]: Can you explain me what the mediator pattern does and when to use it?

Youger Self [irritated]: There’s a mediator between different parts of the code that greatly improves readability.

Older Self: If you don’t know what the mediator pattern is, how are you going to implement it?

Younger Self: Of course, I am not going to implement it myself. There's a framework called MediatR. It has almost the same name, so I cannot be wrong if I use it. This framework will do everything for me.

Older Self [sceptical]: Are you absolutely sure?

Younger Self [self-confident]: Yes, of course. There is a comment from a guy who uses it everywhere because he totally likes it and is totally satisfied by the framework. So I am going to use it everywhere now.

Older Self: Okay, so where do we start?

Younger Self: The service adds a number to a value in a database. I have heard that this is a command. Then the service queries the result. If I separate the command from the query, I can also implement CQRS.

Older Self: Sounds ambitious, but let us first introduce the mediator pattern.

Younger Self: I first need to add a nuget package for MediatR and register it in my Program.cs. That is all I have to do to get all my code injected by dependency Injection.

builder.Services.AddMediatR(
  cfg => cfg.RegisterServicesFromAssemblyContaining<Program>());

I create a command that does the update in the database. It needs the same properties as I had before as parameters. I put the command in its own class.

public class AddDonationCommand : IRequest
{
  public required int Donation { get; init; }
}

Then I create a handler for the command. It will get the database injected as the service before. Each handler has a Handle method. There I do my database query.

// ReSharper disable once UnusedMember.Global
// MediatR will automatically find and use this class somehow, do not remove it!
public class AddDonationCommandHandler : IRequestHandler<AddDonationCommand>
{
  private Database Database { get; }

  public AddDonationCommandHandler(Database database)
  {
    Database = database;
  }

  public async Task Handle(AddDonationCommand request, CancellationToken cancellationToken)
  {
    await Database.AddDonation(request.Donation);
  }
}

Then I do the same with the query. In the service, I just call the mediator and send the command and the query to it.

public async Task<int> UpdateDonation(int donation)
{
  var command = new AddDonationCommand() { Donation = donation };
  await Sender.Send(command);

  var query = new GetTotalDonationsQuery();
  return await Sender.Send(query);
}

KinNeko-De / sample-codingdojo-donation-mediatr

Read my articles to find out how my [younger](https://medium.com/@kinneko-de/344fe6e8e4f6) and [older](https://medium.com/@kinneko-de/0de8a4351da2) selves solved this coding dojo.

Motivation

This is a sample exercise for a coding dojo in C#. It was copied from the donation template

This is the coding dojo solution of my younger selve.

Read my article to find out how my older selve solved this coding dojo.

Challenge

Develop the application further. You are completely free in your decisions

View on GitHub

Older Self: The code now looks much more complex to me.

Younger Self: It’s more simpler because I had don’t have to worry about how the objects are created. The framework does it for me. Now I can just focus on my command and not have to look at the rest.

Older Self: Where do you see CQRS in this specific example?

Younger Self: I separated my query from my command by putting a mediator between them. So its now segregated right?

Older Self: There is an R in CQRS. It stands for responsibility. The responsibility in your code is not segregated. It is still one method, even with all the layers of abstraction in between.

Younger Self [thoughtful]: Hmmmmm [pause] I have the perfect solution for that. We do two teams, one is responsible for the query and one is responsible for the command. Then we do a modular monolith. We can also extract the query to an separate microservice that I call synchronously from the microservice that does the addition.

Older Self [surrenders]: I am at a loss for words. Product owner, we are done.

Product owner: What is the benefit to our company of doing your approach?

Younger Self: Hmm Hmm Hmm

Conclusion

When I was still young, I tried to use as many patterns as possible. Later I realized that I was just adding a lot more complexity to my code. Nowadays, I only use patterns to solve problems that already exist.

I also realized that beautiful code and my personal satisfaction do not pay the bills. So I focus more on business value and the bigger picture than on just code.

If you totally disagree with experience and totally like the approach of my younger self, then stop here. Otherwise, I am happy to share my older self’s aproach with you.

And don’t forget to send me a reaction at this article.

BusinessValue with CQS

Nils Diekmann — Wed, 23 Oct 2024 22:10:57 +0000

simple is harder

Challenge

KinNeko-De / sample-codingdojo-donation

Read my articles to find out how my [younger](https://medium.com/@kinneko-de/344fe6e8e4f6) and [older](https://medium.com/@kinneko-de/0de8a4351da2) selves solved this coding dojo.

Motivation

This is a sample exercise for a coding dojo in C#.

The application consists of a user interface part and a server part with a database. Everything is in one application to avoid unnecessary complexity. API calls are reduced to method calls.

Read my articles to find out how my younger and older selves solved this coding dojo.

Challenge

Develop the application further. You are completely free in your decisions

Architecture in a real world example

In a real world the UI would run on the client. The UI would access the server over an API.

Sources

The headliner image was generated using Gemini's image generation capabilities.

Gemini is a large language model developed by Google AI. It can generate realistic and diverse images based on text prompts. Learn more about Gemini.

View on GitHub

My older self will take on the role of the active developer in this article. Read my other article to find out what my younger self would do instead.

Coding Dojo

Older self: Let’s analyze what the service does first.
It adds the donated money to the project. Then it selects the total amount of donated money. So it does two things.

Order self: Not in general. Maybe you address this because of the the single responsibility principle. In my opinion it adress more code design. In this service two methods are called after each other. But the service and the code must serve the business first, not the other way around.

But I don’t think this service meets the customer’s needs. The donor has to donate money before they can see what other people have donated before them. Maybe it would be better to first show them the money that has already been donated, even without them having to donate money. This might motivate them to donate more money. A second use case for this new query option would be for the project owner to see how much money has already been donated.

Younger self: Which framework do you want to use to solve this problem?

Older self: I don’t need a framework to do this. It is just a matter of rearranging the current code. This is a simple task.
I remove the query part of the service method and add a second method that just selects the total amount of money. I now have two methods in the service.

public async Task<int> GetTotalDonations()
{
  return await Database.GetTotalDonations();
}

public async Task UpdateDonation(int donation)
{
  await Database.AddDonation(donation);
}

I also need to modify the user interface. The user interface can now initially fetch the amount of money donated.

if (firstRender)
{
  await GetTotalDonations();
}

private async Task GetTotalDonations()
{
  Total = await Server.GetTotalDonations();
}

After the user has donated some money, we also need to update the amount. The user needs immediate feedback to feel that their money has been received.

private async Task Donate()
{
  await Server.UpdateDonation(MyDonation);
  [...]
  await GetTotalDonations();
}

KinNeko-De / sample-codingdojo-donation-businessvalue

Read my articles to find out how my [younger](https://medium.com/@kinneko-de/344fe6e8e4f6) and [older](https://medium.com/@kinneko-de/0de8a4351da2) selves solved this coding dojo.

Motivation

This is a sample exercise for a coding dojo in C#. It was copied from the donation template

This is the coding dojo solution of my older selve.

Read my article to find out how my younger selve solved this coding dojo.

Challenge

Develop the application further. You are completely free in your decisions

View on GitHub

Younger self: This will introduce more overhead when we deploy this as a real service. Each call introduces network overhead.

Older self: Yes, you’re right. But separating the functionality into two methods makes our system more flexible. It is more likely to fit business use cases just by combining the calls. So it improves maintainability. If it were really performance relevant, we could just copy the code and both methods could ask for the total amount of the donation. But I don’t think performance is relevant for us in this use case.

Younger self: I see that we separated the query and the command that updates the donations. This is CQRS, right?

Older self: Not really. CQRS is for more complex services where we have multiple services with different databases. And also in these cases you add it only in a later stage when you really want to optimize for more complex queries. But for small projects in early stages it is overkill.
There’s a pattern called CQS, but I did this to add business value to the application and not just to use a pattern.

Younger self: Interesting. Product owner, we are done.

Product owner: I listened to your discussion and it was really interesting. The approach may be interesting. But I want to discuss it with our customers first. We should keep the code in a feature branch for now. Maybe we can also add some more motivational words based on the amount of money already donated.

Conclusion

I realize more and more every day that implementation is not about personal satisfaction. It is about delivering business value to the company, and that is what the company pays me for.

After years of developing and maintaining existing code, I realized that maintenance is the most time-consuming and expensive part of software development. That’s why I prefer simple solutions these days. This includes refining concepts to keep the code as simple as possible. Because simple is harder.

Don’t forget to leave a reaction at this article and then hopefully read about my younger self’s approach.

Angular, Typescript, and CD to GitHub Pages (2024)

Nils Diekmann — Thu, 19 Sep 2024 15:00:00 +0000

How to set up a typescript-based Angular application and deploy it directly to GitHub Pages using a CD GitHub action. A practical guide for 2024.

Motivation

Setting up a new Angular repository in GitHub is not rocket science. The introduction of a CD pipeline is easy too. In this tutorial I will give you a quick and easy step-by-step guide.

Create Angular App

Install the following prerequisites: Git and Node.js.

Angular CLI

I will use Angular CLI to create the initial project setup. I need to install the CLI globally using Node. The CLI version I am using is 18.2.2.

npm install -g @angular/cli

I want to create a template with typescript. I googled that and this leads my to an old and much older link. The older link takes me to a dead page. The trick is, that Angular is generated with typescript by default! So I open a terminal in the root folder where all my git repositories are. The CLI will create a subfolder with the name of my application.

ng new sample-angular-typescript-githubpages

I have some options when I create the application. I choose CSS and no server-side rendering. The template includes a .gitignore and I can push it directly to GitHub.

I can run npm install without any vulnerabilities. But when I run npm ci I get warnings about three deprecated packages. The dependencies are caused by the karma test runner. So I override the version of glob and rimraf in my package.json below devDependencies.

"overrides": {
    "glob": "^9.0.0",
    "rimraf": "^4.0.0"
  }

But I rejoice too soon. Now the tests (ng test) are no longer running. So I roll back the changes and hope for the best. The issue is only one month old.

Run tests locally

When I run the test command a Chrome instance pops up. I have to select a default search engine every time I start the tests. As this is annoying, follow the advice of clever people. First, follow the advice of the second comment, generate a karma.conf.js and insert the snippet from the first comment.

Build and continuous deployment

I create a .github/workflows folder and a builddeploy.yaml with the following content in it. This time I select the right NodeJs version, as 22 is not LTS yet.

name: Deploy Angular to GitHub Pages

on:
  push:
    branches:
      - main # name of the branch you are pushing to

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      - name: Install Dependencies
        run: npm ci
      - name: Build
        run: npm run build -- --configuration production --base-href /${{ github.event.repository.name }}/
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: 'dist/${{ github.event.repository.name }}/browser/.'
  deploy:
    needs: build
    runs-on: ubuntu-latest
    permissions:
      pages: write
      id-token: write
    environment:
      name: github-pages
      url: 'https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}/'
    steps:
      - name: Setup Pages
        uses: actions/configure-pages@v5
      - name: Deploy
        uses: actions/deploy-pages@v4

This workflow uses the GitHub action actions/deploy-pages to deploy the artifact from actions/upload-pages-artifact to GitHub pages. I can also view and download the produced artifact in the workflow run overview.

There is one improvement compared to the react tutorial. I pass two parameters to the npm run build. These parameters are then passed to ng build. The first parameter declares that I want the production configuration and the second is the base-href for my application. It must be the name of the GitHub repository framed by slashes. You can also use a full URL instead.

When I push the file, a workflow run starts automatically . It fails because the GitHub's default configuration is currently to deploy GitHub pages from a specific branch. So I need to change the configuration of my GitHub repository.

I need to change the selection of the drop down box to 'GitHub Actions'. As you can see 'Deploy from a branch' is declared as 'classic', which is usually a nicer word for 'legacy'.

Now I need to re-run the failed jobs of my GitHub Action workflow. GitHub will only execute the 'deploy' job. The already built artifact 'github-pages' will be reused.

After the job re-run succeed, I finally see my GitHub Page. You can see the full code in my GitHub repository.

Conclusion

The setup of an Angular app is more straightforward than in my React tutorial. Angular, GitHub and the community have done a great job making life easy for developer's . Thank you for that ❤.

Versions used

I will try to update this guide to the latest version to keep up with breaking changes in the tools. If you find something that does not work for you, please leave a comment. For a better overview, I list all versions used while writing this article.

Node.js 20.17.0
NPM: 10.8.2
Angular CLI: 18.2.2

Samples

I created a sample application on GitHub:

KinNeko-De / sample-angular-typescript-githubpages

Sample application how to setup a Angular app with typescript and deploy it to GitHub Pages.

Motivation

Sample application of how to set up a Angular app with typescript and deploy it to GitHub Pages.

See my article how to do this yourself.

View on GitHub

Here you can see the deployed version:

kinneko-de.github.io

Gamechanger Protobuf

Nils Diekmann — Thu, 29 Aug 2024 16:33:30 +0000

Protobuf is Google's answer to the need for an effective, efficient, and flexible serialization format as an alternative to JSON or XML. Its benefits pave the way for groundbreaking solutions to the problems faced by modern enterprises in creating a digital landscape in the cloud.

🌎 Lies diesen Artikel in deutsch

Cloud-native means rethinking

Constructing a half-timbered house using traditional methods and tools certainly has its charm and is an achievement not to be sneezed at. However, modern zero-emission houses with a smart home connection cannot be built this way. And let's be honest, the number of customers desperately looking for a half-timbered house is very small.

Cloud-based applications present new and unfamiliar challenges to today's software architects. They need modern processes that are specifically designed to meet the new challenges of the cloud, so that applications can run reliably while conserving resources. Whether it is CPU, memory or disk space, anything that is used less saves money in running the cloud and is also a step towardss green computing.

Numerous challenges

I cannot cover all the challenges in the necessary depth in this article. The text would be so long and complicated that no one would finish reading it. Instead, I will focus on serialization. Serialization is the conversion of structured data into a format suitable for transmission or storage.

In modern distributed systems, the choice of a fast and reliable serialisation format is critical. Serialization itself is not complex, but it is widely used. Choosing the optimal format therefore has a major impact on the overall design of the cloud application.

A well-chosen format ensures that data is transferred over the network quickly and with minimal overhead. In addition, a reliable format is essential to avoid errors during data reconstruction and thus ensuring data integrity.

Automation as a success factor

JSON is currently the serialization format of choice for many. It has overtaken XML over the years. However, the advantage of JSON, as a format that is easy for humans to create and read, does not really come into play when it is used for communication between microservices. Although data and processes are triggered by human interaction, they only run downstream between systems and react to events rather than users.

The argument that developers need to read the serialization format is also no longer valid. Cloud applications run in containers, which are themselves monitored and controlled by systems such as Kubernetes. Manual intervention is not the method of choice for analyzing and resolving problems.

In addition, the application is updated at short intervals via a continuous delivery pipeline to meet market demands for rapid and constant adaptation. There is no alternative to automation in this context, as it ensures secure and, in times of skills shortages, cost-effective operation.

Gamechanger Protobuf

In this article, I will briefly discuss the terms and functionality of Protobuf. I will then explain its novel concepts and features and how they address the shortcomings of JSON. Finally, I will explain why Protobuf is a vast improvement over JSON and the next step in the evolution of serialization formats.

Safe and secure thanks to precise definition and strong typing

In order to serialize data, it must first be described in a structured way. With Protobuf, the data structures are defined in files with a .proto extension. In this proto definition, an object is described by a message. A message may contain any number of fields. A field has an identifier, a type and a number that is unique for this message. The type of a field can in turn be a message, creating a tree structure similar to JSON and XML.

syntax = "proto3";

package example;

message Book {
 string title = 1;
 repeated Page pages = 2;
}

message Page {
 string content = 1;
}

The code example above describes a simplified data structure for a book. The book has a title of type string, which can contain any character string. The book also has an arbitrary number of ("repeated") pages, each of which is stored as a message of type "Page". The pages have the text they contain stored in the "content" field. The explicit specification of types in the proto definition helps to avoid errors and ensures consistency across system boundaries. Protobuf supports common programming language data types. Google also provides a collection of extended types, called WellKnownTyps. You can also create your own data types using self-defined messages.

High productivity with code generation

With other serialization formats, a lot of valuable development time is spent creating the data structures again as objects in the programming language and continually updating them as changes are made. The work is time consuming, but not very challenging, which means that errors creep in quickly. Generic mappers promise to reduce the effort, but they are difficult to configure in all cases where you deviate from the normal use case.

Protobuf relies on automatic and continuous code generation and supports multiple programming languages. The data structure code is generated directly from the proto definition, eliminating much of the effort of other approaches. It is also guarantees that the sender and receiver always use the same implementation as long as they have the same version of the proto schema. The generated source code can also be flexibly extended with additional methods to make the use of custom data types more convenient.

Strictly defined but extendable contract

The proto definition is an interface contract between the sender and the receiver. Protobuf uses only simple structural elements. For example, the keyword "repeated" refers to a list of a messages without specifying its exact number. On the one hand, this means that the sender cannot read possible restrictions on the receiver side from the contract. On the other hand, this can also be seen as an opportunity to make the contract more flexible and therefore easier to extend. The ability to easily change the contract while maintaining the integrity of the application is invaluable in a constantly evolving system landscape.

Simple schema evolution

Protobuf was designed with compatibility in mind. A change is backward compatible if the new version of a receiver can process messages from from an old version of a sender. Conversely, a change is forward compatible if the new version of a sender still works with the old version of the receiver.

By following a few simple rules, Protobuf provides both downward and upward compatibility. For example, the type and number of a field must never be changed. A message and a field cannot be deleted. The proto3 syntax also does not define any mandatory fields, as extensions to a contract must always be optional in order to maintain compatibility.

Compatibility for zero downtime deployment

Backward and forward compatibility is particularly important if it cannot be guaranteed that the sender and receiver and all messages can be updated to a new version of an interface description at the same time. This is an important feature for cloud applications.

Full compatibility makes it possible to roll out a sender and a receiver independently of each other, eliminating the need for time-consuming coordination of the sequence and making continuous delivery possible in the first place. Using a modern software solution for managing containerised applications, such as Kubernetes, also enables automatic rolling updates. This means that a new version of the application can be rolled out without any downtime. Users are unaware of this, which allows for uninterrupted service and an optimal customer experience.

Outstanding efficiency as binary and minimalist

Protobuf serializes messages in binary format. Only the number and value of the field in a message are stored. The proto definition is therefore needed to be able to read the message later. Serialization deliberately avoids unnecessary ballast that is not necessary for the actual benefit. Because of the small amount of data, the creation and reading of messages is very fast.

However, the minimal message storage and high compatibility can lead to unexpected behaviour in the event of errors during development. Under certain circumstances, messages can be deserialized by a foreign proto definition without an error occurring. It is therefore essential to ensure the use of protobuf with automated tests. These can also be used to ensure compatibility with older versions.

Manual error analysis of messages on a case-by-case basis is easily possible by converting the binary information to an object. Messages can also be easily converted to a JSON format. The code required to do this is fully generated according to the proto definition. This means that legacy systems that can only work with JSON can be connected to modern applications. The benefits of Protobuf can be at least partially exploited.

Conclusion

No serialization format, neither Protobuf nor JSON, can be used as a "golden hammer". Protobuf is certainly not an alternative to JSON for all use cases, nor was it designed to be. The strengths of Protobuf make it a flexible and at the same time particularly efficient serialization format. The focus of the application area is primarily on data exchange between software systems. It can therefore be easily used wherever automated processes are created as part of digitization.

The initial adoption of Protobuf may require some training and learning, but this is quickly recouped through the consistent use of the serialisation format across as many use cases as possible. The high degree of automation and code generation allows the development team to focus on what they do best - solving problems.

Now there are no limits to your creativity in designing solutions for your domain based on Protobuf. I look forward to your ideas and suggestions. After all, aren't we all striving to continuously improve our software through first-class design and to make it a success?

In future articles, I will give some concrete examples of how I have used Protobuf in the past. It's best to follow me so that you don't miss any of my future articles on Protobuf. ❤

Gamechanger Protobuf

Nils Diekmann — Thu, 29 Aug 2024 16:32:26 +0000

Protobuf ist Googles Antwort auf die Anforderungen an ein effektives, effizientes und zugleich flexibles Serialisierungsformat als Alternative zu JSON oder XML. Seine Vorteile ebnen den Weg für bahnbrechende Lösungen für die Probleme, mit denen moderne Unternehmen bei der Schaffung einer digitalen Systemlandschaft in der Cloud konfrontiert werden.

🌎 Read this article in english

Cloud native bedeutet neu zu denken

Die Konstruktion eines Fachwerkhauses mit traditionellen Methoden und Werkzeugen hat gewiss seinen Scharm und ist eine nicht zu verachtende Leistung. Moderne Null-Emissions-Häuser mit Smart-Home-Anbindung lassen sich damit allerdings nicht errichten. Und mal ehrlich, die Anzahl der Kunden, die händeringend ein Fachwerkhaus suchen, ist sehr gering.

Cloudbasierte Anwendungen stellen die Softwarearchitekten von heute vor neue und ungewohnte Herausforderungen. Sie benötigen moderne Verfahren die speziell auf die neuen Herausforderungen der Cloud ausgerichtet sind, damit die Anwendungen zuverlässig und zugleich ressourcenschonend operieren. Ob CPU, Arbeits- oder Festplattenspeicher, alles was weniger gebraucht wird, spart Kosten im Betrieb der Cloud und ist zugleich ein Schritt hin zu einer Green-IT.

Zahlreiche Herausforderungen

Ich kann nicht alle Herausforderungen in der erforderlichen Tiefe in diesem Artikel behandeln. Der Text würde so lang und kompliziert, dass niemand ihn zu Ende lesen würde. Stattdessen konzentriere ich mich auf die Serialisierung. Unter Serialisierung versteht man die Umwandlung von strukturierten Daten in ein für die Übertragung oder Speicherung geeignetes Format.

In modernen verteilten Systemen ist es entscheidend, ein schnelles und zuverlässiges Serialisierungsformat zu wählen. Die Serialisierung selbst ist nicht aufwändig. sie wird aber häufig benutzt. Die Entscheidung für ein optimales Format hat also einen großen Einfluss auf das Gesamtdesign der Cloudanwendung.

Ein gut gewähltes Format stellt sicher, dass Daten schnell und mit minimalem Overhead über das Netzwerk übertragen werden. Darüber hinaus ist ein zuverlässiges Format entscheidend um Fehler bei der Datenrekonstruktion zu vermeiden und damit die Datenintegrität sicherzustellen.

Automatisierung als Erfolgsfaktor

Aktuell ist JSON das von vielen präferierte Serialisierungsformat. Es hat im Laufe der Jahre XML den Rang abgelaufen. Der Vorteil von JSON, als für Menschen einfach zu erstellendes und lesbares Format, kommt aber gerade beim Einsatz für die Kommunikation zwischen Microservices nicht wirklich zum Tragen. Daten und Prozesse werden zwar durch menschliche Interaktion angestoßen, laufen danach aber ausschließlich nachgelagert zwischen Systemen ab und reagieren dabei eher auf Events als auf Benutzer.

Auch das Argument, dass Entwickler das Serialisierungsformat lesen müssen ist nicht mehr griffig. Cloud-Anwendungen laufen in Containern, die selbst durch Systeme, wie zum Beispiel Kubernetes, überwacht und kontrolliert werden. Manuelle Eingriffe sind dabei nicht das Mittel der Wahl um Probleme zu analysieren und zu beheben.

Auch das Aktualisieren der Anwendung erfolgt durch eine Continuous-Delivery-Pipeline in kurzen Intervallen, um so den Anforderungen des Marktes nach schnellen und ständigen Anpassungen gerecht zu werden. Die Automatisierung ist in diesem Rahmen alternativlos, gewährleistet sie doch den sicheren und, in Zeiten von Fachkräftemangel, günstigen Betrieb.

Gamechanger Protobuf

Ich werde in diesem Artikel kurz auf die Begriffe und Funktionsweise von Protobuf eingehen. Dann erläutere ich seine neuartigen Konzepte und Eigenschaften und inwiefern diese die Mängel von JSON beheben. Damit begründe ich, warum Protobuf eine überragende Verbesserung im Vergleich zu JSON ist und damit die nächste Stufe in der Entwicklung der Serialisierungsformate darstellt.

Sicher durch präzise Definition und starke Typisierung

Um Daten zu serialisieren, müssen sie zunächst einmal strukturiert beschrieben werden. Bei Protobuf werden die Datenstrukturen in Dateien mit der Endung .proto definiert. In dieser Proto-Definition wird ein Objekt durch eine Nachricht („Message") beschrieben. Eine Message kann eine beliebige Anzahl von Feldern („Fields") beinhalten. Ein Field hat einen Bezeichner, einen Typ und eine, für diese Message, eindeutige Nummer. Der Typ eines Feldes kann wiederum eine Message sein, wodurch eine Baumstruktur entsteht wie sie aus JSON und XML bekannt ist.

syntax = "proto3";

package example;

message Book {
 string title = 1;
 repeated Page pages = 2;
}

message Page {
 string content = 1;
}

Das obrige Codebespiel beschreibt eine vereinfachte Datenstruktur für ein Buch („Book"). Das Buch hat einen Titel von Typ string, der eine beliebige Zeichenkette beinhalten kann. Das Buch verfügt darüber hinaus über beliebig viele („repeated") Seiten, die als je eine Message vom Typ „Page" hinterlegt werden. Die Seiten wiederum haben den Text der auf ihnen steht im Field „content" hinterlegt. Die explizite Angabe von Typen in der Proto-Definition hilft Fehler zu vermeiden und stellt die Konsistenz über Systemgrenzen hinweg sicher. Protobuf unterstützt dabei die gängigen Datentypen von Programmiersprachen. Zudem stellt Google eine Sammlung von erweiterten Typen, die sogenannten WellKnownTyps, bereit. Darüber hinaus können eigene Datentypen über selbst definierte Messages erstellt werden.

Hohe Produktivität durch Codegenerierung

Bei anderen Serialisierungsformaten wird viel wertvolle Entwicklungszeit damit verbracht die Datenstrukturen als Objekte in der Programmiersprache noch einmal anzulegen und bei Änderungen diese kontinuierlich nachzuziehen. Die Arbeit ist aufwändig, aber nicht sehr herausfordernd, was dazu führt, dass sich schnell Fehler einschleichen. Generische Mapper versprechen zwar den Aufwand zu reduzieren, sie sind allerdings in allen Fällen, wo man vom Normalanwendungsfall abweicht, selbst schwer zu konfigurieren.

Protobuf setzt daher auf eine automatische und kontinuierliche Codegenerierung und unterstützt dabei gleich mehrere Programmiersprachen. Aus der Proto-Definition wird direkt der Code für die Datenstruktur generiert, womit ein Großteil des Aufwands anderer Ansätze entfällt. Außerdem ist garantiert, dass Sender und Empfänger immer die gleiche Implementierung nutzen solange sie die selbe Version des Protoschemas haben. Der generierte Sourcecode kann zudem flexibel um zusätzliche Methoden ergänzt werden, um so die Benutzung von eigenen Datentypen komfortabler zu machen.

Streng definierter aber erweiterbarer Vertrag

Die Proto-Definition ist ein Schnittstellenvertrag („Contract") zwischen dem Sender und dem Empfänger. In Protobuf wird ausschließlich auf einfache Strukturelemente gesetzt. So bezeichnet das Keyword repeated eine Liste einer Message ohne Aussagen über deren genaue Anzahl zumachen[vgl. Abb. 1]. Einerseits führt dies dazu, dass der Sender mögliche Einschränkungen auf Empfängerseite nicht aus dem Contract herauslesen kann. Andererseits kann dieses auch als Chance gesehen werden den Vertrag flexibler und damit leichter erweiterbar zu gestalten. Die Möglichkeit den Vertrag leicht ändern zu können und gleichzeitig die Integrität der Anwendung beizubehalten, ist von einem unschätzbaren Wert für eine sich ständig weiterentwickelnde Systemlandschaft.

Einfache Schemaevolution

Bei dem Design von Protobuf wurde ein hoher Wert auf Kompatibilität gelegt. Eine Änderung ist dann abwärtskompatibel, wenn die neue Version eines Empfängers Nachrichten eines Senders einer alten Version verarbeiten kann. Andersherum ist eine Änderung dann vorwärtskompatibel wenn die neue Version eines Senders weiterhin mit der alten Version des Empfängers funktioniert.

Durch die Einhaltung von nur wenigen einfachen Regeln erhält man mit Protobuf sowohl Abwärts- als auch Aufwärtskompatibilität. So darf der Typ und die Nummer eines Fields nie geändert werden. Eine Message und ein Field können nicht gelöscht werden. In der Syntax proto3 wird zudem auch auf die Definition von Pflichtfeldern verzichtet, da Erweiterungen eines Vertrags immer optional sein müssen, um die Kompatibilität zu erhalten.

Kompatibilität für Zero-Downtime

Abwärts- und Vorwärtskompatibilität sind besonders dann wichtig, wenn man nicht gewährleisten kann, dass Sender und Empfänger und alle Nachrichten gleichzeitig auf eine neue Version einer Schnittstellenbeschreibung aktualisiert werden können. Dies ist für Cloudanwendungen eine wichtige Eigenschaft.

Volle Kompatibilität ermöglicht es einen Sender und einen Empfänger unabhängig voneinander auszurollen, wobei auf eine aufwändige Koordinierung der Reihenfolge verzichtet und Continuous-Delivery somit erst ermöglicht wird. Der Einsatz einer modernen Softwarelösung für das Management von containerisierten Anwendungen, wie zum Beispiel Kubernetes, ermöglicht zudem ein automatisches Rolling Update. Dabei wird eine neue Version der Anwendung ohne Downtime im laufenden Betrieb ausgerollt. Die Benutzer bekommen davon nichts mit, was einen unterbrechungsfreien Service und ein optimales Kundenerlebnis erlaubt.

Überragende Effizienz da binär und minimalistisch

Bei Protobuf werden die Nachrichten im Binärformat serialisiert. Dabei werden nur die Nummer und der jeweilige Wert des Feldes einer Nachricht gespeichert. Um die Nachricht später auslesen zu können, benötigt man deshalb die Proto-Definition. Damit verzichtet die Serialisierung bewusst auf unnötigen Ballast, die für den eigentlichen Nutzen nicht notwendig sind. Durch die geringe Datenmenge ist das Erzeugen und Lesen von Nachrichten somit sehr schnell.

Allerdings kann es durch die minimale Speicherung der Nachricht und die hohe Kompatibilität bei Fehlern in der Entwicklung zu unerwartetem Verhalten kommen. So können Nachrichten unter Umständen durch eine fremde Proto-Definition deserialisiert werden, ohne dass ein Fehler auftritt. Deshalb ist es zwingend notwendig die Benutzung von Protobuf durch automatisierte Tests abzusichern. Mit diesen kann auch die Kompatibilität zu alten Versionen sichergestellt werden.

Eine manuelle Fehleranalyse von Nachrichten im Einzelfall ist einfach möglich, indem man die Binärinformationen in ein Objekt umwandelt. Zudem lassen sich Nachrichten einfach in ein JSON-Format konvertieren. Der dafür notwendige Code wird entsprechend der Proto-Definition vollständig generiert. Es können so auch LegacySysteme, die ausschließlich mit JSON arbeiten können, an moderne Anwendungen angeschlossen werden. Die Vorteile von Protobuf können hierbei zumindest teilweise genutzt werden.

Fazit

Kein Serialisierungsformat, weder Protobuf noch JSON, darf als „Goldener Hammer" verwendet werden. Protobuf ist daher sicher keine Alternative zu JSON für alle Anwendungsfälle und wurde auch nicht dafür entwickelt. Die Stärken von Protobuf machen es zu einem flexiblen und zugleich besonders effizienten Serialisierungsformat. Der Fokus des Einsatzgebietes liegt dabei primär auf dem Datenaustausch zwischen Softwaresystemen. Es kann daher besonders einfach überall dort eingesetzt werden, wo im Rahmen der Digitalisierung automatisierte Prozesse geschaffen werden.

Die initiale Einführung von Protobuf mag einen Einarbeitungs- und Lernaufwand bedeuten, der sich aber durch die einheitliche Verwendung des Serialisierungsformats für möglichst viele Anwendungsfälle schnell wieder auszahlt. Durch den hohen Automatisierungsgrad und die Codegenerierung kann sich die Entwicklung besser auf ihre eigentliche Aufgabe - das Lösen von Problemen - konzentrieren.

Jetzt sind deiner Kreativität keine Grenzen mehr gesetzt, um basierend auf Protobuf Lösungen für deine Domäne zu entwerfen. Ich bin sehr gespannt auf deine Ideen und Vorschläge. Denn streben wir nicht letztlich alle danach, unsere Software durch ein erstklassiges Design kontinuierlich zu verbessern und damit zum Erfolg zu führen?

In zukünftigen Artikeln werde ich selbst einige konkrete Beispiele liefern, wie ich Protobuf in der Vergangenheit eingesetzt habe. Am besten folgst du mir, um keinen meiner zukünftigen Artikel zum Thema Protobuf zu verpassen.

Spring cleanup: Outdated Git tags

Nils Diekmann — Wed, 31 Jul 2024 14:25:00 +0000

Imagine your repository is flooded with auto-generated Git tags. Most of them are from feature branches you no longer remember anymore. Just as you strive to keep your code clean, you should also regularly clean up these Git tags.

Motivation

Recently I started programming with Go. I have written some modules and in order to use them in other projects, I need to publish them. Go uses Git tags to mark a specific version. The tags should use semantic versioning. I am already familiar with semantic versioning. In C#, I used this simple approach to give my services and NuGet packages a descriptive version.

I am a lazy person and I like to automate all my work. So I invested blood and tears in automatically tagging my code with corresponding semantic versions. Having even managed to handle multiple modules in one repository, I am now facing the next challenge.

I use feature branches to develop and pre-release my code. The pipeline generates new Git tags with each check-in, which is useful during development. But how do I get rid of them, after I merge the feature branch? I don't want to waste my life manually deleting things.

The full code for the article is available on GitHub. The repository also has an open feature branch for demonstration purposes.

KinNeko-De / sample-git-tag

Sample application how to cleanup outdated Git tags.

sample-git-tag

Sample application how to cleanup outdated Git tags.

See my article how to create this by yourself.

View on GitHub

Step 0: Producing semantic versioning tags

Semantic versioning is a popular versioning scheme that uses a format like "major.minor.patch" to indicate the version of the code base. My GitHub workflow has an environment variable 'MAJOR_MINOR_PATCH' which you must manually update according to the rules of semantic versioning when you develop the code.

For the main branch, the semantic version is determined by the environment variable. For feature branches, a pre-release version is defined by a suffix containing the name of the branch and the build run number: v0.1.0-feature-branch.1. The workflow then pushes the semantic version with actions/github-script as a Git tag. Adding the build run number makes the Git tag unique without manual intervention.

Step 1: Delete outdated tags with a script

When I need to automate my task, I start with writing a script that I can run and debug locally. The first step is to have a good design. I want to delete the git tag from the remote repository. Outdated git tags are defined by git tags, that contain a semantic prerelease version and where the corresponding feature branch was already deleted.

I need to update my local version to the remote version.
I have to fetch all git tags.
I have to fetch all branches.
Determine for each git tag if it is outdated.
If so, delete the git tag on the remote repository.

git fetch --tags --prune --prune-tags will fetch branches and tags from the remote version and delete any local branches and tags that no longer exist on the remote.

git branch -r retrieves all remote branches. Then the names are normalized by removing slashes and whitespaces with sed 's/^*//;s/ *$//. Finally, I filter out my main branch with egrep -v "^main".

git tag - list 'v[0–9]*\.[0–9]*\.[0–9]*-* gives me a list of all git tags, where the name starts with a v followed by the major, minor, and patch version numbers. It also must be followed by a hyphen - and any character that indicates a pre-release version.

I have the convention that all my feature branches start with feature followed by the name of the branch. But the git tag only contains the name of the branch. An example feature/branchname for the feature branch and the corresponding git tag is then v1.2.0-branchname.

[[ $tag =~ ^v[0-9]*\.[0-9]*\.[0-9]*-(.*)\.([0-9]*)$ ]]
featurebranchname=${BASH_REMATCH[1]}

For each of the git tags I cut the feature branch name out (see code block above) and check if a feature branch with a name according to my conversion exists (see code block below).

if [[$existingfeaturebranches =~ "feature/$featurebranchname" ]]

If not so, then the git tag is first deleted on the origin git push origin --delete $tag and then locally git tag -d $tag.

Step 2: Integrate the script into a CI pipeline

Automating tasks with scripts is a valuable first step, but manual execution is time-consuming. Integrating the script into my CI pipeline, triggered on every code push, would be a more convenient approach.

To create this step for my workflow, I ask GitHub Copilot to translate the bash script into a step for my GitHub workflow. Like me, GitHub Copilot prefers to use actions/github-script. Unexpectedly, GitHub Copilot generated JavaScript code instead of Bash. It seems that JavaScript is the preferred language for GitHub actions.

The logic of how the code fetches the branches has changed. GitHub Copilot missed that I was filtering out the main branch. My Git tags for the main branch have the format v1.2.0 and do not include the name of the main branch itself. Other than the fact that the main branch is now logged as an existing feature branch, the change does not affect the result.

const existingFeatureBranches = (await github.rest.repos.listBranches({
  owner: context.repo.owner,
  repo: context.repo.repo,
})).data.map(branch => branch.name);

Another change is that Git tags are no longer filtered to feature branch tags only. Aside from the longer runtime, this does not change the logic, since the other tags do not match the feature branch regular expression anyway. I keep the code as it is and consider this a limitation of the GitHub Rest API.

const tags = await github.rest.git.listMatchingRefs({
  owner: context.repo.owner,
  repo: context.repo.repo,
  ref: 'tags/v',
});

The workflow with the script is triggered every time I push code. I do this a lot to break problems into smaller steps. Since deleting git tags takes time, I do not want to run the script on every push. My repository is configured to automatically delete the feature branch when a pull request completes. After that, the workflow is triggered for the main branch. If I run the action on every run of the workflow for the main branch, it will fit my needs perfectly.

- name: Delete Git Tags
  uses: actions/github-script@v7
  if: github.ref == 'refs/heads/main'

To achieve this goal, I introduce a conditional check within the pipeline using the if keyword. The branch that triggered the current run of the workflow is defined by github.ref. In my case, the name of my main branch is refs/heads/main. The step will be now only executed if the two variables have the same value.

Finally, you can also take a look at my first GitHub action that replicates the functionality of the step inside this sample pipeline. The GitHub action is implemented using TypeScript. It's important to note that this GitHub action is specifically tailored to my conventions and lacks flexibility. Under these miserable conditions, I recommend understanding the underlying concepts before directly using my GitHub action in your projects. Instead, feel free to copy and modify the code to suit your needs.

KinNeko-De / cleanup-outdated-tag-action

GitHub action to cleanup outdated Git tags.

Create a GitHub Action Using TypeScript

Use this template to bootstrap the creation of a TypeScript action. 🚀

This template includes compilation support, tests, a validation workflow publishing, and versioning guidance.

If you are new, there's also a simpler introduction in the Hello world JavaScript action repository.

Create Your Own Action

To create your own action, you can use this repository as a template! Just follow the below instructions:

Click the Use this template button at the top of the repository
Select Create a new repository
Select an owner and name for your new repository
Click Create repository
Clone your new repository

Important

Make sure to remove or update the CODEOWNERS file! For details on how to use this file, see About code owners.

Initial Setup

After you've cloned the repository to your local machine or codespace, you'll need to perform some initial setup steps before you can…

View on GitHub

I'm excited to share my experience and encourage others to explore the potential of GitHub Actions for my automation needs. Thank you for reading my story until the end. ❤

OpenTelemetry Metrics meets Azure

Nils Diekmann — Sat, 20 Jul 2024 20:19:15 +0000

Tutorial featuring Azure Monitor OpenTelemetry Distro for C# and Grafana

Motivation

OpenTelemetry is the future of observability. It is being pushed by Google, Microsoft and Amazon to provide a vendor-neutral solution for traces, metrics and logs. The OpenTelemetry API implementation is available for many programming languages. I will only use metrics and the implementation for the language C# for this tutorial. Let's see how far apart reality and desire are.

I will use the Azure Monitor OpenTelemetry Distro to publish the metrics to the Azure cloud. The way the metrics are collected in the application is independent of this decision. It affects the way how the data is processed by the cloud components. Using the OpenTelemetry Collector as an alternative is also not officially supported by Azure.

Finally, I want to aggregate, transform, and visualize the raw metrics. I will use Log Analytics, Azure's default log analysis solution. Additionally, I will also use Grafana to create a beautiful dashboard. Grafana is available as a managed service on Azure.

OPS side - Infrastructure

The application will push the metrics from the running application to Application Insights. It will use the C# SDK for the connection to Azure Monitor. The data will be stored in the table 'customMetrics' of the Logs database. Log Analytics can query that table. Grafana queries the data from Log Analytics and displays it.

DEV side - Application

My goal is to provide the smallest possible sample to get you started with OpenTelemetry Metrics and Azure. The full source code is also available on GitHub.

KinNeko-De / sample-opentelemetry-azure

Sample application how to publish OpenTelemetry metrics to Azure

Motiviation

Sample application how to publish OpenTelemetry metrics to Azure

See my article how to create this by yourself.

View on GitHub

Metrics in C# are instrumented with classes from the System.Diagnostics.Metrics namespace. The namespace provides an abstraction to separate the instrumentation from the way the metrics are later collected. OpenTelemetry uses a generic data model to define these metrics. For this example, I will use a counter as the simplest type. The metric will track the number of items ordered as an integer counter that only ever goes up.

private static readonly Meter Meter = new("restaurant-order-svc", "0.0.1");
private static readonly Counter<int> ItemsOrderedTotal = 
      Meter.CreateCounter<int>(
         "restaurant-order-svc-items-ordered",
         "item",
         "Items ordered total"
      );

I define my metrics in a specific class called 'Metric'. It is injected as a singleton. The counter is declared as a static field. I prefer not to access the field directly, but to provide a wrapper method instead. With this approach, the class encapsulates how business data is transferred to generic metric data types. Using another layer of abstraction for a simple counter seems like overkill. The advantage becomes more apparent when I use histograms with buckets.

public void ItemsOrdered(int amount)
    {
        ItemsOrderedTotal.Add(amount);
    }

The metric is simulated with an infinite background service. It sends a random number of ordered items at each interval. In a real application, I would instead increase the metric after the order has been successfully stored in the database.

var itemsOrdered = RandomNumberGenerator.GetInt32(1, 9);
Metric.ItemsOrdered(itemsOrdered);

To publish the collected data, I have to configure the application to use OpenTelemetry. I do this within my Program.cs with the service collection extension AddOpenTelemetry(). The extension is available via the NuGet package OpenTelemetry.Extensions.Hosting.

private static void ConfigureServices(IServiceCollection services)
{
    services.AddOpenTelemetry();
}

To publish the data to Azure, I use the service collection extension UseAzureMonitor() service collection extension. This extension is available via the NuGet package Azure.Monitor.OpenTelemetry.AspNetCore. This package has a dependency on OpenTelemetry.Extensions.Hosting so I only need this package as a dependency in my application. Within the AzureMonitor configuration, I need to use the ConnectionString to my Application Insights resource.

private static void ConfigureServices(IServiceCollection services)
{
    var applicationInsightsConnectionString = 
        "InstrumentationKey=00000000-0000-0000-0000-000000000000";
    services.AddOpenTelemetry().UseAzureMonitor(b =>
        {
             b.ConnectionString = applicationInsightsConnectionString;
        })
        .WithMetrics(metricBuilder => metricBuilder
           .AddMeter(Operations.Metric.ApplicationName)
           .AddConsoleExporter(builder => builder.Targets = 
               ConsoleExporterOutputTargets.Console)
         );
}

I also added an OpenTelemetry console exporter to see all the data being generated. It takes some time for the data to appear in Application Insights. Metrics can generally be lost if the application crashes. If you are running the application and you want to be sure that the data is pushed, you should shut down the application gracefully. Locally, you can do this by pressing CTRL+C in the application console.

Configure Azure Infrastructure

Firstly, I need to create an Azure Monitor workspace. I am not sure, if I need it for publishing the metrics. As far as I understood the documentation, it is needed for Grafana to access the stored metrics.

I have also created an Application Insights resource. I need the connection string for the C# SDK. When I create the Application Insights, an associated Log Analytics Workspace is automatically generated.

I had the problem that most of the time I could not query any data through this generated Log Analytics Workspace. The effect was quite random and I was totally confused. It seemed to be solved by creating a new Log Analytics namespace and linking it to the Applications Insights resource. If you encounter the same problem, please leave a comment.

Finally, I created an Azure Managed Grafana resource. I change the pricing tier from 'Standard' to "Essential". This pricing tier is suggested for my use case, but it also has some limitations and a pitfall that I will describe later.

Visualize with Log Analytics

The first challenge I have to overcome is querying the raw data. To do this, I need to go to Application Insight. In the 'Monitor' section, I select the 'Logs' item. The most obvious choice would be 'Metrics', but I cannot find the OpenTelemetry metrics here.

There are several tables within the Logs section. The OpenTelemetry metrics are stored in the 'customMetrics' table. I can now query the data using the Kusto Query Language (KQL).

customMetrics 
| where name == 'restaurant-order-svc-items-ordered'
| order by timestamp asc

Each metric must have a unique name. In this first query, I filter my metric by name and then sort by timestamp. The result of this query is a table where each row is a reported counter value. Because I only increment the value every 15 minutes, the column value is exactly the number of one order.

It is also possible to create a time chart from the raw data. I simply add a render command to the KQL query. There are various types of charts available to visualize the data.

customMetrics 
| where name == 'restaurant-order-svc-items-ordered'
| order by timestamp asc
| render timechart

The result is a time chart that looks very nice. The reason for this is not my query itself, but rather the fact that the application has published exactly one data point every 15 minutes.

To visualize the data, I need to aggregate the values over a specific interval. In this query, I choose a fixed interval of one hour. The timestamp of each data point is rounded to this interval. The values for all data points are summarized to one value per hour.

customMetrics
| where name == 'restaurant-order-svc-items-ordered'
| summarize sum(value) by bin(timestamp, 1h)
| order by timestamp asc
| render timechart

The result is a graph showing the total number of items ordered each hour.

Visualize with Grafana

Once I have created the Grafana resource, I can already enter it and create new dashboards. Within the Azure Portal, I can find the URL to access the Grafana resource.

In order to give Grafana access, I need to link Grafana to the Azure Monitor Workspace. It looked like I could also set the link in Azure Managed Grafana unter ‘Azure Monitor workspaces’, but here I got an error message.

After connecting the Grafana to Azure Monitor, I have a new data source ‘Azure Monitor’ in Grafana. This data source is already configured correctly. Azure also generated a lot of additional dashboards.

Grafana was created with the ‘Essential’ pricing tear because it costs less. The pricing tier has a limit on the number of dashboards, I can only have 20. With the newly created dashboards, I exceed this limit. Whenever I try to save a new dashboard, I get an error message that my quota has been exceeded. I can now change the pricing tier to ‘Standard’, but I can never go back to ‘Essential’. I have fallen into this trap. A much cheaper solution would be to delete all the other dashboards.

I am now ready to start my first visualization. In the ‘Query’ tab, I first select the correct data source to retrieve data from the ‘customMetric’ table. The database source is ‘Azure Monitor’ (1st in the previous picture). I need to change the Service value to ‘Logs’. The value describes the database I want to query from (2nd in the previous picture). I then select as the Resource the Application Insight resource (3rd in the previous picture) on which I previously ran my KQL queries (see the following picture).

I want to select the number of items ordered. To do this, I simply summarize all the values. This is not the total number of items ever ordered, as metrics data is usually purged after some time. It just gives me an idea of the most recent orders and that the application is running smoothly. If the number of orders is lower than expected, it may be an indication that something is wrong.

customMetrics
| where name == 'restaurant-order-svc-items-ordered'
| summarize by timestamp, value
| order by timestamp asc

Summarising by timestamp and value is required in Grafana, otherwise, every single entry in the table will be displayed as a single point in Grafana. Ordering by timestamp is not logically necessary, but the ‘Stat’ graph type I chose displays a nice background image of the individual data points.

My second chart should show the aggregated number of orders in a given interval. To do this, I first declare an interval variable for the dashboard with an enumeration of values.

The user can select the interval in the frontend. The selected value is then passed to the query as a parameter. The query then aggregates the timestamp according to the interval and then summarises all the values.

customMetrics
| where name == 'restaurant-order-svc-items-ordered'
| summarize sum(value) by bin(timestamp, $interval)
| order by timestamp asc

The graph I have chosen to display the data is ‘Time series’. In the top left hand corner of the following image, you can see the selectable interval. The legend of this diagram is not aligned with the selected interval. There are 30 minutes between the legend values, but one hour between the data points. I can change this later.

The variable $interval is defined in Grafana and from the result of the diagram, I see that it works, but KQL’s syntax highlighting does not recognize the variable. It appears as an unknown variable. I did a bit of googling and I think I am not alone with this and I should ignore it for now.

Conclusion

OpenTelemetry is easy to use. I already have experience with metrics and OpenTracing. I am also a bit biased because I really like the concept of metrics. Do you find it easy to use? Please leave a comment below. I really want to know your opinion.

Integrating OpenTelemetry with Azure is fairly straightforward if you know what you need to do. However, what components you need and how they communicate with each other is not very clearly documented. For example, the documentation tells you to use “Azure Monitor”, but then the SDK needs a connection string to Application Insight and not to the Azure Monitor workspace. This, and the fact that there are metrics that are different from ‘customMetrics’, is confusing.

It is time-consuming to put all the pieces together for the first time. The problem with the non-queryable results in Application Insights also took me several hours. When things went wrong, I could not find detailed logs to figure out which component was causing the problem. I had previously used an OpenTelemetry Exporter on my local host, which at least gave me a log of the data received.

The next pain point is the additional cost of the Azure components. If you already have a Prometheus for your Kubernetes cluster, why not store the metrics there? I guess the Azure Monitor OpenTelemetry Distro is aimed at people who don’t have Prometheus and just want to use observability for Azure Container Apps. I think for them the Distro gives an easy first impression of the subject matter. And to make it even easier, I have created this tutorial for you.

Follow me to read more about OpenTelemetry metrics meets Google Cloud later. More claps will motivate me to write it ❤

e1889d6d-945d-436b-b94f-68f268285e7c — WTF? I’m only human

Nils Diekmann — Sat, 13 Jul 2024 12:42:20 +0000

Designing human-friendly identifiers is easy - follow these simple guidelines.

If you are just interested in a quick solution, download the tutorial of my C# and Go examples on GitHub. Read on for the motivation behind the solution.

KinNeko-De / sample-humanfriendly-id-csharp

Sample how to create a human-friendly identifier in C#

sample-humanfriendly-id-csharp

Sample how to create a human-friendly identifier in C#

Generate a human friendly id

dotnet run --project Sample.HumanFriendlyId/Sample.HumanFriendlyId.csproj

Motivation

The motivation behind this project is to described in an article

View on GitHub

KinNeko-De / sample-humanfriendly-id-go

Sample how to create a human-friendly identifier in Go

sample-humanfriendly-id-go

Sample how to create a human-friendly identifier in Go

Generate a human friendly id

go run cmd/humanfriendly-id/main.go

Motivation

The motivation behind this project is to described in an article

View on GitHub

Accept the differences - Humans are not systems and vice versa

The debate is as old as mankind. Some argue that values in systems must be human-readable because the user is human. Others want to force the user to deal with technical values because they are already used by the system.

Have you ever typed a UUID from a letter? If so, I have great respect and at the same time pity for you. UUIDs are designed for systems, not for people. On the other hand, business identifiers, like invoice numbers, are not designed for reliable relationships in systems, especially not in modern distributed microservices.

In computer science, we have the principle of 'separation of concerns'. It says that you should divide your program into distinct sections. If you follow this principle, you benefit from more freedom of use.

It can also save us unnecessary headaches when designing the solution. We can still use UUIDs for internal system IDs. In the rare cases where we need to communicate with a human, use a specially designed human-friendly identifier. I will now give a guideline on how to design it.

Requirements - select what needs to be met

Requirements always depend on the context. I will provide a list of requirements based on my experience. If some of the requirements are not necessary for you, feel free to leave some out. If you have additional requirements, please leave a comment. I would appreciate it if you explain why you have these requirements so that I can learn from you.

(R) It must be easy to read by humans.
(P) Errors of perception must be avoided.
(W) It must be easy to spell, hand-write, and type by humans.
(O) It must be easy to read by optical character recognition.
(S) It must be hard to guess.
(U) It must be unique for the related entity.

R This ensures that users can quickly and accurately interpret the identifier without confusion or difficulty. By prioritizing readability, we aim to improve the user experience and streamline the use of the identifier in different contexts.

P It is important to design the identifier in a way that minimizes misinterpretation. By prioritizing clarity and distinctiveness, we aim to reduce the risk of misreading the identifier.

W This ensures ease of use across multiple communication channels and input methods. We aim to make communication and data entry seamless for users.

O This ensures that the identifier can be accurately scanned and processed by machines, increasing efficiency and automation in various applications. By designing the identifier with OCR readability in mind, we ensure compatibility with modern technology and data processing workflows.

S It is important to make the identifier sufficiently complex and unpredictable to prevent unauthorized access or manipulation. Our goal is to enhance security and protect the integrity of the identifier and associated data.

U By enforcing uniqueness, we maintain data integrity and facilitate accurate referencing and retrieval of information. Each entity is uniquely identified without any overlap or duplication.

From time to time in this article, I will refer to these requirements with the letter at the beginning. This means that I have taken this particular requirement into account when designing a human-friendly identifier.

Characters - make the identifier easy to read

To generate the identifier, we need a specific set of characters. We have to choose this collection carefully because it affects readability (R), perception (P), and writing (W). Ensuring the suitability of these characters is essential to their effectiveness in interacting with humans.

In addition to the characters, the choice of font we use is important in terms of readability (R). Have a look at the appearance of the following characters in print. As you can see in the image below, it is hard to tell the difference even when you see them side by side. Imagine how difficult it would be if you could only see one character. We need to choose a font that makes it easier to distinguish between the characters.

The use of both lowercase and uppercase letters also raises the question of whether the reader needs to use a case-sensitive identifier in a written or typed response. A program cannot easily normalize the input to avoid errors. If the character set contains only uppercase letters, we can easily convert any input to uppercase. This choice also removes the small "l" problem.

We need as many characters as possible to ensure uniqueness (U) and to keep the identifier short (R). By adding numbers, we increase the number of possible characters. On the other hand, it creates a problem with the uppercase "O" and the zero, as they look very similar in any font.

We can remove one of them from the set. As a reader of the identifier, you do not know that we omitted only one character and which one. I have seen several times that there is a note describing whether both zero and "O" or only one of them is used. If we remove both characters, we avoid any kind of confusion for the human (R).

If you live in a country like Germany, where they even print out electronic receipts and still use real paper letters (to the rest of the world: this article was written in 2024!). It is also important to consider that someone will reply handwritten (W).

To OCR (O), a handwritten "1" may look like an "I". Removing just one of them, preferably the "I", is useful in this case because we can normalize the input to replace a recognized "I" with a "1". The number one is more likely to be recognized differently from other characters. By excluding the "I", we also make it easier to recognize the characters as there is no conflict with the small "l" (R).

This leaves us with the following charset:

ABCDEFGHJKLMNPQRSTUVWXYZ12345679

Length - how much do you really need?

In general, it is a good idea to use a wide set of characters as it affects the number of unique identifiers you can generate (U). When you use a narrow character set because of the reason above, you have to increase the length of your identifier to allow more recombination.

The number of possible combinations is calculated as follows:

We have to keep the identifier reasonably short. Long identifiers can be difficult to remember and read (R). If you ever run out of combinations because your business scales up like hell, just increase the length for new identifiers. You can keep the old one instead of setting yourself up for success which will never happen if you use too long identifiers.

With a length of only 8, there are 1,099,511,627,776 combinations, which is enough for most of the time. That is 140 times the number of people in the world. Look at this example, close your eyes, and try to remember it:

AKL1U9ZW

On the other hand, sometimes an identifier needs to be hard to guess (S) when used in a security context. Using a length that is too small makes it more likely that a valid identifier will be found by brute-force attacks. It also makes it more likely that a typo will lead to another valid identifier (U)(W).

Using the character set from the example above and a length of 16 we get the astronomically large number of 1,208,925,819,614,629,174,706,176 combinations. Try to remember the following example as you did it before. Not so easy anymore, right?

AKL1U9ZWKWA6QPAB

Display - the best way to recognize it

The identifier itself is a random, uninterrupted combination of characters. It has no obvious meaning or context. The human brain tends to look for familiar patterns. Since there are none here, it is very difficult to recognize each of the characters.

Remember the tally marks we used when we were kids? After drawing four vertical lines, the fifth line is drawn diagonally across the previous four lines. This groups five of the strokes together to make counting easier.

Grouping items helps people to understand and organize information more quickly and efficiently. This is also known as the "law of proximity". We cannot stroke our characters, but we can use a separator between them. We can use dashes, minuses, hyphens, or slashes, as UUIDs use minus. But when you type or spell the identifier, don't you always wonder if you can leave out the separators?

As children, we would never write the tally marks with a minus or even a slash between them. Such characters were not in our minds at that time. Children always use the simplest and most obvious solution. After five strokes, we leave a little space to group them even more. In the following example the second line is easier to read, right?

We can do the same simple trick with our identifier and use spaces as separators (P). These separators are not even visible to humans. When we process the input for a computer we can normalize it by removing all the spaces between the characters. This makes it as easy as possible for humans to type the identifier. And when you spell the identifier hopefully you will not even notice that there is a hidden separator.

After how many elements should we use a separator? According to "Miller's Law", people can remember about seven elements. The exact number depends on the person, so we will use the lower limit to include everyone (P). Other publications tend to define this lower bound as only four elements. This is exactly the number we used with the tally marks. See the result:

AKL1 U9ZW HKAL BCND

As you may have noticed, we are not the only ones using this rule. IBAN or credit card numbers are grouped the same way. I am pretty sure there are more real-life examples in your cultural context to convince you that this idea is widely adopted.

Random - at least pseudo and safe

The identifier should be randomly generated using a cryptographically secure algorithm (S). This makes it easier for you to pass a pen test, especially in a security-critical scenario. Proactively demonstrating that you have taken this into account during design increases confidence in your overall solution.

Even in non-security scenarios, you want to avoid using auto-incrementing numbers. The recipient of an invoice with invoice number "5" does not need to know that you have only written five invoices this year. Hiding this information is also in the best interest of the business.

Note that the identifier is not globally unique like UUIDs. The generation algorithm is pseudorandom, and the length is shorter so that we have fewer combinations. To make it unique for a given entity, you must ensure this using a shared state between concurrent processes, such as a unique constraint on a database (U).

Summary

Use the character set ABCDEFGHJKLMNPQRSTUVWXYZ12345679
Generate randomly an identifier with a length of 4 or 8
To display it to humans insert a space after four characters
On human input remove spaces and replace "I" with "1"

How do you know all that shit? Are you a real programmer?

Thank you to the University of Paderborn for teaching me that software engineering is more than just coding. Now it is my time to give something back.

Code examples

Last but not least check out the code examples in C# and Go on my GitHub page. I would be happy if you could give them a star. Thanks for reading my article ❤

Launch React from Visual Studio Code

Nils Diekmann — Sun, 07 Jul 2024 19:33:50 +0000

How to launch a React application from Visual Studio Code including Firefox and Chrome using React Script 5.0.1 and Windows

Motivation

I want to launch my React application from Visual Studio Code with a single click. I hate typing a lot in the terminal to launch an application. I prefer to use scripts. There has been a change to react-scripts, so other solutions on the web will no longer work. So here is my current working solution!

Solution

Create different scripts in your package.json to start react-script with different options depending on your current needs. First, set the browser to none to prevent react-scripts from launching the browser itself. Then launch the browser of your choice, followed by starting the application. In your launch.json from Visual Studio Code call the appropriate npm scripts.

Start in Firefox

"scripts": {
  ...
  "startfirefox": "set \"BROWSER=none\" && start firefox http://localhost:3000 && react-scripts start",
  ...
},

Start in Firefox's private Window

"scripts": {
  ...
  "startfirefoxprivate": "set \"BROWSER=none\" && start firefox -private-window http://localhost:3000 && react-scripts start",
  ...
},

Start in Chrome

"scripts": {
  ...
  "startchrome": "set \"BROWSER=none\" && start chrome http://localhost:3000 && react-scripts start",
  ...
},

Start in Chrome incognito

"scripts": {
  ...
  "startchromeincognito": "set \"BROWSER=none\" && start chrome http://localhost:3000 /incognito && react-scripts start",
  ...
},

Launch from Visual Studio Code

Let Visual Studio Code create a launch.json for you. Replace the generated configurations with the following JSON:

"configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Start using Default browser",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["run-script", "start"],
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Start using Chrome",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["run-script", "startchrome"],
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Start using Firefox",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["run-script", "startfirefox"],
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Start using Chrome Incognito",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["run-script", "startchromeincognito"],
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Start using Firefox private",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["run-script", "startfirefoxprivate"],
    },
  ]

Conclusion

This solution works for Windows and the current version of react-script. Since it only uses OS-specific commands, you can easily adapt it to your OS.

Version used

The tutorial was created based on

react-scripts: 5.0.1

Samples

I created a sample application on GitHub:

KinNeko-De / sample-react-vscode

Sample application how to start a React app from Visual Studio Code in different browsers

Motiviation

Sample application how to start a React app from Visual Studio Code in different browsers.

See my article how to create this by yourself.

View on GitHub

Sources

Credits goes to this comment:

https://github.com/facebook/create-react-app/issues/11873#issuecomment-1030708183

React, Typescript, and CD to GitHub Pages (2024)

Nils Diekmann — Thu, 04 Jul 2024 19:44:28 +0000

How to set up a typescript-based React application and deploy it directly to GitHub Pages using a CD GitHub action. A practical guide for 2024.

Motivation

I was new to React, Typescript, and GitHub Pages when I started this. I fell into some traps, mainly due to outdated suggestions, while creating my proof of concept. To avoid this for you, I will give you a quick and easy walkthrough.

Create React app

Install the following prerequisites: Git and Node.js.

Create the React app
I will use create-react-app to create the initial project setup. I start a terminal in the root folder where all my git repositories are located. A subfolder with the name of your application will be created.

npx create-react-app sample-react-typescript-githubpages --template typescript

The cra-template-typescript package does not need to be installed globally. I have uninstalled it for this tutorial. There is also no more switch typescript anymore for create-react-app.

The folder now contains a .git folder. I am removing it to push directly from Visual Studio Code to GitHub. I also renamed my ‘master’ to ‘main’ branch which has implications for the GitHub workflow I will introduce later.

Vulnerability warnings from npm

I get a notice that vulnerabilities have been found.

8 vulnerabilities (2 moderate, 6 high)

Npm suggests that I should run ‘npm audit fix’ to fix this. But this only introduces more vulnerabilities. Instead, I move the ‘react-scripts’ dependency in the package.json to ‘devDependencies’. After this step, I can run npm audit without the dependencies that are only used to develop the page.

npm audit --omit=dev

Build and continuous deployment

I create a .github/workflows folder and a builddeploy.yaml with the following content in it.

name: Deploy React to GitHub Pages

on:
  push:
    branches:
      - main # name of the branch you are pushing to

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '22'
      - name: Install Dependencies
        run: npm ci
      - name: Build
        run: npm run build
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: 'build/.'
  deploy:
    needs: build
    runs-on: ubuntu-latest
    permissions:
      pages: write
      id-token: write
    environment:
      name: github-pages
      url: 'https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}/'
    steps:
      - name: Setup Pages
        uses: actions/configure-pages@v5
      - name: Deploy
        uses: actions/deploy-pages@v4

This workflow uses the GitHub action actions/deploy-pages to deploy the artifact from actions/upload-pages-artifact to GitHub pages. I can also view and download the generated artifact in the workflow run.

When I push the file, a workflow run automatically starts. It fails because the default configuration of GitHub is currently to deploy GitHub pages from a specific branch. So I need to change the configuration of my GitHub repository.

I need to change the selection of the drop down box to ‘GitHub Actions’. As you can see ‘Deploy from a branch’ is declared as ‘classic’, which is usually a nicer word for ‘legacy’.

Now I need to re-run the failed jobs of my GitHub Action workflow. GitHub will only execute the job ‘deploy’ again. The already built artifact ‘github-pages’ will be reused.

The pipeline is now green and I see a link in my execution that I can click to go to my deployed GitHub page. The page is there because I am not getting a 404 error. But the page shows nothing.

Activating npm publish

The reason for this is a configuration in the package.json in my React app that prevents the page from being published. I have to change it to see my page. I also have to add the link to my GitHub Page, even though GitHub Copilot says I do not have to and the problem is somewhere else.

{
...
"private": false,
"homepage": "https://kinneko-de.github.io/sample-react-typescript-githubpages/",
...
}

After pushing and deploying I finally see my GitHub Page. You can see the full code in my GitHub repository.

Conclusion

Even with the traps, it took me only two hours to finally deploy my first React app. React, GitHub and the community have done a great job making the developer's life easy. Thank you for that ❤

Versions used

Node.js 20.12.2
NPM: 9.8.1
cra-template-typescript: 1.2.0
create-react-app: 5.0.1

Samples

I created a sample application on GitHub:

KinNeko-De / sample-react-typescript-githubpages

Sample application how to setup a React app with typescript and deploy it to GitHub Pages.

Motiviation

Sample application how to setup a React app with typescript and deploy it to GitHub Pages.

See my article how to create this by yourself.

View on GitHub

Here you can see the deployed version:

kinneko-de.github.io

Install Module in Powershell without Install-Module

Nils Diekmann — Tue, 02 Jul 2024 21:26:27 +0000

The term ‘Install-Module’ is not recognized as the name of a cmdlet.

Have you ever found yourself in trouble because your Powershell is not working anymore after a colleague told you to just delete every module of Windows Powershell? No problem… you can just use Install-Module to reinstall everything. Oops… Install-Module is part of the module you just deleted. Here is the guide on how to escape this chicken and egg problem.

How to restore Install-Module manually

To restore functionality, you need two modules. PowerShellGet and its dependency PackageManagement. You can get them from the Powershell Gallery. Search for the modules and click on ‘Manual Download’. There you will also find a helpful description of how to install these Nuget packages manually. Follow the instructions step by step. Only in step 5, you should first create a subfolder containing the version of the module. Restart your PowerShell and you can install all other modules now.

Are you a developer and too lazy to do things manually?

Since you already use Powershell, why not just use Powershell?

Finally, copy the two folders PowerShellGet and PackageManagement to your Powershell Module folder. You will get a warning that the files are marked as untrusted. If anyone finds a solution, please leave a comment.

If you are simply smart and pragmatic

Use the Windows Recycle Bin to restore the PowerShellGet and PackageManagement modules.

Please also leave a comment if anyone else is experiencing the same problem. Together we are strong 💪♥️❤♥️

Foto von Tim Marshall auf Unsplash

Don’t trust AI, trust tests

Nils Diekmann — Mon, 01 Jul 2024 23:03:52 +0000

In my very first story, I talked about my experience with AI in the form of GitHub Copilot. It betrayed me again. But I was gently caught by my true lover: UnitTest

I am currently working on code that receives a file using a grpc stream. The file is sent in byte chunks. Go has a nice functionality where you can determine the media type of a file from the first 512 bytes. I do not want to keep all the bytes sent in memory so my goal is to have a byte array of exactly 512 bytes at the end to sniff the media type. All other bytes should be written to a physical file storage and then discarded.

I am not that experienced in working with arrays and slices in Golang, nor in other languages. For my test cases, I choose to test chunks smaller than 512 bytes, exactly 512 bytes, and larger than 512 bytes. If you wonder why, check out what boundary tests are. I have a lot of experience in writing tests.

Not surprisingly, the test with only 4 bytes failed. It took me some time to get deeper into the go standard libraries. I (mis)use tests for this because it is so easy to write, execute, and debug small snippets of code. Here is my learning example:

func TestArray(t *testing.T) {
 //target2 := [6]int{}
 target := make([]int, 6)
 first := []int{1, 2, 3}
 second := []int{4, 5}
 size := 0
 copy(target[size:], first)
 size += len(first)
 copy(target[size:], second)
 size += len(second)
 target = target[:size]
}

AI helps me with explanations and gives me a better understanding of how to use slices in Go. It is always a pleasure for an old man to learn something from the youth full of new ideas.

With the help of GitHub Copilot, my first and second tests pass. Here is the code I used:

var totalFileSize uint64 = 0
sniff := make([]byte, 512)

copy(sniff[totalFileSize:], chunkMessage.Chunk)

The test for more than 512 bytes failed because my slice was out of range. Maybe it is time for me to admit to myself that I still have a lot to learn. GitHub Copilot came up with the following solution:

if totalFileSize < 512 {
    remaining := 512 - totalFileSize
    if len(chunkMessage.Chunk) > remaining {
        chunkMessage.Chunk = chunkMessage.Chunk[:remaining]
    }
    copy(sniff[totalFileSize:], chunkMessage.Chunk)
}

In my arrogance as an old wise man, I thought I could do better. In my defense, chunkMessage.Chunk cannot be modified because all bytes must be copied in the final file. I implemented a shorter version that worked well, at least in my eyes.

if totalFileSize < 512 {
  missingBytes := 512 - totalFileSize
  copy(sniff[totalFileSize:], chunkMessage.Chunk[:missingBytes])
}

I suggested this shorter version to the AI and asked for its opinion on my code. The AI was very pleased with my solution.

…but when I reran the tests, the scales fell from my eyes. GitHub Copilot is right, I do not copy more than 512 bytes. But in the test case where I have less than 512 bytes, this code does not work. The AI chose an answer to please me and avoided pointing out what I’d done wrong. I ended up with the code below. This is the best of both worlds.

if totalFileSize < 512 {
  missingBytes := 512 - totalFileSize
  remaingBytesInChunk := uint64(len(chunkMessage.Chunk))
  if remaingBytesInChunk < missingBytes {
    missingBytes = remaingBytesInChunk
  }
  copy(sniff[totalFileSize:], chunkMessage.Chunk[:missingBytes])
}

I strongly believe that a software engineer has to write tests. Tests are sometimes hard to write, it is stupid boring work and you have to spend time to maintain them. But like a mother, they secure your life and take care of you. With them, I can sleep like a baby without worries. Now the AI does the same.

Foto von Isaac Quesada auf Unsplash

Sleep well, AI. UnitTest loves and protects you.