DEV Community: Pete King

No Need For Docker Anymore

Pete King — Fri, 12 Jul 2024 15:42:33 +0000

Introduction

No need for Docker anymore I hear many people say, well I'd like to express that Docker is incredibly relevant today and into the future - We are not saying goodbye!

Docker

Docker was and still is a game-changer in software engineering, offering a containerisation approach to application development, deployment and management.

Here's a run-down of its significance and the advantages Docker can bring to you and your team:

Consistent Environments: Docker ensure consistent environments across development, testing, and production stages. By encapsulating applications with their dependencies in isolated containers, Docker eliminates environment-related issues that can plague the development lifecycle.
Streamlined Collaboration: Dev Containers, built on Docker's foundation, provide a standardised development environment for your team. Everyone on the product/project gets an identical container with the necessary tools and libraries, fostering a seamless collaboration and eliminating setup discrepancies.
Improved Portability: Docker applications are inherently portable. Containers can run seamlessly on any Linux machine with Docker installed, regardless of the underlying infrastructure. This simplifies development across various environments from local development machines to cloud platforms.
Faster Development Cycles: With Docker, developers can spin-up new environments in seconds. This agility translates to faster development cycles, as developers can quickly test and iterate on their code without lengthly setup times.
Efficient Resource Utilisation: Containers share the host operating system's kernel, making them lightweight and resource-efficient compared to virtual machines. This allows you to run more applications on the same underlying hardware, optimising resource utilisation.
Simplified Scalability: Scaling Dockerised applications is a breeze You can easily add or remove containers based on demand, enabling horizontal scaling for increased processing power or handling spikes in traffic. This leads onto container orchestration platforms such as Kubernetes.
Enhanced Reliability: Docker containers isolate applications from each other and the host system, promoting stability and reliability. If one container malfunctions, it won't impact other containers or the underlying system.

What's not to like hey? 😁

Learning Curve

I acknowledge that there is a learning curve, but trust me (hopefully you can), the initial investment will pay-off in the long term.

We can mitigate by adopting a strategic approach that focuses on practical experience alongside foundational knowledge.

Start with the Basics: Begin by understanding the core concepts like containers, images, and Docker files. Numerous beginner-friendly tutorials and workshops are available online.
Hands on Learning: The best way to solidify new knowledge and understanding is through practical application. Work on small projects, maybe personal ones using Docker. This will help you grasp how Docker functions in real-world applications.
Leverage Online Resources: The Docker community is extensive and supportive. Utilise online forums, communities and of course the official Docker documentation to find answers, troubleshoot issues, and learn from other peoples experiences.
Focus on Practical Use Cases: Instead of getting bogged down by every minute detail, concentrate on how Docker can address your specific development needs. This will make the learning process more engaging and goal-orientated.

Final Thoughts

Docker, along with Dev Containers, streamlines the software development process by ensuring consistency, portability, efficiency, and scalability. By adopting these technologies, you and your team can feel empowered to deliver high-quality software faster and more reliably.

BONUS!

Containerisation is available in different flavours so-to-speak, however, Docker, along with Docker Desktop and its offering is what I would highly recommend if you are doing anything commercially; if you are just doing something yourself you can get away without it. However, if purely personal and not commercial you can still use Docker Desktop without any cost!

The benefits of Docker Desktop, Docker Hub are immense, when you are using base images from external sources you are relying on trust. That's all well and good, however, you can also trust Docker Inc. i.e. if you enable only Docker Official images and Verified Publisher images, you are additionally trusting Docker Inc. as well. I see this as a good thing!

It's all about securing your software supply chain, every link in that chain you would ideally want to be trusted, verified and more. Docker helps you improve in this area directly. Not only this but Docker Inc, has made some strategic partnerships recently, one to mention is Chainguard. I won't go into much of that here, but in essence Chainguard curates minimal, highly-optimised container images and either reduces the open CVE's or gets the CVE count down to zero! Awesome 😎

One last thing, Docker Inc, has Docker Scout which further improves your software supply chain, there's that key phrase again, it's because it's vitally important and not to be underestimated.

More Information

Docker: https://www.docker.com
Docker Docs: https://docs.docker.com
Docker Desktop: https://www.docker.com/products/docker-desktop/
Docker Hub: https://www.docker.com/products/docker-hub/
Docker Scout: https://www.docker.com/products/docker-scout/
Chainguard: https://www.chainguard.dev/
Kubernetes: https://kubernetes.io/
Nomad: https://www.nomadproject.io/
Udemy, Docker Mastery: https://www.udemy.com/course/docker-mastery/

Engineering Metrics Are Overrated

Pete King — Wed, 03 Jul 2024 11:17:51 +0000

Introduction

Engineering metrics are overrated, I don't think so! I think they are vitally important and a valuable tool for gauging the health and progress of software engineering teams.

While some argue that metrics can be misleading or misused, they provide crucial data points that can inform decision-making and identify areas for improvement.

We all know software development is a complex process with numerous moving parts. Measurement is a key way to quantify these parts.

DORA

DevOps Research and Assessment, DORA for short, these metrics focus on outcomes that directly impact software delivery.

However, DORA metrics are lagging indicators, meaning they reveal information after the fact. To gain more real-time insights into development health, leading indicators are essential.

For some quick insight into DORA and its Core Model, please visit DORA is More Than DORA.

Leading indicators 📈

Here are some leading indicators that we commonly use:

Pull Request (PR) size: Smaller PR's are generally easier to review and merge, leading to fast deployment cycles.
Pull Request (PR) review time: Faster PR review times indicate smoother collaboration and knowledge sharing within the team, which can lead to faster deployment cycles; similar to above.
Code coverage: Measures the percentage of code executed by automated tests. High code coverage indicates a lower likelihood of undetected bugs.
Code churn: Tracks the amount of code that is added, deleted, or modified. Low churn suggests a more stable codebase.
Code complexity: Complex code can be difficult to understand and maintain, leading to higher development costs and increased risk of errors. By monitoring code complexity, teams can identify areas for refactoring to improve code quality.

...and there are many more leading indicators that you can add to your team toolbox to continually monitor and improve 😎

How do I get these metrics?

Well that's the hard yet can be easy part, you could build your own integrations and all, crunch this data and present them, leverage open source, however, I would guess this is not your core business. Therefore, I'd argue purchasing a SaaS platform off-the-shelf is the easy way. You'll just need to evaluate the market, ensure it fits your needs and your toolchain etc. There is some open source out there (Apache DevLake) in this space, I'd encourage you to take a look and see if it suits your needs, wants desires.

Final thoughts

The key to using software engineering metrics effectively lies in selecting the right metrics for the specific product/project goals and context. It's also crucial to avoid relying solely on metrics to make decisions, remember, these are data points. Metrics should be used in conjunction with other factors such as team feedback and code reviews to get a holistic view of the team's development process; i.e. data-driven decisions for continuous improvement.

It's not just about lagging metrics such as DORA, but leading metrics for you and your team to monitor, understand, adjust how you go about software engineering and delivering on agreed targets & outcomes 🎯

More information

DORA is More Than DORA: https://dev.to/peteking/dora-is-more-than-dora-22ic
DORA: https://dora.dev

SaaS Platforms

LinearB: https://linearb.io/
Sleuth: https://www.sleuth.io/
Swarmia: https://www.swarmia.com/
Jellyfish: https://jellyfish.co/
Haystack: https://www.usehaystack.io/
AllStacks: https://www.allstacks.com/
Apache DevLake (Open Source): https://devlake.apache.org/

The above is not an extensive list

DORA is More Than DORA

Pete King — Thu, 27 Jun 2024 15:38:09 +0000

Introduction

DORA you hear me say, what's that, and you may already know? Let's take a brief moment to summarise. Nothing to do with Dory I'm afraid, sorry!

First of all, what does DORA stand for? DevOps Research and Assessment, it's a research programme and more, it seeks to understand the capabilities that drive software delivery and operations performance. The data it gathers through its research programme helps teams apply capabilities, leading to better organisational performance.

It's a big undertaking and DORA reports go back to 2014, useful to see trends as well as understand some history.

DORA started as a team at Google Cloud and focused on understanding DevOps performance by using data; metrics. The ultimate goals was to improve performance and collaboration whilst continuing to drive velocity. These metrics as used as a continuous improvement mechanism, teams can understand their current performance and set goals to progress against them.

4-Key Metrics

If you have heard of DORA, you may have heard of the 4-key metrics.

Deployment frequency
Lead time for changes
Change failure rate
Time to restore service

Deployment Frequency: Measure how often code is successfully released to production.

Lead Time for Changes: Measures the amount of time it takes a code change to be committed to production.

Change Failure Rate: Measures the percentage of deployments that result in a failure.

Time to Restore Service: Measure the time it takes to restore service after a deployment failure.

DORA Core Model

Now we've had a general overview, those who have heard about DORA or have even applied it previously may understand those metrics, the journey usually stops there. Teams focus on the 4-key metrics, track where they are, set targets to achieve; hopefully leads to an increase in overall performance, collaboration and velocity.
You'd think that is it... but it isn't.

DORA is more than DORA!

DORA's research programme formulated a model known as DORA Core.
Their research team applied behavioural science methodology to uncover the predictive pathways which connect ways of working, via software delivery performance, to organisational goals and individual well-being.

You can checkout the DORA Core Model here: https://dora.dev/research/

The DORA Core Model is a collection of capabilities, metrics, and outcomes that represent the most firmly-established findings from across the history and breadth of DORA’s research programme. Core is derived from DORA’s ongoing research, including the analyses presented in their annual Accelerate State of DevOps Reports. Core is intended to be used as a guide in practitioner contexts: it deliberately trails the research, evolving more conservatively. The concepts and relationships shown in the Core Model have been repeatedly demonstrated by their research, and have been successfully used by software engineering teams to prioritise continuous improvement.

This is an interactive diagram, I encourage you to checkout: DORA Core Model

Explanation

What does this actually convey? Well, there are a number of capabilities, mostly technical capabilities on the left. You can see how elements contribute to other elements, for instance Documentation quality predicts a whole range of technical capabilities it impacts, this range of technical capabilities contributes to Shift left security and Continuous delivery.

Continuous delivery helps cultivate a Generative organisational culture, and if and when you have Streamlined change approval you'll be able to predict your Software Delivery Performance (DORA Metrics).

Software Delivery Performance, the 4-key metrics are important but are not the be all and end all, i.e. we are software engineering professionals and we have a great deal to do!

As software engineering professionals we care about all the things on the left, including software delivery performance so much that we can at times forget about outcomes. These are vitally important, whatever software product or software capability we are producing there is always a reason, and these metrics predict it will drastically impact an organisation, whether commercially or non-commercially.

Commercial / non-commercial examples:

Profitability
Productivity
Market share
Number of customers
Quantity of products or services
Operating efficiency
Customer satisfaction
Quality of products or services provided
Achieving organisation or mission goals

Well-being

I like to think that happy people are more productive people, no matter what the profession, software engineering is no exception to this. The well-being of your people, your team is always paramount.

When we have less deployment pain, less rework, less burnout, we have a greater chance of success which leads towards achieving our organisational outcomes. Trace those lines back in the DORA Core Model and you'll see Continuous delivery and Streamlined change approval will likely predict the outcomes of Well-being; follow the lines back and you'll see more signs as per the diagram, clear isn't it?

Final Thoughts

I wanted to keep this somewhat short so you can explore more about the model yourself, the title of this article says a lot, DORA is more than DORA, the misconception around it's just 4-key metrics and sometimes no one else around you may care about those metrics either!

DORA is so much more, the DORA Core Model can really help you understand the landscape better, giving you tools, guidance and advice to ultimately achieve the desired for your organisation is seeking. Furthermore, it doesn't stop with DORA's 4-key metrics and its core model either, there are other metrics that can be somewhat leading indicators that can help any software delivery team improve their performance, collaboration and velocity - I may just cover these in a separate article. Stay tuned!

There is a new DORA Core Model V2 available - https://dora.dev/core-v2/

I hope that by reading this you have learnt a little something and are more curious about DORA.

More Information

DORA - https://dora.dev/
DORA Research (DORA Core Model) - https://dora.dev/research/
DORA Capabilities Catalogue - https://dora.dev/devops-capabilities/
DORA Guides - https://dora.dev/guides/
DORA Community of Practice - https://dora.community/

API's From Dev to Production - Part 11 - Pulumi - IAC

Pete King — Mon, 07 Jun 2021 16:30:20 +0000

Series Introduction

Welcome to Part 11 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

Infrastructure as Code
- Pulumi

TL;DR

We got to grips with IaC and made use of our C# skills to build our own infrastructure - Go shift-left! Pulumi has been a joy to use, just so easy, the integration of GitHub Actions and Pulumi has equally been awesome.

We were able to preview our infrastructure changes from a Pull Request using our CI Workflow and achieved C-Deployment by deploying our infrastructure upon merging into main using our CD Workflow.

The Pulumi GitHub Bot works great and the Pulumi Console is simply invaluable. 🤓

GitHub Repository

peteking / Samples.WeatherForecast-Part-11

This repository is part of the blog post series, API's from Dev to Production - Part 11 on dev.to. Based on the standard .net standard Weather API sample.

Requirements

We will be picking-up where we left off in Part 10, which means you’ll need the end-result from GitHub Repo - Part 10 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Don't forget to ensure you have setup Code Climate Quality with your repository.

If you want to follow along from the code from Part 10, that is really ideal, I have everything in order and it should all work fine. If however, you want to skip to the end and run it all, you can too, but there there are some things you'll need to do.

Clone the GitHub Repo (Part-11) into your own GitHub
Install/Obtain Access to Azure CLI
Install Pulumi CLI
Navigate → ./src/Samples.Weatherforecast.Infra
Follow the instructions in this post about: Create a Service Principal
Follow the instructions in this post about: Securely store Azure Service Principal credentials
Execute → pulumi preview

This should create the Pulumi project and its stack.

Introduction

This post is all about Infrastructure as Code, we want to be more autonomous to build and manage our own infrastructure instead of being fully dependant on others. Infrastructure as Code has been around a while and there are many solutions, each cloud provider such as Microsoft Azure, AWS, GCP and more all have their own native solutions, and of course the number one name for some time has been Hashicorp's Terraform (OSS, Cloud, Enterprise). However, they all have something in common... they are all in their own languages, from declarative to DSL's and JSON. Using DSL's can be cumbersome as is learning a new language; however, a purpose built language could have some advantages. Pulumi is one of those solutions that takes a different angle, the ability to get to your end goal of having your infrastructure written as code but in a language you use all the time.

In this post we will dive head-first into, Pulumi!

Let's get started. 🤘

Microsoft Azure

Get access to Azure, this is kind of mandatory, we will be using Microsoft Azure - If however, you don't have an account, you can get one for FREE! 😁

Sign-up to Microsoft Azure.

Install/Obtain access to Azure CLI

Azure CLI on your OS

Azure CLI in Docker

Azure Cloud Shell in BASH

If you already have chocolately installed, you can simply execute:

choco install azure-cli

As long as you can get access, that's all we need, so use whichever method you prefer.

Pulumi

What is Pulumi?

Pulumi is a modern infrastructure as code platform built for engineers; developers and infrastructure.

The beauty of the product is you can build, deploy and manage your applications and infrastructure using languages and tools you are familiar with. Follow your existing engineering practices and guidelines, and take advantage of the same toolchain you use today.

This means if you major in C# for instance, you have all the power of the C# language and all your existing tools. It means you don't need to learn a bespoke language just to manage and provision your infrastructure = Learning curve is minimised.

When you deploy applications, everyone can work together, or you can choose to be isolated; maybe I'm trying something out or I have something very unique. You can work together and share infrastructure packages no matter what language it is written in, deploy with confidence by validating beforehand and automate it all!

If that wasn't enough, with Pulumi you can use policies to provide guardrails for teams, allowing them to feel empowered with autonomy (shift-left), and with the Pulumi Console you can see a full audit history. This is known as Policy as Code.

Pulumi is a breath of fresh air compared to other solutions, now I know what you're thinking, Terraform... Yes, I love it too, and I've worn that T-shirt, it's a good one, everyone has their go-to favourite T. However, I have a preference for Pulumi, its simplicity by using the same language I'm building API's in, whether that is C#, Go, JS/TS, Python etc. The Terraform T-shirt is now washed, ironed, and in the cupboard for a while. Luckily Pulumi can connect to Terraform also, so even if you have a centralised Platform team building foundational infrastructure, initial virtual networking topology, firewalls, SIEM etc., engineering teams can shift-left to build and manage their own components whilst using the outputs of Terraform. If for example I needed the output of a VNET from Terraform, Pulumi can get that and then I can provision my product using Pulumi as a pretty-well autonomous engineering team.

Sign-up

Pulumi has a free tier and they say it's free forever (for individual use) 😁

Sign-up to Pulumi.

Pulumi CLI

Pulumi is controlled primarily using its command line interface (CLI). It works with the Pulumi service to deploy changes to cloud apps and infrastructure. It also keeps a history of who updated what in your team and when; which is awesome! The CLI has been designed for productivity and ultrafast feedback, in addition to the ability to use it in CI/CD scenarios.

Install Pulumi CLI.

(Again) If you already have chocolately installed, you can simply execute:

choco install pulumi

Requirements check

✅ Everything we've done previously, i.e. Part 10 etc.
✅ Azure account
✅ Azure CLI - Version used v2.24.0
✅ Pulumi account (Console)
✅ Pulumi CLI - Version used v2.14.0

Let's get started with Pulumi!

Fire-up your console of choice and navigate to the src directory of the WeatherAPI.

Create a folder called, Samples.WeatherForecast.Infra and navigate to it

mkdir Samples.WeatherForecast.Infra
cd Samples.WeatherForecast.Infra

Inside the infra directory - Create a new Pulumi Project.

pulumi new azure-csharp

From here, you'll be prompted by the Pulumi CLI for some values.

project name: Samples.WeatherForecast.Infra
project description: Infra for WeatherForecast
stack name: dev
azure-native:location: The Azure location to use: uksouth

I've chosen UK South given my location, for you, please choose whatever suits your needs best.

ℹ️ TIP
If you don't know what locations are available, below is a very quick & easy way to find out.

az login

# Once login successful, execute
az account list-locations --output table

Open VS Code by executing code .

You should see a nice, brand new Pulumi C# program! 😄

If you also login to the Pulumi Console, you should see your project there too!

Our API Infrastructure

Steps

Step 1

Open VS Code for the WeatherForecast-API if you haven't already, and Navigate → /src/WeatherForecastApi-Infra/

Step 2

Rename MyStack.cs to AzureStack.cs

Step 3

Inside AzureStack.cs, replace all code with the following

using Pulumi;
using Pulumi.AzureNative.Resources;
using Pulumi.AzureNative.Web;
using Pulumi.AzureNative.Web.Inputs;

class AzureStack : Stack
{
    public AzureStack()
    {
        var config = new Pulumi.Config();

        // Obtain our docker image from config
        var dockerImage = config.Require("docker-image");

        // Resource Group
        var rg = new ResourceGroup("rg-weatherforecastapi-uks-");

        // AppService Plan
        var appServicePlan = new AppServicePlan("appplan-weatherforecastapi-uks-", new AppServicePlanArgs() {
            ResourceGroupName = rg.Name,
            Location = rg.Location,
            Kind = "Linux",
            Reserved = true,
            Sku = new SkuDescriptionArgs
            {
                Name = "B1",
                Tier = "BASIC"
            },
        });

        // WebApp for Containers
        var app = new WebApp("app-weatherforecastapi-uks-", new WebAppArgs() 
        { 
            ResourceGroupName = rg.Name,
            ServerFarmId = appServicePlan.Id,
            SiteConfig = new SiteConfigArgs
            {
                AppSettings = new[] 
                { 
                    new NameValuePairArgs
                    {
                        Name = "WEBSITES_ENABLE_APP_SERVICE_STORAGE",
                        Value = "false"
                    },
                    new NameValuePairArgs
                    {
                        Name = "DOCKER_REGISTRY_SERVER_URL",
                        Value = "https://ghcr.io"
                    },
                    new NameValuePairArgs
                    {
                        Name = "WEBSITES_PORT",
                        Value = "8080" // Our custom image exposes port 8080. Adjust for your app as needed.
                    }
                },
                AlwaysOn = true,
                LinuxFxVersion = $"DOCKER|{dockerImage}"
            },
            HttpsOnly = true
        });

        this.Endpoint = Output.Format($"https://{app.DefaultHostName}/weatherforecast");
        this.HealthEndpoint = Output.Format($"https://{app.DefaultHostName}/health"); 
    }

    [Output] public Output<string> Endpoint { get; set; }

    [Output] public Output<string> HealthEndpoint { get; set; }
}

I appreciate there is a chunk of code there, so let's break it down just a little.

The first couple of lines is simply getting the config from Pulumi. I have chosen the Require method because I would rather it fails miserably if the config setting is not present; without it, we shouldn't continue because we'll have no container image. 😆

Up next is the resource group creation, note that we do not actually specify its location - The location comes from the config file - Pulumi.dev.yaml, it's value is from, azure-native:location. 😉

There are a bunch of other Azure native settings, for more information on these, please see, Azure Native Configuration

Following-on from there is the creation of the Azure AppService Plan, this is essentially our underlying Virtual Machine (VM), I have picked the smallest available WebApp for Containers (Linux) I can; so if you are paying for this I recommend when you test, to destroy afterwards - Still, if you didn't, it would be a under GBP£10 per month anyway.

Then we have the WebApp itself, this is the AppService, there are a few things we need to set here, mainly stating where the Docker registry is, in our case, GitHub Container Registry - Please note that the default is always DockerHub, you can easily point it to ACR (Azure Container Registry) also. We set AlwaysOn and HttpsOnly to true and we of course need to set the Docker Image, this is the, LinuxFxVersion property.

We need a way to output values, we want to output the address of the host as the end of the resource names will be dynamically generated. We set one for the weatherforecast and just to be helpful we set one for the health endpoint too.

Finally, the last 2 lines of the file are the declarations of the output properties, what denotes these as outputs is simply the 'Output' attribute.

Step 4

Modify the Pulumi.dev.yaml file.
Replace its contents with the below:

config:
  azure-native:location: uksouth
  Samples.WeatherForecast.Infra:docker-image: ghcr.io/peteking/samples-weatherforecast-part-11:3dae90cc47eff6861687444a6e91be89a65bed9f

We've added our own config here which is our docker image, therefore, it's configuration instead of C# code.

This should be the location of your Docker image.

If you're using the GitHub Container Registry, it should be a simple swap.

Step 5

We have enough to give it a good manual "interactive" test with the Pulumi CLI, just to make sure everything is working as we expect it to.

Open your terminal and Navigate → /src/WeatherForecastApi-Infra/

Execute:

pulumi up

Here you can see it has executed preview as part of the up command, it highlights what it plans to do, in this case, it will create 4 resources.

Select → yes

Once you've selected yes, Pulumi will go-ahead and create those resources, below is the response.

You can see the Pulumi outputs we created, one that links to the weatherforecast endpoint and one to the health endpoint; we've done this because the resources are dynamic, i.e. at the end of our Azure Resources is a random string to ensure uniqueness. If this doesn't suit your use case, you can ensure consistency instead - For more information, see Pulumi - Autonaming

We now have the opportunity to test our cloud infrastructure as code deployment and see if our API works!

You can use the links from the output to save typing.

The weather forecast gets returned! 🎉

Even the /health endpoint is reporting good news! 🎆

If you want to minimise your costs 💲, you can of course destroy 💥 your resources, and with the powers of Pulumi that is also a breeze with the, destory command.

Execute:

pulumi destory

Select → yes

Once you've selected yes, Pulumi will go-ahead and destroy those resources, below is the output. 👍

ℹ️
You'll notice the output at the bottom stating the stack is still available - All this means is you have all the details, history and everything about your stack in the Pulumi Console; which is all rather helpful, don't you think?

GitHub Actions with Pulumi

All of what we have achieved is great, we have got everything we need infrastructure wise, built it all as code, we've even deployed and tested it for real. However, it's all rather interactive, we need to bake this into both our CI Workflow and our CD Workflow.

Our CI Workflow will execute pulumi preview and utilise the Pulumi GitHub Bot for instant feedback in the Pull Request; by doing this, the person reviewing the PR will be able to see what resources will be created/destroyed/updated - If anything is a miss then you have the opportunity to do something about it.

Our CD Workflow will execute pulumi up and deploy our resources to Azure; but only once all our tests pass, code coverage pass and our container scan is all good too.

In order to interact with Azure in GitHub Actions we will create a Service Principal.

Azure Service Principal

What is it?

A Service Principal is an object in a single tenant or directory in Azure AD. It is the identity of an application instance, it defines access for the application and what resources it can access. Instead of using a user identity, we can create a service identity and manage its security and access.

Create Azure Service Principal

Step 1

Create an, Azure Subscription (if you don't already have one).

Step 2

Obtain the subscription id:

az account list #It's the 'id' field.

Step 3

Set the subscription:

az account set --subscription=<id> #<id> is the 'id' field from above

Step 4

az ad sp create-for-rbac --name sp-weather-api

You should see a result from Step 4 with the following values

{
  "appId": "WWWWWWWW-WWWW-WWWW-WWWW-WWWWWWWWWWWW",
  "displayName": "ServicePrincipalName",
  "name": "http://ServicePrincipalName",
  "password": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "tenant": "YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY"
}

ℹ️ TIP
If you'd rather it in another format to save time later, you can use the --sdk-auth parameter.

Securely store Azure Service Principal credentials

Now we have the Service Principal, we need to be able to store the details securely.

We can store these in GitHub Secrets which is totally fine right, but that means they are not linked to our Pulumi Stack, so another way is to store them inside Pulumi; which can also store secrets

From the output of our Service Principal creation command, the following values can be mapped like so:

Service Principal Value	SDK Name
appId	clientId
password	clientSecret
tenant	tenantId

This is where the --sdk-auth parameter can help, it will automatically map these for you.

Pulumi Secrets ㊙️

By storing these in Pulumi we get multi-user access for deployment which includes using it interactively; if we stored them in GitHub Secrets, we can only use them directly from the pipeline itself; there are pros and cons this of course - Choose what suites you best.

pulumi config set azure-native:clientId "WWWWWWWW-WWWW-WWWW-WWWW-WWWWWWWWWWWW"

pulumi config set azure-native:clientSecret "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" --secret

❗ Important
Note the --secret at the end of the command, this is very important; otherwise, the value will be stored in plane text, whereas we want it to be encrypted and stored as a secret.

pulumi config set azure-native:tenantId "YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY"

pulumi config set azure-native:subscriptionId "ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ"

Once you have executed these commands, you'll notice your Pulumi.dev.yml file has been updated accordingly.

CI Workflow

Let's modify our CI Workflow ./github/workflows/ci-pull-request.yml

Add a new job at the bottom of the Workflow.

infra:
  needs: ci
  runs-on: ubuntu-latest

  steps:
    - name: Checkout repo
      uses: actions/checkout@v2
      with:
        # Disabling shallow clone is recommended for improving relevancy of reporting
        fetch-depth: 0

    - name: Install Pulumi CLI
      uses: pulumi/action-install-pulumi-cli@v1.1.0

    - name: Set Pulumi config values
      working-directory: ./src/Samples.WeatherForecast.Infra
      env:
        PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
      run: |
        pulumi stack select dev
        pulumi config set Samples.WeatherForecast.Infra:docker-image ${{ env.image-name }}

    - name: Pulumi Preview
      uses: pulumi/actions@v3.1.0
      with:
        command: preview
        stack-name: dev
        work-dir: ./src/Samples.WeatherForecast.Infra
        github-token: ${{ secrets.GITHUB_TOKEN }}
      env:
        PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

Where do I get the Pulumi Personal Access Token from?

Navigate → Pulumi Console - https://app.pulumi.com
Navigate → Your profile (top-right)
Click → Access Tokens - Quick link here: https://app.pulumi.com/YOUR_PULUMI_USER_NAME/settings/tokens
Click → Create token
Copy the secret generated
Safely store the secret in GitHub Actions Secrets with the name of PULUMI_ACCESS_TOKEN

Install the Pulumi GitHub Bot

Unless you want to dive into the Actions log to find out how the pulumi preview went, you better install the Pulumi GitHub Bot

Step by step guide on Pulumi Docs

Once you have the bot installed, when you initiate a PR, you'll see something like the below:

Even better though, is you can view more details in the Pulumi Console like so:

CD Workflow

Let's modify our CD Workflow ./github/workflows/build-and-publish.yml

Add a new job at the bottom of the Workflow.

infra:
  needs: build-and-push
  runs-on: ubuntu-latest

  steps:
    - name: Checkout repo
      uses: actions/checkout@v2
      with:
        # Disabling shallow clone is recommended for improving relevancy of reporting
        fetch-depth: 0

    - name: Install Pulumi CLI
      uses: pulumi/action-install-pulumi-cli@v1.1.0

    - name: Set Pulumi config values
      working-directory: ./src/Samples.WeatherForecast.Infra
      env:
        PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
      run: |
        pulumi stack select dev
        pulumi config set Samples.WeatherForecast.Infra:docker-image ${{ env.image-name }}

    - name: Pulumi Up
      uses: pulumi/actions@v3.1.0
      with:
        command: up
        stack-name: dev
        work-dir: ./src/Samples.WeatherForecast.Infra
        github-token: ${{ secrets.GITHUB_TOKEN }}
      env:
        PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

Commit your code

It's time to commit you code, but before you do, let's update our branch protection rule - Now we have an infra job in our Workflow which does a pulumi preview, we most likely want this to be mandatory and pass.

Now that we have our branch protection rules updated and CD Workflow file completed, let's ensure it all works.

Raise your PR and see the CI Workflow kick-off.
Verify everything is fine by looking at the Pulumi GitHub Bot or the Pulumi Console.
Wait until all GitHub Status Checks pass and merge your PR into main.
Verify your Workflow is successful.
Either:
- Go to the Pulumi Console and see your Activity.
- OR see the output from the infra job in the CD Workflow.
Try the output links to verify your API is running.

I prefer the Pulumi Console, see below, you can see the merge of my branch and the creation of the infrastructure once it was merged into main.

I was then able to validate that both the weatherforecast and health endpoints was functional.

What have we learned? 🤔

We have learned all about IaC - Infrastructure as Code, and utilised Pulumi to achieve this - We have leveraged our existing C# skills to shift-left on our infrastructure; therefore, we are now more autonomous.

We've been able to test (preview) and verify our infrastructure changes during Pull Requests as part of CI, and we have achieved C-Deployment by releasing our infrastructure in our CD Workflow.

Relief - That's it! 😌

If you've made it this far, congratulations and thank you 🤝, I know it's a long post and a lot to take in, but I hope you have found it worthwhile.

There is unfortunately or fortunately, depending on how you look at it, still a lot to do 😏 So there will be a more coming soon! 🤪

UPDATE AUGUST 27TH 2021
The below GitHub Issue has now been resolved.

📄 Note
Adding Pulumi to your repo in GitHub will break CodeQL, ~~there is an active GitHub issue open for this.~~

Process failed with exit code 100 #544

peteking posted on Jun 03, 2021

Hi,

I have failure in the codeql-action, it has failed with an exit code of 100, looking at the docs it says it's likely a bug based on this error code.

This is an open source sample project, so all open.

The recent addition made as you can see from the PR is I've added Pulumi for its Azure infrastructure provisioning.

I think it start to look at Pulumi here:

  Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp...
  Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp...
  Pulumi.AzureNative.dll.trap.gz, 9885761: java.lang.OutOfMemoryError: Java heap space

and then slowly goes wrong and fails, out of memory.

Here is the codeql workflow run here: https://github.com/peteking/Samples.WeatherForecast-Part-11/pull/1/checks?check_run_id=2738620747

Part of the log is here for quick reference:

Finalizing csharp
  /opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/codeql database finalize --threads=2 /home/runner/work/_temp/codeql_databases/csharp
  Running pre-finalize script /opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/csharp/tools/pre-finalize.sh in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11.
  Running pre-finalize script /opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/csharp/tools/pre-finalize.sh in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11.
  [2021-06-03 15:44:36] [build-stderr] Scanning for files in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11...
  [2021-06-03 15:44:36] [build-stderr] Scanning for files in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11...
  [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Indexing files in in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11...
  [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Indexing files in in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11...
  [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Running in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11: [/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/xml/tools/index-files.sh, /home/runner/work/_temp/codeql_databases/csharp/working/files-to-index16194344738812882790.list]
  [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Running in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11: [/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/xml/tools/index-files.sh, /home/runner/work/_temp/codeql_databases/csharp/working/files-to-index16194344738812882790.list]
  Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp...
  Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp...
  Pulumi.AzureNative.dll.trap.gz, 9885761: java.lang.OutOfMemoryError: Java heap space
  com.semmle.inmemory.trap.TrapScanner.getTokenString(TrapScanner.java:521)
  com.semmle.inmemory.trap.TrapScanner.getLabelValue(TrapScanner.java:578)
  com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:754)
  Pulumi.AzureNative.dll.trap.gz, 9885761: java.lang.OutOfMemoryError: Java heap space
  com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:495)
  com.semmle.inmemory.trap.TrapScanner.getTokenString(TrapScanner.java:521)
  com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:439)
  com.semmle.inmemory.trap.TrapScanner.getLabelValue(TrapScanner.java:578)
  com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:376)
  com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:754)
  com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:172)
  com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:495)
  com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:129)
  com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:439)
  com.semmle.inmemory.trap.ImportTasksProcessor$$Lambda$234/0x000000010031fc40.accept(Unknown Source)
  com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
  com.semmle.util.concurrent.FutureUtils$$Lambda$236/0x000000010031f440.get(Unknown Source)
  java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
  java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  java.base/java.lang.Thread.run(Unknown Source)
  com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:376)
  com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:172)
  com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:129)
  com.semmle.inmemory.trap.ImportTasksProcessor$$Lambda$234/0x000000010031fc40.accept(Unknown Source)
  com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136)
  com.semmle.util.concurrent.FutureUtils$$Lambda$236/0x000000010031f440.get(Unknown Source)
  java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
  java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  java.base/java.lang.Thread.run(Unknown Source)
  Oops! A fatal internal error occurred.
  Oops! A fatal internal error occurred.
  java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
  java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
    at com.semmle.util.exception.Exceptions.asUnchecked(Exceptions.java:103)
    at com.semmle.inmemory.trap.TrapImporter.run(TrapImporter.java:105)
    at com.semmle.cli2.ql.dataset.ImportCommand.executeSubcommand(ImportCommand.java:98)
    at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
    at com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159)
    at com.semmle.cli2.database.FinalizeCommand.executeSubcommand(FinalizeCommand.java:110)
    at com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:448)
    at com.semmle.util.exception.Exceptions.asUnchecked(Exceptions.java:103)
    at com.semmle.inmemory.trap.TrapImporter.run(TrapImporter.java:105)
    at com.semmle.cli2.ql.dataset.ImportCommand.executeSubcommand(ImportCommand.java:98)
    at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
    at com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159)
    at com.semmle.cli2.database.FinalizeCommand.executeSubcommand(FinalizeCommand.java:110)
    at com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:448)
    at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201)
    at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209)
    at com.semmle.cli2.CodeQL.main(CodeQL.java:96)
  Caused by: java.lang.OutOfMemoryError: Java heap space
    at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201)
    at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209)
    at com.semmle.cli2.CodeQL.main(CodeQL.java:96)
  Caused by: java.lang.OutOfMemoryError: Java heap space
  Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/codeql' failed with exit code 100
  Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/codeql' failed with exit code 100
      at Object.toolrunnerErrorCatcher (/home/runner/work/_actions/github/codeql-action/v1/lib/toolrunner-error-catcher.js:79:19)
      at processTicksAndRejections (internal/process/task_queues.js:93:5)
      at async Object.finalizeDatabase (/home/runner/work/_actions/github/codeql-action/v1/lib/codeql.js:414:13)
      at async finalizeDatabaseCreation (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:74:9)
      at async Object.runAnalyze (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:172:5)
      at async run (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:50:30)
      at async runWrapper (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:96:9)

View on GitHub

Next up

Part 12 in this series will be about:

More Infrastructure as Code with Pulumi
- Environmental support (multiple Stacks)
- Security
  - What can we do on the cheap in Azure?
  - What should we do in a more real scenario?
GitHub Actions
- Clean-up a little and add Environment capabilities

More information

API's From Dev to Production - Part 10 - CodeQL

Pete King — Fri, 21 May 2021 15:08:05 +0000

Series Introduction

Welcome to Part 10 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

SAST - Static Application Security Testing is used to secure software by reviewing/scanning the source code of the software to identify sources of vulnerabilities.

TL;DR

Static Application Security Testing is a very important step in the journey from dev to production. With an abundance of tools out there on the market, there's bound to be one that suits you. With GitHub CodeQL, we can achieve this in little time with little effort.

You can increase your chances of finding security vulnerabilities before they reach production

GitHub Repository

peteking / Samples.WeatherForecast-Part-10

This repository is part of the blog post series, API's from Dev to Production - Part 10 on dev.to. Based on the standard .net standard Weather API sample.

Requirements

We will be picking-up where we left off in Part 9, which means you’ll need the end-result from GitHub Repo - Part 9 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Don't forget to ensure you have setup Code Climate Quality with your repository.

Introduction

As we have discussed in this blog post series and in shift-left, security testing is part of actually shifting-left! We need to carry out security testing, and of course we want to automate it, and ensure its early on in the process. There are many tools out there in the market that can scan your code (statically); the newest entrant is from an acquisition of Microsoft's GitHub - CodeQL.

In this post we are going to go through the steps on how to add SAST using CodeQL to our API.

A bit of history about CodeQL

GitHub acquired Semmle in 2019 for an undisclosed sum, it originally came out of research completed at Oxford University.

The technology is based around queries and its language CodeQL, the analysis engine enables the query of large codebases to identify code patterns and search for vulnerabilities and their variants.

GitHub has now baked this into its platform and has simply made it oh so easy 😁

Why is SAST so important?

SAST makes discovering common vulnerabilities simple, and more critically, automated. SAST is a really good way of improving overall good-practice among your engineers.

Usually engineers outnumber security staff and therefore, it can be rather challenging for any company to find the people to perform code reviews, specifically those with a focus on security. SAST tools provide the ability to analyse 100% of the codebase and are certainly faster than manual code reviews. These tools can easily scan millions of lines of code in a matter of minutes, and automatically identify critical vulnerabilities; such as SQL injection, cross-site scripting, and more with a high level of confidence.

SAST helps integrate security into the early stages of the software development lifecycle and enables the culture of shifting-left; to detect vulnerabilities in the proprietary code in the design stage or the coding stage when they are relatively easier to mitigate.

Step 1

Navigate to your GitHub Repo → Security → Code Scanning Alerts → Click Set up this workflow

Step 2

This will generate a new GitHub Actions Workflow file called, codeql-analysis.yml - You're free to change the name, but I've kept it the same.

The intelligent thing here is that GitHub will detect the languages inside your repo, in our case, it's picked-up C#; which is of course correct 🤓

One thing to note is the cron, for me, it did generate an invalid value according to GitHub. The last digit is the day of the week, 0 = Sunday; for some reason GitHub doesn't like Sunday... In this case, I've changed it to the below, whereby I've set the day of the week to 1 = Monday.

schedule:
    - cron: '44 23 * * 1'

For more information about the crontab format, please see here

Commit your changes and head on over to the Actions tab to see it running.

Here you can see I have a new workflow on the left and it has executed in 4 minutes and 4 seconds!

You'll notice it has done an autobuild; based on its language detection, CodeQL attempts to build your code, its image has loads of the .NET SDK's to cover everything.

It is worth looking through the build output, even just to get a general understanding on what is going on here.

📄 Autobuild
This means it will build our API twice, yes, twice! Is that OK?
Not ideally but... I have no issues with it at present, it will build in parallel to our container build, the key is to scan the code, anything built is discarded anyway, we don't need it, just the results.

It is not optimal given we are using Docker containers, however, there is a way to bake this into Docker, but the documentation is a little light at the moment and requires a decent deep-dive into it - I will most likely investigate it soon, and of course, create a blog post about it.

For more information about CodeQL in containers, please see Running CodeQL code scanning in a container

Step 3

We can configure a few things that are not enabled by default such as additional queries

I have left the default queries and listed some extensions, in our case, I'm adding extended security and security and quality queries, and of course C# 👍

This configuration simply goes into a yaml file.

Create a folder called, codeql under the .github folder.
Copy & paste the following code into a new file called, codeql-config.yml - no need to commit your changes yet, as there is another change in Step 4.

Code
codeql-config.yml

name: "default"

languages: csharp

queries:
  - name: Extended Security
    uses: security-extended
  - name: Security and Quality
    uses: security-and-quality

Step 4

We need to reference our new CodeQL configuration file in our Workflow.

In the Initialize CodeQL Workflow Step → Add a new line to its with clause like so below:

Code
codeql-analysis.yml

config-file: ./.github/codeql/codeql-config.yml

Step 5 (Extra)

If you're like me and really like branch protection rules and GitHub Status Checks, you'll want to create/modify your branch protection rule to have both your workflows as required to pass before merging.

Finally, commit your changes.

Step 6

If you navigate back to Code scanning alerts under Security, hopefully you'll see no alerts!

Here, other tools can publish their results too, by supporting the open SARIF standard.

Which means even if you expand your toolset later, you can have the same GitHub native experience! Pretty cool right?

Step 7 (Extra)

While we are here in the Security tab, I applaud you to ensure you have Dependabot alerts on.

What have we learned?

We have very quickly enabled our API to be scanned by GitHub CodeQL, a SAST analysis engine. It's easy and provides a single pane for you to see any security vulnerabilities in your code. If you end up using another product, these products can push their results into the GitHub Security area of your repository; so long as they integrate with GitHub and the open SARIF standard.

We understand that it will build our API again (but in parallel), but this is not a huge, big deal, as we are just after the results, not the build from that workflow.

Security should always be part of your engineering process.

Dynamic Application Security Testing is another form of security testing, the key point is it being dynamic VS static - We won't cover this here, but there is value to be had with the combination of SAST and DAST. I may cover it in another post in the future 😉

Next up

Part 11 in this series will be about:

Infrastructure as Code
- Pulumi

More information

API's From Dev to Production - Part 9 - SCA

Pete King — Fri, 07 May 2021 10:59:17 +0000

Series Introduction

Welcome to Part 9 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

Static Code Analysis (SCA)

TL;DR

Static Code Analysis is an important step in the journey from dev to production; and continuing into other cycles such as feature development and more. With an abundance of tools out there on the market, there's bound to be one that suits you. CodeCov.io proved useful, but only for code coverage, by taking the next step and using SCA as well as code coverage, your API products will reap the benefits.

Quality should never take a back seat when building mission critical systems.

GitHub Repository

peteking / Samples.WeatherForecast-Part-9

This repository is part of the blog post series, API's from Dev to Production - Part 9 on dev.to. Based on the standard .net standard Weather API sample.

Requirements

We will be picking-up where we left off in Part 8, which means you’ll need the end-result from GitHub Repo - Part 8 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Introduction

We are looking at Static Code Analysis, another must have when we are going from development to production. These days there are a great number of tools out there from self-hosted to SaaS, for me, I'm more comfortable with less to manage and have opted for a SaaS solution; this certainly doesn't mean you have to. I highly recommend you checkout what is on the market and what will suite your needs.

We will be waving goodbye to CodeCov, there was a nasty breach and you can read more about that here and here. However, the move away is not solely because of that reason, the main reason is the SaaS solution we are going to integrate with provides SCA as well as code coverage :)

We will be integrating with Code Climate Quality

Let's get started!

Step 1

Head to their website → Navigate to their Quality product and click → Get Started. This will register Code Climate Quality as a GitHub App.

Once you have complete that journey, you'll be presented with the below - A list of all you public repositories.

Step 2

Add the repo of your choice, in our case, this will be, Samples.WeatherForecast-Part-9.

As soon as you select your chosen repository, Code Climate Quality will starting doing its thing, just wait, have a cuppa tea :)

Once Code Climate Quality has completed its build and scan etc. you'll be informed about it with a nice little pop-up :)

Step 3

Navigate to your repository dashboard.

Checkout the stats - They look pretty good so far!

You'll notice the maintainability rating is currently an A. However, there is no code coverage!

Step 4

Let's fix the code coverage.

Now, Code Climate Quality states in their documentation or technically speaking, doesn't state that C# is supported... Sad face :(

However, let's not give up so quickly, looking at the supported code coverage formats, we should be...covered :D

As we have learnt even in this blog series, lcov is what we are currently using, and we can also generate Cobertura as well; these are supported formats.

We need to feed Code Climate Quality the lcov file, just like we did with our departed CodeCov (don't forget to remove CodeCov from your GitHub Workflows).

Step 4.1

Navigate to Code Climate Quality → Your repo → Repo Settings → Test Coverage.

You'll see your TEST REPORTER ID, we need to put this into our GitHub Secrets area.

You should know how to do this by now, put it into your secrets for your repo and call it, CC_TEST_REPORTER_ID.

Step 4.2

Let's add a new step to our GitHub Action CI Workflow.

Either remove CodeCov.io or before/after the job step, place the following code:

- name: Code coverage [Code Climate]
  uses: paambaati/codeclimate-action@v2.7.5
  env: 
    CC_TEST_REPORTER_ID: ${{secrets.CC_TEST_REPORTER_ID}}
  with:
    coverageLocations: ${{ github.workspace }}/path/to/artifacts/testresults/coverage.info:lcov
    prefix: /code/

We have the environment set to our test reporter id so it know which repo in Code Climate Quality to link it to.

We've specified the coverage file, please note that there is a, :lcov at the end to state that it is indeed an lcov file.

The prefix parameter is purely required because of how we have built with Docker.

Prefix can refer to sub-commands of the test reporter.

We need to use this simply because when we compiled our code inside our Dockerfile, we copied to a local directory inside Docker, we, or more like I, chose to copy to, /code by setting the working directory like so, WORKDIR /code, and then copying the code and building. This means all the paths inside the lcov file will point to /code. However, inside the GitHub repository, that directory does not exist.

Very luckily (and with some trial and error on my part), prefix: /code/ resolves this issue :)

Step 5

Make any commit, or simply trigger the CI workflow.

Head on over to Code Climate Quality and you should have code coverage stats in your dashboard like so:

Step 6

Code Climate Quality has a GitHub Bot.

Let's set this up...

Click → Set up (Pull request comments)

Install the GitHub App...

Confirm the permissions you've set...

Select → Post additional inline comments on new issues

Toggle On → Inline issue comments

Step 7

Let's test out Code Climate Quality by making some obvious issues and see how it handles it, plus we can test out the developer experience.

Step 7.1

Modify the, WeatherForecastController.cs
Change → Get() method...

[HttpGet]
public IEnumerable<WeatherForecast> Get()
{
    var rng = new Random();

    var temps1 = Enumerable.Range(1, 5).Select(index => new WeatherForecast
    {
        Date = DateTime.Now.AddDays(index),
        TemperatureC = rng.Next(-20, 55),
        Summary = Summaries[rng.Next(Summaries.Length)]
    })
    .ToArray();

    var temps2 = Enumerable.Range(1, 5).Select(index => new WeatherForecast
    {
        Date = DateTime.Now.AddDays(index),
        TemperatureC = rng.Next(-10, 45),
        Summary = Summaries[rng.Next(Summaries.Length)]
    })
    .ToArray();

    return Enumerable.Concat(temps1, temps2);
}

All we've done here is duplicate the amount of responses, but it's slightly different. Take note of the TemperatureC value in each. Hopefully this will be caught by Code Climate Quality as a duplication.

Step 7.2

For the next modification, let's change the WeatherForecast.cs file.

Add → New property called, TemperatureFahrenheit.

public int TemperatureFahrenheit  => 32 + (int)(TemperatureC / 0.5556);

This is identical, except for the property name, hopefully Code Climate Quality will pick this up as a duplication also.

Step 8

Raise a PR for all of your changes and with your new CI workflow you should see some action taken by Code Climate Quality...

As you can see from the image above, the Code Climate Quality GitHub Bot has made inline comments as it found a duplication, whoohoo!

However, don't jump for joy just yet, as this is the only duplication found, it seems it has not picked-up on the property body duplication :(

Step 9

Let's enable GitHub Status Checks.

Navigate to Code Climate Quality → Your repo → Repo Settings → GitHub → Pull request status updates → Click Edit → Toggle On

Back in GitHub, head over to your repository → Branches → Branch Rule → Check the below items so GitHub knows what required:

A handy thing in Code Climate Quality is you can see issues within its product itself, not just GitHub, so you have 2 views to look at.

By resubmitting the PR, you will see the GitHub Status Checks shine - Note I still have CodeCov.io in there, but it will be gone in subsequent posts.

Given the status checks are true/false - Code Climate Quality has flagged the duplication issues and now I need to approve this impact to the codebase. For this example, I will approve so we can see what happens.

This is a key moment to take note of, by using this workflow we can easily assess the impact and approve or deny, keeping track of codebase health has now never been more simple :)

Codebase Impact

As we expected, we have made things worse! We have duplicated code, our maintainability has gone down, therefore, our rating has dropped from A to B; naturally our test coverage has dropped down as well.

Technical Debt View

A nice view in Code Climate Quality is the Technical Debt graph, it needs to be taken with a pinch of salt in terms resolution time and all, however, I do see how this can be useful from time to time.

What have we learned?

We have learned about why Static Code Analysis is important and how an optimised workflow using SaaS tools such as Code Climate Quality can impact an engineering teams' daily lives. This is part of the journey to production, releasing without knowing elements such as maintainability and code coverage can so easily create issues down the line, especially with missions critical systems.

Understanding maintainability and test coverage are vitally important to the quality of the products engineering teams' produce.

CodeCov.io has now been removed.

Next up

Part 10 in this series will be about:

SAST - Static Application Security testing is used to secure software by reviewing the source code of the software to identify sources of vulnerabilities.

More information

API's From Dev to Production - Part 8 - Status Checks

Pete King — Thu, 01 Apr 2021 14:58:15 +0000

Series Introduction

Welcome to Part 8 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

More code coverage
- GitHub Status checks

TL;DR

We take advantage of GitHub Status Checks, ensuring our codebase doesn't decrease against our predefined code coverage target; blocking PR's (pull-requests) from being merged. We set a target of 80% knowing our current code coverage is only 36%, we take decisive action to exclude classes from coverage where in this case we believe it is safe. We integrate it all together step by step and achieve our desired outcome of a new CI workflow on PR's with this code coverage gate, and boost code coverage to 95%.

GitHub Repository

peteking / Samples.WeatherForecast-Part-8

This repository is part of the blog post series, API's from Dev to Production - Part 8 on dev.to. Based on the standard .net standard Weather API sample.

Introduction

Code coverage is an important metric, out of context it can be a disaster metric, if you are collecting code coverage for a repository, ensuring you have an agreed target in the engineering team and protecting your codebase is of upmost importance.

How do we ensure there is a target, how to ensure our codebase doesn't get worse with every PR (pull-request)?

We can take advantage of GitHub Status Checks, with Codecov and other tools, there is tight integration with status checks.

In this post, let's explore how we can achieve our desired outcome.

Requirements

We will be picking-up where we left off in Part 7, which means you’ll need the end-result from GitHub Repo - Part 7 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Don't forget to ensure you have setup Codecov.io with your repository, including the Codecov GitHub Bot - For details on how to set it up, please see Part 7

Steps

CI Workflow

Create a new branch, call it say, codecov.
Add a new yaml file called, ci-pull-request.yml under your .github\workflows folder.
Add the following code to your new workflow:

name: CI Pull Request

on:
  pull_request:
    branches: [ main ]

  workflow_dispatch:

env:
  image-name-unit-tests: unit-tests:latest

jobs:
  ci:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repo
        uses: actions/checkout@v2
        with:
          # Disabling shallow clone is recommended for improving relevancy of reporting
          fetch-depth: 0

      - name: Unit tests [build]
        run: docker build --target unit-test -t ${{ env.image-name-unit-tests }} .

      - name: Unit tests [run]
        run: docker run --rm -v ${{ github.workspace }}/path/to/artifacts/testresults:/code/test/Samples.WeatherForecast.Api.UnitTest/TestResults ${{ env.image-name-unit-tests }}

      - name: Code coverage [codecov]
        uses: codecov/codecov-action@v1.2.1
        with:
          files: ${{ github.workspace }}/path/to/artifacts/testresults/coverage.info
          verbose: true

You'll notice this is pretty similar to our other main workflow, albeit smaller - We are simply building the unit test image and running it and sending the coverage to Codecov

Code coverage configuration

Add the following to your codecov.yml file:

coverage:
  status:
    project:
      default:
        target: 80%    # the required coverage value
        threshold: 1%  # the leniency in hitting the target

Create pull request

Create a PR (pull request)
Using GitHub / GitHub Desktop / Git cmdline, push your PR

Once the PR is opened, it will kick-off our new workflow, and it will wait for Codecov to come back.

Our open pull request

Once Codecov comes back, the status is updated.

We have set the minimum target of 80%, and unfortunately we have a low of 36%.

It will block our pull request.

This can be a great CI workflow to ensure coverage is at your minimum expectations. This however, cannot tell if your unit tests are good unit tests vs bad unit tests though!

Why is the coverage so low?

As you saw in the previous post, we have no unit tests covering Program.cs and Startup.cs. This skews our actual coverage, and for us I'm happy to not cover these with unit test, so I'd rather exclude those for now.

We could however, cover them with integration tests.

There is an attribute in .net to denote that a class etc. is excluded from coverage. This is called, ExcludeFromCodeCoverageAttribute. For more information, please visit ExcludeFromCodeCoverageAttribute Class

Place this attribute on both, Program.cs and Startup.cs.

using System.Diagnostics.CodeAnalysis;

namespace Samples.WeatherForecast.Api
{
    [ExcludeFromCodeCoverage]
    public class Program
    {
        ...
    }
}

using System.Diagnostics.CodeAnalysis;

namespace Samples.WeatherForecast.Api
{
    [ExcludeFromCodeCoverage]
    public class Startup
    {
        ...
    }
}

Test locally

Kick-off your local unit-tests by executing .\unit-test.ps1, you should see a big improvement in coverage as we have decisively and specifically excluded items.

We can see that the is now a difference between what was covered previously and what is covered now.

Given we are above the target of 80%, the GitHub Status Check will allow it to pass.

Our code coverage goes up as indicated, we can also see this directly in Codecov.io; a nice graph, and who doesn't love a nice graph!

Our files are excluded from coverage, hence the code coverage boost as we expected.

Our badge proudly displays our code coverage percentage.

Go one step further

As you can see from our changes, we've created a new workflow and we have GitHub Status Checks to ensure we have quality gates with our code based on a code coverage target. We can go one step further with this PR workflow by including most of our other main workflow...

We can add the App [build], App [scan] steps, one thing we should not do though, is publish to our container registry, we certainly don't want to do that; as we are only at the PR stage!

The benefit of doing this is running the container scan on a final build image, and we will break our build if the scan fails based on our configuration for it. We can also introduce another GitHub Status Check to ensure the ci workflow completes successfully.

Let's give this a go...

...
env:
  image-name: ghcr.io/peteking/samples-weatherforecast-part-8:${{ github.sha }}
...

...
      - name: App [build]
        run: docker build -t ${{ env.image-name }} .

      - name: App [scan]
        uses: azure/container-scan@v0
        with:
          image-name: ${{ env.image-name }}
          severity-threshold: MEDIUM
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_CR }}

Next step

Repository Settings → Branch Rule → Check for ci to mark it as Required

Final step

Commit → Push → Create Pull-Request

You should now see your CI workflow run and go through the status checks. If the CI workflow fails, for example, the App [scan] step finds a CVE and it is deemed we do not pass, this will stop engineers from merging in code that will end include an open CVE.

All status checks pass!

We have some duplication

Yes, we now have some duplicated yaml code in our CI workflow and our build and push workflow. We can extract and share this out potentially with composite GitHub Action Workflows. However, I have not covered this in this post. This may be covered in another post or it may be tackled down the line in this blog series.

What have we learned?

We have learned GitHub Status Checks are powerful tools and how to take advantage of them. We have configured a new CI workflow that acts as a quality gate for code coverage with a minimum of 80% to pass; we can be somewhat happy that our codebase won't degrade its code coverage over time with each PR.

We extended our CI workflow with our build and scan steps to protect ourselves from potential CVE's before merging; and finally building and pushing our image.

If we see an open CVE when doing our PR, we can investigate, rectify if needed and re-submit the PR. Long-live shift-left!

Next up

Part 9 in this series will be about:

Static code analysis (SCA)

More information

API's From Dev to Production - Part 7 - Code Coverage

Pete King — Thu, 25 Mar 2021 15:23:02 +0000

Series Introduction

Welcome to Part 7 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

Code coverage

TL;DR

We understood code coverage is an important metric, but a metric that shouldn't looked at in isolation.

Achieving a high percentage of code coverage is a great goal to shoot for, but it should be paired with having a robust, quality, high-value test suite.

We utilise Coverlet to output code coverage files (lcov format), setup and configure Codecov, update our Dockerfile and GitHub Actions Workflow for code coverage, and finally, a little insight into the Codecov settings file to set a target.

Codecov.io is was a great solution and easy to integrate, I can highly recommend the product; I set it up in less than 10 minutes!

GitHub Repository

peteking / Samples.WeatherForecast-Part-7

This repository is part of the blog post series, API's from Dev to Production - Part 7 on dev.to. Based on the standard .net standard Weather API sample.

Introduction

We would all like to write perfect, bug free code, but we know in reality this will not happen, we are human after all. So, we write tests whereby we arrange, act and assert a scenario, in turn, this uses our code that will effectively be running in production, servicing the needs of our customers. A good metric to track is code coverage, it can help assess the quality of our suite of tests; how much of our tests actually execute lines of code (and more) that will be servicing those needs of our customers?

Code coverage is a measurement used to express which lines of code were executed by tests. Typically there is the use of three primary terms to describe each line executed.

Hit indicates that the source code was executed by a test.
Partial indicates that the source code was not fully executed by the a test; there are remaining branches that were not executed.
Miss indicates that the source code was not executed by tests.

There are many products that offer solutions, here we are going to look into one of them, it's free for open source, and it has a paid plans of course too.

What percentage of coverage should I aim for?

I'm afraid I have some bad news... There is no magic formula or silver bullet. A very high percentage of coverage could still be problematic if certain critical parts of the application are not covered, or if the tests are not robust enough to capture failure when they execute; either locally or as part of CI.

Having said that, some general views are it's about 80%, however, you must be aware that it is a target and it is not something to achieve immediately and to cut corners in order to hit the target. Some product teams may feel under pressure to meet targets and create broad tests to meet those targets. Don't fall into this trap, you don't want to have laser focus trying to hit every line of code, instead, the focus should be on the business requirements of your application.

Code coverage reports

As you'll see in this post, code coverage reports provide a way to identify critical misses in your testing. You API/application will have many tests, and can at times, be hard to navigate, all you'll know is some of your tests are failing, and here is the list. Code coverage reports provide a way to dig into the details and find actions to take; find out what is not tested.

Code coverage does not equal good tests

Remember, just because code is covered with tests, it does not mean everything is all good. If your tests are not the right tests, i.e. the test does not test the right elements in the right way, it means your tests are of low quality and value. However, your code coverage is high... so what? You see, code coverage is a metric that must be viewed in context with other metrics and other practices.

Achieving a high percentage of code coverage is a great goal to shoot for, but it should be paired with having a robust, quality, high-value test suite.

Requirements

We will be picking-up where we left off in Part 6, which means you’ll need the end-result from GitHub Repo - Part 6 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Coverlet

Coverlet is the number one way (in my view) to add cross platform code coverage to .NET, with support for line, branch and method coverage. It works with .NET Framework on Windows and .NET Core on all supported platforms. In short, it's great!

Let's set it up in our project.

Step 1

To add Coverlet to our project, we need to add the, coverlet.msbuild package to our unit test project.

Open your terminal, cd to the folder where your unit test project is located, and run the following command:

dotnet add package coverlet.msbuild

If you're using Visual Studio you can simply add a new reference using the UI.

This command will use Nuget and update your .csproj file like so:

    <PackageReference Include="coverlet.msbuild" Version="3.0.2">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>

Step 2

Update our dotnet test command to collect code coverage metrics.

-p:CollectCoverage=true \
-p:CoverletOutput="TestResults/coverage.info" \
-p:CoverletOutputFormat=lcov

Dockerfile - [Unit test runner] section

This is what our modified Unit test runner section looks like now with the additional parameters for code coverage.

# Unit test runner
FROM build AS unit-test
WORKDIR /code/test/Samples.WeatherForecast.Api.UnitTest
ENTRYPOINT dotnet test \
    -c Release \
    --runtime linux-musl-x64 \
    --no-restore \
    --no-build \
    --logger "trx;LogFileName=test_results_unit_test.trx" \
    -p:CollectCoverage=true \
    -p:CoverletOutput="TestResults/coverage.info" \
    -p:CoverletOutputFormat=lcov

We have chosen the lcov format simple because the tools we are going to use next require this format.

You can output other formats and even multiple formats at the same time.

As a quick example, if you wished to output lcov and opencover formats, you could modify the last two lines like so:

-p:CoverletOutput="TestResults/" \
-p:CoverletOutputFormat=\"lcov,opencover\"

Step 3

We now need to build locally, we have our .\unit-test.ps1 script, run that and we should have output from Coverlet on our code coverage, let's double-check just to make sure.

Look in your /TestResults directory, in there you should have an additional file called, coverage.info.

Codecov

If you are looking for a nice elegant solution, Codecov has got you covered; no pun intended... or was there... :P

Let's setup Codecov.

Navigate to Codecov, and sign up

I've used my GitHub account.

Once connected you'll see you have no repositories setup like the screenshot below:

Select → Add repository

For this example, I'm going to Select the
Samples.WeatherForecast-Part-7 repository.

New step

We need to add a new step to upload our coverage results to Codecov, and as you've seen previously, this is the power of GitHub Actions - There is a pre-built Action from Codecov to do that for us!

For more information, please see, GitHub Action for Codecov.

Immediately after the Unit test [publish] step, please add a new step with the following code:

- name: Code coverage [codecov]
  uses: codecov/codecov-action@v1.2.1
  with:
    files: ${{ github.workspace }}/path/to/artifacts/testresults/coverage.info
    verbose: true

Commit time for Codecov

If you've carried out this exercise locally like myself, and ensured the coverage.info is generated by running .\unit-test.ps1.

It's time you committed your changes, your build should kick-off and Codecov should know all about it.

Once your build job has executed the Codecov action, your results should be available to see in the Codecov platform.

If we dive into the breakdown of the code, you should see something similar below:

Looking at our code coverage, it isn't that fantastic, overall we have achieved 36%, but at least we can take some action. We have our sample unit test, which is incredibly basic, and given we have no tests covering program.cs and startup.cs it will hurt our overall coverage percentage.

Code coverage badge

Everyone likes a badge on their 'readme.md', let's add our Codecov bage.

Add the following code into your, readme.md file, of course, yours will be slightly different to mine.

[![codecov](https://codecov.io/gh/peteking/Samples.WeatherForecast-Part-7/branch/main/graph/badge.svg?token=KZW5MORPPY)](https://codecov.io/gh/peteking/Samples.WeatherForecast-Part-7)

Where can I get the code you need, well, that's easy...

Navigate to your project in Codecov

Click → Settings (top-right)

In the Sidebar Click → Badges

Codecov Repository Configuration

First of all, install the Codecov Bot in GitHub using the following link:

GitHub App - Codecov Bot

You can configure Codecov with various different configurations, for more information, please see, About the Codecov yaml

Here we have set the target coverage of 30%, this is quite low as we know, but this is just an example to show how it works.

Commit your code, and wait for your results to show-up, it should all *pass fine.

coverage:
  status:
    project:
      default:
        target: 30%    # the required coverage value
        threshold: 1%  # the leniency in hitting the target

We will take advantage of the above settings in the next blog post :)

What have we learned?

We have learned how to add code coverage, in particular Codecov.io to our API. In order to achieve these results, we have seen how to add Coverlet and output our coverage files (specifically lcov). We've adapted our Dockerfile, GitHub Actions Workflow, configured a code coverage target (albeit a low one) and even added a badge to our GitHub repository readme file.

📄 UPDATE - CodeCov.io
We end up removing CodeCov.io from our API in Part 9, however, it's still worthwhile going through this process. Even so, CodeCov.io could be OK for you, don't let me stop you! 👍

Next up

Part 8 in this series will be about:

More code coverage - We will set a realistic code coverage target.
- GitHub Status checks - How to protect your code?

More information

API's From Dev to Production - Part 6 - Unit Tests

Pete King — Fri, 19 Mar 2021 15:02:25 +0000

Series Introduction

Welcome to Part 6 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

Unit Testing - with Docker and GitHub Actions

TL;DR

We optimised the image layering by ensuring we COPY our .csproj files separately in order for the Docker build engine to use its build cache more effectively.

We add a basic unit test (in principle), use the .NET Core Test Explorer for ease of use within VS Code.

We build 2 images, one for unit testing and our final image, we achieved this by using the --target option on the docker build command. This enabled us to run Docker for our unit tests separately (as it needs the SDK to run).

We add the same capability to our GitHub Actions' Workflow and persist the unit test results as an artifact, and use a GitHub Action to publish a pretty test report against the build job. GitHub Actions - Test Reporter

GitHub Repository

peteking / Samples.WeatherForecast-Part-6

This repository is part of the blog post series, API's from Dev to Production - Part 6 on dev.to. Based on the standard .net standard Weather API sample.

Introduction

In Part 5 we were able to get our healthcheck in our API and even our Dockerfile. This is a natural progression to move onto other key capabilities we should have in our API engineering flow - Unit tests!

We won't go so in depth into the good practices around uniting testing per se, nor will I describe the different types of unit testing; socialable and solitary unit testing.

What we will cover is:

How to add unit testing to our out-of-the-box weather API that we have?
How that can work with our Docker setup?
How we can run unit tests locally?
How we can run unit tests in GitHub Actions?
How we can report the unit test results in GitHub Actions?

Requirements

We will be picking-up where we left off in Part 5, which means you’ll need the end-result from GitHub Repo - Part 5 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Additional Requirements

We will need one further VS Code extension that will prove useful:

.NET Core Test Explorer - https://marketplace.visualstudio.com/items?itemName=formulahendry.dotnet-test-explorer

Add new unit test project

Open any terminal such as Windows Terminal.

Navigate to the root folder of the project.

In my case, I've put it into a GitHub folder under a folder called, 'Samples.WeatherForecast'.

Execute:

dotnet new xunit -o ./test/Samples.WeatherForecast.Api.UnitTest

Add reference

We now need to add a reference to our API to our unit test project.
Staying in the same terminal session; ensuring you are still in the root directory.

Execute:

dotnet add ./test/Samples.WeatherForecast.Api.UnitTest/Samples.WeatherForecast.Api.UnitTest.csproj reference ./src/Samples.WeatherForecast.Api/Samples.WeatherForecast.Api.csproj

Open VS Code

Let's get started and open VS Code.

TIP

In your terminal, execute: code .

You should now have your API project as before, but now you should have a test folder and your new xUnit test project too.

Navigate to the test project and Click → UnitTest1.cs.

Unit test

Let's rename the file from, UnitTest.cs to WeatherForecastControllersTests.cs.
Change the namespace from, UnitTest1 to Samples.WeatherForecast.Api.UnitTest.
Add a new (overwrite existing test) with a new test called, ShouldReturnAListOfValues().

Full code

using System;
using Xunit;
using Microsoft.Extensions.Logging.Abstractions;

namespace Samples.WeatherForecast.Api.UnitTest
{
    public class WeatherForecastControllerTests
    {
        [Fact]
        public void ShouldReturnAListOfValues()
        {
            // Arrange
            var logger = new NullLogger<Samples.WeatherForecast.Api.Controllers.WeatherForecastController>();
            var service = new Samples.WeatherForecast.Api.Controllers.WeatherForecastController(logger);

            // Act
            var result = service.Get();

            // Assert
            Assert.NotNull(result);
        }
    }
}

Your VS Code screen should look like mine below:

Now, this is not the best unit test in the world, but the principle is still the same.

This is to demonstrate how you can set it up and where to put your unit tests.

Navigate to the .NET Core Test Explorer

You'll notice that it says, "Test1", this was the original test name in our UnitTest.cs file, it needs to re-scan to find tests.

Click Refresh

You should see our new test method popup like below:

Click the Play button

You should see a nice green tick in two places, on the test explorer, and in your code against the test method.

Dockerfile

Now we have the unit test(s) working, we need to modify our Dockerfile, and I'm sorry to say, but we are going to change it quite a lot...

We need to add in the unit test stage, which will introduce a new ENTRYPOINT.
Whilst we are here, we can optimise the build process a little too, and I'll explain why this is important - it goes back to Part 2 where we optimised for size, and we went through how the Docker build cache works. However, we haven't taken complete full advantage of it here... and we should!

Let's start with the optimisations

We are going to start with the optimisations simply because for our unit tests to run, we will need some of those changes.

We will go from line 1 to the end of the file just so it's all nice & clear.

Feel free to clear your Dockerfile.

ARG VERSION=5.0-alpine

FROM mcr.microsoft.com/dotnet/runtime-deps:${VERSION} AS base
WORKDIR /app
EXPOSE 8080
# HEALTHCHECK --interval=60s --timeout=3s --retries=3 \
#    CMD wget localhost:8080/health -q -O - > /dev/null 2>&1

Here we set a variable for the version and we setup our base image, set our working directory to /app, EXPOSE port 8080, and we have our commented-out HEALTHCHECK; this we don't really need, but it's nice to have it on hand if we do.

This was at the end of our Dockerfile previously, but I've moved it to the top as it's a common thing to see in multi-stage builds.

FROM mcr.microsoft.com/dotnet/sdk:${VERSION} AS build
WORKDIR /code

# Copy and restore as distinct layers
COPY ["src/Samples.WeatherForecast.Api/Samples.WeatherForecast.Api.csproj", "src/Samples.WeatherForecast.Api/Samples.WeatherForecast.Api.csproj"]
COPY ["test/Samples.WeatherForecast.Api.UnitTest/Samples.WeatherForecast.Api.UnitTest.csproj", "test/Samples.WeatherForecast.Api.UnitTest/"]

Here we are using the dotnet SDK and we label that as our build image.

We set the WORKDIR to /code - I didn't want to call it /app as I wanted a clear difference between the code to build and the final deploy folder.

The most important thing to note is that we are now specifically using COPY to copy our API project, and our Unit Test project

Why copy each one instead of what we had before, which was using COPY . . ?

Well, it's because if any files change, the entire layer will need to be re-created each time, if we copy each one at a time, Docker will create a new layer for each COPY command. Therefore, Docker can use its build cache to see if it needs copying or not; Docker will highly optimise the build process.

RUN dotnet restore "src/Samples.WeatherForecast.Api/Samples.WeatherForecast.Api.csproj" -r linux-musl-x64
RUN dotnet restore "test/Samples.WeatherForecast.Api.UnitTest/Samples.WeatherForecast.Api.UnitTest.csproj" -r linux-musl-x64
COPY . .

We then need to run dotnet restore for each project, similar to what we had before.

Finally, we execute COPY . . to copy any other files that could have changed.

# Build
RUN dotnet build \
    "src/Samples.WeatherForecast.Api/Samples.WeatherForecast.Api.csproj" \
    -c Release \
    --runtime linux-musl-x64 \
    --no-restore    

RUN dotnet build \
    "test/Samples.WeatherForecast.Api.UnitTest/Samples.WeatherForecast.Api.UnitTest.csproj" \
    -c Release \
    -r linux-musl-x64 \
    --no-restore

Now we build each project using dotnet build, ensuring we set, --no-restore , just like we did before.

# Unit test runner
FROM build AS unit-test
WORKDIR /code/test/Samples.WeatherForecast.Api.UnitTest
ENTRYPOINT dotnet test \
    -c Release \
    --runtime linux-musl-x64 \
    --no-restore \
    --no-build \
    --logger "trx;LogFileName=test_results_unit_test.trx"

This is a big change to note here, this is our unit test runner.

We use FROM build as we need to use the .NET SDK, we label this one as, unit-test.

Set the WORKDIR

We create a new ENTRYPOINT and execute dotnet test

This is where the magic happens!

We have another ENTRYPOINT - How can that work I hear you ask?

Well, it comes down to how Docker processes the Dockerfile - The last ENTRYPOINT command is the one it will run; kind of like a default if you will.

So now I hear you ask, how can we make use of 2 ENTRYPOINT commands then...?

Part of the docker build command is an option called, --target.

For more information about the docker build command, please see, Docker Docs - docker build - target

--target will target a specific build stage, the label we need to use is defined using the FROM statement, it's the AS we need - in our case here, we have FROM build AS unit-test, therefore, the build stage name is, unit-test.

When we wish to execute our unit test runner using Docker, we can simply build our Dockerfile up to this build stage; the remaining instructions in the Dockerfile are ignored.

Build [unit-test]

To build our docker image, we can execute the following command in your terminal of choice:

docker build --target unit-test -t samples-weatherforecast-unit-test:v6 .

Run [unit-test]

If we wish to execute our unit tests, we can run the following command:

docker run --rm samples-weatherforecast-unit-test:v6`

This will run our instructions in that build stage, which in our case are our unit tests using, dotnet test.

Test Results

When you execute docker run on this unit-test image, our dotnet test command is outputting test results.

However, the test results that dotnet test creates are stored in the in the container, we need to get them out to our host; otherwise, we can't see them.

This is where bind volume mount option comes into play - we can use the -volume OR -v option for short to map the container directory to our host directory when we use, docker run.

For more information about bind volume mount please see, Docker Docs - docker run - Bind Volume Mount

Our docker run command can be modified to something like the following:

docker run --rm -v "${pwd}\TestResults:/code/test/Samples.WeatherForecast.Api.UnitTest/TestResults/" samples-weatherforecast-unit-test:v6

Info

pwd = Print Working Directory

Now when we execute our docker run command, we should see the test results file; called, test_results_unit_test.trx.

Please ensure this works before continuing.

Let's make execution easier

It is a little bit repetitive executing these commands from time to time, sure you can use the test explorer quite a lot, but having to remember these commands and type them out without any typos is rather annoying.

The best approach is to script it, you can use pretty much anything you want, here as an example, I'm going to use trusty old PowerShell, but feel free to use BASH or even a MAKE file.

Script running of unit tests

In your root directory, create a new file called, unit-test.ps1.

Paste in the following code:

$IMAGE_NAME_AND_TAG="samples-weatherforecast-unit-test:v6"

Write-Output "Unit tests [build]"
docker build --target unit-test -t $IMAGE_NAME_AND_TAG .

Write-Output "Unit tests [run]"
docker run --rm -v "${pwd}\TestResults:/code/test/Samples.WeatherForecast.Api.UnitTest/TestResults/" $IMAGE_NAME_AND_TAG

As you can see, it's pretty simplistic, we are just trying to make things easier for ourselves.

Script the running of the build

Let's do the same thing to our build so we don't have to type docker build all the time.

In your root directory, create a new file called, build.ps1.

Paste in the following code:

$IMAGE_NAME_AND_TAG="samples-weatherforecast:v6"

Write-Output "App [build]"
docker build -t $IMAGE_NAME_AND_TAG .

How do I use these scripts?

Unit tests

Open your terminal and execute: .\unit-test.ps1.

This will build your test container, and run your unit tests.

Run this now please...

If all goes well, you should see that the unit test(s) have passed.

Build

Open your terminal and execute: .\build.ps1.

Run this one now too please...

Again, if all is fine, you should have your container image built.

Please make sure your container has been built by double-checking it's there: docker image ls is your friend here.

In addition, run your the container and test with Postman or your preferred method.

docker run -it --rm -p 8080:8080 samples-weatherforecast:v6`

GitHub Actions

We have everything working locally and we are happy, let's move onto our CI; lovely GitHub Actions.

We will need to add in a some steps to our workflow that we had before.

In our env section in our build-and-push.yaml file, let's add a new variable called, image-name-unit-tests, code below:

env:
  image-name: ghcr.io/peterjking/samples-weatherforecast-part-6:${{ github.sha }}
  image-name-unit-tests: unit-tests:latest

Next up is adding a new step, just below the, checkout repo step. We need to build our unit-test runner image like so below:

- name: Unit tests [build]
        run: docker build --target unit-test -t ${{ env.image-name-unit-tests }} .

Immediately after this step, we need to run the unit tests, please add the step below:

- name: Unit tests [run]
        run: docker run --rm -v ${{ github.workspace }}/path/to/artifacts/testresults:/code/test/Samples.WeatherForecast.Api.UnitTest/TestResults ${{ env.image-name-unit-tests }}

You'll notice we are doing the bind volume mount, -v, but it's different from our local script, this is because of how GitHub Actions works, it seems we cannot make use of pwd, but thankfully, GitHub provides built-in variables we can take advantage of, in our case, we will use the, github.workspace.

The GitHub workspace directory path. The workspace directory is a copy of your repository if your workflow uses the actions/checkout action. If you don't use the actions/checkout action, the directory will be empty. For example, /home/runner/work/my-repo-name/my-repo-name.

For more information, please see, GitHub Docs - Environment variables

We have the test results file in trx format on this GitHub Actions Runner Host. It would be nice if we can persist this against this build job... and we can! We can use the upload artifact v2 action.

- name: Unit tests [results]  
        uses: actions/upload-artifact@v2
        if: always()
        with:
          name: unit-test-results
          path: ${{ github.workspace }}/path/to/artifacts/testresults/test_results_unit_test.trx

Commit your changes now, and ensure everything works.

As soon as you commit your workflow file, the build should kick-off, your unit tests image should be build, run the image, output the tests results, and store the test results against the build job itslef.

You should see a screen like this:

We can do better!

Can we? Yes, we sure can... It's great we have an artifact stored against the job, but it's rather painful and annoying to download it, and open the trx file, or look through the build log.

There are a bunch of GitHub Actions that can help us out here, for me I've chosen, GitHub Actions - Test Reporter

Immediately after our artifact upload step, let's create another one like so:

- name: Unit tests [publish]
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: Unit tests
          path: ${{ github.workspace }}/path/to/artifacts/testresults/test_results_unit_test.trx
          reporter: dotnet-trx
          token: ${{ secrets.GITHUB_TOKEN }}

This action will carry out some magic for us.

The token: ${{ secrets.GITHUB_TOKEN }} is an in-built, predefined secret that gives access to the build job.

Please commit this change, you'll build should kick-off once again, and hopefully you'll see the test results nicely formatted.

What have we learned?

We have learned how to subtly change our image layering to ensure the Docker build cache is as optimized as possible; yes, it's more code to write, but you'll save build time in the end, especially if your solution grows.

We've added a new unit test project using the commandline and hooked up the references, build an incredibly basic unit test (in principle).

We have taken advantage of the --target option to build a container image up to a certain point in our Dockerfile; in our case, we have our unit test runner. We have even scripted it to make our lives a litter easier too with a few lines of good old Powershell :)

On top of this we have modified our GitHub Actions' Workflow in a similar vein to how we are executing locally; we build the unit test image, run the unit test using Docker, and build our final image.

Finally, we have persisted the unit test results as an artifact and have a nicely formatted report against our build job.

Next up

Part 7 in this series will be about:

Code coverage

More information

API's From Dev to Production - Part 5 - Healthchecks

Pete King — Fri, 12 Mar 2021 17:45:07 +0000

Series Introduction

Welcome to Part 5 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at CIS Issues:

Add a healthcheck to the Dockerfile
Add an item to the CIS allowedlist.yaml

TL;DR

Healthchecks are important to ensure you are aware of the state of your services, particularly when you have monitoring solutions that inform you if something is not quite right. No one wants customers telling you your product doesn’t work when there are adequate ways to deal with incidents before they could potentially impact customers.

It can be good to have the Docker healthcheck instruction, but not strictly necessary when we are not running Docker Swam, but can be handing with Docker Compose - Therefore, I recommend adding it, but maybe comment it out, add CIS-DI-006 to the allowedlist.yaml so it is ignored by the container scan.

GitHub Repository

peteking / Samples.WeatherForecast-Part-5

This repository is part of the blog post series, API's from Dev to Production - Part 5 on dev.to. Based on the standard .net standard Weather API sample.

Introduction

In Part 4 we were able to get into security concerns and looked into both container scanning and CIS. We solved the the very important CIS-DI-001. Here we will look further into CIS-DI-006.

Requirements

We will be picking-up where we left off in Part 4, which means you’ll need the end-result from GitHub Repo - Part 4 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Add HEALTHCHECK

Healthchecks are an important piece of surfacing health of components of a system, you might ask why you’d want to know? Well, if you service is not servicing requests you’d want to know before your customers staff calling you!

.NET 5 has some cool built in functionality to help with all the boilerplate of adding healthchecks, but remember, the Dockle tool itself is not checking for that. It’s actually checking for the HEALHCHECK instruction in the Dockerfile.

….BUT, we still need a healthcheck in our API, otherwise, there’s nothing to actually check!

In the src folder, the .NET 5 API, you should be familiar with the Startup.cs file.

Below is an snippet, and we need to add in line 4.

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddHealthChecks();

    services.AddSwaggerGen(c =>
    {
        c.SwaggerDoc("v1", new OpenApiInfo { Title = "Samples.WeatherForecast.Api", Version = "v1" });
    });
}

Another place we need to modify is further down, I’ve added in line 19.

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
        app.UseSwagger();
        app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "Samples.WeatherForecast.Api v1"));
    }

    app.UseHttpsRedirection();

    app.UseRouting();

    app.UseAuthorization();

    app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllers();
        endpoints.MapHealthChecks("/health");
    });
}

That’s all there is to it on the .NET side of things, however, Docker won’t call it yet - So we need to update our Dockerfile.

# Final stage/image
FROM mcr.microsoft.com/dotnet/runtime-deps:${VERSION}

RUN addgroup -g 1000 dotnet && \
    adduser -u 1000 -G dotnet -s /bin/sh -D dotnet

WORKDIR /app
COPY --chown=dotnet:dotnet --from=publish /out .

ENV ASPNETCORE_URLS=http://*:8080

HEALTHCHECK --interval=60s --timeout=3s --retries=3 \
    CMD wget localhost:8080/health -q -O - > /dev/null 2>&1

USER dotnet
EXPOSE 8080
ENTRYPOINT ["./Samples.WeatherForecast.Api"]

Above is a snippet (the lower half of our Dockerfile), we’ve added in line 12 & 13.

For more information about the HEALTHCHECK instruction, please see, Docker Docs - Builder - Healthcheck

From the options we are using, we have set the interval to 60 seconds, a timeout of 3 seconds, and we retry 3 times if it fails. On the continued line 13 we execute a command.

You may be wondering why we are simply not just using curl, well curl would be great, however, it’s isn’t even installed! This is how stripped-back this container image is, if we wanted to use curl we’ve have use apt-get etc.

So we are using wget instead, where we call our /health endpoint and pipe out the response check the values for output.

Build & test time!

docker build -t samples-weatherforecast:v5 .

docker run -it --rm -p 8080:8080 samples-weatherforecast:v5

Once it’s running you’ll want to check the status of the container, you can do this with the following command:

docker container ls

Remember, you have a minute before it starts checking!

If you’re quick, you’ll see something like the below, note the status is, starting.

After 60 seconds, we should see a status of healthy.

It works!

What have we learned?

We have learned the very basics and simple steps to add a minimal healthcheck to our API, and ensure Docker will call this and mitigate CIS-DI-006; however, the big question is, do we need the HEALTHCHECK instruction if we are going to be hosting this on anything other than Docker Swarm OR Docker Compose - My view is we actually don’t!

But… We should have a healthcheck in our API, because whatever we host it in, in our case it will be Azure’s Web App for Containers, we will need a healthcheck.

So you may think this is a pointless exercise, but not really, we’ve got the API healthcheck, but we can safely remove the HEALTHCHECK OR comment-out the instructions from our Dockerfile.

Up next

Part 6 in this series will be about:

Unit Testing - with Docker and GitHub Actions

More information

API's From Dev to Production - Part 4 - Image Scanning

Pete King — Fri, 05 Mar 2021 12:20:11 +0000

Series Introduction

Welcome to Part 4 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

Add container scanning to our GitHub Action workflow
Resolve security vulnerabilities

TL;DR

We add the Azure Container Scan GitHub Action to our actions' workflow file, before pushing the container image to our registry. Add a group and user to our Dockerfile and use the USER command to specify the user for the process to run; ensuring our copied files can be accessed by this user. The security attack surface is reduced by running as non-root, and this is raised in CIS; the linting provided by Dockle. We resolved some slightly older CVE’s as examples; these can be raised by Trivy and can be allowed through or not based on configuration. Both Trivy and Dockle are wrapped by the Azure Container Scan GitHub Action.

GitHub Repository

peteking / Samples.WeatherForecast-Part-4

This repository is part of the blog post series, API's from Dev to Production - Part 4 on dev.to. Based on the standard .net standard Weather API sample.

Introduction

In Part 3 we were able to get into GitHub Actions and build and push our Docker image to the GitHub Container registry. This is a great start, we are producing a Docker image in CI and it’s a consistent image, however, what about vulnerabilities?

Requirements

We will be picking-up where we left off in Part 3, which means you’ll need the end-result from GitHub Repo - Part 3 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Vulnerabilities

A software vulnerability is a glitch, flaw, or weakness that is present in software or in an Operating System. CVE (Common Vulnerabilities and Exposures), contains a list of records each containing an identification number, a description, and at least one public reference - for publicly known cybersecurity vulnerabilities.

CVE Records are used in numerous cybersecurity products and services from around the world, including the U.S. National Vulnerability Database (NVD).

There are many open source and commercial tools specifically designed for CI purposes; to be used before pushing a Docker image to a container registry. This can ensure any CVE’s of certain thresholds can be caught a build time.

Commercial tools can also scan containers at runtime and apply custom mediating actions.

CIS (Center for Internet Security)

CIS is a community of cybersecurity experts, who have developed CIS Benchmarks: more than 100 configuration guidelines across 25+ vendor product families to safeguard systems against today’s evolving cyber threats.

CIS also produced a list of hardened images - https://www.cisecurity.org/cis-hardened-image-list/

Why is this important?

It’s important because the software we design, build, and ship needs to be secure, we want to ensure safety of our end users' data and be responsible. We must take security seriously and both CVE and CIS provide a trusted source of information.

The open source community has done an amazing job of utilising the information and provided fantastic tooling we can all take advantage of.

What can we do?

In terms of CVE’s, AquaSec has created not only a great commercial product but an open source product called Trivy. This is a Docker image scanner that detects vulnerabilities of OS packages (Alpine, RHEL, CentOS, etc.) and application dependencies (Bundler, Composer, npm, yarn, etc.). Trivy is easy to use, you can just install the binary and scan, all that’s required is to specify a target such as an image name of the container.

For CIS, there is an open source tool called Dockle - Dockle is a Container Image Linter for Security, Helping build the Best-Practice Docker Image, which is ideal for CI.

GitHub Actions to the rescue

Microsoft has create a GitHub Action which wraps both Trivy and Dockle called container-image-scan, you can find it on the GitHub Actions Marketplace and the open source repo, Azure/container-scan.

As of writing, the GitHub Container Registry does not scan images you push there, but other container registries do, such as the Azure Container Registry (ACR). It’s good that images will be scanned once they are in the registry, however, in terms of shifting-left, it is hugely beneficial to scan images during CI, if a vulnerability is found during build, engineers in the team are notified immediately, they can fix it themselves or consult with experts to help them.

This promotes a faster feedback loop and potentially stops images being pushed to a registry if that have vulnerabilities.

Let’s use Azure Container Scan

In out workflow file we previously called, build-and-publish.yml, it’s located in the .gihub/workflows folder. We can add an extra step, we’ve given it a name, Scan docker image, specified the azure/container-scan@v0 GitHub Action, and set the required and optional parameters. We’ve set the severity-threshold: MEDIUM, this is completely up to you, for this example, I’ve simply chosen MEDIUM. Please read the documentation on the Azure Container Scan repository and set it to what you believe is adequate for you specific situation.



- name: Scan docker image
  uses: azure/container-scan@v0
  with:
    image-name: ${{ env.image-name }}
    severity-threshold: MEDIUM
    username: ${{ github.repository_owner }}
    password: ${{ secrets.GH_CR }}

Full Workflow file



name: Build and Publish

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  workflow_dispatch:

env:
  image-name: ghcr.io/peterjking/samples-weatherforecast-part-4:${{ github.sha }}

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout the repo
        uses: actions/checkout@v2

      - name: Build docker image
        run: docker build . -t ${{ env.image-name }}

      - name: Scan docker image
        uses: azure/container-scan@v0
        with:
          image-name: ${{ env.image-name }}
          severity-threshold: MEDIUM
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_CR }}

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_CR }}

      - name: Push docker image
        run: docker push ${{ env.image-name }}

If you commit the changes, your GitHub Action will kick-off and build and scan.

If you have not specified your repository secret GH_CR because you may be using a new repository perhaps for this, don’t forget to add the secret to your repo.

Scan results

CVE

Depending on when you are running this and what the state of the base image will determine what you see. For me at the time of writing, I have no CVE’s, which is great! Only a week before writing this, I made a change for a similar workflow and had a HIGH CVE, it was all to do with OpenSSL in Alpine Linux. The screenshot is below for when it came up.

We can check what CVE this is with its number, a quick Google will reveal some interesting information.

For this particular issue, I found the following link:

alpine: bump 3.12.3/3.11.7 (CVE-2020-1971) #9295

ncopa posted on Dec 16, 2020

View on GitHub

Whilst writing this blog series, this issue was open.
Since then, the issue has been closed and merged :)

It was essentially a simple update, OpenSSL needed to be updated in Alpine Linux to resolve this vulnerability.

Since we have zero known vulnerabilities at the moment, from another API, I did come across another issue, again, with Alpine but earlier in December 2020. The scan results are below as another example:

In early December this came up, and this is because I was using the same base image we are using in this blog post. You should always investigate CVE and determine if they apply to you, and the reason why I’m showing you this is for that reason. Let’s go through it and see what it is.

Again, a quick Google on CVE-2020-28928, and I decide to settle on the information over at AquaSec:

https://avd.aquasec.com/nvd/cve-2020-28928/

You can see it is indeed MEDIUM, 5.5 severity, and when it was published, November 24th 2020.

If you scroll down the page you’ll see a nice thoughtful list on mitigations. If you look even closer you can see the potential mitigation is essentially memory management. Luckily for us, we are using a language and framework which prides itself on being memory safe and good memory management (on our behalf of course). Given we are using C# in this case and we are not using C or C++ for instance and using string.h etc. We can safely ignore this.

How do we ignore this CVE you ask? Well, if you check the documentation on the Azure Container Scan repository, it shows you how…

Simply create a new folder in your .github/workflows/ called containerscan, in there create a new YAML file called, allowedlist.yaml.



general:
  vulnerabilities:
    - CVE-2020-28928
  bestPracticeViolations:

Good Practice

If you do this, you need to have a regular review going forward of what you are allowing through.

Given the noted good practice above, this is a suburb example, Alpine Linux resolved this issue. How did I know this? Well, looking at Alpine’s GitHub repo tells all in this GitHub issue:

alpine:latest has CVE-2020-28928 #123

jerome-laforge posted on Dec 04, 2020

musl : 1.1.24-r9 - Layer: sha256:ace0eda3e3be35a979cec764a3321b4c7d0b9e4bb3094d20d3ff6782961a8d54

CVE-2020-28928 In musl libc through 1.2.1, wcsnrtombs mishandles particular combinations of destination buffer size and source character limit, as demonstrated by an invalid write access (buffer overflow). Solution: Upgrade musl to 1.1.24-r10

References: http://www.openwall.com/lists/oss-security/2020/11/20/4 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28928 https://lists.debian.org/debian-lts-announce/2020/11/msg00050.html https://musl.libc.org/releases.html

View on GitHub

Now, for my repo or a repo that had this vulnerability, I can safely remove it from the, allowedlist.yaml file.

CIS

In terms of CIS, we have a single 1 x WARN, and 2 x INFO.

Personally, I believe CIS-DI-0001 should really be higher, you really should in my view run your process as non-root. It’s easily solved too!

When the processes inside the container run as root, the same as the host machine, this give unprecedented privileges; we should always opt for least-privilege as a matter of practice. For example, if a user is able to gain access to the host as root

By default, root in a container is the same root (uid 0) as the host machine. If a user succeeds in breaking out of an application running as root in a container, this malicious user may be able to gain access to the host with the same root user.

Good Practice
It is good practice to launch processes with a non-root user.

TIP

Some official images already create a user for you to use, it’s always good to inspect the Dockerfile your using as your base to see.

In our case, a user is not created for us, so let’s fix that now, you can add the following code to your Dockerfile.



RUN addgroup -g 1000 dotnet && \
    adduser -u 1000 -G dotnet -s /bin/sh -D dotnet

Once we do this, when the process runs, it should be running as this, dotnet user. Therefore, we should also make sure when we COPY files we change the owner to the dotnet user using chown like so:

COPY --chown=dotnet:dotnet --from=publish /out .

If we do not do this, we run the risk of the dotnet user not being able to see the files and execute - That would be bad for us.

Full Dockerfile



ARG VERSION=5.0-alpine

FROM mcr.microsoft.com/dotnet/sdk:${VERSION} AS build
WORKDIR /app

# Copy and restore as distinct layers
COPY . .
WORKDIR /app/src/Samples.WeatherForecast.Api
RUN dotnet restore Samples.WeatherForecast.Api.csproj -r linux-musl-x64

FROM build AS publish
RUN dotnet publish \
    -c Release \
    -o /out \
    -r linux-musl-x64 \
    --self-contained=true \
    --no-restore \
    -p:PublishReadyToRun=true \
    -p:PublishTrimmed=true

# Final stage/image
FROM mcr.microsoft.com/dotnet/runtime-deps:${VERSION}

RUN addgroup -g 1000 dotnet && \
    adduser -u 1000 -G dotnet -s /bin/sh -D dotnet

WORKDIR /app
COPY --chown=dotnet:dotnetgroup --from=publish /out .

USER dotnet
EXPOSE 8080
ENV ASPNETCORE_URLS=http://+:8080
ENTRYPOINT ["./Samples.WeatherForecast.Api"]

Now you have a slightly refactored Dockerfile, let’s give this a go locally to make sure all is well.

Build the image locally:

docker build -t samples-weatherforecast:v4 .

Run the image locally:

docker run -it --rm -p 8080:8080 samples-weatherforecast:v4

Test the image in Postman or REST Client - https://marketplace.visualstudio.com/items?itemName=humao.rest-client

Once you’ve verified that it works, it’s now time to commit those changes. This should kick-off the GitHub Action and build, scan and push the Docker image container registry.

We can verify our API is running under our dotnet user but getting inside the container.

How you ask? Well, there are two ways, the CLI way or you can use Docker Desktop.

I’ll show off the Docker Desktop feature.

Open Docker Desktop → Move you mouse of your running container → Click, CLI.

A command prompt should open and you should be inside your container!

Now, execute, whoami and it should come back saying, dotnet; this is your USER.

You can also execute the top command, let’s do this too:

Cool! All running as dotnet.

Build results

Packages → Container Registry
Let’s double check that our image was pushed by checking our GitHub Packages area.

Success!

In addition, I have also pulled this image down and tested it too just like we did in API's From Dev to Production - Part 3

What have we learned?

We have learned how to scan our container images before we end up pushing them to our container registry. We understand how important this is and why we should do it, the open source tools that are available to you. We have also learned how to decrease our security attack surface by running our process as a user by adding a group and user in our Dockerfile.

There are many open source tools that are very capable, however, at times you may wish to opt for a paid product.

Here is a short list:

Snyk
Acqua
Prisma Cloud (formally Twistlock)
Anchore
...and many more, don't forget some container registries also scan images.

Up next

Part 5 in this series will be about CIS Issues:

Add a healthcheck to the Dockerfile
Add an item to the CIS allowedlist.yaml

More information

API's From Dev to Production - Part 3 - GitHub Actions

Pete King — Fri, 26 Feb 2021 11:56:33 +0000

Series Introduction

Welcome to Part 3 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

GitHub Actions:
- Build the docker image
- Publish the docker image to the GitHub Container Registry

TL;DR

Instead of building locally like we did previously, we utilised GitHub Actions and the GitHub Container Registry to store our Docker image. GitHub Actions workflow is basic YAML, and we used a few actions to achieve what we needed. We checked-out the repository, docker build, logged into our GitHub Container Registry and executed a docker push command. We used a Personal Access token and stored it in GitHub secrets to use in our workflow, and finally, we pulled down our public image from the GitHub Container Registry and tested locally using Postman.

GitHub Repository

peteking / Samples.WeatherForecast-Part-3

This repository is part of the blog post series, API's from Dev to Production - Part 3 on dev.to. Based on the standard .net standard Weather API sample.

Requirements

We will be picking-up where we left off in Part 2, which means you’ll need the end-result from GitHub Repo from - Part 2 to start with.

What are GitHub Actions?

GitHub Actions makes it easy to automate all kinds of software workflows, right inside GitHub. You can build, test, and deploy your code and more. Workflows can be triggered off many GitHub events like push, issue creation and more. There is support for Linux, macOS, Windows, ARM and of course containers Even matrix builds - so you can build the same software across multiple OS’s. There are live logs, secret store and more, plus the UX is rather nice too! All of this is driven by YAML files, nice and neat, and stored under source control management.

There is also GitHub Packages which has various package management providers like npm, nuget and more. For containers, there is a new GitHub Container Registry, so images no longer need to be stored in GitHub Packages but there is a real container registry - We will use this in this blog post :D

Let’s get started

If you navigate to GitHub Actions in your repository, you’ll be presented with the above screen. There are various starter workflows or you can simply start from scratch.

Build your Workflow from scratch

We’re going to build it from scratch, there isn’t much to it, and it’s easier to learn and see what happening this way.

Select → set up a workflow yourself

You will see a default workflow created for you with some helpful, guiding comments.

I have changed the workflow file name to build-and-publish.yml and the name (line 3) to, Build and Publish.

You have the on section where you can set triggers for various GitHub events.

They are fairly self-explanatory, but if you’re struggling, there is some decent documentation on GitHub about about it here: GitHub Actions - Workflow Syntax

For now, let’s remove all the comments as it is slightly more verbose for my liking, but please make sure you do understand what commands we are utilising. If not, things will become more clear as we edit this workflow now.

If we remove all the comments and cut out some of the default template code, you should be left with something like this, fairly barebones:

name: Build and Publish

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  workflow_dispatch:

jobs:
  build-and-publish:
    runs-on: ubuntu-latest

    steps:

The above is telling us that when a push, or a pull request event happens on the main branch, execute this workflow, which when then run the jobs, and within jobs there are steps.

Let’s focus on the steps and add the checkout of the git repository.

      - name: Checkout the repo
        uses: actions/checkout@v2

The uses is key to understand, it’s an action, and you can even have a look at the code in the GitHub Marketplace

You’ll find useful documentation on what it does and how to use it, and you can even create your own GitHub Action to do whatever you want - and guess what? They too can be based on Docker containers!

For more information about how to create custom GitHub Actions and use them to your advantage, or you want to share yours with the rest of us, please visit, GitHub - Creating Actions

Next up is the Docker build.

      - name: Build docker image
        run: docker build . -t ghcr.io/peterjking/samples-weatherforecast:${{ github.sha }}

Here we are simply running docker build, our Dockerfile is in the current directory, hence the period . and we set the image tag to my GitHub Container Registry : github.sha.

Each GitHub Account has a Container Registry, AKA GitHub Packages, for more information please visit - https://ghcr.io

You image tag must be lowercase - if you have a GitHub username like mine where there is uppercase letters, you can either hardcode it here like I have for the moment, store this in a GitHub Secret or convert to lowercase dynamically.

Now we have built the Docker image, we now would like to push it to our GitHub Container Registry, for this, we need valid credentials. For something like this, we can use a GitHub Access Token and securely store it in our GitHub Secrets area of our repo. Let’s go set this up now.

Navigate to Settings (on your profile)

Select Developer Settings

Select Personal Access Tokens

Click Generate new token

You’ll see a new screen whereby you can set the name of the PAS (Personal Access Token) and set the OAuth scopes for it. In our case, we only need write:packages, read:packages and delete:packages.

Give the access token a name, I’ve called mine, ghcr.

Copy your access token.

Once you’ve generated the token, you won’t be able to get the value again, however, you can always regenerate a new value.

Navigate to your GitHub repository Settings → Secrets → Click New repository secret.

Enter a name, paste in your value, and click → Add secret.

Now we have successfully stored our Personal Access Token as a secret, we can use that to access our GitHub Container Registry and push our image.

Below we use a GitHub Action called login-action@v1 and set it with some variables.

For more information about this action, please visit, GitHub Actions Marketplace - docker-login

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_CR }}

You’ll notice we are using an another internal GitHub environment variable called repository_owner, this contains the repository owner, which equates to my username, and therefore, my GitHub Container Registry.

Next there is the secret we created earlier, GH_CR.

Now we have login action setup, we can finally do our docker push.

      - name: Push docker image
        run: docker push ghcr.io/peterjking/samples-weatherforecast:${{ github.sha }}

That’s it!

However, we can make a minor enhancement given we are using the image tag in two places.

I’ve added an env section which enables variables within my workflow, I’ve called it, image-name, and used it in the docker build and docker push commands.

Full Dockerfile

name: Build and Publish

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  workflow_dispatch:

env:
  image-name: ghcr.io/peterjking/samples-weatherforecast:${{ github.sha }}

jobs:
  build-secure-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout the repo
        uses: actions/checkout@v2

      - name: Build docker image
        run: docker build . -t ${{ env.image-name }}

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_CR }}

      - name: Push docker image
        run: docker push ${{ env.image-name }}

Make sure you workflow looks like the above, except the username in the env image-name variable of course; this should be your username.

Now you’re ready to press → Start commit!

Once the GitHub Action kicks into Action - You’ll see it queued and then move into in progress.

The time delay between Queued to In Progress, if you’re used to an older style build system where you can be waiting minutes or even hours in some cases, you’ll be happy to learn this is ultrafast!

The build has now started and In progress.

All green... It worked!

Below we can now see the summary, the build has taken 1 minute and 31 seconds.

By clicking ‘build-secure-push’ you will see the steps and their logs.

Now, let’s check that our Docker image has actually been stored in our GitHub Container Registry by navigating to Packages.

Click the package link.

You can see a helpful section at the top where there is a sample commandline.

Copy this now as we'll need it in the very next step to test.

Let’s test this image

docker pull ghcr.io/peterjking/samples-weatherforecast-part-3:e5f0710a08eae6d19c39a8ef04dbddff211dcd88`

Note your image name will have a unique SHA1 as the Docker tag.

Copy the docker pull from your package, head-on over to the command line and execute.

If you are working on a private package, you will most certainly receive unauthorized.

One way to solve this is to open your package to the public by changing the visibility settings in the Package Settings.

Once you make the package public, you cannot go back.

The alternative is to login to your GitHub Container Registry first with docker login.

For now, I’ve simply made the package public, for more information on docker login, please see, Docker Docs - Commandline - Login

Docker run

docker run -it --rm -p 8080:8080 ghcr.io/peterjking/samples-weatherforecast-part3:e5f0710a08eae6d19c39a8ef04dbddff211dcd88`

Send a request

What have we learned?

Well, we’ve learned a bunch here haven’t we? We have learned the very basics of GitHub Actions, how to write your own workflow, pretty much from scratch, generate a Personal Access Token and store that token in your repository Secrets area.

In our workflow we added the docker build, logging into the GitHub Container registry, and finally the docker push command.

We’ve seen GitHub Actions in Action! The workflow summary and the workflow job output too.

Finally, we’ve pulled down our Docker image from our public GitHub Container Registry and tested it locally. There is different compared to how we’ve done it before in this blog series, previously we’ve built the docker image locally and it’s already in our local Docker repository. We are now using the same flow as the base images you’ve used in our Dockerfile, and what other companies / individuals have stored in Docker Hub. Equally, you do not have to use the GitHub Container Registry, you could simply sign-up to Docker Hub. Not to forget there are other 3rd party solutions such as:

...and the list goes on!

Up next

Part 4 in this series will be about:

Add container scanning to our GitHub Action workflow
Resolve security vulnerabilities