DEV Community: Nico Meisenzahl

Azure Kubernetes Service — Next level persistent storage with Azure Disk CSI driver

Nico Meisenzahl — Wed, 13 Oct 2021 17:01:06 +0000

When talking about persistent storage with Kubernetes Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) are our tools of choice. We can use them with Azure Disks and Azure Files for a long time now. But now it is time to bring them to the next level!

This post mainly focuses on Azure Disks (don’t miss to review the links below to get more details on Azure Files).

What’s new?

With Azure Disk Container Storage Interface (CSI) driver for Azure Disk, we are now getting some great features that help us to run our stateful services much smoother on Azure Kubernetes Service:

Zone Redundant Storage (ZRS) Azure Disks
ReadWriteMany with Azure Disks
Kubernetes-native volume snapshots
Volume resizing
Volume cloning

Azure Disk Container Storage Interface driver

Since Kubernetes 1.21 (on Azure Kubernetes Service) we can use the new Container Storage Interface implementation which is available for Azure Disk and Azure Files.

Wait, what is Container Storage Interface?

The Container Storage Interface (CSI) is an abstraction layer that allows third-party storage providers to write plugins exposing new storage systems in Kubernetes. CSI is generally available for some time now and replaces the previous volume plugin system. With CSI third-party providers can develop and maintain their plugins outside of the Kubernetes project lifecycle which brings them more flexibility. CSI is the future for any storage integration and is already used in many scenarios.

How to use the Azure Disk CSI driver with Azure Kubernetes Service?

You will need at least Kubernetes 1.21 or newer to be able to use the Azure Disk Container Storage Interface driver. Besides this you will also need to enable some Azure features gates to be able to use all of the above-mentioned features:

az feature register --name SsdZrsManagedDisks --namespace Microsoft.Compute
az feature register --name EnableAzureDiskFileCSIDriver --namespace Microsoft.ContainerService
az feature list -o table --query "[?contains(name, 'Microsoft.Compute/SsdZrsManagedDisks')].{Name:name,State:properties.state}"
az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/EnableAzureDiskFileCSIDriver')].{Name:name,State:properties.state}"
az provider register --namespace Microsoft.Compute
az provider register -n Microsoft.ContainerService

You will then need to spin up an Azure Kubernetes Cluster with the CSI driver feature enabled:

az group create -g aks-1-22 -l westeurope
az aks create -n aks-1-22 \
  -g aks-1-22 \
  -l westeurope \
  -c 3 \
  -s Standard_B2ms \
  --node-zones 1 2 3 \
  --aks-custom-headers EnableAzureDiskFileCSIDriver=true
az aks get-credentials -n aks-1-22 -g aks-1-22

Furthermore, you will need to create a StorageClass and VolumeSnapshotClass to use all mentioned features:

cat <<EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azuredisk-csi-premium-zrs
provisioner: disk.csi.azure.com
parameters:
  skuname: Premium_ZRS
  maxShares: "3"
  cachingMode: None
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
  name: azuredisk-csi-vsc
driver: disk.csi.azure.com
deletionPolicy: Delete
parameters:
  incremental: "true"
EOF

The StorageClass is configured to use the new Premium_ZRS Azure Disks that allow to mount the Disk to multiple Pods across multiple Availability Zones.

The VolumeSnapshotClass allows us to use the Kubernetes-native volume snapshot feature.

We are now ready to create a PersistentVolumeClaim based on the above StorageClass:

cat <<EOF | kubectl apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-azuredisk-csi-premium-zrs
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 256Gi
  volumeMode: Block
  storageClassName: azuredisk-csi-premium-zrs
EOF

Kubernetes-native volume snapshots

With the Azure Disk CSI driver, we are able to create volume snapshots. You will find more details on the volume snapshot feature itself here. Use the following code snippet to create a snapshot of the above create PVC:

cat <<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: azuredisk-volume-snapshot
spec:
  volumeSnapshotClassName: azuredisk-csi-vsc
  source:
    persistentVolumeClaimName: pvc-azuredisk-csi-premium-zrs
EOF

You can now verify the snapshot with the following kubectl command:

kubectl describe volumesnapshot azuredisk-volume-snapshot

Volume cloning

Volume snapshots themselves are great but we can also use them to clone an existing PVC. For example to attach it to another workload. This can be done with the following manifest:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-azuredisk-snapshot-restored
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: azuredisk-csi-premium-zrs
  resources:
    requests:
      storage: 265Gi
  dataSource:
    name: azuredisk-volume-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
EOF

Once again you can review it with kubectl:

kubectl get pvc,pv

Volume resizing

The CSI driver also supports resizing an existing PVC. Some important notes on this one:

The PVC needs to be unmounted for resizing
The PV will be resized directly, the PVC after mounting it

You can resize the above-cloned PVC via kubectl by executing the following command:

kubectl patch pvc pvc-azuredisk-snapshot-restored --type merge --patch '{"spec": {"resources": {"requests": {"storage": "512Gi"}}}}'

ReadWriteMany across Availability Zones

As already mentioned above we are now also able to mount our PVC to multiple workloads via ReadWriteMany. With the new ZRS Azure Disks, we are even further able to use them across Availability Zones.

Let’s test these new features with the below Deployment manifest:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          volumeDevices:
            - name: azuredisk
              devicePath: /dev/sdx
      volumes:
        - name: azuredisk
          persistentVolumeClaim:
            claimName: pvc-azuredisk-csi-premium-zrs
EOF

As you see above are spinning up a deployment with three replicas across our spread Kubernetes nodes mounting the same Azure Disk with read-write access. You can review it by executing the following kubectl command:

kubectl get pods,volumeattachments -owide

That said, (as you may have already seen) ZRS only supports Block devices so far.

More details

Don’t miss to review the following links to get some more insights and further documentation:

Your Attackers Won't Be Happy — How GitLab Can Help You Secure Your Cloud-Native Applications!

Nico Meisenzahl — Sat, 26 Sep 2020 21:22:18 +0000

In the cloud-native ecosystem, decisions and changes are made on a rapid basis. Applications get adapted and deployed multiple times a week or even day. Microservices get developed decentralized with different peoples and teams involved. In such an environment, it is crucial to ensure that applications are developed and operated safely. This can be done by shifting security left into the developer lifecycle but also by using DevSecOps to empower operations with enhanced monitoring and protection for the application runtime.

In this article, I would like to show you how GitLab can help you streamline your application security from a code and operations point of view by providing you with real-world examples. Before we deep dive into the example, let me first introduce you to the GitLab Secure and GitLab Defend product portfolio which are the foundation for this. GitLab Secure helps developers to enable accurate, automated, and continuous assessment of their applications by proactively identifying vulnerabilities and weaknesses and therefore minimizing security risk. GitLab Defend, on the other hand, supports operations by proactively protecting environments and cloud-native applications by providing context-aware technologies to reduce overall security risk. Both are backed by leading open-source projects that have been fully integrated into developer and operation processes and the GitLab user interface (UI).

The attack

Let’s assume we have an application hosting a web interface that allows a user to provide some input. The application is written in Golang and executes the input as part of an external operating system command (os/exec). The application does not contain any validation or security features to validate the input, which allows us to inject additional commands that are also executed in the application environment.

The application is running as containerized microservices in a Kubernetes cluster. The Kubernetes Cluster is shared across multiple teams and projects, allowing us to inject and read data in another application running next to ours. In our example, we will connect an unsecured Redis instance in a different Namespace and read/write data.

Now let us take a closer look at how GitLab can help us detect the attack, permit its execution, and finally help us find and fix the root cause in our code.

Container Host Security

Container Host Security helps us to detect an attack in real-time by monitoring the pod for any unusual activity. It can then alert operations with detailed information on the attack itself.

Container Host Security is powered by Falco, an open-source runtime security tool that listens to the Linux kernel using eBPF. Falco parses system calls and asserts the stream against a configurable rules engine in real-time. The Falco deployment used by Container Host Security can be deployed and fully managed using GitLab Managed Apps.

In our example, Falco detects the injected redis-cli command, which is used to read/write data into the unsecured Redis instance.

Falco can now alert operations who can use those valuable insights to define and execute further steps.

Container Network Security

A first step to permit access to the unsecured Redis instance would be to permit traffic between the application in our Kubernetes cluster. This can be done by using Container Network Security. Container Network Security is again fully managed by GitLab Managed Apps and can also be configured within the GitLab project user interface.

Container Network Security is powered by Cilium, an open-source networking plugin for Kubernetes that can be used to implement support for NetworkPolicy resources. Network Policies can be used to detect and block unauthorized network traffic between pods and to/from the Internet.

Implementing Network Policies for our application will block the underlying network traffic generated by the attack. The policies can be enabled within the GitLab project UI:

Web Application Firewall

With Container Network Security in place, our attack isn’t able to talk to the Redis instance anymore, but it is still possible to execute other network unrelated attacks using the command injection. Web Application Firewall (WAF) can now help us to increase the security and detect and block the attack at the Kubernetes Ingress level.

The Web Application firewall is also powered by open-source. It is based on the ModSecurity module, a toolkit for real-time web application monitoring, logging, and access control. It is preconfigured to use the OWASP’s Core Rule Set, which provides generic attack detection capabilities. Like the other integrations, Web Application Firewall is also fully managed by GitLab using GitLab Managed Apps.

In our example, the Web Application Firewall detects the attack and is also able to block it:

Blocking the attack at the Ingress level will help us to deny the traffic before it hits our application. To do so, we can enable the Web Application Firewall blocking mode directly from the GitLab UI:

In addition to Container Host Security, we could have used the Web Application Firewall to detect the attack using the Thread Monitoring dashboard within our GitLab project:

The Thread Monitoring dashboard also provides us with useful insights and metrics of our enforced Container Network Policy.

Static Application Security Testing

We have now successfully protected our application runtime and ensured that no additional attacks can be executed. But we should also find and fix the root cause to ensure that such incidents are not recurring in the future. This is where Static Application Security Testing (SAST) can help us. Static Application Security Testing can be easily integrated into our project using GitLab CI/CD and then allows us to analyze our source code for known vulnerabilities.

In our case (a Golang application) the code scanning is executed using the open-source project Golang Security Checker. The results are displayed in the Security dashboard of our GitLab project for easy access:

In our example, the code scan has identified the root cause and provides us with detailed information about the vulnerability, the line of code that needs to be fixed, and the ability to easily create an issue to fix it.

Finally, of course, we should also talk to the team running the other application to make sure that their Redis instance gets secured too. We should also verify how the other GitLab Secure features can help to further improve the overall security of the application.

GitLab Defend and Secure in action

If you like to get more insights on GitLab Secure and Defend and want to see it in action, you are welcome to check out Wayne, Philippe and myself in our session “Your Attackers Won't Be Happy! How GitLab Can Help You Secure Your Cloud-Native Applications!” at GitLab Commit where you can gain further insights on Container Host Security, Container Network Security, Web Application Firewall (WAF), and Status Application Security Testing (SAST).

Azure Service Operator - manage your Azure resources with Kubernetes

Nico Meisenzahl — Mon, 29 Jun 2020 07:01:43 +0000

Before I introduce you to Azure Service Operator and how it helps you to manage your Azure resources with Kubernetes let me briefly start with why you should use it and where it can help. Let me give you two examples:

Think of a common cloud-native application. Some microservices running on Kubernetes, using Redis for caching and a database to persist state. In such a scenario a common practice is to store and manage the application and its dependencies together. Until now you might have packed your microservices into a Helm chart for easier deployment and also created some Terraform code to deploy and manage the Redis and database. But those are still not linked together and are also deployed via two different continuous delivery pipelines. You could now argue that for example, the Terraform Helm provider could fix this issue by combining your application and infrastructure. But do you like your developers to learn and use another tool - mainly used for infrastructure management? Wouldn't it be better to just manage the application dependencies together with the application itself in a Helm chart? This is where Azure Service Operator can help!

Another example would be GitOps. With GitOps, Git is the single source of truth. Applications and infrastructure are defined in a declarative manner, stored in Git, and automatically updated and managed by Kubernetes (to be more precise by Kubernetes Controllers). With Azure Service Operator in combination with, for example, FluxCD Kubernetes can also manage Azure resources using the GitOps approach.

What Azure Service Operator is and how it works

So now that we know how the Azure Service Operator project can help us, we should talk about what it is exactly and how it works.

Azure Service Operator is an open-source project by the Microsoft Azure. It is a pretty new project that got announced last week. The whole project, as well as the roadmap, is available on GitHub.

The Azure Service Operator consists of:

Custom Resource Definitions (CRDs) for each of the Azure services. CRDs are Kubernetes API extensions. This enables us to create Kubernetes resources of kind RedisCache or SQLServer.
A Kubernetes controller that watches for changes of the CRDs and then acting (creates, update, delete the Azure resources) on them.

Beside these Azure Service Operator also depends on:

Cert-manager to manage internal certificates. Cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources. Cert-manager is not part of the Azure Service Operator and needs to be installed upfront.
Azure AD (AAD) Pod Identity is used to manage authentication against Azure when managed identities are used. AAD Pod Identity is part of the Azure Service Operator and is provided as a Helm subchart.

With this abstraction, Azure Service Operator is not limited to be used with Azure Kubernetes Service, but can also be used with any Kubernetes cluster — regardless of whether it runs in a public or private cloud.

Further technical details on how Azure Service Operator works can be found here.

So far Azure Service Operator supports the following Azure resources:

Resource Group
Event Hubs
Azure SQL
Azure Database for PostgreSQL
Azure Database for MySQL
Azure Key Vault
Azure Cache for Redis
Storage Account
Blob Storage
Virtual Network
Application Insights
API Management
Cosmos DB
Virtual Machine
Virtual Machine Scale Set

Get started with Azure Service Operator

Below you will find all the steps necessary to get started with Azure Service Operator. A more detailed guide is available here.

First of all, we need to install Cert-Manager:

kubectl create namespace cert-manager
kubectl label namespace cert-manager cert-manager.io/disable-validation=true
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --set installCRDs=true

Before we can install the Azure Service Operator, we need to create a service principal that is used for authentication against Azure. We will then assign it contributor access on a subscription level (as mentioned above, it is also possible to use managed identities, which requires AAD Pod identity and AKS):

NAME=aso-sp
AZURE_TENANT_ID=$(az account show --query '[tenantId]' -o tsv)
AZURE_SUBSCRIPTION_ID=$(az account show --query '[id]' -o tsv)
AZURE_CLIENT_SECRET=$(az ad sp create-for-rbac -n $NAME --role contributor --year 99 --query '[password]' -o tsv)
AZURE_CLIENT_ID=$(az ad sp list --display-name $NAME --query '[].appId' -o tsv)

Now we are ready to install Azure Service Operator using Helm:

export HELM_EXPERIMENTAL_OCI=1
helm chart pull mcr.microsoft.com/k8s/asohelmchart:latest
helm chart export mcr.microsoft.com/k8s/asohelmchart:latest --destination .
helm install aso -n azureoperator-system --create-namespace \
  --set azureSubscriptionID=$AZURE_SUBSCRIPTION_ID \
  --set azureTenantID=$AZURE_TENANT_ID \
  --set azureClientID=$AZURE_CLIENT_ID \
  --set azureClientSecret=$AZURE_CLIENT_SECRET \
  --set createNamespace=true \
  --set image.repository="mcr.microsoft.com/k8s/azureserviceoperator:latest" \
  ./azure-service-operator

At this point we are ready to test our installation by creating a first resource group:

cat <<EOF | kubectl apply -f -
apiVersion: azure.microsoft.com/v1alpha1
kind: ResourceGroup
metadata:
  name: aso-test-rg
spec:
  location: "westeurope"
EOF

The output of kubectl get resourcegroups aso-test-rg will give us details about the status of resource creation. Once we see successfully provisioned the resource is available.

kubectl get resourcegroups
NAME           PROVISIONED   MESSAGE
aso-test-rg    true          successfully provisioned

Bundle your application with its infrastructure dependencies

As we now know how Azure Service Operator works, we are going to look at how we can use it to bundle our application together with our infrastructure. We will use the Azure voting app as an example. It consists of a Python microservice provided as a deployment, a LoadBalancer service to publish it externally, and Azure Cache for Redis to persist the state. Let’s take a look at the manifest:

apiVersion: azure.microsoft.com/v1alpha1
kind: ResourceGroup
metadata:
  name: azure-vote-rg
spec:
  location: westeurope
---
apiVersion: azure.microsoft.com/v1alpha1
kind: RedisCache
metadata:
  name: azure-vote-redis
spec:
  location: westeurope
  resourceGroup: azure-vote
  properties:
    sku:
      name: Basic
      family: C
      capacity: 1
    enableNonSslPort: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-front
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-front
  template:
    metadata:
      labels:
        app: azure-vote-front
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: azure-vote-front
        image: microsoft/azure-vote-front:v1
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 80
        env:
        - name: REDIS_NAME
          valueFrom:
            secretKeyRef:
              name: azure-redis
              key: redisCacheName
        - name: REDIS
          value: $(REDIS_NAME).redis.cache.windows.net
        - name: REDIS_PWD
          valueFrom:
            secretKeyRef:
              name: azure-redis
              key: primaryKey
--------
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: azure-vote-front

We will now take a closer look at the above definitions. In the first step, we declare a resource group as before. Next, we declare an Azure Cache for Redis by specifying a name, a location, a resource group, and several Redis-related parameters. If we take a closer look at the deployment itself, we may notice the following:

        env:
        - name: REDIS_NAME
          valueFrom:
            secretKeyRef:
              name: azure-redis
              key: redisCacheName
        - name: REDIS
          value: $(REDIS_NAME).redis.cache.windows.net
        - name: REDIS_PWD
          valueFrom:
            secretKeyRef:
              name: azure-redis
              key: primaryKey

Azure Service Operator will not only provide the resources for us but will also ensure that we can discover the secret and the name by putting them into a Kubernetes secret (you can also use Azure Keyvault for that). In this way, we can now easily inject them into our containers.

Manage your Azure resources with a GitOps approach

Now that everything above is in place, we are ready to talk about using Azure Service Operator in a GitOps approach. In the following example, we will use FluxCD to achieve this.

First of all, you need to install the fluxctl CLI. Afterward, we install FluxCD and provide our Git repository containing our manifests (you can reuse my repository by forking it):

helm repo add fluxcd https://charts.fluxcd.io
helm repo update
helm upgrade -i flux fluxcd/flux \
  --set git.url=git@github.com:nmeisenzahl/aso-fluxcd-sample.git \
  --set git-path=workloads \
  --namespace flux

After the installation is complete, we have to give FluxCD access to our repository. This is done with an SSH key, which you can get with fluxctl:

fluxctl identity --k8s-fwd-ns flux

Once you add the SSH key to your profile, FluxCD will immediately start to create your resources based on the manifests. FluxCD will now ensure that all changes are applied.

Scaffold a production-ready Terraform project on Azure

Nico Meisenzahl — Wed, 17 Jun 2020 06:45:59 +0000

This post is an updated version of my previous post “Using Terraform with Azure” that I published some time ago. Now, nearly one year later, I have learned a lot and also optimized and extended the examples and code snippets here and there. As a result, we decided to publish all code in this public GitHub repository. This post should provide you with some further details on the project and any details around it. All below code snippets are related to this project.

The project

As mentioned above, we decided to publish everything needed to scaffold a new production-ready Terraform project on Azure. The project is called “Terraform scaffold for Azure” and is available here.

To get started you need to make sure to meet the following requirements:

a Unix-Shell (we might support Powershell Core in the future, you can use Azure Cloud Shell until then)
Azure CLI
Owner access on the target Subscription (to allow Terraform to add future Service Principals as contributors)

If those requirements are met, you only need to execute a single script called up.sh. This will then provision the needed resources:

a Service Principal used to run Terraform on behalf
a Storage Container used to store the Terraform state file
a Key Vault containing all secrets to allow easy and secure access

After executing the script you are ready to create your first Terraform project on Azure. But before that, I would like to give you some details about the project itself.

Where to store the state store

It is best practices to store the Terraform state file in a secure and central location. Secure, because it contains sensible data like secrets. Central, because you may like to work with your entire team on the same project.

Speaking of Azure the best location to store it is in a Storage Container as Blob which is supported by Terraform by default. The needed Storage Container will be created by the up.sh script.

You then need to link the Storage Container within your Terraform project by defining some setting in your main.tf configuration file:

provider "azurerm" {
  version = "~> 2.1"
  features {}
}

terraform {
    backend "azurerm" {
        key = "azure.tfstate"
    }
}

Why you should use a Service Principal

It is best practice to run Terraform on behalf of a Service Principal. That allows you to either restrict access rights to those needed for your particular deployment. But, you can also give your Service Principal extended privileges that your personal account might not be allowed to have.

The Service Principal will be also be created by the up.sh script. By default, it will be added with owner access rights to allow it to enable future Service Principals which might be created within your Terraform project. This can, of course, be adjusted to your personal needs.

To use a particular Service Principal, we need to make the Terraform CLI aware of it by providing the following environment variables in advance:

export ARM_SUBSCRIPTION_ID=<your-subscription-id>
export ARM_TENANT_ID=<your-tenant-id>
export ARM_CLIENT_ID=<your-service-principal-id>
export ARM_CLIENT_SECRET=<your-service-principal-secret>

Secret management

As you may have noticed, the above setup depends heavily on secrets for various authentications. I already mentioned above that you should never store secrets within your code. In order to simplify the use of this setup but still ensure a high level of security, it is recommended to store and retrieve them from a vault.

Since you are working with Azure, we use Azure Key Vault for this. The up.sh script also takes care of this. It will create the Key Vault itself but also directly store all needed secrets in it. With this in place, you can easily retrieve the needed secrets when needed. Regardless of whether you run the CLI manually, as a script or in a CI/CD pipeline.

This is a sample code snippet that you can use to retrieve the secrets locally or as a task within your CI/CD pipeline:

export subscriptionId="00000000-0000-0000-0000-000000000000"
export rg="my-rg"

# sets subscription
az account set --subscription $subscriptionId

# get vault
export vaultName=$(az keyvault list --subscription=$subscriptionId -g $rg --query '[0].{name:name}' -o tsv)

## extracts and exports secrets
export saKey=$(az keyvault secret show --subscription=$subscriptionId --vault-name="$vaultName" --name sa-key --query value -o tsv)
export saName=$(az keyvault secret show --subscription=$subscriptionId --vault-name="$vaultName" --name sa-name --query value -o tsv)
export scName=$(az keyvault secret show --subscription=$subscriptionId --vault-name="$vaultName" --name sc-name --query value -o tsv)
export spSecret=$(az keyvault secret show --subscription=$subscriptionId --vault-name="$vaultName" --name sp-secret --query value -o tsv)
export spId=$(az keyvault secret show --subscription=$subscriptionId --vault-name="$vaultName" --name sp-id --query value -o tsv)

# exports secrets
export ARM_SUBSCRIPTION_ID=$subscriptionId
export ARM_TENANT_ID=$tenantId
export ARM_CLIENT_ID=$spId
export ARM_CLIENT_SECRET=$spSecret

# runs Terraform init
terraform init -input=false \
  -backend-config="access_key=$saKey" \
  -backend-config="storage_account_name=$saName" \
  -backend-config="container_name=$scName"

With this, you are now ready to build your first production-ready and secure Infrastructure-as-code deployment with Terraform on Azure. As mentioned above, all scripts and further details are available here.

Containerize your .NET Core app – the right way

Nico Meisenzahl — Tue, 16 Jun 2020 06:04:13 +0000

There are already many articles out there that provide you with details on how to containerize your .NET Core application. Nevertheless, I still saw the need to write a bit more detailed post which helps you to build a production-ready container image based on container and .NET Core best practices.

For better understanding, I will explain everything in detail based on a small sample ASPNET Core web application. You will find more details on the application itself here. Of course, the shared best practices are not limited to .NET Core. You can adjust them and use them with any of your projects.

There are millions of use cases out there and that’s why there isn’t a one fits all solution. I would like to introduce you to two different options I use most. You will get all the details to decide which of them works best for you. Let us start with the basics first.

The common way

This is a common example Dockerfile which comes across my way quite often:

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
WORKDIR /app
COPY /app/output .
EXPOSE 8080
ENTRYPOINT ["dotnet", "sample-mvc.dll"]

There is nothing particularly wrong with it. It will work, but it is not tuned or optimized at all. Neither for performance nor for security-related issues and therefore not optimal for a production environment. Some examples are:

containers executing their process as root
an inefficient sorting order that results in slower build times * due to invalid image layer caches
an improper image layer management that affects the final image size
slower builds due to missing .dockerignore file

Let’s have a closer look at another Dockerfile example:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1
WORKDIR /app
ADD /src .
RUN dotnet publish \
  -c Release \
  -o ./output
EXPOSE 8080
ENTRYPOINT ["dotnet", "sample-mvc.dll"]

This example uses a containerized build. This means that the application build itself is moved into the Docker build process. Doing this a pretty good pattern that allows you to build in an immutable and isolated build environment with all dependencies build in. But as a downside, you need to build your image based on a bigger SDK image. The SDK image provides the needed dependencies to build the application but wouldn’t be needed to execute it afterward. Luckily, there is a solution that addresses this particular issue.

Multi-stage builds

If you are using a similar version of the above Dockerfiles you might not have heard about a feature called multi-stage build. Multi-stage builds allow us to split our image build process into multiple stages.
The first stage is used to build our application that requires that we need to provide the required dependencies. In the second stage, we are copying the application artifacts into a smaller runtime environment which then is used as our final image. This corresponding Dockerfile could look like this:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-env
WORKDIR /app
ADD /src .
RUN dotnet publish \
  -c Release \
  -o ./output
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
WORKDIR /app
COPY --from=build-env /app/output .
EXPOSE 8080
ENTRYPOINT ["dotnet", "sample-mvc.dll"]

Let’s take a closer look at the individual steps:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-env
...
RUN dotnet publish \
  -c Release \
  -o ./output
...

Our first stage is based on the SDK image which provides all dependencies to build our app.

...
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1
...
COPY --from=build-env /app/output .
...

In the second stage, we define a new base image that this time only contains our runtime dependencies. We then copy the application artifacts from the first stage into the second one.

With this in place, we are now able to build a smaller and more secure container image because it only contains the dependencies needed to execute the application. But we still have room for further improvement which we talk about in the next paragraph.

If you like to learn more about Dockerfile best practices I would recommend you check out this page of the official Docker documentation.

You have the choice

As already mentioned above, there isn’t a single best practice. It varies from use case to use case. With the examples below, you will get two blueprints as well as their pros and cons, which you can then use to adapt them according to your needs.

A good starting point

The below Dockerfile is an optimized version of the above multi-stage example and should be a good fit for most scenarios.

ARG VERSION=3.1-alpine3.10
FROM mcr.microsoft.com/dotnet/core/sdk:$VERSION AS build-env
WORKDIR /app
ADD /src/*.csproj .
RUN dotnet restore
ADD /src .
RUN dotnet publish \
  -c Release \
  -o ./output
FROM mcr.microsoft.com/dotnet/core/aspnet:$VERSION
RUN adduser \
  --disabled-password \
  --home /app \
  --gecos '' app \
  && chown -R app /app
USER app
WORKDIR /app
COPY --from=build-env /app/output .
ENV DOTNET_RUNNING_IN_CONTAINER=true \
  ASPNETCORE_URLS=http://+:8080
EXPOSE 8080
ENTRYPOINT ["dotnet", "sample-mvc.dll"]

Again, we take a closer look at the individual steps:

ARG VERSION=3.1-alpine3.10
FROM mcr.microsoft.com/dotnet/core/sdk:$VERSION AS build-env
...

We are first defining our base image tag using an ARG instruction. This helps us to easily update the tag instead of changing several lines. As you may have noticed, we use a different tag. The tag3.1-alpine3.10 states that this image contains the ASPNET version 3.1 and is based on Alpine 3.10.

Alpine Linux is a Linux distribution designed for security, simplicity, and resource efficiency use cases. In this stage, Alpine Linux already can help us to reduce the footprint of our build stage.

...
FROM mcr.microsoft.com/dotnet/core/aspnet:$VERSION
...

Because we are using a multi-stage build we also need to define the image used in our final stage. Once again we will use the Alpine based ASPNET runtime as our base image. As already said, building our image based on Alpine allows us to build a smaller and more secure container image.

ADD /src/*.csproj .
RUN dotnet restore
ADD /src .
RUN dotnet publish \
  -c Release \
  -o ./output

Unlike in the above example, we this time splitting the build process into multiple pieces. The dotnet restore command uses NuGet to restore dependencies as well as project-specific tools that are specified in the project file. The dependencies restore is also part of the dotnet pubish command but separating it allows us to build the dependencies into a separate image layer. This shortens the time needed to build the image and reduces the download size since the image layer dependencies are only rebuilt if the dependencies get changed.

...
RUN adduser \
  --disabled-password \
  --home /app \
  --gecos '' app \
  && chown -R app /app
USER app
...

To secure the runtime of our application we need to execute them without any root privileges. Because of this, we are creating a new user and changing the user context using the USER definition.

...
ENV DOTNET_RUNNING_IN_CONTAINER=true \
  ASPNETCORE_URLS=http://+:8080
EXPOSE 8080
...

Because we run our app without any root privileges we need to expose it on a port higher 1024. In this example, 8080 was chosen. With the ENV definition, we are exposing further environment variables to our application process. DOTNET_RUNNING_IN_CONTAINER=true is only an informal environment variable to let a developer/application know that the process is running within a container. ASPNETCORE_URLS=http://+:8080 is used to provide the runtime with the information to expose the process on port 8080.

Smaller, smaller, smaller

As already mentioned, the above example should fit for most of the scenarios. The following example describes a way to build the smallest possible container image. A possible use-case might be for IoT Edge use cases or environments that need optimized start times. Unfortunately, we also get some disadvantages which I will talk about in detail below.

ARG VERSION=3.1-alpine3.10
FROM mcr.microsoft.com/dotnet/core/sdk:$VERSION AS build-env
WORKDIR /app
ADD /src .
RUN dotnet publish \
  --runtime alpine-x64 \
  --self-contained true \
  /p:PublishTrimmed=true \
  /p:PublishSingleFile=true \
  -c Release \
  -o ./output
FROM mcr.microsoft.com/dotnet/core/runtime-deps:$VERSION
RUN adduser \
  --disabled-password \
  --home /app \
  --gecos '' app \
  && chown -R app /app
USER app
WORKDIR /app
COPY --from=build-env /app/output .
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 \
  DOTNET_RUNNING_IN_CONTAINER=true \
  ASPNETCORE_URLS=http://+:8080
EXPOSE 8080
ENTRYPOINT ["./sample-mvc", "--urls", "http://0.0.0.0:8080"]

Once again, we take a closer look at the individual steps:

...
RUN dotnet publish \
  --runtime alpine-x64 \
  --self-contained true \
  /p:PublishTrimmed=true \
  /p:PublishSingleFile=true \
  -c Release \
  -o ./output
...

The big difference to the upper one is that we will build a self-contained application. Providing the parameter --self-contained true will force the build to include all dependencies into the application artifact. Wich includes the .NET Core runtime. Because of this, we also need to define the runtime we would like to execute the binary in. This is done with the --runtime alpine-x64 parameter.

Since the final image should be optimized for size we are defining the /p:PubishTrimmed=true flag that advises the build process to not include any unused libraries. The /p:PublishSingleFile=true flag allows us to speed up the build process itself. As a downside, you will have to define dynamically loaded assemblies upfront to make sure that required libraries aren’t trimmed and therefore not available in the image. More details on this are available here.

A second disadvantage of having a smaller image is that code changes result in a bigger change. This is because the code and runtime are packed together in a single image layer. Every time the code changes the whole image layer needs to rebuild and also redistributed to the system running the code.

...
FROM mcr.microsoft.com/dotnet/core/runtime-deps:$VERSION
...

Because the application artifact is self-contained we do not need to provide a runtime with the image. In this example, I have chosen the runtime-deps image based on Alpine Linux. This image is stripped down to the minimum native dependencies needed to execute the application artifact.

...
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 \
  DOTNET_RUNNING_IN_CONTAINER=true \
  ASPNETCORE_URLS=http://+:8080
...

Another image size improvement is to use the globalization invariant mode. This mode is useful for applications that are not globally aware and that can use the formatting conventions, casing conventions, and string comparison and sort order of the invariant culture. The globalization invariant mode is enabled via the DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 environment variable. If your application requires globalization you will need to install the ICU library and remove the above environment variable. This will increase your container image size by about 28 MB. You will find more details on the globalization invariant mode here.

...
ENTRYPOINT ["./sample-mvc", "--urls", "http://0.0.0.0:8080"]

For self-contained applications, we need to change the ENTRYPOINT definition to run the binary itself.
The size of this image will be around 73 MB (including my sample application). Let’s compare this to other images:

an image based on a common multi-stage Dockerfile: 250 MB
an image based on the above multi-stage Dockerfile: 124 MB

As already mentioned above: Which Dockerfile is most suitable for you depends on your use case. Smaller is not necessarily better.

If you are planning to deploy your application to Azure Kubernetes Service or Azure Container Instances you might also think about storing your images in the Azure Container Registry. Azure Container Registry also supports to build container images. You can include it in your build pipeline or just call it manually using the az acr build command.

I hope that these details help you to containerize your .NET Core application. As already mentions the above examples are best practices and might need to be customized to fit your needs. Check out Michael Dimoudis post on details about how to harden your .NET Core container images.

Policy and Governance for Kubernetes

Nico Meisenzahl — Mon, 15 Jun 2020 17:27:11 +0000

Before I talk about Policy and Governance for Kubernetes let’s briefly talk about policy and governance in general. In short, it means to provide a set of rules which define a guideline that either can be enforced or audited. So why do we need this? It is important because in a Cloud ecosystem decisions are made decentralized and also taken at a rapid pace. A governance model or policy becomes crucial to keep the entire organization on track. Those definitions can include but are not limited to, security baselines or consistency of resources and deployments.

So, why do we need Governance and Policy for Kubernetes? Kubernetes provides Role-based Access Control (RBAC) which allows Operators to define in a very granular manner which identity is allowed to create or manager which resource. But RBAC does not allow us to control the specification of those resources. As already mentioned this is a necessary requirement to be able to define the policy boundaries. Some examples are:

whitelist of trusted container registries and images
required container security specifications
required labels to group resources
permit conflicting Ingress host resources
permit publicly exposed LoadBalancer services
…

This is where Policy and Governance for Kubernetes comes in. But let me first introduce you to Open Policy Agent. Open Policy Agent is the foundation for policy management on Kubernetes or even the whole cloud-native ecosystem.

Open Policy Agent

Open Policy Agent (OPA) is an open-source project by styra. It provides policy-based control for cloud-native environments using a unified toolset and framework and a declarative approach. Open Policy Agents allows decoupling policy declaration and management from the application code by either integrating the OPA Golang library or calling the REST API of a collocated OPA daemon instance.

With this in place, OPA can be used to evaluate any JSON-based inputs against user-defined policies and mark the input as passing or failing. With this in place, Open Policy Agent can be seamlessly integrated with a variety of tools and projects. Some examples are:

API and service authorization with Envoy, Kong or Traefik
Authorization policies for SQL, Kafka and others
Container Network authorization with Istio
Test policies for Terraform infrastructure changes
Polices for SSH and sudo
Policy and Governance for Kubernetes

Open Policy Agent policies are written in a declarative policy language called Rego. Rego queries are claims about data stored in OPA. These queries can be used to define policies that enumerate data instances that violate the expected state of the system.

You can use the Rego playground to get familiar with the policy language. The playground also contains a variety of examples that help you to get started writing your own queries.

OPA Gatekeeper — the Kubernetes implementation

Open Policy Agent Gatekeeper got introduced by Google, Microsoft, Red Hat, and styra. It is a Kubernetes Admission Controller built around OPA to integrate it with the Kubernetes API server and enforcing policies defined by Custom Resource Definitions (CRDs).

The Gatekeeper webhook, gets invoked whenever a Kubernetes resource is created, updated, or deleted which then allows Gatekeeper to permit it. In addition, Gatekeeper can also audit existing resources. Polices, as well as data, can be replicated into the included OPA instance to also create advanced queries that for example need access to objects in the cluster other than the object under current evaluation.

Get started with Gatekeeper

This post will not cover how to install Gatekeeper. All steps as well as detailed information are available here.

After installing Gatekeeper you are ready to define your first policies. But let’s first review what we got:

Gatekeeper consists out of two deployments:

gatekeeper-controller-manager is the Admission Controller monitoring the Kubernetes API server for resource changes, verifying policy definitions, and permitting actions.
gatekeeper-audit is used to verifying and auditing existing resources.

Besides these two deployments, also two Custom Resource Definitions (CRDs) got created:

ConstraintTemplate is used to define a policy based on metadata and the actual Rego query.
Config is used to define which resources should be replicated to the OPA instance to allow advanced policy queries.

A first example

In this example, we want to make sure that all labels required by the policy are present in the Kubernetes resource manifest.

To do this, we first have to build our Rego query:

package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

It consists of a package containing a violation definition. The violation defined the input data, the condition to be matched, and a message which gets returned in case of a violation.

We now need to make Gatekeeper aware of our definition. This is done by wrapping our Rego query into a ContrainTemplate resource:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
        listKind: K8sRequiredLabelsList
        plural: k8srequiredlabels
        singular: k8srequiredlabels
      validation:
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

The ConstraintTemplate resource contains out of the Rego query as well as some metadata. As soon as we apply the ConstraintTemplate resource a new CRD will be created. In our case, it is called K8sRequiredLabels. This allows us to easily interact with our policy later on.

We now defined our policy but you may have noticed that we not yet defined which label we require our Kubernetes resources to have. We also did not yet define on which Kubernetes resources we like to have this policy applied too. To do so, we must first create another manifest called Constraints:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-gk
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["gatekeeper"]

The Constraints manifest allows us to enforce our above-defined policy by providing our parameters as well as the Kubernetes resource it should act on. In our example, we want to enforce this policy for all namespace resources and make sure that they all contain a gatekeeper label.

If we now apply the above manifest, we cannot create a namespace without providing a gatekeeper label. Instead, our configured message is returned. If we provide a label the namespace gets created.

The Gatekeeper repository provides some more examples which can be used to get started.

Policy auditing

Let’s talk about auditing. Why do we need it? Well, we may have created resources before we created policies, and we would like to review them to make sure they meet our policies. Or maybe we just want to audit our cluster without enforcing the policies.

For this we can use our K8sRequiredLabels Custom Resource Definition we created above and retrieve our auditing information from the Kubernetes API server using kubectl or any other tool of our choice. In the below example I used kubectl describe K8sRequiredLabels ns-must-have-gk to get the following output:

Furthermore, Gatekeeper provides us with Prometheus endpoints that can be used to collect metrics and build dashboards and alerts based on them. Both, gatekeeper-controller-manager as well as gatekeeper-audit are exposing those endpoints which can be used to gather the following metrics:

gatekeeper_constraints: Current number of constraints
gatekeeper_constraint_templates: Current number of constraint templates
gatekeeper_constraint_template_ingestion_count: The number of constraint template ingestion actions
gatekeeper_constraint_template_ingestion_duration_seconds: Constraint Template ingestion duration distribution
gatekeeper_request_count: The number of requests that are routed to admission webhook from the API server
gatekeeper_request_duration_seconds: Admission request duration distribution
gatekeeper_violations: The number of audit violations per constraint detected in the last audit cycle
gatekeeper_audit_last_run_time: The epoch timestamp since the last audit runtime
gatekeeper_audit_duration_seconds: Audit cycle duration distribution

All you have to do is make your Prometheus instance aware of it. The easiest way is to customize both deployments and add the well-known annotations to the pod specification to enable Prometheus scraping:

annotations:
  prometheus.io/port: "8888"
  prometheus.io/scrape: "true"

With this in place, you can collect the above Gatekeeper metrics and use them to build auditing dashboards as well as alerting.

Happy auditing! 😃