DEV Community: boncheff

Table Driven Unit Tests in Go

boncheff — Sun, 13 Jun 2021 14:45:20 +0000

Whether you have been writing code for some years, or just getting started in software engineering, table driven tests have something to offer to you.
In this article I will introduce you to table driven tests, and show you a few ways you can incorporate them in your codebase.

Let's get started!

Testing your code, in particular with unit tests, can save you a lot of time!

As the popular expression goes, "6 hours of debugging can save you 5 minutes of reading the documentation". We have all been there! But would we really need to spend 6 hours debugging our code if we had written good tests in the first place? Probably, not. There are plenty of articles on how to write good unit tests - this article is not one of them - so now that we got this out of the way, let me take you on a little journey.

Imagine you work in a bank, and are writing the code to perform a transfer of funds from one bank account to another (i.e. checking account to savings account). For the sake of simplicity, we can only use USD as a currency:

I want to transfer 150.99 USD from my checking account to my savings account.



type transferRequest struct {
     amount               float64
     currency             string 
     originAccountID      string 
     destinationAccountID string
}

func validateTransferRequest(request transferRequest) error {
       if request.amount <= 0 {
              return errors.New("invalid amount")
       }

       if request.currency != "USD" {
              return errors.New("invalid currency: must be USD")
       }

       if request.originAccountID == "" {
              return errors.New("empty origin account ID")
       }

       if request.destinationAccountID == "" {
              return errors.New("empty destination account ID")
       }

       return nil // request is valid so we return nil
}

How do you go about testing this function? I can think of a few ways:

Approach 1: if you do not mind code repetition, you can write a test case to validate each request field
Approach 2: if you are in a rush, you can write an OK table test, removing some of the repetition from the first approach
Approach 3: if you are a perfectionist, you can write a table test on steroids which makes your test code more readable and extensible

Let's write some code for each approach and see how it looks.

Approach 1 (code duplication):



import (
       "errors"
       "testing"

       "github.com/stretchr/testify/assert"
       "github.com/stretchr/testify/require"
)
func TestValidator(t *testing.T) {
       t.Run("error due to invalid amount", func(t *testing.T) {
              transferRequest := transferRequest{
                     amount:               0,
                     currency:             "USD",
                     originAccountID:      "checking",
                     destinationAccountID: "savings",
              }

              err := validateTransferRequest(transferRequest)
              require.Error(t, err)
              assert.Equal(t, "invalid amount", err.Error())
       })

       t.Run("error due to invalid currency", func(t *testing.T) {
              transferRequest := transferRequest{
                     amount:               150.99,
                     currency:             "INR",
                     originAccountID:      "checking",
                     destinationAccountID: "savings",
              }

              err := validateTransferRequest(transferRequest)
              require.Error(t, err)
              assert.Equal(t, "invalid currency: must be USD", err.Error())
       })

       t.Run("error due to an empty origin account ID", func(t *testing.T) {...}

       t.Run("error due to an empty destination count ID", func(t *testing.T) {...}
}

You get the idea - the test cases read nicely but there is a lot of repetition, and although readable, it can be quite overwhelming especially if your test file has hundreds or even thousands of lines of code!

Approach 2 (table driven tests):



import (
       "errors"
       "testing"

       "github.com/stretchr/testify/assert"
       "github.com/stretchr/testify/require"
)
func TestValidatorShouldError(t *testing.T) {
       type errorTestCases struct {
              description   string
              input         transferRequest
              expectedError string
       }

       for _, scenario := range []errorTestCases{
              {
                     description: "invalid amount",
                     input: transferRequest{
                            amount:               0,
                            currency:             "USD",
                            originAccountID:      "checking",
                            destinationAccountID: "savings",
                     },
                     expectedError: "invalid amount",
              },
              {
                     description: "invalid currency",
                     input: transferRequest{
                            amount:               150.99,
                            currency:             "INR",
                            originAccountID:      "checking",
                            destinationAccountID: "savings",
                     },
                     expectedError: "invalid currency: must be USD",
              },
              {
                     description: "invalid origin account ID",
                     input: transferRequest{
                            amount:               150.99,
                            currency:             "USD",
                            originAccountID:      "",
                            destinationAccountID: "savings",
                     },
                     expectedError: "empty origin account ID",
              },
              {
                     description: "invalid destination account ID",
                     input: transferRequest{
                            amount:               150.99,
                            currency:             "USD",
                            originAccountID:      "checking",
                            destinationAccountID: "",
                     },
                     expectedError: "empty destination account ID",
              },
       } {
              t.Run(scenario.description, func(t *testing.T) {
                     err := validateTransferRequest(scenario.input)
                     require.Error(t, err)
                     assert.Equal(t, scenario.expectedError, err.Error())
              })
       }
}

This is already looking a lot better! We removed some of the boilerplate and still ended up with a readable and maintainable test file!

Approach 3 (table driven tests on steroids):



import (
       "errors"
       "testing"

       "github.com/stretchr/testify/assert"
       "github.com/stretchr/testify/require"
)
func TestValidatorShouldError(t *testing.T) {
       for scenario, fn := range map[string]func(t *testing.T){
              "invalid amount":                 testInvalidAmount,
              "invalid currency":               testInvalidCurrency,
              "invalid origin account ID":      testInvalidOriginAccountID,
              "invalid destination account ID": testInvalidDestinationAccountID,
       } {
              t.Run(scenario, func(t *testing.T) {
                     fn(t)
              })
       }
}

func testInvalidAmount(t *testing.T) {
       transferRequest := transferRequest{
              amount:               0,
              currency:             "USD",
              originAccountID:      "checking",
              destinationAccountID: "savings",
       }

       err := validateTransferRequest(transferRequest)
       require.Error(t, err)
       assert.Equal(t, "invalid amount", err.Error())
}

func testInvalidCurrency(t *testing.T) {
       transferRequest := transferRequest{
              amount:               150.90,
              currency:             "INR",
              originAccountID:      "checking",
              destinationAccountID: "savings",
       }

       err := validateTransferRequest(transferRequest)
       require.Error(t, err)
       assert.Equal(t, "invalid currency: must be USD", err.Error())
}

func testInvalidOriginAccountID(t *testing.T) {...}

func testInvalidDestinationAccountID(t *testing.T) {...}

Whoa! Take a moment to let it sink in before you start cursing me!

It might look like all we have done is moved the duplication further down the page, but in reality this approach is a lot more powerful than the previous two.

First, it allows you to quickly scan through the test case definitions and see what we are testing, without getting burdened with the how.

Second, if you have more complicated business logic with a lot of edge cases for each test case (i.e. you only allow transfers where the currency of the origin and destination accounts match, or there is only a specific list of allowed currencies) you can abstract this in the test case function without burdering yourself with how exactly it is being tested, and only dig into each function if required.

So here you have it folks - here are 3 of my favourite ways of writing table driven tests in Go.

Please let me know if you follow any of these approaches yourself, or if you happen to write table tests differently, please let me know so I can learn a different approach!

I hope that you have learned something new or that you simply enjoyed reading it!

CKAD - Revision - Services & Networking

boncheff — Sun, 09 Feb 2020 20:57:18 +0000

A service provides an abstract way to expose an application running on a set of Pods as a network service.

For example, suppose you have a set of Pods that each listen on TCP port 9376 and carry a label app=MyApp:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376

Services can be one of the following four types:

ClusterIP
NodePort
LoadBalancer
ExternalName

ClusterIP is the default and only provides access internally

NodePort is great for when static IP addresses are necessary, such as opening a specific IP through a firewall. It is a simple connection from a high-port routed to a ClusterIP using iptables or ipvs. The creation of a NodePort generates a ClusterIP by default. The traffic is routed from the NodePort to the ClusterIP.

LoadBalancer was created to pass requests to cloud provider like AWS. The address is made available to public traffic and packers are spread among the Pods in the deployment automatically.

ExternalName allows the return of an alias to an external service. It is used for services not yet brought into the Kubernetes cluster (a bit like a placeholder). Once those services are ready we can then change the type and traffic would flow to the internal objects.

kubectl proxy creates a local service to access a ClusterIP which can be useful for troubleshooting

Exposing an application with a Service

We can use kubectl expose command which exposes a resource as a new Kubernetes service.

Ingress

An ingress is an API object that manages external access to the services in a cluster, typically HTTP.

Ingress can provide:

load balancing
SSL termination
name-based virtual hosting

Service Mesh

We can implement a service mesh for more complex connections or resources, such as service discovery, rate limiting, traffic management and advanced metrics.

A service mesh consists of edge and embedded proxies communicating with each other and handling traffic based on rules from the a control plane.

Typically we use Envoy and Istio

CKAD - Revision - Observability

boncheff — Thu, 06 Feb 2020 21:11:26 +0000

Liveness Probes

Many applications running for long periods of time eventually transition to broken states, and cannot recover except by being restarted. Kubernetes provides liveness probes to detect and remedy such situations.

Here is an example of a pod using a liveness probe:

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5 // how long to wait before performing first probe
      periodSeconds: 5 // how often to run the liveness probe

Readiness Probes

Sometimes, applications are temporarily unable to serve traffic. For example, an application might need to load large data or configuration files during startup, or depend on external services after startup. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.

Readiness probes are configured similarly to liveness probes. The only difference is that you use the readinessProbe field instead of the livenessProbe field.

readinessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

CKAD - Revision - Configuration

boncheff — Wed, 05 Feb 2020 21:15:21 +0000

ConfigMaps

Data in ConfigMaps in kubernetes is not encoded or encrypted and contains key-value pairs or plain configuration files in any format.

Here are a few uses of ConfigMaps:

Pod env vars from single or multiple ConfigMaps
Use ConfigMap values in Pod commands
Populate Volume from ConfigMap
Add ConfigMap data to specific path in Volume
Set file names and access mode in Volume from ConfigMap data
Can be used by system components and controllers.

Creating ConfigMaps

ConfigMaps can be created in one of the three following ways:

kubectl create configmap myconfigmap \
--from-literal=city=London \             
--from-file=./myconfigmapfile.txt \
--from-file=./myconfigmapdirectory/

which results in the following ConfigMap:

k get configmap myconfigmap -o yaml

apiVersion: v1
data:
  city: London
kind: ConfigMap
metadata:
  creationTimestamp: "2020-01-12T11:22:43Z"
  name: myconfigmap
  namespace: default
  ...

Security Context

A security context defines privilege and access control settings for a Pod or Container so we can limit what processes running in containers can do. For example we can limit:

the user ID of the process (UID)
the Linux capabilities
filesystem groups

If we want to enforce that containers cannot run their process as root user we can add runAsNonRoot: true to the pod spec. Or we can define a PodSecurityPolicy to that effect.

To automate the enforcement of security contexts, we can define PodSecurityPolicies (PSP)

Pod Security Policies are cluster-level rules that govern what a pod can do, what they can access, what user they run as...

For a PSP to be enabled we must first configure the admission controller of the controller-manager to contain PodSecurityPolicy.

Service Accounts

Service accounts are used by processes to access the API (a service account provides an identity for processes than run in a pod)

CKAD - Revision - Pod Design

boncheff — Wed, 29 Jan 2020 21:31:46 +0000

Labels, Selectors and Annotations

Labels

Labels are key/value pairs that are attached to objects, such as pods. Labels allow for efficient queries and watches and are ideal for use in UIs and CLIs. Non-identifying information should be recorded using annotations.

Here is an example on how to create a pod using kubectl and add a label to it

kubectl run test_pod --image=nginx --restart=Never --labels="country=united_kingdom" --dry-run -o yaml'

which results in the following yaml file

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    country: united_kingdom
  name: test_pod
spec:
  containers:
  - image: nginx
    name: test_pod
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

Selectors

Unlike names and UIDs, labels do not provide uniqueness. In general, we expect many objects to carry the same label(s).

Via a label selector, the client/user can identify a set of objects. The label selector is the core grouping primitive in Kubernetes.

Here is an example of a nodeSelector which selects nodes with the label accelerator=nvidia-tesla-p100:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-test
spec:
  containers:
    - name: cuda-test
      image: "k8s.gcr.io/cuda-vector-add:v0.1"
      resources:
        limits:
          nvidia.com/gpu: 1
  nodeSelector:
    accelerator: nvidia-tesla-p100

Annotations

You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata.

For example, here’s the configuration file for a Pod that has the annotation imageregistry: https://hub.docker.com/:

apiVersion: v1
kind: Pod
metadata:
  name: annotations-demo
  annotations:
    imageregistry: "https://hub.docker.com/"
spec:
  containers:
  - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80

Deployments

A deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state and the deployment controller changes the actual state to the desired state at a controlled rate.

To create a deployment you can run:

kubectl create deployment my-dep --image=busybox

or create using a file using kubectl create -f <path_to_file.yaml>

The following is the resulting yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: test
  name: test
spec:
  replicas: 1
  selector:
    matchLabels: <----- defines how the Deployment finds which Pods to manage
      app: test
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: test
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}

Rolling updates

We can do a rolling deployment update using
kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1 --record command, which results in the following output: deployment.apps/nginx-deployment image updated

Another way is to run kubectl edit deployment <my_dep_name> and update the image manually.

Once an update command is issued, we can keep track using kubectl rollout status deployment/<my_dep_name>

We can check status of rollout using kubectl rollout status deployment/<my_dep_name> or check history using kubectl rollout history deployment/<my_dep_name>.

Rollbacks

We can roll back a deployment using kubectl rollout undo deployment/<my_dep_name>

Jobs

A Job creates one or more pods and ensures that a specified number of them successfully terminate.

We can create a job using kubectl create job myjob --image=nginx --dry-run -o yaml -- /bin/sh -c 'echo hello;sleep 100;' which results in the following job being created:

apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  name: myjob
spec:
  template:
    metadata:
      creationTimestamp: null
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - echo hello;sleep 100;
        image: nginx
        name: myjob
        resources: {}
      restartPolicy: Never
status: {}

Once created we can use kubectl describe job myjob and do kubectl get pods to get the list of running pods for that job.

# CronJob

A Cron Job creates a time-based schedule.

CKAD - Revision - Multi-Container Pods

boncheff — Tue, 28 Jan 2020 20:58:14 +0000

See https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns/ for mode details on this topic.

Sidecar Container - Adds some functionality not present in the main container such as logging - this allows for a decoupled and scalable approach rather than bloating main container code. Prometheus and Fluentd logging use sidecar containers to collect data
Adapter Container - Used to modify (adapt) the data either on ingress or egress to match some other need. An adapter would be an efficient way to standardise the output of the main container to be ingested by the monitoring tool without having to modify the monitor of the containerised application. An adapter container transforms multiple applications to a singular view
Ambassador - an ambassador allows for access to the outside world without having to implement a service or another entry in an ingress controller:
- Proxy local connection
- Reverse proxy
- Limit HTTP requests
- Re route from main container to the outside world. (Open Source Kubernetes-Native API Gateway built on the Envoy Proxy)

CKAD - Revision - Core Concepts

boncheff — Sun, 26 Jan 2020 18:10:32 +0000

Pods

A pod is a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers. A Pod’s contents are always co-located and co-scheduled, and run in a shared context.

Pods share networking and storage:

containers within a Pod share an IP address and port space, and can find each other via localhost
a pod can specify a set of shared storage volumes, allowing containers to share data.

Restarting a container in a Pod should not be confused with restarting
the Pod. The Pod itself does not run, but is an environment the containers
run in and persists until it is deleted.

Here is an example of a pod template:

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox
    command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600']

Services

A service is an abstraction which defines a logical set of Pods and a policy by which to access them (sometimes this pattern is called a micro-service). The set of Pods targeted by a Service is usually determined by a selector.

Services allow for decoupling - if a set of "frontend" pods needs to talk to a set of "backend" pods, this is possible due to the Service abstraction.

Here is an example of a service template that would match the above pods:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:    <--------------------- this is what selects pods
    app: myApp <--------------------- 
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376

Kubernetes assigns this Service an IP address (sometimes called the “cluster IP”), which is used by the Service proxies.

Kubernetes ServiceTypes allow you to specify what kind of Service you want. The default is ClusterIP.

Type values and their behaviors are:

ClusterIP: exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType.
NodePort: exposes the Service on each Node’s IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You’ll be able to contact the NodePort Service, from outside the cluster, by requesting :.
*LoadBalancer: exposes the Service externally using a cloud provider’s load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.
ExternalName: allows the return of an alias to an external service. It is used for services not yet brought into the Kubernetes cluster (a bit like a placeholder). Once those services are ready we can then change the type and traffic would flow to the internal objects.

Volumes

On-disk files in a Container are ephemeral, which presents some problems for non-trivial applications when running in Containers. First, when a Container crashes, kubelet will restart it, but the files will be lost - the Container starts with a clean state. Second, when running Containers together in a Pod it is often necessary to share files between those Containers.

The Kubernetes Volume abstraction solves both of these problems.

A volume has the same lifetime as the pod that encloses it

Namespaces

Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces.
Namespaces are intended for use in environments with many users spread across multiple teams, or projects.

Deployment

A Deployment provides declarative updates for Pods and ReplicaSets.

ReplicaSet

A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.

You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate.

DaemonSet

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod such as cluster storage daemon, logs collection daemon or monitoring daemon.

StatefulSet

StatefulSet is the workload API object used to manage stateful applications.

Manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods.

CKAD - Service Types

boncheff — Mon, 20 Jan 2020 20:49:02 +0000

Services can be one of the following four types:

ClusterIP
NodePort
LoadBalancer
ExternalName

ClusterIP is the default and only provides access internally

LoadBalancer was created to pass requests to cloud provider like AWS. The address is made available to public traffic and packers are spread among the Pods in the deployment automatically.

kubectl proxy creates a local service to access a ClusterIP which can be useful for troubleshooting

Exposing an application with a Service

We can use kubectl expose command which exposes a resource as a new Kubernetes service.

Ingress

An ingress is an API object that manages external access to the services in a cluster, typically HTTP.

Ingress can provide:

load balancing
SSL termination
name-based virtual hosting

Service Mesh

We can implement a service mesh for more complex connections or resources, such as service discovery, rate limiting, traffic management and advanced metrics.

A service mesh consists of edge and embedded proxies communicating with each other and handling traffic based on rules from the a control plane.

Typically we use Envoy and Istio

CKAD - Security Notes

boncheff — Sat, 18 Jan 2020 18:13:28 +0000

To perform any action in Kubernetes cluster we need to access the API and go through three main steps:

Authentication
Authorisation (RBAC or ABAC)
Admission Control

See the official documentation for more details:

Authentication

Authentication is done with certificates, tokens or basic auth
Users are not created by the API and should be managed by external system
Service accounts are used by processes to access the API (a service account provides an identity for processes than run in a pod)

Authorisation

Once a request is authenticated, it needs to be authorised to be able to proceed through the Kubernetes system. There are three main modes of authorisation:

ABAC (attribute based access control)
RBAC (role based access control)
WebHook

Admission Controller

Accesses the contents of the objects being created by the requests. It can modify the content or validate it and potentially deny the request.

Security Context

A security context defines privilege and access control settings for a Pod or Container so we can limit what processes running in containers can do. For example we can limit:

the user ID of the process (UID)
the Linux capabilities
filesystem groups

If we want to enforce that containers cannot run their process as root user we can add runAsNonRoot: true to the pod spec. Or we can define a PodSecurityPolicy to that effect.

To automate the enforcement of security contexts, we can define PodSecurityPolicies (PSP)

Pod Security Policies are cluster-level rules that govern what a pod can do, what they can access, what user they run as...

For a PSP to be enabled we must first configure the admission controller of the controller-manager to contain PodSecurityPolicy.

Network Security Policy

A network policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints.

NetworkPolicy resources use labels to select pods and define rules which specify what traffic is allowed to the selected pods.

CKAD - Deployment Configuration Notes

boncheff — Wed, 01 Jan 2020 21:32:30 +0000

Deployment Configuration

Volumes

a volume is a directory (possibly pre-populated) made available to containers in a Pod
a Kubernetes volume shares at least the Pod lifetime, not the container within
if you want your storage lifetime to be distinct from a Pod, use Persistent Volumes. These allow for volumes to be claimed by a Pod using PersistentVolumeClaims.
a volume can persist longer than a pod, and can be accessed by multiple pods using PersistentVolumeClaims. This allows for state persistency
a volume can be made available to multiple pods, with each given an access mode to write
we can pass encoded data to a pod using Secrets and non-encoded data using a ConfigMap - we can pass things like SSH keys, passwords etc...
there is no concurrency checking, which means data corruption is probable unless outside locking takes place
examples of volume types:
- rbd
- NFS
- gcePersistentDisk

There are three access modes for volumes:

RWO(ReadWriteOnce) - allows read-write by a single node
ROX(ReadOnlyMany) - allow read-only by multiple nodes
RWX(ReadWriteMany) - allows read-write by multiple nodes

When a volume is requested, the local kubelet uses the kubelet_pods.go script to:

map raw devices
determine and make mount point for container
create symbolic link on the host filesystem to associate the storage with the container

If no particular StorageClass was requested, only access_mode and size are used as params to create a volume.

Volume types

There are many different types of volumes such as GCEpersistentDisk or awsElasticBlockStore (GCE and EBS respectively)

An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node. As the name says, it is initially empty. Containers in the Pod can all read and write the same files in the emptyDir volume, though that volume can be mounted at the same or different paths in each Container. When a Pod is removed from a node for any reason, the data in the emptyDir is deleted forever.
A hostPath volume mounts a file or directory from the host node’s filesystem into your Pod. This is not something that most Pods will need, but it offers a powerful escape hatch for some applications.

Secrets

Secrets in kubernetes are not encrypted by default - only base64 encoded.

Config Maps

Data in ConfigMaps in kubernetes is not encoded or encrypted and contains key-value pairs or plain configuration files in any format.

Here are a few uses of ConfigMaps:

Pod env vars from single or multiple ConfigMaps
Use ConfigMap values in Pod commands
Populate Volume from ConfigMap
Add ConfigMap data to specific path in Volume
Set file names and access mode in Volume from ConfigMap data
Can be used by system components and controllers.

Creating ConfigMaps

ConfigMaps can be created in one of the three following ways:

kubectl create configmap myconfigmap \
--from-literal=city=London \             
--from-file=./myconfigmapfile.txt \
--from-file=./myconfigmapdirectory/

which results in the following ConfigMap:

k get configmap myconfigmap -o yaml

apiVersion: v1
data:
  city: London
kind: ConfigMap
metadata:
  creationTimestamp: "2020-01-12T11:22:43Z"
  name: myconfigmap
  namespace: default
  ...

Scaling and Rolling Update

To scale a deployment we can use

kubectl scale deploy/mydeployment --replicas=4

and then to downscale

kubectl scale deploy/mydeployment --replicas=1

Deployment rollbacks

Let's say we want to update a container's image using the following command

kubectl set image deployment/test nginx=nginx:0.9 --record

Let's say we now want to update to an even newer version of nginx using the following command

kubectl set image deployment/test nginx=nginx:66.6 --record

If you now run kubectl get pods you will see that the pod is stuck in ImagePullBackoff state so we want to immediately rollback the changes to a previously working version. We can do this using the following command

kubectl rollout undo deployment/test

To view a history of all events we can run

kubectl rollout history deployment/test

which shows the following output:

deployment.apps/test 
REVISION  CHANGE-CAUSE
1         kubectl set image deploy/test test=nginx --record=true
4         kubectl set image deployments/test nginx=nginx:0.8 --record=true
5         kubectl set image deployments/test nginx=nginx:0.9 --record=true
6         kubectl set image deployments/test nginx=nginx:0.66 --record=true
7         kubectl set image deployments/test nginx=nginx:66.66 --record=true
8         kubectl set image deployments/test nginx=nginx:66.66 --record=true

CKAD notes

boncheff — Fri, 27 Dec 2019 16:57:00 +0000

This article will contain a series of notes I made while preparing for the CKAD exam.
I intend to use these notes to refresh my knowledge on the topic periodically and hope fellow engineers find them useful during their foray into Kubernetes.

Please come back to read them periodically as I will add new notes as I work through the preparation material.

Let's dive straight in!

General background information on Linux:

systemd - system startup and service management (replaces service package)
journald - manages system logs (this is a systemd service that collects and stores logging data - it creates and maintains structures, indexed journals based on logging information that is received from a variety of sources.
firewalld - firewall management daemon (provides a dynamically managed firewall with support for network / firewall zones to define the trust level of network connections or interfaces - it has support for IPv4, IPv6 firewall settings and Ethernet bridges.
ip - network display and configuration tool (this is part of the networking tools package, and is designed to be a replacement for the ifconfig command. The ip command will show or manipulate routing, network devices, routing information and tunnels.

What is Kubernetes?

...an open source system for automating deployment, scaling and management of containerised applications.

started at Google 15 years ago
containers provide a fine-grained solution for packing clusters efficiently

Trivia:

Kubernetes in greek means 'the helmsman' or pilot of the ship

Kubernetes offers:

continuous integration - a consistent way to build and test software. Deploying new packages with code written each day or every hour instead of quarterly. Tools like Helm or Jenkins are often part of this with Kubernetes.
continuous delivery - an automated way to test and deploy software into various environments. Kubernetes handles the lifecycle of containers and connection of infrastructure resources to make rolling upgrades and rollbacks easy.

Kubernetes approaches these software issues by deploying a large number of small web services called microservices. One way of visualising this is - instead of deploying a large Apache web server running httpd daemons responding to page requests, there would be many smaller nginx servers handling traffic and requests.

Kubernetes Architecture

kubectl - is used to talk to the API server (or we can write our own client)
kube-scheduler - sees the requests for running containers coming to the API and finds a suitable node to run that container on
each node in a cluster runs two processes - a kubelet and kube-proxy. A kubelet receives requests to run containers, manages any necessary resources and watches over them on the local node. A kube-proxy creates and manages networking rules to expose containers on the network.

Resource types

Orchestration is managed through a set of watch-loops knows as operators or controllers.

Each controller interrogates the kube-apiserver for a particular object state, modifying the object until the declared state matches the current state.

a Pod is a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers. A Pod’s contents are always co-located and co-scheduled, and run in a shared context
a Deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate
a ReplicaSet is a controller which deploys and restarts Docker containers until the requested number of containers is running
a DaemonSet ensures that a single pod is deployed on every node (often used for logging and metrics)
a StatefulSet can be used to deploy pods in a particular order, such that following pods are only deployed if previous pods report a ready status
we can use labels which are part of object metadata
nodes can have taints to ensure pods are not scheduled on inappropriate nodes unless the pod has a toleration in its metadata

kubectl describe nodes | grep -i taint
Taints:             node-role.kubernetes.io/master:NoSchedule

This means that no pod will be able to schedule onto this node.

Annotations are used by third party agents and other tools nut not by Kubernetes

Kubernetes Master Node

runs various server and manager processes for the cluster
kube-apiserver, kube-scheduler and etcd database are all components of the master node

Kube-apiserver is central to the operations of a kubernetes cluster.
All calls are handled via this agent. All actions are accepted and validated by this agent and it is the only agent that connects to the etcd database. It acts as a master process for the cluster. Each API call goes through authentication, authorisation and several admission controllers.

Kube-scheduler uses an algorithm to determine which node will host a Pod of containers. Views available resources such volumes and then deploys based on availability and success.

Etcd database stores the state of the cluster, networking and other persistent info. Values are always appended to the end. Previous copies of a data are marked for future removal by compaction process. There is a master database along with several followers. Starting from v1.15.1 kubeadm allows easy deployment of multi master clusters with stacked etcd clusters.

Kube-controller-manager is a core control loop daemon which interacts with the kube-apiserver to determine the state of the cluster. If it does not match, the manager will contact the necessary controllers to match the desired state. There are several controllers to use such as endpoints, namespaces and replication.

Worker Nodes

all worker nodes run a kubelet and kube-proxy as well as a container engine such as Docker
kubelet interacts with underlying Docker engine to ensure containers that need to run are actually running - it works to configure the local node until the specifications (in PODSPEC) has been met. Also sends back status to kube-apiserver for persistence
kube-proxy manages network connectivity to the containers using IPTABLES entries. Also has userspace mode, in which it monitors Services and Endpoints using a random high number port to proxy traffic

Pods

smallest unit of work - we do not (or rarely) directly interact with containers
represents a group of co-located containers with some associated data volumes
containers in a pod share the same network namespace
sidecar containers perform helper tasks like logging

Each pod is made up of:

one or more containers
one or more volumes
a pause container

The pause container is used to get an IP address and then ensure all containers in a pod use its network namespace - containers can die and come back and they still use the same network namespace.

Services

A Service is an abstraction which defines a logical set of Pods and a policy by which to access them (sometimes this pattern is called a micro-service)

Controllers

In Kubernetes, controllers are control loops that watch the state of a cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state. Endpoints, namespace and serviceaccounts are examples of controllers.

Kubernetes Networking

Every pod gets its own IP address (Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration.
All containers within a pod behave as if they are on the same host with regard to networking. They can all reach each other's ports on localhost.
Pods are assigned IP address prior to container being started. The service object is used to connect pods within the network using CLUSTERIP addresses. From outside the cluster it uses NODEPORT addresses and using a load balancer if configured with a LOADBALANCER service. See here for more details
- ClusterIP: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType.
- NodePort: Exposes the Service on each Node’s IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You’ll be able to contact the NodePort Service, from outside the cluster, by requesting :.
- LoadBalancer: Exposes the Service externally using a cloud provider’s load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.
Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. - An Ingress can be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name based virtual hosting. An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.
```
 internet
     |
[ Ingress ]
--|-----|--
[ Services ]
```
kubeadm - kubernetes cluster bootstrapping tool - uses CNI (container network interface) as the default network interface mechanism.
CNI is an emerging specification that configure container networking and remove allocated resources when the container is deleted.

Creating a simple pod / service

kubectl create -f <service.yaml>

Services and Pods are "linked" as follows:

pods use labels (key:value pairs) as metadata i.e. labels -> type -> webserver
the service uses a selector (with same key:value pairs) to match those pods i.e. spec -> selector -> webserver

Using a single container per pod allows for the most granularity and decoupling. There are still some reasons to deploy multiple containers, sometimes called composite containers, in a single pod. The secondary containers can handle logging or enhance the primary, the sidecar concept, or acting as a proxy to the outside, the ambassador concept, or modifying data to meet an external format such as an adapter. All three concepts are secondary containers to perform a function the primary container does not.

Creating a simple deployment

kubectl create deployment <deployment_name> --image=<image_name>

kubectl create deployment example --image=busybox

Creating a pod does not take advantage of orchestrating abilities of Kubernetes. Deployments on the other hand give us scalability, reliability and updates.

Container runtime

A container runtime is the component which runs the containerised application upon request. Docker Engine remains the default for Kubernetes, though cri-o or rkt, and others are gaining community support.

The Open Container Initiative (OCI) was created to help with portability across systems and environments. Docker donated their lib container project to form a new codebase called runC to support these goals.

Here is how to create a a Dockerfile

Start Dockerfile with FROM i.e. FROM busybox
Then build it with docker build -t myapp
Check that image built successfully with docker images
Then run the container with docker run myapp
Then push to the registry with docker push

We can also build a local repo to push images to - this adds privacy and reduces bandwidth
Once it is configured, we can docker build, tag and push

To create a deployment with image version

kubectl create deployment <name> —image=<repo>/<app_name>:<version>

i.e.

kubectl create deployment test-img —image=1.2.3.4:5001/myapp:v2.2

Probes

LIVENESS PROBE - when to restart a container (deadlocked etc) (if under a controller, it is restarted, else terminated)

READINESS PROBE - when it is available to accept traffic

Considerations for good design

decoupled resources - instead of hard coding a resource in an application, an intermediary, such as a Service, enables connection and reconnection of other resources, providing flexibility.
transience - each object should be developed with the expectation that other components will die and be rebuilt. (This allows us to terminate and deploy new versions easily)
flexible framework - many independent resources work together, but decoupled from each other, and without expectations of individual or permanent relationship. This allows for greater flexibility, higher availability and easy scalability.

Resource Usage

Pods usually use as much CPU and memory as the workload requires. By using resource requests the scheduler will only schedule a pod to a node if the resources exist to meet all requests on that node.

CPU
- (spec.containers[].resources.[ limits | requests ] .cpu
- If pod uses more CPIU than allowed it WON’T be evicted
RAM
- (spec.containers[].resources.[ limits | requests ] .memory)
- If pod uses more memory than required, it may be restarted or evicted.
Ephemeral Storage - when a container crashes, kubelet will restart it and all the files will be lost. (Container starts with a clean state). When running containers in a pod, it is often necessary to share files between containers (this is where VOLUMES come in).
- If it uses more storage then specified it will be EVICTED.

Patterns for composite containers

See https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns/ for mode details on this topic.

Sidecar Container - Adds some functionality not present in the main container such as logging - this allows for a decoupled and scalable approach rather than bloating main container code. Prometheus and Fluentd logging use sidecar containers to collect data
Adapter Container - Used to modify (adapt) the data either on ingress or egress to match some other need. An adapter would be an efficient way to standardise the output of the main container to be ingested by the monitoring tool without having to modify the monitor of the containerised application. An adapter container transforms multiple applications to a singular view
Ambassador - an ambassador allows for access to the outside world without having to implement a service or another entry in an ingress controller:
- Proxy local connection
- Reverse proxy
- Limit HTTP requests
- Re route from main container to the outside world. (Open Source Kubernetes-Native API Gateway built on the Envoy Proxy)

Kubernetes Jobs

Jobs are part of the batch API group. They are used to run a set number of pods to completion. If a pod fails, it will be restarted until the number of completion is reached.
A job spec has a PARALLELISM (# of pods to run) and a COMPLETION ( # of pods to run successfully for the job to be considered done).

Deployment Configuration

Volumes

a volume is a directory (possibly pre-populated) made available to containers in a Pod
a Kubernetes volume shares at least the Pod lifetime, not the container within
if you want your storage lifetime to be distinct from a Pod, use Persistent Volumes. These allow for volumes to be claimed by a Pod using PersistentVolumeClaims.
a volume can persist longer than a pod, and can be accessed by multiple pods using PersistentVolumeClaims. This allows for state persistency
a volume can be made available to multiple pods, with each given an access mode to write
we can pass encoded data to a pod using Secrets and non-encoded data using a ConfigMap - we can pass things like SSH keys, passwords etc...
there is no concurrency checking, which means data corruption is probable unless outside locking takes place
examples of volume types:
- rbd
- NFS
- gcePersistentDisk

There are three access modes for volumes:

RWO(ReadWriteOnce) - allows read-write by a single node
ROX(ReadOnlyMany) - allow read-only by multiple nodes
RWX(ReadWriteMany) - allows read-write by multiple nodes

When a volume is requested, the local kubelet uses the kubelet_pods.go script to:

map raw devices
determine and make mount point for container
create symbolic link on the host filesystem to associate the storage with the container

If no particular StorageClass was requested, only access_mode and size are used as params to create a volume.

Volume types

There are many different types of volumes such as GCEpersistentDisk or awsElasticBlockStore (GCE and EBS respectively)

An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node. As the name says, it is initially empty. Containers in the Pod can all read and write the same files in the emptyDir volume, though that volume can be mounted at the same or different paths in each Container. When a Pod is removed from a node for any reason, the data in the emptyDir is deleted forever.
A hostPath volume mounts a file or directory from the host node’s filesystem into your Pod. This is not something that most Pods will need, but it offers a powerful escape hatch for some applications.