DEV Community: Katia HIMEUR

Introduction to Kustomize - How to customize Kubernetes objects

Katia HIMEUR — Thu, 08 Jul 2021 19:00:49 +0000

What is Kustomize?

Kustomize is a tool used to customize Kubernetes objects in a template-free way. It provides several features that allow us to customize the application’s configuration.

We can use Kustomize in two ways: use the standalone version of Kustomize or use kubectl. Kustomize is a part of Kubectl since version 1.14.

Kustomize is easy to learn and use because the customization file is the same as the Kubernetes manifest. It is very handy when you work with Kubernetes. That makes the learning curve low.

One advantage of Kustomize is that it uses a kustomization.yaml file to customize Kubernetes manifests. That avoid us editing directly the manifests. So we can use the original manifests without needing Kustomize.

We will see above, the main features of Kustomize.

Features

1. secretGenerator and configMapGenerator

With Kustomize, we can generate secrets and configMaps from literals or files and rolling out changes. This is possible through the use of secretGenerator and configMapGenerator.

Example : Using Kustomize to generate Kubernetes secret to store tls certificate and key file.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
 - namespace.yaml

secretGenerator:
- name: my-tls
 files:
 - cert/tls.cert
 - cert/tls.key
 type: "kubernetes.io/tls"
 namespace: my-app

Example: Using Kustomize to generate Kubernetes secret to store database password.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

secretGenerator:
- name: database-password
 literals:
   - password=pass

For secretGenerator, as we see, we can specify the namespace where we want to store the secrets. We can also specify the type of secret and add labels and annotations.

Example : Generate a configMap YAML:

# config-file.cnf
character-set-server=utf8mb4

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
- name: database-config-file
  files:
  - config-file.cnf

2. Container image

Kustomize allows us to override container's name and version. We can specify a tag or a digest for container's version.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  template:
    spec:
      containers:
      - name: app-one
        image: app-one:latest
      - name: app-two
        image: app-two:latest
      - name: app-three
        image: app-three:latest

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

images:
- name: app-one
  newName: main-application
- name: app-two
  newTag: 1.0.1
- name: app-three
  digest: sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

resources:
- deployment.yaml

3. Namespaces and names

We can use Kustomize, to set for all resources within a project or for a group of resources, namespace, name prefix, or name suffix.

If a namespace is already set, Kustomize will override it.

Example: Set namespace with Kustomize

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: backend-services
resources:
- deployment.yaml

Example: Prepends the value to the names of all resources and references.

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namePrefix: staging-

resources:
- deployment.yaml

Example : Appends the value to the names of all resources and references

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

nameSuffix: -beta

resources:
- deployment.yaml

4. Set labels and annotations

We can use Kustomize to set labels and annotations for a group of resources. To do that, use commonLabels and commonAnnotations.

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
  environment: staging
commonAnnotations:
  imageregistry: "https://hub.docker.com/"
resources:
- deployment.yaml

5. Bases

When we use Kustomize, we need a directory, called the base. In this directory, we put a set of resources and a kustomization.yaml file.

To avoid rewriting the base content and to enable reusability, the base content can be versioned in a remote repository. Make sure there is a kustomization file inside the repository.

# kustomization.yaml
bases:
# GitHub URL
- github.com/example/kustomize/bases/staging/?ref=v1.1.1

7. Overlays

An overlay is a directory with a kustomization.yaml that refers to one or multiple bases directory.

8. Inline patches

Kustomize uses patches to introduce environment specific changes on an already existing standard config file without disturbing it.

There are 3 ways for patching a kustomization file :

Strategic Merge patch

# kustomization.yaml
patchesStrategicMerge:
- |-
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: deploy
  spec:
    template:
      spec:
        containers:
        - name: nginx
          image: nginx:1.20.0-alpine
        - $patch: replace

Json patch

# kustomization.yaml
patchesJSON6902:
- target:
    group: apps
    version: v1
    kind: Deployment
    name: deploy
  patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/image
      value: nginx:1.20.0-alpine

A list of patches

Conclusion

There are many ways to customize Kubernetes objects, and the purpose of this post is to introduce Kustomize and show how to customize Kubernetes objects with kustomization files.

Why and How we should include Database Delivery in CI/CD processes?

Katia HIMEUR — Fri, 18 Sep 2020 06:52:01 +0000

1. Context

Often, in development teams, the Database changes in the production environment are stressful. If something goes wrong, rollbacks can be very painful and difficult, especially when these changes result in data loss or unavailability of applications.

Once the database is deployed, its state will constantly change. In case of failure, you can’t simply deploy a new database and start from scratch. There is data to back up and you should be able to restore the database if needed. You should be able to identify the risks associated with database modifications and be able to react very quickly if an outage occurs. Especially when the frequency of deployment is high.

To go further, we will see two scenarios that show why we should include Database Delivery in CI/CD processes.

Scenario 1:

A new version of an application must be deployed in production. This version requires a change in the database’s schema. The team used to do all modifications manually. After the deployment, they notice that there is a bug and they have to do a rollback because the application is unavailable. Everyone is very stressed. The developer has to restore the database to its previous state in a difficult context. The risk of making a mistake increase.

Scenario 2:

The development team implemented the Canary Release pattern to reduce the risk of deploying a new version of the application in production. They are used to deploy new versions with confidence. However, this time, the source code includes a migration script to update the database schema. The script is automatically executed after the deployment. This new application's version has a bug and all the traffic is routed to the previous version. But now, the database has been changed and it is no longer compatible with the previous version of the application causing unavailability of production.

These two scenarios show us why databases can be an important point of failure and why we should take database delivery into account when we design CI/CD processes. If the database changes are not well managed, the productivity of development teams decreases. Conversely, the risk and duration of downtime in production increase, quite the opposite of what we are trying to achieve by adopting the DevOps philosophy and agile methodologies.

In what follows, we will see how we can include Database Delivery in CI/CD processes and what best practices we can follow to secure database changes

2. Database Delivery

The first thing that we have to do before thinking about industrialization or how we can include database delivery in our CI/CD pipelines is choosing a database delivery approach: State-based or Migration-based deployment. To be able to choose, I will give you a description of these two mechanisms.

2.1. State-based deployment

In this approach, every database object (tables, views...) is described in the SQL file. If you want to modify the database, you need to modify the concerned SQL files. You have to choose a Compare tool that will generate automatically the ALTER requests to modify the database.

The source of truth is the source code. If you want to know the state of the database, you can read the SQL files.

With this approach, it is difficult to work simultaneously on the development of more than one version of the same application because you can’t modify simultaneously the same files without encounter conflicts.

2.2. Transformational or Migration-based deployment

The most common approach. You start with the script creating the database. Then, each time you want to modify the database, you create a new script that will migrate its state from a version to another. Over time, we end up with several migration scripts. These scripts must be executed incrementally.

With this approach, we encourage small changes, encouraged by DevOps philosophy, which, unlike big changes, reduce the risks associated with deploying new versions. To further reduce the risk of errors in performing migrations, automate its execution. In this way, you will be able to integrate this step into your CI / CD pipelines.

These migration scripts keep track of all changes made to the database since its creation.

This approach fits with parallel development.

The system of truth is the database itself. So, to determine the state of the database, you have to connect to it.

3. Best practices

3.1. Collaboration

If there is a DBA team, they should collaborate closely with developers.

3.2. Backup before any modification

Always backup your database before performing any modification.

3.3. Avoid manual changes

It is well known; automation reduces the risk of human errors and makes the deployment faster. So, automate the execution of the databases related scripts.

3.4. Every change in the database should be done with a versioned script

All databases artifacts should be version controlled. And every change in the database should be done with a versioned script. This allows you to keep track of the changes made.

3.5. Follow the Code Review Process

The human is not infallible. This is why it is important to follow the Code Review process with the migration scripts to limit the risk of errors. As for the applications. If there is a DBA team, make them participate in the code review process

3.6. Test before release

Always test before releasing in production. However, make sure that your staging data are as similar as possible to your production database to avoid nasty surprises.

3.7. Allow only additive changes

If you have to remove a database object, like deleting a column or table, the rollback is more difficult because of the loss of data. Consider the strategy consisting of allowing only additive changes.

3.8. You should be able to recreate the database if needed

Databases are not recreated after each deployment. Modifications are made on the existent database. But what if it is accidentally deleted? How we can restore it? Make backups of databases and have an action plan to restore it.

3.9. Consider Blue/Green deployment for big changes

If you need to make big changes on the database, consider the Blue/Green deployment pattern to reduce risks. You will run two environments side by side and configure your application to use the new one.

4. Conclusion

If you want to gain agility and productivity, it is very important to quickly integrate database delivery into the design of your CI/CD processes.

Provision ephemeral Kubernetes clusters on AWS EKS using Terraform and Gitlab CI/CD

Katia HIMEUR — Wed, 15 Jul 2020 20:07:45 +0000

1. Introduction

In software development, tests are very important. Tests help detect bugs before they occur in production environments. They reduce the risk of regression when the software is updated. They are a great tool to secure the delivery and deployment processes. Today, we cannot dissociate continuous deployment or continuous delivery from tests.

Testing locally is good, but testing in an environment as similar to production as possible is even better. Having a staging environment seems like a minimum to have. However, what if you want to test several versions of the same application at the same time?

Also, add to that, many companies want to reduce IT costs, and they see in digital sobriety a way to achieve this goal. So, having one or more staging environments running when we don't need them is not acceptable.

This is why we will see how we can provision ephemeral Kubernetes clusters on AWS EKS using Terraform and Gitlab CI/CD.

2. Provision Kubernetes clusters on AWS EKS with Terraform

Terraform is described by its creators as a tool for building, changing, and versioning infrastructure safely and efficiently. It is very simple to learn and use. This is why I choose it t to manage our infrastructure.

The infrastructure in this example consists of:

A VPC (Virtual Private Cloud);
An Amazon EKS cluster;
An Amazon Elastic Container Registry (ECR) to store Docker images;
A DNS record that routes traffic to the deployed application.

You can find the source code here: https://gitlab.com/ephemeral-aws-eks-clusters/terraform-ephemeral-aws-eks-clusters

And, if you want to learn more about Terraform, you can begin with the official documentation: https://www.terraform.io/docs/index.html

3. Build the application example

We will build a custom Nginx docker image with an index.html file. This application will display the name of the git branch from which it was built. We need to define the index.html and the Dockerfile.

Content of index.html file

<!doctype html>
<html lang=en>
  <head>
    <title>Hello from BRANCH_NAME</title>
  </head>
  <body>
    Hello, <br><br>
    The version of this app is: BRANCH_NAME
  </body>
</html>

Content of Dockerfile

FROM nginx:1.19.0-alpine

ARG BRANCH_NAME=master

COPY index.html /usr/share/nginx/html/

RUN sed -i "s/BRANCH_NAME/$BRANCH_NAME/g" /usr/share/nginx/html/index.html

The complete source code is here: https://gitlab.com/ephemeral-aws-eks-clusters/example-app-for-ephemeral-eks-clusters

4. Deployment workflow

We develop a web application with a single index.html file. We will package this application in a Docker image. For every new feature, we create a new git branch. Every time we create a merge request, we want to be able to test this new version of the application in an Amazon EKS Cluster. When the request is merged, we want to be able to destroy this cluster. Below the deployment workflow described above:

The Amazon EKS cluster is created;
The Docker image’s application is built;
The new Docker image is pushed in Amazon Elastic Container Registry (ECR);
The new version of our application is deployed in the Kubernetes cluster with a “kubectl” command.
A record DNS is created or updated with the load balancer hostname that routes traffic to our application

5. Continuous Delivery with Gitlab CI/CD

Now, after we define the deployment workflow, we need to automate it. Gitlab CI/CD is an open-source project integrated to Gitlab. It allows us to configure CI/CD pipelines with a “.gitlab-ci.yml” file located at the repository’s root directory.

We will define two “.gitlab-ci.yml” files: one in application repository and another in the Terraform repository. We will use triggers.

.gitlab-ci.yml file of Terraform's repository

image:
  name: hashicorp/terraform:0.12.28
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

stages:
  - provision
  - deploy
  - clean

cluster:provision:
  stage: provision
  rules:
  - if: '$CI_PIPELINE_SOURCE == "trigger" && $AWS_ELB_HOSTNAME == null && $CLEAN_ENV == null'
  script:
  - echo "Cluster name $CLUSTER_PREFIX_NAME"
  - terraform init -var=cluster_prefix_name=$CLUSTER_PREFIX_NAME
  - terraform workspace new "$CLUSTER_PREFIX_NAME" || terraform workspace select "$CLUSTER_PREFIX_NAME"
  - terraform apply -var=cluster_prefix_name=$CLUSTER_PREFIX_NAME -auto-approve

elb:provision:
  stage: provision
  rules:
  - if: '$CI_PIPELINE_SOURCE == "trigger" && $AWS_ELB_HOSTNAME != null && $CLEAN_ENV == null'
  script:
  - echo "Cluster name $CLUSTER_PREFIX_NAME"
  - terraform init -var=cluster_prefix_name=$CLUSTER_PREFIX_NAME
  - terraform workspace new "$CLUSTER_PREFIX_NAME" || terraform workspace select "$CLUSTER_PREFIX_NAME"
  - terraform apply -var=cluster_prefix_name=$CLUSTER_PREFIX_NAME
    -var cluster_prefix_name="${CI_COMMIT_REF_SLUG}"
    -var aws_elb_hostname=${AWS_ELB_HOSTNAME}
    -var aws_zone_id=${AWS_ZONE_ID}
    -var route53_record_name=${ROUTE53_RECORD_NAME}
    -target=aws_route53_record.www -auto-approve

deploy:app:
  stage: deploy
  rules:
  - if: '$CI_PIPELINE_SOURCE == "trigger" && $AWS_ELB_HOSTNAME == null && $CLEAN_ENV == null '
  before_script:
    - apk add -U curl
  script:
  - curl -X POST -F token="${PIPELINE_TRIGGER_TOKEN}"
    -F ref=${BRANCH_NAME}
    "${PIPELINE_TRIGGER_URL}"

cluster:clean:
  stage: clean
  rules:
  - if: '$CI_PIPELINE_SOURCE == "trigger" && $CLEAN_ENV != null && $CLUSTER_PREFIX_NAME != null'
  script:
    - terraform init -var=cluster_prefix_name=${CLUSTER_PREFIX_NAME}
    - terraform workspace new ${CLUSTER_PREFIX_NAME} || terraform workspace select "$CLUSTER_PREFIX_NAME"
    - terraform destroy -auto-approve -var=cluster_prefix_name=${CLUSTER_PREFIX_NAME}

.gitlab-ci.yml file of the application's repository

stages:
  - provision
  - build
  - release
  - deploy
  - clean

variables:
  CLUSTER_NAME: ephemeral-eks
  AWS_REGION: eu-west-1
  DOCKER_IMAGE_NAME: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/my-app
  DOCKER_IMAGE_TAG: ${CI_COMMIT_REF_SLUG}

provision:cluster:
  stage: provision
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  before_script:
    - apk add -U curl
  script:
    - curl -X POST
      -F token="${PIPELINE_TRIGGER_TOKEN}"
      -F variables[CLUSTER_PREFIX_NAME]="${CI_COMMIT_REF_SLUG}"
      -F variables[BRANCH_NAME]="$CI_COMMIT_REF_NAME"
      -F ref=master "${PIPELINE_TRIGGER_URL}"

build:
  stage: build
  rules:
    - if: '$CI_PIPELINE_SOURCE == "trigger"'
  script:
  - docker build --build-arg BRANCH_NAME="${DOCKER_IMAGE_TAG}" -t ${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG} .

release:
  stage: release
  rules:
    - if: '$CI_PIPELINE_SOURCE == "trigger"'
  before_script:
    - apk add -U --no-cache python3 py-pip
    - pip3 install awscli --upgrade
    - aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
  script:
    - docker push ${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}

deploy:
  stage: deploy
  variables:
    ROUTE53_RECORD_NAME: ${CI_COMMIT_REF_SLUG}.${DOMAIN_NAME}
  rules:
    - if: '$CI_PIPELINE_SOURCE == "trigger"'
  before_script:
    - apk add --no-cache -U curl jq python3 py-pip gettext
    - pip3 install awscli --upgrade
    - curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
    - chmod +x ./kubectl && mv ./kubectl /usr/local/bin/kubectl
    - mkdir -p $HOME/.kube
    - echo -n $KUBE_CONFIG | base64 -d > $HOME/.kube/config
    - aws eks --region ${AWS_REGION} update-kubeconfig --name ${CLUSTER_NAME}-${CI_COMMIT_REF_SLUG}
  script:
    - cat myapp-service.yaml | envsubst | kubectl apply -f -
    - export AWS_ELB_HOSTNAME=$(kubectl get svc my-service -n my-namespace --template="{{range .status.loadBalancer.ingress}}{{.hostname}}{{end}}")
    - curl -X POST -F token="${PIPELINE_TRIGGER_TOKEN}"
      -F variables[CLUSTER_PREFIX_NAME]=${CI_COMMIT_REF_SLUG}
      -F variables[AWS_ELB_HOSTNAME]=${AWS_ELB_HOSTNAME}
      -F variables[AWS_ZONE_ID]=${AWS_ROUTE53_ZONE_ID}
      -F variables[ROUTE53_RECORD_NAME]=${ROUTE53_RECORD_NAME}
      -F ref=master
      "${PIPELINE_TRIGGER_URL}"

destroy:cluster:
  stage: clean
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: manual
  before_script:
    - apk add --no-cache -U curl jq python3 py-pip gettext
    - pip3 install awscli --upgrade
    - curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
    - chmod +x ./kubectl && mv ./kubectl /usr/local/bin/kubectl
    - mkdir -p $HOME/.kube
    - echo -n $KUBE_CONFIG | base64 -d > $HOME/.kube/config
    - aws eks --region ${AWS_REGION} update-kubeconfig --name ${CLUSTER_NAME}-${CI_COMMIT_REF_SLUG}
  script:
    - cat myapp-service.yaml | envsubst | kubectl delete -f -
    - curl -X POST
      -F token="${PIPELINE_TRIGGER_TOKEN}"
      -F variables[CLUSTER_PREFIX_NAME]="${CI_COMMIT_REF_SLUG}"
      -F variables[CLEAN_ENV]=true
      -F ref=master "${PIPELINE_TRIGGER_URL}"

6. Deploy a new staging environment

We pushed a new git branch “feature/one” and we create a merge request. A new pipeline is automatically created. The first job running trigger another pipeline in the infrastructure repository’s code. The first step is to provision a new Amazon EKS Cluster. A clean job is created. We will need it only in the end when we will want to destroy this staging environment. It will be run manually.

It takes about fifteen minutes to have an active Kubernetes cluster on AWS.

After the Kubernetes cluster is active and the Amazon ECR is created, a second pipeline is triggered. Three new jobs are created for building, release, and deploy the new Docker image app.

At last, A record DNS is created or updated if already exists with the load balancer hostname that routes traffic to our application.

The web application is now accessible to other users from a custom DNS name:

7. Spot instances

To go further, we can use spot instances to reduce costs. However, keep in mind that with spot instances, there is a risk to have no instance available if Amazon EC2 doesn't have the capacity in the Spot Instance pool.