Reducing Lead Time for Changes

Peter Wong — Mon, 03 Oct 2022 05:39:01 +0000

Our previous post describes how we have implemented a Dynamic Feature Environment that would allow engineers at Prenetics to spin up/down a fully functional independent K8s cluster that runs our full set of microservices, offering an internal Environment as a Service (EaaS). The aim is to provide the necessary tools for our engineers to easily build and test locally.

At Prenetics Engineering we also look at improving our delivery efficiency. Lead Time for Changes (LT) is a one of the four DORA metrics and it measures the amount of time a commit would take to get into production. Reducing LT would mean faster Time to market.

We split our LT improvements into the following phases:

Provision
Build
Deploy
Test
Release

In the following we look at how we have been optimising the provision phase.

Provision

We use CodeBuild to execute build and test steps, and to deploy our digital assets

k8s changes via AWS Cli and kubectl,
web changes via AWS Cli to S3
mobile changes via Expo Cli
infrastructure changes via Terraform

Each of our CodeBuild projects executes the required steps inside its specified Docker container. AWS offers a number of standard Docker images. However, these images do not have a number of applications needed in our build, deploy and test steps. We have therefore defined our own CodeBuild Docker image (see https://gist.github.com/peteryhwong/7bbbe461b028f5040c81cdcea2f256af is a snippet of the Dockerfile), Our custom Docker image defines from an Linux Alpine based "Docker in Docker" image and pre-installed the tools such as jq, terraform, kubectl, and AWS Cli that are needed in our build, deploy and test steps. This custom image is smaller in size and it means our steps do not need to download the same external dependencies in every execution.

This image is stored in our AWS ECR and is provisioned when CodeBuild projects execute.

Note that unlike the AWS standard images, containers created from our custom image would not the Docker daemon (dockerd) running by default, hence we would need to start Docker daemon manually when the container starts. See https://gist.github.com/peteryhwong/a51c39d69647c233808339c62d7b8f28 for details.

On average we see a reduction of up to a minute during the provision phase of our CodeBuild projects and our typical microservice's build phase is normally less than 5 minutes.

In our next post we will look at how we have been optimising the build phase.

Dynamic Kubernetes based Feature Environment on AWS

Peter Wong — Mon, 12 Sep 2022 17:14:02 +0000

At Prenetics, our mission is to decentralise healthcare by offering affordable and accessible healthcare services across the pillars of Prevention, Diagnostic and Personalised Care to the public.

Engineering at Prenetics focuses on delivering robust, secure and scalable capabilities to our digital products and services. We deploy our API as microservices onto K8s (AWS EKS) and have implemented an EFK stack for centralised logging, monitoring and alerting. We have implemented CI/CD pipelines using CodePipeline and have been practicing Continuous Delivery.

Our core engineering principle is "You build it; you run it" and in order for our engineers to fully embrace this DevOps model, they must be able to easily build and test locally, thereby reducing the build and test feedback loop to the minimum. To this end we have developed the Dynamic Feature Environment.

Dynamic Feature Environment allows engineers to spin up/down a fully functional independent K8s cluster that runs our full set of microservices.

Precisely, a dynamic feature environment consists of an EC2 instance running Minikube and connecting to a small RDS instance that is created from a snapshot of our integration environment's RDS.

There are two EC2 Image Builder pipelines. The Minikube Ingress pipeline creates a AMI based on an Amazon Linux 2 base image. The pipeline installs Minikube, enables its ingress (minikube addons enable ingress), and also installs tools needed to deploy our microservices. The Minikube Service pipeline creates a AMI based on the output of Minikube Ingress and caches the latest Docker images (from AWS ECR) of our microservices. This pipeline is triggered daily.

There is a Lambda function (lambda-devlauncher) that our engineers can invoke to spin up an EC2 instance using the latest AMI from Minikube Service. This EC2 instance sits on the public subnet (with a public IP address) and is only accessible within our internal network (VPN). The EC2 instance's user data fetches the latest Docker images (from AWS ECR) of our microservices as well as the images of the services under test. The user data exposes the APIs, the cluster's K8s dashboard (minikube dashboard), and a centralised logviewer (minikube addons enable logviewer).

The Lambda function also creates a small RDS instance to be used by the feature environment. It is created from the latest snapshot of our integration environment's RDS. In practice we always have n+1 small RDS instances, where n is the number of feature environments running, so the time needed to spin up a dynamic feature environment is reduced to the minimum.

There is an EventBridge rule that invokes the lambda function daily to terminate any old feature environments and to create 1 small RDS instance for usage.

The time from invoking the lambda function to having the environment ready is around 12 minutes.

DEV Community: Peter Wong

Reducing Lead Time for Changes

Provision

Dynamic Kubernetes based Feature Environment on AWS