DEV Community: Eddie Webbinaro

Kubernetes deployments with Cloudflare

Eddie Webbinaro — Mon, 15 Oct 2018 13:35:00 +0000

This post is the second in a series that covers multi-cloud deployment. The first post, “Dockerizing Java apps with CircleCI and Jib,” addressed using Google’s Jib project to simplify wrapping a Java application in Docker. This post covers multi-cloud deployment behind a global load balancer.

In the first post, we talked about building containerized applications and published a simple Java Spring Boot application with Google’s Jib. That process is efficient at building and assembling our application, but we did not address distribution. In this post, we will look at a method of deploying our Docker application to Kubernetes and enabling global access with Cloudflare’s domain name system (DNS) and proxy services.

We will be using several technologies in this blog, and a basic understanding of CircleCI is required. If you are new to CircleCI, we have introductory content to help you get started.

The code used in this post lives in a sample repository on Github. You can reference https://github.com/eddiewebb/circleci-multi-cloud-k8s for full code samples, or to follow along. Here is the workflow that we will be building:

Credential management

I brought credential management to the top of this post on deployments because it is fundamental to a successful CI/CD pipeline. Before we start pushing our configuration, we are going to set up our access to Docker Hub (optional), Google Cloud Platform (GCP) and Cloudflare.

Authenticating with Docker Hub

My sample workflow uses Docker Hub as the registry for our containerized application. As an alternative, you can push them to Google or Amazon's registries. How you authenticate will depend on your application. This sample app uses Maven encryption to authenticate Google Jib's connection with Docker Hub, as described in our first post.

Authenticating with Google Cloud Platform

GCP is the IaaS provider we chose to host our Kubernetes cluster. To get started authenticating with GCP, you will need to create a service user key as described in the Google docs. Choose the new .json format when it prompts you to download a key. You will also need to assign roles to the service account. To successfully create a cluster and deployment, your service account will need ‘Kubernetes Engine Cluster Admin’ and ‘Kubernetes Engine Developer.’ Those roles can be granted on the main identity and access management (IAM) tab in Google’s console.

The key file contains sensitive information. We never want to check sensitive information into the source code. In order to expose it to our workflow, we will encode the file as a string, and save it as an environment variable:

base64 ~/Downloads/account-12345-abcdef1234.json

Copy the output and save it as an environment variable in the CircleCI project configuration:

Later, when we need that within our deployment job, we can decode it back to a file with base64 again:

echo "${GCP_KEY_FILE}" | base64 --decode >> gcp_key.json

I feel it necessary to issue the standard precaution: base64 is not encryption, it adds no security nor true obfuscation of the sensitive values. We use base64 as a means to turn a file into a string. It is paramount that you treat the base64 value with the same level of protection as the key file itself.

Authenticating with Cloudflare

One of the last steps in our deployment is the use of Cloudflare to route traffic for justademo.online back to our various clusters. We need a Cloudflare API key, a DNS zone, and an email address. Since these are all simple strings they can be added as is.

CLOUDFLARE_API_KEY - Can be found in profile and domain dashboard
CLOUDFLARE_DNS_ZONE - Can be found on domain dashboard
CLOUDFLARE_EMAIL - Email used to create Cloudflare account

Deploying Docker containers to Kubernetes with CircleCI

I found Google Cloud's Kubernetes support intuitive and easy to get started. Their gcloud command line interface (CLI) even provides a means to inject the required configuration into Kubernetes kubectl CLI. There are many providers out there and, although not covered in this blog post, the sample code does include a deployment to Amazon Web Services (AWS) using kops.

Configuring the command line tools

CircleCI favors deterministic and explicit configuration over plugins. That enables us to control the environment and versions used right within our configuration. The first thing we will do is install our CLI tools:

Remember: You can copy text from our sample project's configuration.

Because the steps above install the CLI on each build, it does add ~10-15 seconds to our job. For tools that you use often, it is worth your time to package your own custom Docker images instead.

With the gcloud and kubectl tools installed, we can begin creating our cluster and deployment. The general flow in GCP’s Kubernetes environment is:

Create a cluster that defines the number and size of VMs to run our workload.
Create a deployment that defines the container image to use as our workload.
Create a load balancer service that maps an IP and port to our cluster.

Before gcloud will do any of that, we need to provide it with credentials and project information. This is the step where we "rehydrate" our GCP key into a readable file:

Running your first deployment

The gcloud CLI now has access to our service account. We will use that access to create our cluster:

gcloud container clusters create circleci-k8s-demo --num-nodes=2

This command should block until the cluster creation succeeds. That allows us to provide our container image as a workload. The gcloud CLI will also save the required permissions into kubectl automatically for us:

kubectl run circleci-k8s-demo --image=${DOCKER_IMAGE} --port 8080

The variable DOCKER_IMAGE defines the full coordinates of our container in standard registry format. If you followed our previous post on versioning Java containers with Jib, the DOCKER_IMAGE would be: eddiewebb/circleci-k8s-demo:0.0.1-b8.

Unlike the create cluster command, the command to create or update the deployment is asynchronous. We need to run some basic smoke tests against our cluster. We use the rollout status command to wait for a successful deployment. This command is perfect because it waits for the final status, and will exit with a 0 on a successful deployment:

kubectl rollout status deployment/circleci-k8s-demo

Our container is now running on several nodes, with various IPs, on port 8080. Before we move on from GCP, the last thing to do is set up the local load balancer to send traffic from a single IP to all of the containers in the cluster. We use kubectl's expose command for that:

kubectl expose deployment circleci-k8s-demo --type=LoadBalancer --port 80 --target-port 8080

Just to make sure everything is started and routing successfully, we can run a primitive health check:

As you may notice in the Smoke Test job above, we do not know the IP address that Google will assign ahead of time. In a production environment, you would likely choose to purchase and allocate a reserved IP, instead of using a floating IP. You can see an example of that in our AWS deployment that is part of the sample project.

Routing your domain with Cloudflare

Each cloud provider exposes ways to get your dynamic cluster IP assigned to a known domain name. I want to demonstrate how we might serve our application from multiple cloud providers and dynamically address distribution and load. In addition to providing a straightforward API to manage our DNS, Cloudflare allows us to route traffic through their proxy for additional caching and protections. Additionally, since this is running in its own step, we need to grab the CLUSTER_IP from kubectl again. We will then use that IP in a POST call to Cloudflare's DNS:

Note: Because we want to use this sub-domain in our load balancers later, it is important that we leave "proxying" disabled.

That will map the GCP cluster to a subdomain. This subdomain will be used by our global load balancer to distribute traffic among multiple cloud providers.

Cool! Did it work? If you have followed along, you will have a live sub-domain similar to k8sgcp.justademo.online routing through Cloudflare to your Kubernetes cluster in Google Cloud.

Updating existing Kubernetes deployments on GCP with CircleCI

Our example above created everything fresh. If you tried to rerun that deployment, it would fail! Google will not create a cluster, deployment, or service that already exists. To make our workflow behave properly, we have to augment our original solution to consider the current state.

We can use get and describe commands to determine if our stack already exists, and run the appropriate update commands instead.

Create cluster OR get credentials:

Run deployment OR update the image:

Our sample project uses logic that will create or update all aspects, depending on the current state.

Alternately, and for more robust deployments, you will want to explore the use of Kubernetes deployment definitions that can be defined and applied, capturing the configuration as a .yml file. More information can be found in the Kubernetes examples.

Adding AWS deployments

AWS provides several options for deploying containers, including Fargate, ECS, and building your own cluster from EC2 instances. You can look at the deploy-AWS job within our sample project to explore one approach. We used the kops project to provision a cluster from EC2 and Elastic Load Balancers (ELB) to route traffic to that cluster.

What is germane to the topic in this blog post is that we now have the same application running across multiple cloud providers. Next we will implement a global load balancer to distribute traffic among them and setup our DNS. Setting up DNS will be a little different due to kops creating an ELB for us.

Tie it together with Cloudflare

As with Cloudflare’s DNS service, their Traffic service exposes a full API for managing the load balancer configuration. This makes it straightforward for us to automate that task in CircleCI.

Create a load balancer in Cloudflare

I opted not to script this part since it truly is a one-time thing. Create a load balancer that is associated with the primary domain you want sitting in front of all your clouds:

You will need to create at least one origin pool. If you have already run a deployment, you can specify the sub-domain we created above for our GCP cluster:

You will also be asked to define a health monitor. I chose a simple GET to a /build-info endpoint on the application, expecting a 200 response:

Once your initial load balancer and GCP origin pool are created, we can move on to scripting the AWS integration that updates a second origin pool with the dynamic AWS ELB created on each deployment.

Updating our Cloudflare origin pools on deployment

As mentioned above, when using kops in AWS it will create an ELB routed to the Kubernetes cluster automatically. Rather than setting up an AWS domain in Cloudflare, we will just use the one provided by Amazon. We will need to pass that along to Cloudflare. The ELB's name can be obtained using kubectl get service:

Because the ELB may not be immediately ready, we wrap that logic in a loop:

The result should be two healthy origin pools, running in separate cloud providers, all behind a single domain:

We are using proxying and SSL enforcement on our top-level domain, so go ahead and check out https://justademo.online/.

Summary

This post explored the creation of a Docker application and deployment across multiple cloud providers. We used Google's native support for Kubernetes cluster, while in AWS we created our cluster from scratch. Finally, we used Cloudflare as a global load balancer and DNS provider, allowing us to achieve a multi-cloud deployment that is transparent to our end users. Please check out the sample project with full sample code for everything covered in this post.

Happy shipping!

Dockerizing Java apps with CircleCI and Jib

Eddie Webbinaro — Tue, 02 Oct 2018 20:14:00 +0000

This post is the first in a series that includes multi-cloud deployment behind a global DNS. It was originally posted on CircleCI's blog.

The evolution of deployment platforms and orchestration engines has been a tremendous accelerant for application and infrastructure teams alike. Clear boundaries and well defined interfaces between the layers of our technology stack means we can rapidly iterate and deploy with repeatable results. For many teams, Docker has become a key ingredient in this architecture, allowing for deployment to platforms like Kubernetes.

Wrapping an application in Docker is a powerful way to enforce the correct dependencies and operating system. Unfortunately, it requires additional knowledge to properly configure and package the container images. In this blog post, I’ll discuss the use of Jib, an open source project from Google, which aims to simplify that effort for Java applications.

We will be using several technologies in this blog and a basic understanding of CircleCI is required. If you’re new to CircleCI, we have introductory content to help you get started.

What is Jib?

Jib builds Docker and OCI images for your Java applications and is available as a plugin for Maven and Gradle.

Jib has a few interesting value propositions. They are described in more detail on their project page. Namely, it reduces the complexity and time required to turn a stand-alone Java application into a fully-portable Docker image. Our sample application is a Spring Starter web app, but the project supports any Java based application with great support for any required customizations. In addition to the core libraries, the team is sharing plugins for Maven and Gradle to integrate seamlessly into your existing build process.

Prerequisites

To follow along with this demo, you need an account with a Docker registry. I’ll be using Docker Hub, but many others are supported including Artifactory, Nexus, and GCR.

Our sample application

I’m using a sample Java application from Spring Initializer. I have shared the full project at https://github.com/eddiewebb/circleci-jib-demo if you just want to jump into the code or follow along. Before moving forward, confirm your application runs locally. For Spring Boot applications, this is as simple as:

mvn spring-boot:run

The sample project and blog post use the Maven plugin, but as noted above, Gradle is fully supported.

Running Jib

Jib reduces the boilerplate configuration needed by setting some reasonable defaults that include a base distroless image. Jib will even infer the appropriate class to use as an entrypoint when starting your application.

If you want to test locally, you can build right into your Docker daemon.

mvn clean compile com.google.cloud.tools:jib-maven-plugin:0.9.7:dockerBuild

You can spin it up with:

docker run -p 8080:8080 java-jib-demo:0.0.1-SNAPSHOT

Note: 8080 is the default port for Spring Boot applications. It may vary based on your starting point.

Pushing Jib images to a registry

One of the perks of using Jib is that we actually don’t need a Docker daemon. We can build and publish an image directly to a public/private registry. Since registries require authentication, let’s setup our Maven project to use our Docker Hub credentials. (Docker Hub is the default registry, see Configuration Guide if you are using an alternate).

In your Maven’s settings.xml (typically found in <HOME>/.m2/settings.xml), define a server block for the Docker Hub registry that includes your Docker username and password:

Setting passwords in clear-text is never a good idea and certainly not something we want in our code repository. In the “Enabling CircleCI Deployments” section below, we will setup Maven Encryption.

With credentials configured, the only other information you need to provide is the resulting Docker image name to use by passing -Dimage.

Let’s give is shot by saving the created image locally. For that we will use the jib:build target:

mvn clean compile com.google.cloud.tools:jib-maven-plugin:0.9.7:build -Dimage=eddiewebb/hello-spring

Once the build completes, you will find a brand new image sitting in your registry.

Configuring Jib

With anything beyond trivial applications, you will need to make modifications to the generated image. Fortunately, Jib provides several configuration parameters that can be dropped right into your pom.xml.

To keep some discipline around our versioning, we can’t release every image with the default “latest” tag. Using the <to> element of our configuration we can fully specify the location and tagging.

The property build.number is something we will pass in that correlates to the CircleCI build identifier. Add a default value to keep local development simple, i.e., 000.

You will notice several other properties including the commit hash and workflow ID. These are just a few of the environment variables made available to every CircleCI job. We will use them as additional properties that we pass into the application to display important build information from a running application. In order to set these values within the JVM, we will use another configuration block, <jvmFlags>.

You can find additional configuration parameters explained on the project page.

Let’s validate our configuration one last time before configuring CircleCI.

Now, visit http://localhost:8080/build-info and you will see how the demo application exposes the additional JVM flags we passed in.

Let’s use CircleCI to automate this process for repeatable and deterministic builds, enabling end-to-end traceability.

Encrypting Maven credentials

The first thing to address is credential management. We’re not going to check any sensitive credentials into source code, but our CircleCI builds will need access to them. There are a few ways to solve this, and my example uses a combination of Maven Encryption and CircleCI Environment Variables.

Encrypt a “master” password:

mvn --encrypt-master-password
Master password: <<type some random but lengthy string as input>>
{sqtg2346i10Hf2Z1u9bYgyDxooiIa6AXlBY92E5x+2dvqsCU7kI+iM9b8lI42Thh}

Save the results in a file named settings-security.xml in your Maven home directory, i.e. ~/.m2/settings-security.xml.

Encrypt your Docker Registry password:

mvn --encrypt-password
Password: <<your Docker registry credentials>>
{9Z0In9ZySLsH7re5LlnGsvstLmfGfHL+7vi92maAXGiZ4oBP1iGwdoXR7QElP3TU}

Update your settings.xml with the generated cipher:

At this point you can run

mvn clean compile jib:build

again to confirm the encrypted credentials are working.

Setting up CircleCI

Add your project to CircleCI by visiting https://circleci.com/dashboard and following the prompts to “Start Building” your project’s repository.

The first build will fail since we haven’t finished configuring the project. To get it passing, you will need to visit the project’s configuration page and add a variable named maven_security_master that contains your master password from the previous steps. That variable is used by our .circleci/config.yml to recreate the critical settings-security.xml file in our builds.

You will also want to copy your ~/.m2/settings.xml to the projects .mvn/wrapper/ folder. It is important that both the settings.xml and master password are made available to CircleCI to properly upload.

With the settings.xml file in place, commit and push your latest changes to your repository. CircleCI should start building and successfully publish your Docker image.

Once published, if you run our newly generated image you will see all the values we passed from CircleCI’s build environment exposed on the /build-info url, giving your application end-to-end traceability!

If you’d like to take your CircleCI workflow to the next level, and start deploying your containerized application to ECS or Kubernetes, be sure to checkout our next blog post in the series and watch our previously recorded webinar with our Solutions Engineer Chris Black - Docker Deployments 102.

Happy building!