Continuous Development with Java and Kubernetes

Zoltan Polgar — Fri, 01 Mar 2019 12:26:04 +0000

If you are developing multiple applications in the same time for Kubernetes, you will realise that running and debugging these during the daily work is very sophisticated and time consuming, since we have to repeat the following steps:

Building the application
Building the docker image
Pushing the container image to a docker registry
Creating/Refreshing the Kubernetes' objects

It's enough to have a database and an MQ connection in addition, and you are in a serious trouble. I'm not saying that It's impossible to test everything together, but I'm absolutely sure that if you need to do the steps above by hand, it will break your productive flow.

Fortunately Google recently announced Jib 1.0.0 which combined with Skaffold able to solve this problem. In this article I'm going to show you a workflow with these tools alongside Spring Boot and Helm.

A Spring Boot application

First we need to have a basic Spring Boot application. I suggest to use Spring Initiallizr which can help you bootstrap a Spring Boot application in a few seconds. In this article I'm using Spring Boot 2.1.3 with gradle, however you will find the maven related files in the repository too.

Just download the generated project and let's create a RestController

@RestController
public class TestController {

    @GetMapping(value = "/echo/{text}")
    public ResponseEntity test(@NotNull @PathVariable String text) {
        return ResponseEntity.ok(text);
    }

    @GetMapping
    public ResponseEntity hello() {
        return ResponseEntity.ok("HELLO");
    }
}

During the article I want to demonstrate Jib's customizability, so let's change the default port number to 55000. Create an application.properties and put this line into the file

server.port=55000

Open your favorite shell and run ./gradlew bootRun. If both http://localhost:55000 and http://localhost:55000/echo/test seems to work, let's continue with the containerisation phase.

Using Jib for docker image creation

Jib is implemented in Java and runs as part of your Maven or Gradle build. You do not need to maintain a Dockerfile, run a Docker daemon. You just need to open your build.gradle file, append the plugins section with id 'com.google.cloud.tools.jib' version '1.0.0' and add the following piece of code to the end

jib {
    to {
        image = 'com.github.pozo/spring-boot-jib'
    }
    container {
        ports = ['55000']
    }
}

It should be noted that the configuration above is not mandatory. However, without these lines Jib would produce an image named spring-boot-jib:0.0.1-SNAPSHOT where the first part is the value of the rootProject.name from settings.gradle and the second one is provided by the version variable from build.gradle. In order to build a more advanced image, I suggest to look over the available options.

The above configuration’ equal would look like this with Dockerfile

FROM gcr.io/distroless/java:latest

COPY dependencyJars /app/libs
COPY snapshotDependencyJars /app/libs
COPY resources /app/resources
COPY classFiles /app/classes

COPY src/main/jib /

ENTRYPOINT ["java", jib.container.jvmFlags, "-cp", "/app/resources:/app/classes:/app/libs/*", jib.container.mainClass]
CMD [jib.container.args]

Jib uses distroless java as the default base image, which seems to use container related JVM flags by default. The multiple copy statements are used to break the app into layers, allowing for faster rebuilds after small changes.

If you want to specify a different base image, just add a from section

jib {
    from {
        image = 'java:8-jre-alpine'
    }
    to {
        image = 'com.github.pozo/spring-boot-jib'
    }
    container {
        ports = ['55000']
    }
}

If we want to build the container image we have two options. Using jib task which pushes a container image for your application to a container registry, or jibDockerBuild which uses the docker command line tool and requires that you have docker available on your PATH or you can set the executable location via the dockerClient object.

Open a terminal and execute

./gradlew jibDockerBuild

I'm going to explain why we need to use this task instead ofjib in the next paragraph.

Check the output of docker images command. If you wonder why your image created by ~49 years ago, It's for reproducibility purposes, Jib sets the creation time of the container images to 0. In order to use the current time just add useCurrentTimestamp = true inside of the jib.container. For more advanced questions check out the Jib's FAQ.

After the build phase we can run our image with

docker run -p 8080:55000 com.github.pozo/spring-boot-jib

If both http://localhost:8080 and http://localhost:8080/echo/test seems to work, we can continue with our Kubernetes objects.

Running the application image inside of Kubernetes

If you are not familiar with Kubernetes I recommend to start with the official documentation.

First of all we need an up and running Kubernetes cluster. Fortunately we have several options nowadays. For instance we can use Minikube which runs a single-node Kubernetes cluster inside a VM on your laptop.

An another option is Docker for Desktop Kubernetes. It runs a single-node cluster locally within your Docker instance. I believe this one is the most comfortable way to hack around with Kubernetes, so I suggest to use this one.

We also need kubectl which is a command line interface for running commands against Kubernetes clusters.

If you decided to use Minikube then start the cluster with

minikube start

This command creates and configures a Virtual Machine that runs a single-node cluster. This command also configures your kubectl installation to communicate with this cluster. You also need to run

eval $(minikube docker-env)

The command minikube docker-env returns a set of Bash environment variable exports to configure your local environment to re-use the Docker daemon inside the Minikube instance. (source)

This means you don't have to build on your host machine and push the image into a docker registry, you can just build inside the same docker daemon as Minikube. So after

./gradlew jibDockerBuild

Minikube will be able to reach out the image com.github.pozo/spring-boot-jib. Don't forget to run eval $(minikube docker-env) every time when you prompt a new terminal.

To enable the Kubernetes cluster in case of Docker for Desktop just follow the platform specific instructions.

To run our image in the cluster we need to define a Deployment first. Deployment represent a set of multiple, identical Pods with no unique identities. A Deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive. In addition we need an externally exposed Service whereby we can reach our application from outside of the cluster. Here is a diagram about what we want to achieve

Let's Create a kubernetes directory under the project's root and create a file spring-boot-jib.yaml with the following content under the freshly created directory

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spring-boot-jib
  template:
    metadata:
      labels:
        app: spring-boot-jib
    spec:
      containers:
        - name: spring-boot-jib-pod
          image: com.github.pozo/spring-boot-jib
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 55000
---
apiVersion: v1
kind: Service
metadata:
  name: spring-boot-jib-service
spec:
  type: LoadBalancer
  ports:
    - port: 8080
      targetPort: 55000
      protocol: TCP
      name: http
  selector:
    app: spring-boot-jib

A few important things to mention here.

The Deployment's spec.selector.matchLabels.app value must be the same as the spec.template.metadata.labels.app value
Service's spec.selector.app value must be the same as the Deployment's spec.selector.matchLabels.app value. So the service can find our Pod, and hand over every request to it.
The Service's spec.ports.port value should be what we want to expose to the outside world, in our case 8080
The Service's spec.ports.targetPort must be the same as the Deployment's spec.template.spec.containers.ports.containerPort so the Service will redirect everything to the container's port number 55000.
Finally the Pod's spec.containers.imagePullPolicy must be IfNotPresent or Never. The default value Always would produce an error, since there is no such com.github.pozorepository exist.

As we previously used jibDockerBuild we have our image locally, and because of Docker for Desktop's cluster uses our host's docker instance, it will able reach the image by default. In case of Minikube, due to eval $(minikube docker-env) the image built by Minikube's docker daemon, and it will able reach the image too.

Open a terminal and run

kubectl create -f kubernetes/

The create command iterates over the kubernetes directory and creates Kubernetes resources from it's content.

Open a browser and try to reach http://localhost:8080 and http://localhost:8080/echo/test again. In case of a Minikube execute minikube service list to find out the deployed service address. If we are getting status code 200 then we did a great job, and we have a running application in our cluster.

Put everything together with Skaffold

We almost crossed the finish line! Currently we have a containerised application and we can deploy it anytime into the cluster by hand. At this point we must repeat the building and deploying steps after every changes.

Skaffold going to help us to eliminate this handwork. Go to their website and follow the installation instructions. If everything set, create a skaffold.yaml into the project's root directory with the following content

apiVersion: skaffold/v1beta4
kind: Config
build:
  local:
    push: false
  artifacts:
    - image: com.github.pozo/spring-boot-jib
      jibGradle: {}
deploy:
  kubectl:
    manifests:
      - kubernetes/*.yaml

This is our brand new Skaffold pipeline file.

The build.local.push: false enforces Skaffold to use jibDockerBuild.
The build.artifacts is a list of the actual images we're going to be build.
The build.artifacts.jibGradle configures Skaffold to use Jib during the image building phase.
The deploy.kubectl.manifests value set the folder name where we already have our kubernetes manifest files. If we skip this the default directory name would be k8s.

If you are looking for more advanced pipeline configuration I recommend to check out this well annotated example.

Open a terminal and run

skaffold dev --trigger notify

This command runs our pipeline file in development mode, which means every change in our codebase will trigger Skaffold to call Jib to build the image, and kubectl to deploy it. Sounds cool right ?

Change the return value of TestController's hello method to "GOODBYE" and see what happens in the terminal. Refresh the browser after a few seconds, and you will see "GOODBYE" instead of "HELLO".

Using helm

If you are not familiar with helm I suggest to take a look at the official quick starter guide first.

I must admit It's absolutely not mandatory to use Helm for development, and someone suggest you to think twice before using it, however according to my experience helm making the application deployment easy, standardised and reusable, especially when you have to work with several applications.

Create a helm chart for our application with

helm create spring-boot-jib

The helm's create command will generate a directory structure with several files. In order to make it cleaner what do we have in this folder, rename it to helm. The most important files are in the templates directory and the values.yaml itself.

Change the generated service.yaml's spec.ports.targetPort to

targetPort: {{ .Values.service.containerPort }}

The deployment.yaml's spec.template.spec.containers.image value to

image: "{{ .Values.image.repository }}{{ if .Values.image.tag }}:{{ .Values.image.tag }}{{ end }}"

And the deployment.yaml's spec.template.spec.containers.ports value to

ports:
  {{- toYaml .Values.container.ports | nindent 12 }}

Everything between {{ }} came from the values.yaml or _helper.tpl files. In fact we are using Go templating. And change the values.yaml file like this

replicaCount: 1

image:
  repository: com.github.pozo/spring-boot-jib
  tag: latest
  pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: "spring-boot-jib"

service:
  type: LoadBalancer
  port: 8080
  containerPort: 55000

container:
  ports:
    - name: http
      containerPort: 55000
      protocol: TCP

As we want to use Helm instead of kubectl, we need to adjust the Skaffold pipeline accordingly

apiVersion: skaffold/v1beta4
kind: Config
build:
  local:
    push: false
  artifacts:
    - image: com.github.pozo/spring-boot-jib
      jibGradle: {}
deploy:
  helm:
    releases:
      - name: spring-boot-jib
        chartPath: helm
        values:
          image.repository: com.github.pozo/spring-boot-jib
        setValues:
          image.tag: ""

We must configure the chartPath, the image.repository, and we must set image.tag value to empty string for Helm, so Skaffold will be able to manage custom tags on the deployment.

If everything set, run

skaffold dev --trigger notify

If we are using Minikube we don't need to execute eval $(minikube docker-env) anymore, since Skaffold will take care of it. If you want to see what happens under the hood, just add the -v debug switch.

skaffold dev --trigger notify -v debug

Live debugging

During the development It's a natural demand to setup a breakpoint and check the application's state while It's running. Locally It's a very easy process but what if we have everything in the cluster ? Actually It's easier as you might think, we need to adjust just a few things.

Add 5005 to the list of port under the jib configuration in the build.gradle
Add the agentlib related configuration inside of jvmFlags property

jib {
    to {
        image = 'com.github.pozo/spring-boot-jib'
    }
    container {
        useCurrentTimestamp = true
        ports = ['55000', '5005']
        jvmFlags = [ '-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005' ]
    }
}

Extend values.yaml's container.ports with

- name: debug
  containerPort: 5005
  protocol: TCP

Create a remote configuration for the debugger

Then run the pipeline. Skaffold will automatically forward every listed ports, however be careful, after a change Skaffold might pick a random port if the requested one isn't available. Set a breakpoint, and run the previously created Remote configuration, call the corresponding endpoint and voilà.

I hope you enjoyed reading the article as much as I enjoyed writing it. I'm not a proficient writer, so if you have a comment, remark feel free to share it in the comments section. The source code of this article is available on GitHub.

I want to thank Gergő Szabó and Dániel Szabó for all the help.

If you are interested in this topic I suggest to look over these articles as well

DEV Community: Zoltan Polgar