Containerising and Deploying with k8s

This started less an article or blog, more a journal of my many failed attempts at containerising and deploying a small app. I was keen to practice with the technologies I use every day, but without the guardrails of a fantastic platform team with all the tools and support they provide.

The following describes the (eventual!) successful steps to...

• Package a Kotlin web application with environment secrets in a Docker image

• Upload the image to a container registry, and run both manually and as part of a CI build pipeline

• Create a Kubernetes (k8s) cluster and deploy the app, both manually and as part of the CI build pipeline

• Expose the application to the internet

The code for the Tempo-Valence application used can be seen here

---

1. Package app in a docker image

• Google Kubernetes Engine (GKE) accepts Docker images as the application deployment format.

• Using gcr.io container registry (after failing with other registry providers - see out takes below!)

  • Enable Cloud Registry API in Google Cloud Platform (GCP) console
  • Set up Service Account in GCP Service Accounts - IAM & Admin so that this can be used to access the Google container registry (gcr). This service account will need to have a Storage Admin role
  • Download key from created GCP service account and add it to a gitlab-ci variable - called GOOGLE_CREDENTIALS in my example, which is accessed in the gitlab-ci.yml file below
  • (The key can also be saved to a local, non-committed keyfile somewhere if you wish to run the jib docker image locally to check)
  • Update build step of gitlab-ci.yml to...

build:
    <<: *gradle-image
    stage: build
    variables:
        CLIENT_KEY: $CLIENT_KEY
        GOOGLE_CREDENTIALS: $GOOGLE_CREDENTIALS
    before_script:
        - echo "$GOOGLE_CREDENTIALS" > keyfile.json
    script:
        - export CLIENT_KEY="$CLIENT_KEY"
        - gradle jib -Djib.console=plain
        - gradle clean build
    only:
        - master

• Add jib config in build.gradle.kts of the project as follows...

jib {
    container {
        mainClass = application.mainClassName
        environment = mapOf(Pair("CLIENT_KEY", System.getenv("CLIENT_KEY")))
        ports = listOf("8000")
    }
    from {
        image = "openjdk:11"
    }
    to {
        image = "gcr.io/<gcp-project-name>/tempo-valence:latest"
        auth {
            username = "_json_key"
            password = file("keyfile.json").readText()
        }
    }
}

(The CLIENT_KEY environment variable above refers to a secret env var required to access an external api from the application. The value of this env var is stored as a Variable in Gitlab CI so the pipeline can access to run the app.)

---

2. Upload docker image to a container registry

• See previous step

• To run locally...
  

docker run --rm -p 9000:9000 gcr.io/<gcp-project-name>/tempo-valence:latest

# in new terminal tab
curl http://localhost:9090
# Welcome to TempoValence!

---

3. Create a k8s container cluster

Following manual commands create cluster, deploy application and expose on an external IP

gcloud auth list

gcloud config set account [e-mail]

gcloud config set project <gcp-project-name>

gcloud config set compute/zone europe-west2-a

gcloud container clusters create tempo-valence-cluster

gcloud compute instances list

kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest

kubectl get deployment

kubectl get pods

kubectl scale deployment tempo-valence --replicas=2

kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5

kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
# the ports here matter - think it has something to do with how we've set up the application in the image

kubectl get service

# and use the EXTERNAL_IP of tempo-valence-service to access, e.g...
http://35.234.150.122:9000/

---

4. Deploy

• See step 3 for manual deployment

• In order to deploy from the gitlab-ci pipeline, the gitlab-ci.yml looked like this for the deploy stage...
  

deploy:
stage: deploy
variables:
    CI_SERVICE_ACCOUNT: $CI_SERVICE_ACCOUNT
image:
    name: kiwigrid/gcloud-kubectl-helm
before_script:
    - echo "$CI_SERVICE_ACCOUNT" > key.json
    - gcloud auth activate-service-account --key-file=key.json
    - gcloud config set project <gcp-project-name>
    - gcloud config set container/cluster tempo-valence-cluster
    - gcloud config set compute/zone europe-west2-a
    - gcloud container clusters get-credentials tempo-valence-cluster --zone europe-west2-a
script:
    - kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest
    - kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
    # Wait for the deployment to be applied
    - kubectl rollout status deployment tempo-valence --watch

Few things to note here!

• Set up another service account in GCP, to be used by gitlab ci to authenticate against both the container registry and k8s cluster (may be possible just to amend the roles of previous service account, but I added a new one to keep it separate)

  • Give the service account 2 project roles -> Kubernetes Engine Developer and Storage Object Admin
  • Download the key associated to the account and add in to a new CI variable in gitlab

• The image is the image used to run the deployment - bit of trial and error here, as this image needed to have both kubectl and gcloud cli tools loaded (and I couldn't be bothered to create one and store it somewhere that didn't need authentication!) The image included above is open and does the job.

• For this to work, the tempo-valence-cluster of course needs to be up and running

---

---

Steps to run

As this will not be continually running here are the steps needed before this will be available on the internet

---

To setup

gcloud container clusters create tempo-valence-cluster

# Run ci pipeline, or...
kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest

kubectl scale deployment tempo-valence --replicas=2

kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5

kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000

---

To teardown

kubectl delete service tempo-valence-service

gcloud container clusters delete tempo-valence-cluster

---

---

Links and article credits

• A lot of ideas from this Google k8s tutorial

• And this GCP tutorial

• One of our own from Matt Dowds on Kubernetes deployment form Gitlab CI

---

---

Outtakes - my many failed attempts at step 1!

Package app in a docker image

• Google Kubernetes Engine (GKE) accepts Docker images as the application deployment format. To build a Docker image, you need to have an application and a Dockerfile (in that application)
• Add Dockerfile to root of project. 1st attempt looked like this...

# use base image
FROM openjdk:11

# set working directory
WORKDIR /

# copy the runlocal.sh file (contains env vars and gradle run command)
COPY runlocal.sh .

# Inform Docker that the container is listening on the specified port at runtime
EXPOSE 8080

#Run the specified command within the container
RUN ./runlocal.sh

# Copy the rest of your app source code from your host to your image filesystem
COPY . .

• Run the following command to build and test docker image locally...

docker image build -t tempo-valence:1.0 .

Sadly this exited in error...

Step 5/6 : RUN ./runlocal.sh
---> Running in 7d1ddc474a6f
./runlocal.sh: line 5: ./gradlew: No such file or directory
The command '/bin/sh -c ./runlocal.sh' returned a non-zero code: 127

• So next I tried working with a gradle base image...

FROM gradle:5.4.1-jdk11 AS build

Gradle started, but sadly a config error was returned...

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project ''.

• Then looked in to --env-file and using other ways of getting the relevant client key into the image without exposing it. Finally added the env_var to a .env file and accessed in Dockerfile as follows...

FROM openjdk:11

WORKDIR /

EXPOSE 8080

ENV CLIENT_KEY .env

COPY . .

This successfully builds and tags the image, but my concern is now that this .env file is only available locally, as it is .gitignore'd

• Running the image as a container using

docker container run --publish 8000:8080 --detach --name tv tempo-valence:1.0

# --publish 8000:8080 asks Docker to forward traffic incoming on the host’s port 8000, to the container’s port 8080. Containers have their own private set of ports, so if you want to reach one from the network, you have to forward traffic to it in this way. Otherwise, firewall rules will prevent all network traffic from reaching your container, as default

# --detach asks Docker to run this container in the background

# --name specifies a name with which you can refer to your container in subsequent commands, in this case tv

didn't run the app, as we have no current run command in the Dockerfile

So, added a couple of bits back in the Dockerfile...

FROM gradle:5.4.1-jdk11

WORKDIR /

EXPOSE 8080

ENV CLIENT_KEY $CLIENT_KEY

CMD [ "gradle", "run" ]

COPY . .

This successfully runs the container, but I cannot access the app via the browser on http://localhost:8000 as expected

Some links being used so far...

• https://docs.docker.com/get-started/part2/
• https://www.reddit.com/r/docker/comments/7dhszj/dockercompose_how_do_i_expose_a_port_to_my/

Some useful commands...

• docker exec tv env (lists env vars set in the container)

---

New attempts... (13.03.20)

• Tried using jib to containerise the app

• Added the following into build.gradle file...
  

plugins {
    id 'com.google.cloud.tools.jib' version '2.1.0'
}

jib {
    container {
        mainClass = application.mainClassName
        environment = [CLIENT_KEY:System.getenv("CLIENT_KEY")]
        ports = ['8000']
    }
    from {
        image = "openjdk:11"
    }
    to {
        image = "matttea/matttea-images/tempo-valence:latest"
    }
}

• Run the following commands to build...

export CLIENT_KEY=<client_key_value>

gradle jib

• Returned error related to authentication...

Containerizing application to matttea/matttea-images/tempo-valence...
Base image 'openjdk:11' does not use a specific image digest - build may not be reproducible
The credential helper (docker-credential-desktop) has nothing for server URL: registry-1.docker.io

Got output:

credentials not found in native keychain

...

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Build image failed, perhaps you should make sure your credentials for 'registry-1.docker.io/matttea/matttea-images/tempo-valence' are set up correctly. See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-unauthorized for help

• Using this link

• See also this journal entry on docker credentials helper

---

New attempts... (25.05.2020)

• tried changing job image to detail to image = "https://index.docker.io/v1//matttea/matttea-images/tempo-valence:latest" based on docker logout command url response

• didn't work...
  

> Task :jib FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> Invalid image reference https://index.docker.io/v1//matttea/matttea-images/tempo-valence:latest, perhaps you should check that the reference is formatted correctly according to https://docs.docker.com/engine/reference/commandline/tag/#extended-description
For example, slash-separated name components cannot have uppercase letters

(Originally posted on https://logical-progression.matttea.com)