DEV Community: Erik Lindblom

Preventing your Kubernetes volume from being deleted in GKE

Erik Lindblom — Tue, 30 Jun 2020 00:00:00 +0000

The story

This one caught off guard. I have this Kubernetes cluster I managed, and well, one day I was doing my thing. And I noticed that my laptop was low on resources, so I decided to clean things up. Cleared out a few namespaces and everything was good to go. Well, not 3 seconds later, someone pings on a chat channel that their dev environment is gone. Well...crap. I double-check my kubectl context, and yup, sure enough. I was not connected to my local instance, I was connected to the dev Kubernetes cluster. Oops!

Thankfully this was only dev, and not production or anywhere else important. But it still stood I blew up the cluster, and needed to rebuild it, and quickly. This post isn't how I brought that cluster back online (spoiler: we have tools and pipelines to do that quickly), but how I figured out preventing this from happening in the future.

What are PVCs

Let's make sure we are on the same page. If you don't know. Persistent volume claims (PVC) are a way for you to get permanent disk storage in your Kubernetes cluster. This means if your pod dies, or is deleted, you can have your data persist. This is all fine and dandy but by default. Those PVCs are a namespaced resource. Which means if you delete the namespace. That resource is also deleted.

Using persistent disks in GKE

To prevent your data from getting lost if the namespace is deleted. You need to attach that data to a disk that lives outside of Kubernetes control. To do this, we will use a disk created in Google cloud, and attach to that disk inside our Kubernetes environment. This is documented in GKE but let me give you the easy steps of getting there:

First, let's create the disk:

gcloud compute disks create my-disk --type=pd-ssd --size=1G

Next, let's attach to that disk in a pod:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: demo
spec:
  capacity:
    storage: 1G
  accessModes:
    - ReadWriteOnce
  claimRef:
    namespace: default
    name: demo
  gcePersistentDisk:
    pdName: my-disk
    fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1G

Lots of stuff right? Some import bits are that we supply both the PersistentVolume and the PersistentVolumeClaim. Usually, we only had the claim itself. On the volume, the gcePersistentDisk identities the name of the disk to attach to, and fail if it not found. Also, the claimRef needs to match up with both the namespace and the name for this to be bound correctly. Also, be careful that the storage also lines up correctly, otherwise again, things won't be bound.

Migrate to use that disk

So, creating a disk is rather straight forward, but what if you already have a disk and want to migrate over to this? We can do that. All PVCs in GKE already have a disk attached to them. They are just managed outside of the cluster itself. So here is the general workflow you will need to go through to migrate over to something more permanent:

Scale down your pods
Take a snapshot of your current disk
Delete your current deployment
Create a disk from the snapshot
Deploy again with the PVC attacked to the disk

Step: Scaling down the pods

To ensure that all the data is done being written to the disks, and nothing changes the data mid-flight. It is a good idea to make sure all pods that are using the disks are scaled down to 0. If you are using a deployment, this is as easy as kubectl scale --replicas=0 deployment/my-stuff. Then verify that all the pods are gone before proceeding on to the next step.

Step: Snapshotting your current disk

To get us to a new disk, we need to go through and create a snapshot of the data you currently have, to seed the permanent disk. Whats need is that we can actually script this next part out. Here is what this script would look like:

# Capture properties of disk
VOLUME_NAME=$(kubectl get pvc my-pvc -o=jsonpath='{.spec.volumeName}')
DISK_NAME=$(gcloud compute disks list --filter="name~''$VOLUME_NAME''" --format="value(name)")
DISK_ZONE=$(gcloud compute disks list --filter="name~''$VOLUME_NAME''" --format="value(zone)")

gcloud compute disk snapshot --zone $DISK_ZONE --snapshot-names my-snapshot $DISK_NAME

First, we had to capture data from the PVC. In this example, mv-pvc would be the name of the PVC you create in your cluster. We capture the volume name which is the name of the disk created outside of the cluster.

Next, we need to capture some information about the disk itself. We need to know the full disk name, along with which zone it was created in.

From there, we then have all the information we need to create our snapshot my-snapshot (all these names can be tweaked, and are given as examples).

Step: Deleting your deployment

Next, let's go ahead and delete your deployment. If you want a simple one-liner, you can use kubectl to delete everything: kubectl delete all,pvc,pv --all-namespaces -l app=my-app. The label (-l) assumes you have some kind of annotation attached to your resources.

Step: Creating the disk

Now that we have everything cleaned up, let's create the new disk.

gcloud compute disks create my-disk --size=1G --source-snapshot=my-snapshot --type=pd-ssd

Fairly straight forward. This will create the disk with a 1G limit. We use the SSD disk type because we are under 100GB for this example. But tweak it for your needs.

Step: Deploying to the attached disk

Now that we have the disk ready, we need to redeploy our service back into Kubernetes. But this time, we need to ensure that your PVC attaches to the disk. We will use a manifest like this:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: demo
spec:
  capacity:
    storage: 1G
  accessModes:
    - ReadWriteOnce
  claimRef:
    namespace: default
    name: demo
  gcePersistentDisk:
    pdName: my-disk
    fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1G
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: demo
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: demo
    spec:
      containers:
        - name: task-pv-container
          image: nginx
          ports:
            - containerPort: 80
            name: "http-server"
          volumeMounts:
            - mountPath: "/usr/share/nginx/html"
            name: task-pv-storage

If all goes well, all your data is still present and you have no migrated over to a more permanent disk.

Terraform, Google Cloud, and Kubernetes working together

Erik Lindblom — Sun, 07 Jun 2020 00:00:00 +0000

Here is the situation: I want a secret (let's say a database password or any Kubernetes resource) stored inside a Kubernetes secret. The infrastructure is controlled by Terraform. So, how do I go about to keep those two things in sync?

Well, recently I found out a neat pattern! You can connect Terraform with both GCP infrastructure, and Kubernetes at the same time, and you can keep them in sync! How do you say? Well, let me show you!

Let's look at the full-blown example and then we will break the interesting bits down:

provider "google" {
}

data "google_client_config" "default" {
}

provider "kubernetes" {
  load_config_file = false
  host             = "https://${google_container_cluster.primary.endpoint}"
  token            = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(
    google_container_cluster.primary.master_auth[0].cluster_ca_certificate
  )
}

resource "kubernetes_namespace" "test" {
  metadata {
    name = "test"
  }

  depends_on = [google_container_node_pool.primary]
}

Ok, ok. This isn't a full-blown example, but these are the most important pieces. The full example can be found in my example repository here. Here I am just highlighting the pieces you probably already want and need.

The first big piece you will need is the google_client_config. This piece is the one that led me down this road in the first place. This gives information back to Terraform on how to access the Google API and how to use other pieces. The most important piece here is the access token. This is a short-lived token that will be used to communicate with the cluster with whatever privilege the gcloud command (or Terraform provider context) has.

As you may notice, the google_container_cluster gives us the other pieces. It gives us the host, along with the CA certificate of the host. This then gives you full access to the Kubernetes instance and allows you to start managing the instance with Kubernetes!

Now the next bit that may trip people up is the depends_on. Because the provider doesn't naturally give you dependencies. Terraform can't figure out the dependency graph on its own, so it needs some help. We specifically target the node pool because we want actual node workers in kubernetes for certain things to work (i.e. namespace is a good example that needs a worker node to work fully). If you don't have this, destroying and or other massive changes might leave you in a stuck state.

Now that we've tied things together using a basic namespace. Let's see how we would tie a secret. In our case, this will be the password to the database user. This would look like this:

resource "random_password" "user" {
  length = 16
}

resource "google_sql_user" "users" {
  name     = "user"
  instance = google_sql_database_instance.instance.name
  password = random_password.user.result
}

resource "kubernetes_secret" "user" {
  metadata {
    name      = "user-password"
    namespace = kubernetes_namespace.test.metadata[0].name
  }

  data = {
    password = random_password.user.result
  }
}

We use Terraform to generate the password, the link the two other resources together. In this case, we create a SQL user and the secret to store that same password for a pod to use to connect to the database.

The trickiest part of all of this is to ensure that the dependency graph that Terraform generates is accurate. So using depends_on and other things may need to be used more often.

Step by Step: A simple Node.js, Docker, and Kubernetes setup

Erik Lindblom — Sat, 16 Nov 2019 00:00:00 +0000

I've now been playing with Node.js, Docker, and Kubernetes for quite some time. And it just so happen that recently someone needed a good introduction to Node.js, Docker, and Kubernetes. However, after searching online I couldn't find one that just had a few simple things to walk through. So, here this is. Hopefully this blog post will demonstrate how to create a simple Node.js, create a Docker container, demonstrate it running, then deploy that Docker container to a local Kubernetes setup. There will be light touches on what exactly all of those parts are and hopefully give you a starting point to start exploring these technology stacks.

Step 0: Prerequisites

I am going to assume a few things in this blog post. First, you have Node.js installed. I prefer to use nvm as my manager of my node instance, but there are several out there that can do the trick. For this blog post, I will be using the latest LTS Dubnium release 10.16.3. I will also be using yarn as the Node.js package manager.

Next, we will need Docker installed. If you are using Mac or Windows, go ahead and get the wonderful Docker for Mac/Windows tools. This will give you a wonderful set of tools to use Docker on those platforms. For Linux, go aheads and get a Docker CE from what ever distro package you have. For this blog post, I will be running Docker for Mac 2.1.3.0. I will also verify that it works on Linux, but sadly don't have a way to verify Windows at this time. There isn't anything too complicated here so I have some confidence that it should work across platforms fairly easily.

Next, we will need a Kubernetes instance running locally. For Mac and Windows, that is built into the Docker for Desktop tool. For Linux, I recommend Minikube.

That should be all of the base tools you will need. Hopefully those are all fairly easy to install, but if you run into issues, please reach out to me and I'll attempt to help and add notes to this blog post for future visitors.

Step 1: A basic node server running

First thing first, let's setup our environment with a very basic Node.js Express server and get it running. Get to a blank directory and run the following command:

> yarn init -y

Next, let's get our Express library. We do that by running the following command:

> yarn add express@4.17.1

Rant: Now, if you are familiar with the Node.js ecosystem you may find it very odd that I added a specific version of the express library. First, you should definitely try and lock your packages down to as specific version as you can. Personally, I've been bitten far too many times by drifting dependencies. Yes, the lock files help this, but it still happens from time to time. So try and lock things down to as specific as possible. I hope you will thank me later, and I'm sad that the Node community uses fuzzy versions far too often in my opinion.

This should install the Express library and create a yarn.lock file and a node_modules folder with all the files needed for that library. Now that we have Express, let's create a very simple server. Here is what you want in the file index.js:

const express = require('express');

const app = express();

app.get('/', (request, response) => response.send('Hello World'));

app.listen(8080, () => console.log('Running server'));

Let's go ahead and run this file by running the following in a command prompt: node index.js. You should get the Running server output on the console and then you can visit http://localhost:8080 and see the Hello World text in the web browser. If you do, congratulations! We have a very simple web server up and running. If not, double check that you have the package installed correctly, and that your index.js is in the same folder as the package.json and node_modules folder. Please reach out if you need help getting past this step so I can help troubleshooting steps.

Step 2: Dockerize

Now that we have some working code, let's go ahead and get this application stuffed into a Docker container. Create a file named Dockerfile and put this inside of it:

FROM node:10.16.3 as builder

WORKDIR /build
COPY . .
RUN yarn install
RUN yarn install --production

FROM node:10.16.3-slim

WORKDIR /app

COPY --from=builder /build/node_modules ./node_modules/
COPY --from=builder /build/index.js .

CMD node index.js

Let's go through this line by line to understand what we are doing:

Line 1: Very first thing you do in a Dockerfile is define where the starting point is. For us, we are going to use the Node with our locked in version. Now, something you may not be familiar with is the as builder. We are going to use what is called a multi-stage build. This is slightly overkill for our example, but this is a framework for future work. We are going to use a builder that will build up our application. Then we will copy over the smallest amount of bits we absolutely need for a production system. This way we have the smallest image we need to ship into production. Also from a security perspective, we are shipping the smallest amount of thing so our foot print is as small as possible.

Line 3: The WORKDIR command sets our default working from and also sets where we are currently working from. We are going to use a folder at the root called build and work from there

Line 4: First we are copying over everything into our Docker container with a neat little trick of COPY . .. Now, this may look funny so let me explain what kind of magic this is doing. Remember that we are asking the Docker system to copy things into the Docker environment. So the first parameter in COPY is referencing from the filesystem relative to the Dockerfile. The second parameter is referencing in relation to where in the Docker container it should put those files. For us, we are asking to copy everything from our project, into the Docker container. It's a neat trick I employ instead of trying to copy different folders. If I need to exclude things, you will use the .dockerignore file.

Line 5-6: Now, this looks VERY odd, but just hang in there with me. First we use yarn install to get all of the dependencies. While, yes, the very next line we do yarn install --production, I do this for a good reason. More likely then not, you will want a build step to do something. Either packing, compiling, transpiling, take your pick. You can add any step in between those two yarn install commands to get the right build system setup that you need.

Now that we have a docker image, let's just go through and test this docker image and make sure things work just like they did in the last step. First, let's build the docker image by running docker build . -t myimage. The -t myimage tags the image with a name we can easily use.

To run the image you just built run docker run --rm -it -p 8080:8080 myimage. You should be able to hit http://localhost:8080 and get the same Hello World text like you did in the last time. hit ctrl+c to stop the image.

Step 3: Pushing a docker image and prep work for kubernetes

In this tutorial, I am going to assume you have a kubernetes instance up and running somewhere. If you don't, you can either use Docker for Desktop which has Kubernetes built in for both Mac and Windows. Or, you can use minikube.

No matter where you have it running. This tutorial will assume you have kubectl pointed to a running Kubernetes instance and that you also have a registry you can upload your docker image.

Let me actually go into detail a little bit about that last thing. We need to push the Docker image to a registry for your Kubernetes instance to pull down. Now, there a wide range of place you can do that. And that require a wide variety of different methods to do it. I am going to assume that you can docker push some kind of image somewhere and that is accessible to your Kubernetes cluster. If you and running the Docker for Desktop tool, a docker build will suffice. If you are running Minikube, you will need to reuse the Docker daemon. If you are running a cluster in the cloud somewhere, you will have to make sure that Kubernetes is setup to pull from that registry.

Step 4: Deploying that image to Kubernetes

With your image now ready to deploy, lets go through what that would require. For this tutorial we are going to create a deployment and a service.

A deployment is a Kubernetes object that defines how to create "pods". A pod is a single (but can be multiple) runner Docker instance. A deployment controls how many pods are currently running and has all of the logic built into making sure there are enough pods to satisfy your requirements. It also helps control roll outs as you update your image. This means that as you roll out a new image, it will bring a new pod up, make sure the pod is running, and then kill off old pods in a controlled manner. Deployments are usually your bread and butter, but they aren't the only objects that control pods. There are a few different types of controllers out there but this tutorial will only be focused on the deployment variety.

So, if a deployment controls whats running inside of Kubernetes, how do we expose that pod to network traffic? Like maybe public internet traffic? That is where services come in. A service is a Kubernetes object that controls how network connections are made to the pods. A service defines which ports are open and are connected, and whether the pods should be exposed internally to the Kubernetes instance or externally. Services can also do load balancing if you desire.

Now, while this glossed over a lot of details, I think this should make you dangerous enough to start with. Let's look at how a deployment and service object are created and deployed to Kubernetes now. Let's take a look at this file:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: myimage
        imagePullPolicy: Never
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

Holy crap batman thats alot! Let's walk through what all of this means.

Line 1 & 24: For this example I put both objects inside of one file. Not always a normal thing todo, but its an option. The --- is a YAML file separator for multiple YAML objects inside of a file. Just want to point this out first if you see these files separated in the wild. Thats fine, I just wanted to give you one file to play with instead of multiple.

Line 2, 3, 25 & 26: This describe the type of Kubernetes object. There are two parts to this. The apiVersion, and the kind of object. These set of properties let's Kubernetes define a whole host of options and let's them version out behavior for certain objects. You can find which objects are support by running kubectl api-resources and the versions of those with kubectl api-versions. The resources list which API group is used, which you cross-reference to which version you should use. If the resource is listed blank, its part of "core" which is usually just v1. You usually don't fiddle with this much, and just copy from project to project. But its better to be aware of why this is here then just blindly copying it.

Line 4 - 7: This section describes the metadata for the deployment. Metadata is just that, information about the object. For a deployment there are two main parts, a name which is exactly that, and is required. Then some kind of label. The label is important because this gives you the ability to "select" this deployment depending on what kind of values you give the object. This will become important later on in our service.

Line 8: This starts the meat of the deployment object, the spec or specification of what you want to deploy.

Line 9: The replicas is the number of instances you want running.

Line 10 - 12: This section describes what pods the deployment controls. Usually this means you create a selector that has the same matching labels as your template section. I personally haven't come across a case where this didn't match up with what I had in the template section, but I'm sure there are cases out there.

Line 13: This is the start of the template section. The template section will describe what each pod will have. This includes the image of the container, along with any environment variables, files, etc that are needed to run that pod.

Line 14 - 16: This section contains the metadata for each pod that is run. Again, usually this just contains the a label that has information for your selector in the above section.

Line 17: This defines the spec for a pod. In this example we will have only 1 container, but this is the section we would add information for an initContainer or side car containers.

Line 18 - 23: This is the meat of the pod. We define a name, a image, and the ports that are exposed. The name can be whatever you want, it doesn't necessarily have to match the deployment name, but usually does for making life easier later. The image is the location of the docker image. In this example I am assuming that you are using the Docker for Desktop tool, which means we can give it the same name as the last step (myimage). I also added a imagePullPolicy because the Kubernetes instance inside of that tool should not try and reach out to the internet for this image. I would recommend reading up on which image pull policy is right for your situation. We list the ports that are exposed next. This isn't completely necessarily but usually added for documentation proposes.

Line 29: This section defines our service and how it operates. Let's dig into this section now.

Line 30 - 31: This defines what pods should be exposed through this service. This usually matches very closely to what the deployment had in its selector as well.

Line 32: Since we want to expose this service we want to put a type on it. There are a couple of types, and the one we are interested in is the LoadBalancer. This is because we want to expose this service outside of Kubernetes, and that requires a load balancer for that.

Line 33 - 36: This defines the ports that are going to be exposed from this service. For our example, we are going to take the pods port 8080 (targetPort) and expose it to the outside world on that same port 8080 (port). We could have exposed it on port 80 if we wanted too. But for this instance, we just went for the easy route of aligning those numbers up.

Phew, that is a lot. So what should I do with all of this now? Well, let's deploy it. To do that we would run kubectl apply -f deploy.yaml. This of courses assumes that all of the above is in a file called deploy.yaml. Kubectl would then submit that file to Kubernetes and the magic starts to happen on creating the pods. To see your pods up and running we would run kubectl get pods and hopefully you would see something like this:

> kubectl get pods
NAME                    READY   STATUS        RESTARTS   AGE
my-app-bb697dc4-q6vl7   1/1     Running       0          14s
my-app-bb697dc4-qpjgf   1/1     Running       0          14s
my-app-bb697dc4-vsxcv   1/1     Running       0          14s

As you can see, you see the name attribute come through. Along with a deployment number (bb697dc4 in this example) and a pod number (q6vl7, qpjgf, and vsxcv in this example).

If everything is running, we should then be able to hit the service. To view the status of the service we would run kubectl get service and see something like this:

> kubectl get service
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
my-service   LoadBalancer   10.106.118.92   localhost     8080:32361/TCP   44m

If we hit that External-IP with the port, we should see the same Hello World we saw in the above 2 examples.

Conclusion

Well, we made it! I know there is a lot in here, and there is definitely a lot more, but hopefully this gives you enough pieces that you can start putting your own software together that can run on Kubernetes. Always feel free to reach out to me if you have question or comments.

Setting up a Docker container on a Container-Optimized OS running on the Google Cloud

Erik Lindblom — Mon, 11 Mar 2019 00:00:00 +0000

Wow, that title looks like a mouth full doesn't it? So why do this? Isn't there Kubernetes for running Docker containers? Well, sometimes I only need just one container running and I really don't want to add the overhead of having to maintain another Kubernetes clusters. Lame excuse, but this works well for a small installation where I just don't need the power of a full Kubernetes cluster.

What exactly is a Container-Optimized OS (COS)? Well, this OS is built by Google to be optimized for being the base operating system of a Kubernetes cluster or even just a simple Docker container. Its built upon the Chromium OS and has a lot of security features baked into it from the onset. For example, the root filesystem is mounted as read only. So the binaries that are on the system don't change, and can't be changed. Go ahead and read all about it, the project is a great one.

Here is what I will show you in this blog post. Throughout this blog post, we will use a base Nginx Docker image as the container we want to run. We will first use some short cuts the gcloud command provides to run a container on this image. We will then go through what would be required to get a image from a private repository up and running. Let's get started!

All source code is hosted on github as well so you don't have to retype if you don't want too.

Perquisites

This post will assume you have a Google cloud account all setup and a Google cloud project already carved out. If you haven't maybe follow the quickstart tutorial, then come back.

Warning: I will be showing commands that create resources inside of a Google Cloud environment. That means real resources that cost real money. Be careful if you don't want to be charged.

Making gcloud do all the work

Let's do something very basic, we will start up a container using the gcloud commands to run a container on top of the COS VM. Here is what that gcloud command looks like:

gcloud compute instances create-with-container nginx-vm --container-image nginx:1.15.8 --tags http-traffic

Take note of the external IP address that is spit out after running that command. We will need that in the next step.

Next we will need to expose that instance with a firewall rule:

gcloud compute firewall-rules create allow-http-traffic --target-tags http-traffic --allow tcp:80

Once that is complete, go ahead and try and hit the external IP address of that image. You should now see a "Welcome to nginx!" screen.

Before the next part, if you want to delete that image, go ahead and run gcloud compute instances delete nginx-vm.

Well that doesn't look hard does it? We can take images from docker hub and push them up, and bam, we have a running docker container running in the cloud. However, what happens if we want our own code, or code not hosted on the Docker hub? Let's create a private docker image and show you how to authenticate and pull that image down.

Setting up a custom Docker image

Let's establish a private docker image that we would like to be running on the COS image. We will take the base nginx image, and modify it with a custom index page to demonstrate we can build and deploy the custom image.

Here are the two files you will need, first the new index.html file:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to My Custom nginx!</h1>
</body>
</html>

And here is what the Dockerfile would look like:

FROM nginx:1.15.8

COPY index.html /usr/share/nginx/html

If you would like to test that locally, run docker build -t test . && docker run --rm -p 8080:80 test and point the browser to http://localhost:8080, you should see the "Welcome to My Custom nginx!" banner.

Let's now take our image and push it up into the private repository. This is a little beyond the scope of this blog post, but go ahead and pick somewhere you can upload a docker image that can be accessed over the public internet. I've used gitlab but any registry really would work. Just make sure its accessible to the public. If you need help with that part, reach out to me and I'll attempt to assist.

Authenticating with the registry

Now that we have our image up in a registry, we need some way to authenticate with that registry on our VM. If you were doing this locally, I bet you ran a docker login command at some point to get authenticated with that registry. Well, thats exactly what we are going to do with this VM as well.

So how do we going about running that command? Well, COS has a toolkit installed called cloud-init. Cloud-init is a set of tools to manage cloud images. These help provide ways to create startup scripts to get your cloud image running in exactly the way you want. They are used on most cloud providers and most distros have hooks or tools that can be used. For this particular case we will be interested in providing user scripts that can run at startup. For this, we will be providing a user-data variable that provides a cloud-config block. This block will create a user, a service definition, as well as run a startup script to start that service. Sounds easy right? Well, let's go!

Let's take a look at what this cloud-config configuration file may look like here:

#cloud-config

users:
- name: cloudservice
  uid: 2000

write_files:
- path: /etc/systemd/system/myservice.service
  permissions: 0644
  owner: root
  content: |
    [Unit]
    Description=My Service
    Requires=docker.service network-online.target
    After=docker.service network-online.target

    [Service]
    Environment="HOME=/home/cloudservice"
    ExecStartPre=/usr/bin/docker login %REGISTRY% -u %USERNAME% -p %PASSWORD%
    ExecStart=/usr/bin/docker run --network=host --rm --name=myservice %DOCKER_IMAGE%
    ExecStop=/usr/bin/docker stop myservice
    Restart=on-failure
    RestartSec=10

    [Install]
    WantedBy=multi-user.target

runcmd:
- iptables -A INPUT -p tcp -j ACCEPT
- systemctl daemon-reload
- systemctl enable --now --no-block myservice.service

This configuration file is used as a template that will generate us a configuration file for our cloud-config process. Let's point out some of the more important lines then I'll explain what the variables are:

Line 1: This flags this file as a cloud config file. If you don't have this, the system won't pick it up and process it. It is VERY important to have it exactly this way.

Line 3 - 5: Sets up a user dedicated to running this service. This creates a security sandbox because this user will not have any permissions assigned to it except to run this container.

Line 7 - 26: This is the file contents of /etc/systemd/system/myservice.service. As the path suggests, this is a systemd service file that will run our docker container. This section has information about the file permissions as well as the actual content of the file embedded in this configuration.

Line 12 - 15: Sets the description of the service as well as any requirements the service needs to run. In this case, we want the network to be online and docker to be running before this service starts.

Line 17 - 23: Sets how the service starts and stops. This is some of the more important bits. First, we are setting up the home directory to be the user that we had created earlier. This kinds of sandboxes and isolating where this is running. Next the ExecStartPre is where some of this magic starts to come into place. This is where the docker login command is executed. All of the parameters are currently variables and can be replaced but this is where the magic happens. Next the ExecStart actually runs the docker run command. One important thing to note is the --network=host flag. We want the docker container to be using the host's network interfaces instead of creating the normal isolated network stack. This way the container can act just like the host on the network and have all ports exposed without any further magic.

Line 22 - 23: This sets up the service to restart on failure and wait 10 seconds between each retry

Line 28 - 31: This section runs once the configuration has been read and all other parts are completed. This acts like a startup script. First we setup the internal firewall to accept TCP connections using iptables. Then we reload the systemd daemon to read in our configuration files, then we enable and queue up startup of our service. This is really important to note,the --now --no-block commands are important here because we want the service to start up now, but we also want it queued in the systemd process so that it waits for the Docker service and network connects to be online. Otherwise our container may not start right because Docker isn't ready or we can't reach out to the internet to get our container.

Now, what are all these variables for:

%REGISTRY%: This defines the root domain of where the registry is. For example, in testing this, I used gitlab so the registry would have been registry.gitlab.com.

%USERNAME%: This is the username to be logged in as. Keeping with the gitlab theme, I used a deploy token and the username was given to me by gitlab.

%PASSWORDD%: This is the password or token that can be used to log in.

%DOCKER_IMAGE%: This is the docker image path. I.e. registry.gitlab.com/myimage/test:1.

Now, how do I run this? Well, replace all of the variables with the right values and then run this command:

gcloud compute instances create test-nginx --image cos-stable-72-11316-136-0 --image-project cos-cloud --tags http-traffic --metadata-from-file user-data=cloud-init-config.yml

If you take that IP address that is spit out by that command, you should see the "Welcome to My Custom nginx!" banner. If not, maybe read through the troubleshooting section. If everything worked, congratulations!

Notes on troubleshooting

Alright, the site didn't come up. Now what? Well, let's first SSH into the box so we can start investigating what happened. To do that, let's make sure we have the firewall open by running this command: gcloud compute firewall-rules create allow-ssh-traffic --allow tcp:22. This should open up the firewall to allow you to SSH into the box by executing the following command gcloud compute ssh <box name, i.e. test-nginx>.

Now that you are on the box, what exactly are you looking for? Well, for our docker example we want to make sure we have the docker container up and running so we would issue a docker container ls command to verify it is up and running. If it is up and running, then you may have a firewall issue. Visit the console and inspect the network interface and see if the ingress is setup correctly. What we would like to see is that port 80 is open and working right. If it is, maybe the iptables didn't run as expected so verify on the VM instance itself you can do something like curl localhost. If that works, verify iptables is correctly setup.

But what if the container isn't even running? Where do I go from there? Well, we need to investigate the logs and start sifting through what may have happened. For this the sudo journalctl command is what you start poking around in. This gives you all of the logs of the box as it starts. Another command for our example is the sudo systemctl status myservice this shows you the raw status of the service, if its up or down, and what may be happening right now with the service.

Another thing to check that I ran into a lot is first very that the cloud init script is even running correctly. A great way to check for that is to verify that the files we were expecting are even there. In this example, we would expect to see a file at /etc/systemd/system/myservice.service. If that isn't there, usually the problem is that the very first line doesn't read #cloud-config EXACTLY. Double check your config and consult the logs for user-data parsing and retrieving. There is usually a log message that the file isn't understood from the variable and is ignoring the data.

Conclusion

Hopefully I've given you a taste of how to setup a single Docker container instance using the Container-Optimized OS on the Google Cloud. I've found this a useful feature when I don't really need all the horse power of a kubernetes instance. This lighter weight infrastructure for a few containers provides to be useful and cost effective. But also be warned, these instances are obliviously not replicated in anyway, so if one goes down, it is all down. Be careful in the cloud because every stack should be built on the assumption that cloud resources could come and go at will.

All of this code and examples are hosted on my github account.

Making Inspec with SSH work with Packer on the Google Cloud Platform

Erik Lindblom — Fri, 11 Jan 2019 00:00:00 +0000

This entry was originally published on my blog at blog.baens.net

Background

This blog post aims to help anyone trying to make a Packer image, tested with Inspec, on the Google Cloud Platform. I have recently been using the these tools for my day-to-day and I wanted to make these three tools work together, which was surprisingly difficult. This blog post will hopefully help anyone out there that may be trying todo the same.

What I want from these tools is have Packer spin up a new VM, run all the build steps, then test that VM with Inspec. Also, I didn't want to clutter up the VM with anything more installed then it really needed, so I didn't want Inspec running with all of its tools installed on the new VM. That means I wanted Inspec to run from my box then connect through SSH to the new box. Sounds simple right? Well, there were enough gotchas and lack of search resources out there that I felt I needed to write this up.

So enough yap, let's see what the code looks like. If you want to see everything put together, code repository here.

The problems I faced

So let me walk through the problems I encountered along this journey that I will try and demonstrate answers to.

Problem #1: The first problem I hit was that I needed Inspec to talk to the new VM, but Packer doesn't provide those variables very easily. What I needed was the host IP of the VM running in the cloud. Since Packer didn't provide that out of the box, I had to figure out a way to get around that.

Problem #2: Once I did establish a connection, I then needed to authenticate Inspec with the VM. Inspec could natively connect over SSH, so creating and establishing a SSH key to login with was the obvious choice. However, Packer again didn't provide a convenient way of doing this so I need to establish that key myself and set a few manual things to make that work.

Solving Problem 1: Getting the IP of the Packer VM in GCP

This sounds easy enough but is surprising difficult: how do I get the IP of the currently running VM that Packer is communicating with? One approach might ping some remote web site that tell you your public IP, but that seems ridiculous to introduce another layer outside of your control. My search stumbled upon a number of examples of how to do this on AWS but very little on GCP. However, people using AWS did give me an idea of what todo, or at the very least what to search for.

What people use on AWS often is called the Metadata server. The Metadata server is a service for every VM running on AWS (and GCP) that can tell you about the instance you are running on. This was the thing I needed, I found the documentation for GCP and sure enough, it had information on all of the exposed IP addresses. The particular address I needed was located at this URL: http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip. And whats even better was that this was generic across instances so I didn't have to do some fancy logic depending on the box.

Now that I have the HTTP address of where I can get the external IP address, how would I exactly retrieve that information from the command line. Thankfully, curl was already installed so I crafted this command:

curl  -H \"Metadata-Flavor: Google\" metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip

And executed that and viola, the IP address I needed.

Now, something to be pointed out. That HTTP header we added (Metadata-Flavor: Google) is required. This is a safe guard to protect you incase you accidentally point something wrong to this location.

Now that I have my IP address, how do I use it? Again, long searches didn't show a way to pipeline variables from one Packer step to another so what was I going todo? I'm not 100% sure how I stumbled upon but I found a very neat trick of downloading the file after you created it. Here is what that will look like in Packer's provisioning steps:

"provisioners": [
    {
        "type": "shell",
        "inline": ["curl -H \"Metadata-Flavor: Google\" metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip > /tmp/ip"]
    },
    {
        "type": "file",
        "direction": "download",
        "source": "/tmp/ip",
        "destination": "./host"
    }
]

And that solved the problem. Just have to create a file with data from a special internal endpoint, then download that file for use in next steps. Easy! Or at least easy once you work it out. Go figure!

Solving Problem 2: Communicating over SSH

Now that I had the IP address, I need to communicate from Inpsec to that IP. Inspec has several ways of communicating with the machine. It can do it locally or even execute on a remote machine. I wanted to execute locally but communicate on the remote machine so that I didn't have to install anything on the remote box. This lead me down the path that I needed to use SSH to communicate with that remote box. By default SSH requires a username and password, and since I was going to be using this in an automated build environment. Having something prompt for a username and password wasn't going to fly, never mind the fact that isn't very secure either. So I need to setup a SSH key somehow in the automated process to get logged into the remote box.

Google cloud provides a way for you to provide a SSH key that will get installed on the box for you to use. So first let's generate the key. The way Google parses the key information is it looks for comments inside of the key for which user is associated with this. Here is the command I use to generate the key:

ssh-keygen -f inspec-key -C packer -N '' -m PEM

The command will output a set of SSH key files called inspec-key with a comment of packer and the -N flag is the password to be used (blank in this case). You need to specify the PEM format because some of the Ruby modules that will be loaded can't parse newer OpenSSH key formats.

Alright, I now have my SSH key, now I need to place that SSH key for Packer to use when it creates the VM instance. For that you need to modify the builder with a few parameters. Here is what the builder turns into:

"builders": [
        {
            "type": "googlecompute",
            ...
            "metadata": {
                "ssh-keys": "packer:{{user `inspec-key`}}"
            }
        }
    ],

And here is what the command would look like to run Packer:

packer build -var inspec-key=$(cat inspec-key.pub) image.json

The metadata/ssh-keys is the important part. You add the metadata with the username, then the contents of the key. This tripped me up at first because I thought the key had already been setup with all of the relevant information, but trial and error found the right format. And just to be clear, the format is <username>:<ssh public key data>.

Alright, now that we have our new instance up and running, how do we connect Inspec? Well, again, I didn't really want to have to install another tool so let's use docker to run the actual tool. The docker command will look like this:

docker run --rm -v $(pwd):/workspace -w /workspace chef/inspec:3.2.7 detect -t ssh://packer@$( cat host ) -i inspec-key

Few things to explain in this:

-v $(pwd):/workspace -w /workspace - Take your current directory and mount it at /workspace and make the current directory when the docker container is running inside of that directory.
chef/inspec:3.2.7 detect - The docker image and the command to run. For the demo I used detect just to show you it connects. When you actually want to run tests you run exec.
-t ssh://packer@$( cat host ) -i inspec-key - Here are a few of the magic bits. Tell Inspect to connect through ssh with the data from the host file we downloaded in the previous step. Next, use the inspec-key file we created earlier.

The resulting Packer file and the test run

Alright, all together this will look like this:

{
    "builders": [
        {
            "type": "googlecompute",
            "project_id": "{{user `gcp-project`}}",
            "source_image_family": "ubuntu-1804-lts",
            "zone": "us-central1-a",
            "image_description": "Image demo for SSH keys",
            "ssh_username": "packer",
            "tags": "packer",
            "image_name": "test-image-{{isotime | clean_image_name}}",
            "metadata": {
                "enable-oslogin": "false",
                "ssh-keys": "packer:{{user `inspec-key`}}"
            }
        }
    ],
    "provisioners": [
        {
            "type": "shell",
            "inline": ["curl metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip -H \"Metadata-Flavor: Google\" > /tmp/ip"]
        },
        {
            "type": "file",
            "direction": "download",
            "source": "/tmp/ip",
            "destination": "./host"
        },
        {
            "type": "shell-local",
            "command": "docker run --rm -v $(pwd):/workspace -w /workspace chef/inspec:3.2.7 detect -t ssh://packer@$( cat host ) -i inspec-key"
        }
    ]
}

The builder is setup with our keys for us to connect. The tool then downloads the IP of the server, and we use that with Inspec to connect to the host. It should look something like this when it runs:

Link to gif of the build, something messed up on dev.to

All source code can be found on github if you want to run this yourself.

Dockerize the Kotlin toolchain

Erik Lindblom — Mon, 31 Dec 2018 00:00:00 +0000

This entry was originally published on my blog at blog.baens.net

Am I crazy? Yes, yes I am. For what ever reason, I love the idea of a project that I can literally move from computer to computer very few things installed and be able to be productive in that code base. By this, I mean I could install, one, maybe two tools and then I can be productive in a code base. Right now, Docker feels like maybe I can achieve this crazy dream.

With this crazy idea in mind, I decided if I could fully dockerize the Kotlin tool chain. In theory I can setup a workstation by installing an editor, git, docker, and bam. I'm now 100% ready to code on this machine. Alright, with this, off we go.

Step 1: Getting the basic build working

Now, to make things easy, I am going to reuse my Kotlin base and recreate this project with this tool chain. First step, let's replace the gradlew file with a docker executed gradle file. It will look something like this:

#!/usr/bin/env sh

docker run --rm -it -v $(pwd):/workspace -w /workspace gradle:5.0.0-jdk11 gradle $@

This is a fairly straight forward docker command, but just in case you have never used docker before let me touch on some of the basics:

docker run - we are going to run a docker image
--rm -it - remove (rm) once completed and add standard input pipes (it)
-v - mount our current workspace into the directory inside the container at /workspace
-w - make the current working direction /workspace (i.e. cd /workspace at start)
gradle:5.0.0-jdk11 - use gradle 5.0.0 with the JDK 11 image (see a whole list of tags on the gradle docker hub)
gradle - execute the gradle command
$@ - pass all arguments from the parent shell to this command

Alright, let's test this and see what happens!

Now, don't be too alarmed if your output is not exactly the same. I had already ran it so the docker download process isn't going, but it should be fairly close after that.

Looks like we have some jars, let's get them into a container and run them!

Step 2: Put into jar into Docker

Now, I currently had a Dockerfile that built the jar, then created the docker image, I am going to continue that idea. Here is what the Dockerfile looks like:

FROM gradle:5.0.0-jdk11 as BUILD

RUN mkdir -p /home/gradle/src
WORKDIR /home/gradle/src
COPY . /home/gradle/src
RUN gradle --no-daemon shadowJar

FROM openjdk:11.0.1-jre

COPY --from=BUILD /home/gradle/src/build/libs/step-by-step-kotlin-all.jar /bin/runner/run.jar
WORKDIR /bin/runner

CMD ["java","-jar","run.jar"]

I'm not going to go much into this file since I already explained it in the other blog post, but let me point out that line 3 was special and needed. The permissions were a little wonky inside of the container and I had to make sure the folder was already created correctly. There are many many work around for this, but I didn't want to demonstrate that.

Let's build and run this now!

Excellent! It worked as expected. But dang is it slow. Let's try really quick and fix that.

Step 3: build time improvements

Ok, obviously the build loop on this is a tad bit ridiculous. Let me show you one quick improvement for build times. If you watched closely to the video, the downloading of plugins and other artifacts is by the largest amount of time spent. This should be easily fixed if we just cached those artifacts and let gradle discover them each time. The gradle tool does this natively by storing in your home directly this folder, so why don't we just create a folder that can live between each gradle run? Well, here's what the gradlew turns into if we want to do exactly that:

#!/usr/bin/env sh

[ -d .build_cache ] || mkdir .build_cache
docker run --rm -it -v $(pwd):/workspace -v $(pwd)/build_cache:/home/gradle/.gradle -w /workspace gradle:5.0.0-jdk11 gradle --no-daemon $@

Nothing wild. Just create the build_cache folder and then mount that folder inside of the docker image where it would be expected. My quick tests show we go from 30 seconds to 8 second builds. Not bad.

Step 4: .....Profit??.... Evaluating if this is practical

Well, now that we have this sort of working and are producing builds, is this really practical? Well, honestly, its pretty close. It works, you don't need many things installed. There are some further speed improvements you could get with some tweaks. I would use (and I sort of use a setup for some projects like this now) this setup as my own setup, but I may not push it on other devs to use it unless they wanted to. But I'm starting to like this kind of setup. Again, there are many hiccups with this approach, but it gets pretty close.

So, this has at least been a fun adventure of seeing if this was possible and I think it shows promise into where things may go.

Special Notes

If you checkout the code, you may notice a .dockerignore file. This file tells docker what not to include when it is moving files over to the docker context. This is pretty important because the .gradle folder doesn't share well across the machines and things start breaking.

Setting up a development workspace for Postgres and vault

Erik Lindblom — Wed, 13 Jun 2018 00:00:00 +0000

This entry was originally published on my blog at blog.baens.net

My current line of work has led me to start getting into docker more and more. One of the neat things I've been getting to play with is using Postgres as my database and Vault for secrets management. I figured it was time to showcase what a basic development workspace looks like to help get someone up and running quickly.

I've hopefully boiled this down into the smallest pieces needed to make things work. This allows a new developer on a project to run docker-compose up which will bring up a working development environment.

I will guide you first through some setup scripts that are needed, then tie them all together in the docker-compose.yml file.

Setting up Postgres with SSL

One of requirements I've had to fill is that I need a running Postgres instance with SSL turned on. To create the needed SSL files I've created this script to create all the necessary files:

#!/bin/bash
# Creates SSL certificates need for postgres

OUTPUT_FOLDER="$(dirname "$0")/../run/ssl-files"

mkdir -p $OUTPUT_FOLDER

OUTPUT_FOLDER="$(cd "$(dirname "$OUTPUT_FOLDER")"; pwd)/$(basename "$OUTPUT_FOLDER")"

echo "SSL files output folder [$OUTPUT_FOLDER]"

rm -rf $OUTPUT_FOLDER/*

OPENSSL_COMMAND="docker run --rm -it -w /out -v $OUTPUT_FOLDER:/out svagi/openssl"

echo -e "\n\n### Create CA ###"
$OPENSSL_COMMAND req -new -x509 -nodes -newkey rsa -out ca.pem -keyout ca-key.pem -set_serial 01 -batch -subj "/CN=ssl-server"
$OPENSSL_COMMAND rsa -in ca-key.pem -out ca-key.pem

echo -e "\n\n### Create postgres server key ###"
$OPENSSL_COMMAND req -new -newkey rsa -keyout server-key.pem -out server-req.pem -passout pass:abcd -subj "/CN=postgres-ssl-server"
$OPENSSL_COMMAND rsa -in server-key.pem -out server-key.pem -passin pass:abcd
$OPENSSL_COMMAND x509 -req -in server-req.pem -CA ca.pem -CAkey ca-key.pem -set_serial 02 -out server-cert.pem

echo -e "\n\n### Create client key ###"
$OPENSSL_COMMAND req -new -newkey rsa -keyout client-key.pem -out client-req.pem -passout pass:abcd -subj "/CN=postgres-ssl-server"
$OPENSSL_COMMAND rsa -in client-key.pem -out client-key.pem -passin pass:abcd
$OPENSSL_COMMAND x509 -req -in client-req.pem -CA ca.pem -CAkey ca-key.pem -set_serial 03 -out client-cert.pem

chmod 0600 $OUTPUT_FOLDER/*

This file assumes we are in a subfolder and will be creating a run/ssl-files folder above it and drops all the commands in there. To make things easier and more portable, I'm using Docker as the tool to bring in openssl command instead of getting it installed locally.

Configuring Postgres

The next script I have is to configure postgres correctly. This will add those SSL files and make sure only SSL connections are accepted. This looks like this:

#!/bin/sh

echo "Setting up postgres hba settings"
cat <<EOM > ${PGDATA}/pg_hba.conf
local     all  all       trust
hostnossl all  all  all  reject
hostssl   all  all  all  password
EOM

echo "Adding SSL settings to postgresql.conf file"
cat <<EOM >> ${PGDATA}/postgresql.conf
ssl = on
ssl_cert_file = '/var/ssl/server-cert.pem'
ssl_key_file = '/var/ssl/server-key.pem'
ssl_ca_file = '/var/ssl/ca.pem'
EOM

This script is intended to be run inside of the docker setup routine which we will get to in a minute. The pg_hba.conf file is to ensure that only ssl connections are accepted and the postgresql.conf file is adding the SSL settings needed for that.

This script will be run when we start up postgres and will be configured in our docker-compose.yml file. We will get to that in just a minute.

Seeding vault

The next script is related to Vault. This script will start vault then configure vault to be able to create secrets for our database. The script looks like this:

#!/bin/sh
# Run a vault dev server and configure that instance for the database

# Capture the token defined in the environment and make it our dev server token so they match
export VAULT_DEV_ROOT_TOKEN_ID=$VAULT_TOKEN

# Start vault server and place in background
vault server -dev &

# Wait for server to start
sleep 2

echo "Enabling database plugin"
vault secrets enable -path=dbs database

echo "Adding postgres connection information"
vault write dbs/config/mydb \
  plugin_name=postgresql-database-plugin \
  connection_url='postgresql://{{username}}:{{password}}@database:5432/mydb' \
  allowed_roles=mydb-admin \
  username="admin" \
  password="secret" \
  verify_connection=false

echo "Adding admin role"
vault write dbs/roles/mydb-admin \
  db_name=mydb \
  default_ttl=5m \
  max_ttl=1h \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
                         VALID UNTIL '{{expiration}}'; \
                         GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"

# Wait for vault server to finish
wait

This script is intended to be run when the vault container starts. This means that this script is responsible to ensure the server starts and stays running. Once that is complete, we then write out our configuration using the vault command. After that is all done, we let the script hang with the vault server portion with the wait command. This is because the script is what actually started the server and when this script ends, so will the docker container.

One of the first bits in the script is to take VAULT_TOKEN variable that is passed and make that our dev server token. Now, a Vault token is your key in getting things out of vault. To ensure that we know the key, instead of doing the default behavior of generating a random key, we tell the server which token to use. And to make it even easier, we let the docker container pass us one token through VAULT_TOKEN and we tie that to the VAULT_DEV_ROOT_TOKEN_ID so we don't have to define that twice. And just to be extra clear, VAULT_TOKEN is what a client uses, the VAULT_DEV_ROOT_TOKEN_ID is what the server itself uses. If we didn't provide that dev token, the server would generate a random token that makes it a lot harder to development against. This is the difference between our dev Vault instance and a production one. In dev, we can define that token up front, in production, those are randomly generated for you.

The next interesting bits are the connection and role setup. One quick thing I want to point out that the paths follow a pattern. The name dbs comes from where we enabled the database plugin. There are then 3 sub paths of that, config, roles and creds. The config path is where the configuration to our database is stored. Next we add to the roles part which defines what roles can create credentials to our database with the config. Then there is a creds key that actually creates the credentials when it is read, more on using that later.

To reiterate we store configuration in dbs/config/mydb, we store role information in dbs/roles/mydb-admin. This automatically creates a path at dbs/creds/mydb-admin that we will use later to get our credentials.

The postgres credentials use and management is worth a blog post all together but let me talk about this really quick. With Vault, we are creating temporary roles that will come and go at will. When we use those credentials from Vault to also manage the schema (e.g. creating tables), we need to pay extra close attention on who actually owns those tables. When you create a table, who ever the role is that you are currently working as is the role that will be the owner of said table. Then, your next role will have issues actually accessing that table unless permissions and ownership have been setup correctly. I have a dba.stackexchange question related this that will have a longer blog post later.

Docker-compose setup

The docker-compose.yml file that binds all of these together looks like this:

version: '3'
services:
  database:
    image: postgres:9.6
    volumes:
      - ./run/ssl-files:/var/ssl/
      - ./scripts/setup-postgres:/docker-entrypoint-initdb.d/setup.sh
    ports:
      - 5432:5432
    environment:
      - POSTGRES_USER=admin
      - POSTGRES_PASSWORD=secret
      - POSTGRES_DB=mydb
  vault:
      image: vault:0.10.1
      ports:
        - 8200:8200
      environment:
        - VAULT_TOKEN=secret-token
        - VAULT_ADDR=http://127.0.0.1:8200/
      volumes:
        - ./scripts/setup-and-run-vault:/setup-and-run.sh
      command: /setup-and-run.sh

Let's step through this line by line.

Line 1 & 2 are just normal Docker compose file format stuff.

The next 3 sections are the 3 different services we will be running. Postgres the database is defined in lines 3 - 13 and the Vault service is defined in 14 - 23.

The database section is a fairly straight forward postgres docker setup. We add the SSL files to the docker container (line 6) and the script to be run on startup (line 7). We also expose the ports to other apps that I am developing can actually hit this when it is running (line 9). The last bits are environment variables to setup the initial database (line 11 - 13). Those last ones you can tweak to what ever you desire, these are just some defaults I've picked.

The vault section is mostly straight forward. In the environment variables (starting line 20) we have two definitions, one for our token, another one for the address of the running vault instance. The first is the token we use for our dev server as well as accessing that server (remember: we define the token and use it in our script for the dev server). The next variable needs to be defined because by default vault will try and use https. Looks kind of silly that you need to define the local server, but that's alright. The volume section adds our script to the container from our local disk, and then we override the default command to run that script.

Actually using this

Alright, we now have all of this crazy stuff, how exactly do I use any of this? Well, let's demonstrate that with running inside of this local project, not connecting any code. Just making sure things are running and that we can demonstrate they are running.

First, how do I start this up? Well run docker-compose up. This brings the whole system up. The whole point of what we did before with the seeding and such is that one command will give you a working system. If you go read through all of the output that gives you, you will see that the database was started and initialized and vault was started and initialize.

Next, let's go ahead and see how to connect to the database. One neat way is we can actually get connected to our running instance and execute psql through that. You can do that by docker-compose exec database psql -U admin -d mydb. If you ran that, you should now be inside of psql prompt that is connect to the database.

However, that isn't the fun part with vault. What if I wanted to try out using the whole pipeline of vault creating a credential, then getting into the database with those credentials? Well, its a bit more complicated but here is a script that goes through that process:

VAULT_SETTINGS=$(docker-compose exec vault vault read --format=json dbs/creds/mydb-admin)
VAULT_USERNAME=$(echo $VAULT_SETTINGS | docker run --rm -i colstrom/jq -r '.data.username')
docker-compose exec database psql -U $VAULT_USERNAME -d mydb

The first part is capturing a vault token by executing vault read inside of one of the vault container. This will create a JSON object that looks something like this:

{
  "request_id": "92b63ef7-9ed1-c669-cb36-16ac8110f7c2",
  "lease_id": "dbs/creds/mydb-admin/af6bcdaf-b20e-4927-6eff-c5c93aef3e2e",
  "lease_duration": 300,
  "renewable": true,
  "data": {
    "password": "A1a-0684tz5r1zpz3vys",
    "username": "v-token-mydb-adm-29r3ryxwxs3ryrvy9010-1528801793"
  },
  "warnings": null
}

As you can see, this gives you a username and password that you could use to login to your postgres instance. This is a temporary credential that will expire after a default of 5 minutes.

We take that VAULT_SETTINGS then (using Docker of course!) we extract the username from that JSON Object using the jq command. Notice how you can pipe data into a docker container. This makes docker an extremely useful tool in your toolkit because you can now use it like you would use it for normal Linux commands and create a nice data pipeline.

From there we then login with that username, and we can verify that by running \c.

So there you have it, a vault and postgres development workspace. I have the code over on github if you just want to download it and use it as you need it.

Adding tests to Kotlin & JAX-RS (Jersey) project

Erik Lindblom — Mon, 28 May 2018 00:00:00 +0000

This entry was originally published on my blog at blog.baens.net

I have a blog post about setting up a JAX-RS with Kotlin and one thing I neglected to show was how to test. So, let's go back and fix this. For this, I will be using JUnit 5 for the test runner, AssertJ as the assertion library, and Mockito as the mocking framework.

Let's go over quickly what tests I've found valuable as I've created JAX-RS services. Well, in reality, there's only two: one that exercises the HTTP interface (so, "integration test") and one that tests the class and methods in isolation (so, "unit tests"). I find it more valuable to tell which parts of the system are being exercised in the tests rather than trying to identify what type of common test level they are at. Just a weird quirk, as you will see I have plenty of them.

Setting up the dependencies in gradle

First, we need to make sure we are on gradle 4.6 or greater. I personally use the wrapper version, but just verify that you have it somehow. To get that specific wrapper version, you can run gradle wrapper --gradle-version 4.6 and then all things should be good there.

For the actual code dependencies, here is what you will need in the build.gradle file:

def junitVersion = "5.1.0"

dependencies {
    // junit dependencies
    testCompile "org.junit.jupiter:junit-jupiter-api:${junitVersion}"
    testRuntime "org.junit.vintage:junit-vintage-engine:${junitVersion}"
    testRuntime "org.junit.jupiter:junit-jupiter-engine:${junitVersion}"

    // jersey test dependencies for HTTP testing
    testCompile "org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-grizzly2:${jerseyVersion}"

    // assertJ
    testCompile 'org.assertj:assertj-core:3.10.0'

    // mockito
    testCompile 'org.mockito:mockito-core:2.18.3'
    testCompile 'org.mockito:mockito-junit-jupiter:2.18.3'
    testCompile 'com.nhaarman:mockito-kotlin:1.5.0'

}

Kind of messy isn't it? So a note on some of these. Any of the org.junit ones are the core JUnit libraries you include for JUnit 5. We've include the junit-vintage-engine because for our tests which exercise the HTTP tests are still based upon the older JUnit 4 tests from the jersey-test-framework-provider. We will get into that more later.

I've included AssertJ as the assertion library because I like the syntax better than the out of the box JUnit 5 style.

Mockito is the mocking framework, along with a few nice to have from mockito-kotlin that turn the any() option into an actual instance instead of a null object. And mockito-junit-jupiter which allows us to run Mockito with JUnit 5.

Setting up the structure

First, let's do a little bit of work to setup the inversion of control system that is used the JAX-RS version I used (Jersey), which happens to be H2. We are going to create a place we can inject bindings and determine if these are test bindings, or if they are production bindings. This gives us a layer of control of which systems to use and makes it easier for testing.

A basic binding implementation for H2 looks like this:

class Bindings : AbstractBinder() {
    override fun configure() {
        // bind(<implementation class>).to(<interface>)
    }
}

This is where we will house all of our bindings inside of that configure method. We will get into more of what that looks like a little bit later.

Now, let's actually get something useful. Let's create an interface that will hand back a list of data. So, something like this:

data class HelloJson(val prop1: Int, val prop2: String)

interface DataService {
    fun all() : List<HelloJson>
}

Nothing fancy, just a method that would give us back a list of data objects.

Notes on how to make IntelliJ work with this

So, IntelliJ doesn't like the source along side the tests files. And honestly, I've found that to be a much better project structure in every language I'ved used. So, to make this work (though very kludgy) here is the special stuff you need to add to the build.gradle file:

apply plugin: 'idea'

// This is here to fix the intellij's issues with
// the complicated source mappings. Use the 'idea' job
// to generate a intellij project
// Ticket to make this just work out of the box: https://youtrack.jetbrains.com/issue/IDEA-188436
idea {
    module {
        scopes.COMPILE.plus += [ configurations.testCompile ]
    }
}

This is intended only for IntelliJ. I will try and test Eclipse here soon and report back but I've honestly haven't touched that in years.

HTTP Tests

One of the very first tests I write when creating end points is that I can get a 200 back from the end point and get JSON back as the content-type. These tests will be my happy path test, so it should always pass. As requirements grow for this end point I will go back and add things that may be required to make this test pass by default. This is what these tests may look like:

class `HTTP HelloWorldResource should` : JerseyTest(Application()) {
    @Test
    fun `returns 200`() {
        val statusCode = target("helloWorld").request().get().status
        assertThat(statusCode).isEqualTo(200)
    }

    @Test
    fun `returns JSON type`() {
        val mediaType = target("helloWorld").request().get().mediaType
        assertThat(mediaType).isEqualTo(MediaType.APPLICATION_JSON_TYPE)
    }
}

Note: that @Test annotation is from the JUnit 4 namepsace, the import looks like this: import org.junit.Test. If you don't have this and instead use the newer JUnit 5 namespace, the test will not run. That is because the JerseyTest base class assumes that certain things run that JUnit 5 doesn't do anymore (think static class setup process and such). You can be really neat and rename the JUnit 4 import like this: import org.junit.Test as TestV4 to avoid confusion if you put these types of tests in the same file.

As you can see, this test isn't that horrible. It is fairly straight forward and the boilerplate around the test isn't bad. Let's go into a more complicated example with bindings.

Setting up a null binding as your first implementation

When I start adding layers to my application, I usually start with drilling down from my "upper" layers to the "lower" layers. Because of that I usually don't have an implementation per say to use for testing or for anything much else. I kind of like this approach because I can test out the API to see if I indeed want this abstraction and what kind of behavior I expect. With all of that in mind, usually when I create an interface I will always have a null object right along side it. This will make it easier to start off with some default behavior and makes it easier to test the system down the road.

For example, a the null object for the DataService defined above may look like this:

class NullDataService : DataService {
    override fun all(): List<HelloJson> = emptyList()
}

Method level testing

Now that we have our interface and it's null object, let's say we need to add functionality to our helloWorld method so that it returns the data from this service. With a TDD approach, the first test maybe will just check that the method returns the same number of items. The next test may ensure that we actually have an item contained in the returning list. These tests are fairly straight forward so let's look at the code for it:

@ExtendWith(MockitoExtension::class)
class `HelloWorldResource Should` {

    fun createHelloJson(prop1: Int = 1, prop2: String = "test") = HelloJson(prop1, prop2)

    @Mock
    lateinit var mockDataService: DataService

    @InjectMocks
    lateinit var helloWorldResource: HelloWorldResource


    @Test
    fun `returns same number of elements as DataService`() {
        whenever(mockDataService.all()).thenReturn(listOf(createHelloJson(), createHelloJson()))

        assertThat(helloWorldResource.helloWorld()).hasSize(2)
    }

    @Test
    fun `contains item from returned list`() {
        val expectedObject = createHelloJson(prop1 = 2)

        whenever(mockDataService.all()).thenReturn(listOf(expectedObject))

        assertThat(helloWorldResource.helloWorld()).contains(expectedObject)
    }
}

Few things to point out here. I am using the MockitoExtension that allows us to reduce on the setup parts of the tests. This allows us to create properties with the @Mock and @InjectMocks annotations. This makes reuse in the test class VERY easy. The lateinit part is because Kotlin really wants classes to show when they will be initialized. This statement flags to the compiler that we will be doing it behind the scenes later and not to worry about things.

I've also added a helper method to make creating test data easier. As you see at the top, we have a createHelloJson function that creates the data object. Then if we want to override different properties, we can easily (as demonstrated in the 2nd test) add just the property we want to override. This allows the objects to have sane defaults, and if that object grows, makes it easy to add more fields without having to change 30 tests.

Making the HTTP tests work after we add the DataService constructor dependency

Now, if you are following along you will notice that the above two tests work just fine, but if you run the HTTP test now, things break. That's because we now require the DataService to be injected at runtime from H2. To fix that we need to make sure we have our bindings correct, so here is what the test binding and injection will look like

// in our Application.kt file
class TestBindings : AbstractBinder() {
    override fun configure() {
        bind(NullDataService()).to(DataService::class.java)
    }
}

// in our HelloWorldResource.kt file
class HelloWorldResource
@Inject constructor(private val dataService: DataService)

// in our HelloWorldResource.tests.kt file
class `HTTP HelloWorldResource should` : JerseyTest(Application(TestBindings()))

First we needed to add the @Inject property to the constructor of our resource. This will signal to Jersey that this constructor will need outside dependencies and to use H2 to find them. Our dependency is defined in our TestBindings class, with the NullDataService as our default. Then we add that test binding to our JerseyTest creation making this all work.

Wrapping up

Now, if any of this didn't make sense and you want to see it all put together, go over to the github repository where this is housed and you should be able to take a look at the full thing working. I've created a static list implementation of that data service for one to look at what an actual implementation may look like.

Bonus: How to log exceptions in Jersey

If you check out the code on github, you will notice another class way down in the Application.kt file. This class hooks into Jersey's event system so if a request errors out in an exception, we can observe it. Take note this is just for observing if an exception has happened, not actually doing anything with the request. If we wanted to handle that, we could create a request mapper, but that is outside of the scope of this particular blog post.

Notes from experimenting with GraphQL and Kotlin

Erik Lindblom — Thu, 22 Mar 2018 00:00:00 +0000

This entry was originally published on my blog at blog.baens.net

What is this GraphQL thing

I have finally got time to try out GraphQL. I've been reading about it more and more and wanted to get a sense of how to build a data API with it. Now, if you don't know what GraphQL is, here is my take on it. It is a way to create data APIs that allow clients to tell you what data they actually need. Instead of you trying to design up front all of your use cases and force clients to use that. You let them tell you what data they need, when they need it. This makes it so you as the data designer just expose the data that you have and expose ways to filter the data and the relationships that are inside the data. You don't care how the client uses it, you just expose the data as you have it. I believe this is how a data API should be designed. This decouples the why from the how which usually empowers users of the API and makes it more useful longer term.

Getting started with GraphQL on Kotlin

My first approach was to create a Jersey web application that exposed the correct end points for GraphQL. Apparently, this is the wrong approach. The library I've found (graphql-java) doesn't work that way. It works by adding a servlet directly to the server. Sorry if that is totally greek. Let's just say I was taking the approach of trying to build out a REST end point that just happens to have some GraphQL functionality. This was the wrong approach and what I needed to do was actually just add an HTTP route to a bit of logic and the libraries handle everything for me.

The "How to GraphQL with Java" tutorials were extremely helpful. This showed me how to wire up the library correctly and it gave me enough snipplets to figure out what needed to go where. So let's get started.

Server Configuration

So for my approach I wanted to try and use the embeddded Grizzly Java server because that is what I've been using lately as a server inside of my JVM projects. This made things a little bit more complicated because it was intended to be used as a JAX-RS server. Which meant that all the information currently out there has information on how to set that up. But for this particular case I actually needed a servlet. This was definitely going off the well beaten path. A few posts later and some tinkering it does work.

The first part I figured out was getting the GraphQL schema parsed. This file, which is embedded inside of the application, needs to be parsed out to figure out what data you have. Along with that, you then provide a root resolver that will contain the logic of how a request is intercepted and executed. This looks like this:

// Setting up graphql schema, servlet, and bindings to server
val graphqlSchema = SchemaParser.newParser()
        .file("schema.graphqls")
        .resolvers(Query()) // we can add any number of resolvers here
        .build()
        .makeExecutableSchema()

OK, that is fairly straight forward. Now we are going to take that schema and create a servlet that we can add to our server. This looks like this:

val graphqlServlet = SimpleGraphQLServlet
            .builder(graphqlSchema)
            .build()

That isn't rocket science either. However, the next part is what took me the longest. For Grizzly, there is a not well documented class that helps you setup servlets. The WebAppContext was the magical find and it really isn't well documented out there unless you look through Java source code (...yuck!...). This class sets up the servlet with all of the things it needs to run (context and bindings if you know servlets). I was hoping for something that just required a few annotations here and there but Grizzly doesn't support that (or at least I couldn't get it working). If you follow along the GraphQL how to, they setup a servlet using that annotation magic. But here, we have to do a little bit more work ourselves. This is how it ended up looking:

val webappContext = WebappContext("Graphql Context", "/")

webappContext.addServlet("GraphQL Endpoint", graphqlServlet)
             .addMapping("/graphql")

val server = HttpServer.createSimpleServer()

webappContext.deploy(server)

I've included the server portion as well because you have to "deploy" the webapp context to a server. For these few lines it took a surprising amount of work to get it there. But in the end I like it. It is a straight forward setup that anyone could follow along if they need too.

Now the next thing I wanted is to provide an interface that will allow users to play with the API. This is like the swagger interface people are familiar with for REST end points. For GraphQL the interface usually used is GraphiQL. To setup GraphiQL is relatively easy. You copy the index.html file from their github repository, place it in your code base. Tweak the file to point to a CDN, and bam. Done. But, to get the Grizzly server to serve up that file took a little bit of coxing. Again, you need to add a servlet that will serve up that static HTML file correctly. This actually took a StackOverflow question to get me on the right path. From that it looks like I needed the CLStaticHttpHandler class which looks like this:

server.serverConfiguration
          .addHttpHandler(CLStaticHttpHandler(Thread.currentThread().contextClassLoader), "/")

This really isn't ground breaking but there it is. The hard part is getting the pieces together in the right order. But that's why there are blog posts. To help spread this kind of information.

And that's it! We now have our server configured and ready to rock and roll.

Embedding the schema and html files in the JAR

So, we have our Java code working fine, now we need to embeded those HTML files and GraphQL schema files inside our jar for the services to use. This is really straight forward but I wanted to point out that I personally don't like the default folder structure of Java projects. So I of course have to make life hard and tweak everything. Here is what I tweaked in gradle to make HTML files be in the folder src/html and graphql schema files be at src/graphql

sourceSets {
    main.resources.srcDirs += ['src/html', 'src/graphql']
}

This sets the resources directories manually (in gradle, a resource is anything that isn't code) and places them correctly in the jar file.

The whole shabang

So here is the final repository if you want to see everything working: https://github.com/baens/blog-experimental-graphql-servlet

You can download and fire up the server through ./gradlew run or even use the Docker image and use that. In the end all this does is do the same thing the how to GraphQL site does, but the pieces above are where the work was at. The resolver and repository are nothing magical at this point and follow the examples laid out everywhere else.

Note: magic is currently needed to make a few things work

So, hopefully this section won't be needed much longer but I do want to point out that there is a hack to make things work. I stumbled upon a bug that would crash the GraphQL processing with this particular server. The details are a little messy, but until this issue has been resolved we need to get a custom build of the servlet library. Now this is where things are really neat. There is a service out there (how the hell do people do this for free?!) that will take any github repository and build a library for you. Jitpack.io is just absolutely amazing and I want to shout out that this service is just incredible.

So the magic little bits I had to sprinkle inside of gradle file were

repositories {
    maven { url 'https://jitpack.io' }
}

dependencies {
    compile 'com.github.briancullen:graphql-java-servlet:master-SNAPSHOT'
}

Fairly straight forward. Point to the jitpack repository, and point to the repository that has the fix in it.

Step-by-step: Kotlin, JAX-RS (Jersey), and Docker

Erik Lindblom — Sat, 17 Feb 2018 00:00:00 +0000

This was originally published on my personal blog

Hello Again and welcome to another blog post in my ongoing series on step-by-step tutorials. This one will actually build atop the last Hello Kotlin one and make a JAX-RS web service. This particular implementation of JAX-RS will be Jersey. Jersey is actually the reference implementation from Oracle, but there are a few out there that I've honestly haven't given enough time.

Step 0: Prerequisites

For this one, I am going to start off with my Kotlin Hello World repository. This already has a Gradle wrapper setup, along with some basic things. I've changed mainClassName to Server.kt and archivesBaseName to match this projects name.

Step 1: Web Server

First step will be to get a working web server up and running. This project will be using the grizzly web server, which I recommend, as the embedded web server. It has a decent enough history of performance and from my usage it does a decent job.

So let's take a look at how a quick server can be setup so we can verify we have the HTTP pipeline correct.

src/Server.kt

import org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpServerFactory
import javax.ws.rs.core.UriBuilder

fun main(args: Array<String>) {
    val url = UriBuilder.fromUri("http://0.0.0.0/")
            .port(8080)
            .build()

    val httpServer = GrizzlyHttpServerFactory.createHttpServer(
            url,
            true
    )

    if (System.getenv()["SHUTDOWN_TYPE"].equals("INPUT")) {
        println("Press any key to shutdown")
        readLine()
        println("Shutting down from input")
        httpServer.shutdownNow()
    } else {
        Runtime.getRuntime().addShutdownHook(Thread {
            println("Shutting down from shutdown hook")
            httpServer.shutdownNow()
        })

        println("Press Ctrl+C to shutdown")
        Thread.currentThread().join()
    }
}

Line 6: This creates a URI for the server to bind to. This particular one makes it so http://localhost:8080 will work. Probably overkill but that is what the Grizzly Server factory wants.

Line 9: This creates the Grizzly Server and the 2nd parameter (true) makes the server start immediately.

Lines 14 - 27: Now, let me explain this because this is sort of magic. When running under Gradle (./gradlew run), Gradle captures all of the input and passes it along to the application. So for example, on the command line you usually stop applications by hitting Ctrl+C. This however doesn't work when you run it under Gradle because Gradle captures the signal and stops the daemon that is running the application. Sometimes, the Gradle daemon doesn't stop properly, or clean everything up and produces a mess the next go around.

So my hack to fix all of this was to have a feature flag in an environment variable. When running under Gradle (hold on one sec and I'll show you how) we will just wait for input. When running under everything else (i.e. docker, java -jar) we will wait for Ctrl+C or a kill signal from somewhere else.

Now for the toolchain bits, we need to add the dependencies and setup the run bits to work with out shutdown hooks.

build.gradle (only parts)

...
dependencies {
    compile 'org.jetbrains.kotlin:kotlin-stdlib-jre8'
    compile "org.glassfish.jersey.containers:jersey-container-grizzly2-http:2.26"
}
...
// Properties for `./gradlew run`
run {
    standardInput = System.in
    environment = ["SHUTDOWN_TYPE" : "INPUT"]
}

Line 4: We add the grizzly http server and Jersey container.

Line 8 - 11: This is how we make ./gradlew run work. The standardInput we set to the normal System.in (it defaults to an empty set which breaks things). Then we setup the environment variables with the environment configuration.

We need to add a way for the port to be exposed by Docker to the outside world. Since we are binding on port 8080 we need to have that exposed as well. So in the Dockerfile we will add one little line before the CMD statement.

Dockerfile (only parts)

EXPOSE 8080

CMD ["java","-jar","run.jar"]

So verify the two ways to run the application and view http://localhost:8080.

First run ./gradlew run and if you hit any key, it will shutdown.

Second run the docker command docker build -t myserver . && docker run -p 8080:8080 myserver. This you can run and then hit Ctrl+C to shut it down.

Quick aside on the docker command: there are two parts to it, first you build the image you want to run, then you run it. The -t you pass in the build command is a tag that makes it easier to look up later. In the run part the -p is the ports we will bind to the host. For this example we are taking the containers's 8080 port mapping it to the hosts 8080 port.

Step 2: Add Simple JAX-RS endpoint that prints "Hello World"

For this let's create the needed pipeline to get the needed resources. There is a little bit of yak shaving needed for this setup to get all of the right pieces in the place but I think I've boiled it down to the simple parts.

First let's see what the resource (these are what the groups of end points are called) look like for our first end point.

src/jaxrs/resources/HelloWorld.kt

package jaxrs.resources

import javax.ws.rs.GET
import javax.ws.rs.Path

@Path("helloWorld")
class HelloWorldResource{
    @GET
    fun helloWorld() = "Hello World"
}

There really isn't much to talk about here. The @Path is where the end point will be (so in this case /helloWorld). This will be retrieved from a HTTP GET.

Alright, the next step is to get the system setup to register these resources automatically. So let's setup the config needed for that.

package jaxrs

import org.glassfish.jersey.server.ResourceConfig

class Application : ResourceConfig() {
    init {
        packages("jaxrs.resources")
    }
}

Again, this is fairly straight forward. The ResourceConfig is where all of base application is setup. In this example. we scan the jaxrs.resources namespace for any resources or other things that can register with JAX-RS. This is a specific to Jersey but it makes it easier to setup things.

Alright, now that we have all of the configuration setup for JAX-RS and a resource, lets make the server recognize it. So to the Server.kt file added this.

Server.kt (parts)

val httpServer = GrizzlyHttpServerFactory.createHttpServer(
            url,
            Application(),
            true
    )

Line 12: This is all you need to add. A new instance of the Application class and the Grizzly server does all the rest for you.

Now, one last thing, we need to add a dependency for Jersey. This one will make all of the magical dependency injection happen behind the scenes. The Jersey team have made this configurable dependency, so if you want a different type of injection system you can. The specific one implementation we will use is HK2 as opposed to Guice or others. This was just the default that I've used and have had no problems with it so I recommend it. So here is the line that needs to be added.

build.gradle (parts)

dependencies {
    compile 'org.jetbrains.kotlin:kotlin-stdlib-jre8'
    compile "org.glassfish.jersey.containers:jersey-container-grizzly2-http:2.26"
    compile "org.glassfish.jersey.inject:jersey-hk2:2.26"
}

Go ahead and run to see the output on http://localhost:8080/helloWorld.

Step 3: Add JSON handling

Now that we have a simple web server up, let's get JSON handling. This isn't configured out of the box but it is straight forward. So let's get it setup and working and have an end point setup to demonstrate that.

Let's setup our resource to have the parts needed to show JSON output.

src/jaxrs/resources/HelloWorld.kt

@GET
fun helloWorld() = "Hello World"

data class HelloJson(val prop1: Int, val prop2: String)

@GET
@Path("json")
@Produces(MediaType.APPLICATION_JSON)
fun helloJson() = HelloJson(1, "test")

Line 13: This is why I love Kotlin, this is a data class. A beautifully pure and simple structure to pass data around. We have two properties that will be exposed to the JSON interface. One an integer, the next a string.

Lines 15-18: This is how you setup an endpoint that will produce a JSON object (well, force it too, but that's for another day, or go read the Jersey documentation). We simply just return an instance of the HelloJson data class.

Now, we need to tweak our dependencies and add the JSON handling. While we do this, we will also be violating the Rule of 3, and we are going to refactor out the common things that may change. Let's see the result and we will talk about what exactly that means.

build.gradle (parts)

def jerseyVersion = "2.26"

dependencies {
    compile 'org.jetbrains.kotlin:kotlin-stdlib-jre8'
    compile "org.glassfish.jersey.containers:jersey-container-grizzly2-http:${jerseyVersion}"
    compile "org.glassfish.jersey.inject:jersey-hk2:${jerseyVersion}"
    compile "org.glassfish.jersey.media:jersey-media-json-jackson:${jerseyVersion}"
}

Line 22: We added this line because the parts that could change between all of these jersey dependencies is the Jersey version. So let's do the work now to make it so we can make the version change in one place. A small application of the Rule of 3, but hey, it works.

Line 28: This is adding Jackson to do our JSON processing. This is the magic dependency that we will need to make this all work.

At this point you can now run it and hit http://localhost:8080/helloWorld/json and you should see JSON output.

And with that, you should now have the basics to be dangerous with. You can setup a HTTP REST server and go to town with what ever you need. But hey, let me throw in one more little bonus step that may help you out.

Bonus Step 4: Add verbose logging

Just to make it easier to see requests coming in and out of your service on the command line. Which can help with proving you are either insane or sane depending on the circumstances, let's add a logger that will output each request.

Thankfully this fairly straight forward. We just need to tweak the Application.kt setup. So here is what you need.

src/jaxrs/Application.kt

register(LoggingFeature(Logger.getLogger(LoggingFeature.DEFAULT_LOGGER_NAME),
        Level.INFO,
        LoggingFeature.Verbosity.PAYLOAD_ANY,
        Integer.MAX_VALUE)
)

These are the lines you need to add to the init block. Fairly straight forward (make sure to add all of the right import statements!). This should now display all of the requests and responses on the command line so you can verify things are going in and out as expected.

Step-by-step: Kotlin and Docker

Erik Lindblom — Sun, 11 Feb 2018 00:00:00 +0000

This was originally published on my personal blog

This is a step-by-step guide on how to get Kotlin running inside of a Docker container. We will get a basic Kotlin project running locally then demonstrate how to get that same project being built inside and running inside of a Docker container.

The companion repository is here.

Step 0: Prerequisites

I am going to assume you have the following installed and already running. We will be using the local gradle wrapper instance after the initial setup, but you do need it for the initial creation.

The other one I am going to assume is having a running docker instance.

Java (JDK 1.8.0_144)
Gradle (to create the initial wrapper)
Docker (17.12.0-ce-mac49 (21995))

Step 1: Gradle

The goal of this step will be to get a gradle wrapper up and running so that we can lock in the gradle version for everyone that will be using this repository.

Run the following:

> gradle wrapper --gradle-version 4.5

Verify it works by running:

> ./gradlew -v

------------------------------------------------------------
Gradle 4.5
------------------------------------------------------------

Build time:   2018-01-24 17:04:52 UTC
Revision:     77d0ec90636f43669dc794ca17ef80dd65457bec

Groovy:       2.4.12
Ant:          Apache Ant(TM) version 1.9.9 compiled on February 2 2017
JVM:          1.8.0_144 (Oracle Corporation 25.144-b01)

That's it! You can double check that commit that I have associated with this step to see what I add to a git repository for this but this really is an easy step.

Step 2: Kotlin Hello World locally

Our end goal for this step will be to make it print "Hello World" from the ./gradlew run command.

Let's first setup our tool chain. This will be just a gradle file with all of the plugins and dependencies that we will need.

build.gradle:

/**
 * This section is for all of the plugins we need to make this work. 
 * Kotlin, the fat jar builder, and flag it 
 * as an application so `./gradlw run` will work
 */
plugins {
    id 'org.jetbrains.kotlin.jvm' version '1.2.21'
    id 'com.github.johnrengelman.shadow' version '2.0.2'
}

apply plugin: 'application'


/**
 * Standard dependency section for gradle. 
 * Define the kotlin standard libray for
 * Java 8.
 */
repositories {
    mavenCentral()
}

dependencies {
    compile 'org.jetbrains.kotlin:kotlin-stdlib-jre8'
}

/**
 * Personal preference. I hate having src/main/kotlin
 * be the root, so I change it that 'src'
 * is the root of my source directory.
 */ 
sourceSets {
    main.kotlin.srcDirs += 'src'
}

// Define the main startup class and jar name
mainClassName = 'MainKt'
archivesBaseName = 'step-by-step-kotlin'

// tell the jar which class to startup in.
jar {
    manifest {
        attributes 'Main-Class': 'MainKt'
    }
}

Let's go through some of those lines again and draw out whats going on.

Lines 6 - 11: These are just plugin lines. Gradle has a couple of ways to define plugins. One is called the plugin DSL the other is called script plugin. I'm not going to try and explain the difference here, please go read the plugins doc to get a better idea. One plugin to note is the shadow plugin. This one is to create Fat Jars, or in other words, jars that contain every dependency that is needed to run. Instead of having to make sure things are on your classpath or some other nonsense, I can just give you this jar file, and it will run. This will make it running inside of docker easier.

Line 24: Here we are defining the kotlin standard for Java 8. This is newer in Kotlin 1.1 and you can read about it here. There are a few ways to do this, but this was specific to what was needed for Java 8. I've run into issues using the backwards compatible one and try and use the Java 8 one so that the newer features actually work

Lines 32 - 34: This is entirely optional, but this is how I prefer to setup my projects. I hate the default of src/main/java or something silly like that. That is 3 folders I have to navigate just to get to my source. PLUS then I have to navigate down through the folders of my namespace. Rarely do projects have to have all of these different types of languages so you can eliminate the java folder at the very least. And 2nd, having to maintain a test source folder and a main source folder is the PITA. I much prefer to have my tests and source side by side. It just makes it easier to find my tests that are associated with each file. Again, this is all personal choice but I've been doing this long enough to feel strongly about this one now. Give me one folder with all my source and don't make me nest things that I don't want too!

Line 37: This is a simple line but I did want to call out something. MainKt is special because it will be the name of the file with kt at the end of it. I will explain a little bit more when we get to that source. But this is a convention Kotlin has when there is just a method inside of the file. It will automatically enclose that method in a class for the JVM to use. So be careful about that kt.

Now for the actual, very simple, hello world code.

src/Main.kt:

fun main(args: Array<String>) {
    println("Hello World")
}

I want to point out and congratulate the Kotlin team on this. If you have ever done any Java programming, you know that you always need a class per file. Kotlin has a few conventions to make certain things easier and I think this is brilliant. I love things like this that just remove the pomp and circumstance. This is especially helpful when trying to get new developers into the language. Every time I've had to teach someone Java, the public class Whatever was foreign and we had to go down a rabbit hole even before we had anything running.

You should now be able to run ./gradlew run and see this:

>./gradlew run

> Task :run
Hello World


BUILD SUCCESSFUL in 5s
2 actionable tasks: 2 executed

Step 3: Kotlin hello world inside docker

Update as of 2018-02-15: Thammachart Chinvarapon was gracious enough to point out that I was using the wrong Java docker container. I corrected this and moved it to the openjdk container. Thanks Thammachart!

The next step will be to run this inside of a docker container. While this would be fairly straight forward, I want to demonstrate how to separate your build docker image from your running docker image. While this adds a little bit of complexity, this will actually be more like how you really use these kind of things.

Since we have everything working, we just need to get a Dockerfile created and running.

Dockerfile

ARG VERSION=8u151

FROM openjdk:${VERSION}-jdk as BUILD

COPY . /src
WORKDIR /src
RUN ./gradlew --no-daemon shadowJar

FROM openjdk:${VERSION}-jre

COPY --from=BUILD /src/build/libs/step-by-step-kotlin-all.jar /bin/runner/run.jar
WORKDIR /bin/runner

CMD ["java","-jar","run.jar"]

Line 3 - 9: This is the "builder" section.

Line 9 - 14: This it the section that will be actually running.

This is an awesome way to wrap up in one docker file how to build and create the image inside of itself, as well as run it. What I try and do is make it so you can create this docker image from nearly anywhere. This builder image includes all of the tools needed to build things out. And since we have this gradle wrapper, the toolchain is also there.

The 2nd part is what is actually running. The COPY --from=BUILD is the magic that we can pull from the last image (note: BUILD is from line 3 as BUILD).

I'm not going to go too much of what exactly docker is doing but do want to point out a couple of quick things. We basically do a copy, setup what is the working directory (think cd if you were using command line), then do the work or run.

Run this and the image will build and run:

docker build -t kotlin . && docker run kotlin