I recently found an interest in Kubernetes and learned about it at night, while working at something not web-related at day. As part of my learning journey I wanted to quickly see and experience how Kubernetes actually works in action. So I decided to write a few services that can be used to trigger and observe certain behavior of Kubernetes. I started with Load Balancing, Self Healing of Services and Auto Scaling depending on CPU utilization.
In this blog post I will explain how each service works and how Kubernetes behaves in practice. Note that this was the first time I wrote Go, so I take no guarantee that the code is not shitty or against Go conventions and best practices that I do not yet know. :-)
This blog post addresses anyone who already has a Kubernetes cluster up and running. You should know what a Pod is, a Replica Set or a Deployment, and can build Docker containers and have used a Docker Registry.
If you do not run a Kubernetes cluster already, I can recommend the book "Kubernetes: Up & Running" and/or the blog post "How to Build a Kubernetes Cluster with ARM Raspberry Pi" by Scott Hanselman. If you don't know what Kubernetes is or how it works, you can read the excellent series of blog posts "Understanding Basic Kubernetes Concepts" by Puja Abbassi from giantswarm.io.
For the impatient
Before running kube-alive on your cluster, make sure that you have the following:
- kubectl installed and configured to a running cluster (check that "kubectl get nodes" gives you a list of at least one node in state "Ready")
- bash
- Your cluster runs on Linux on amd64 or ARM CPUs
If you do not have a cluster up and running already, I recommend the already mentioned article by Scott Hanselman to start a cluster on Raspberry Pis, or you can use Minikube to run a local cluster on your PC or Mac.
If you just want to deploy kube-alive to your cluster and see it in action, you can do this with this single command:
curl -sSL https://raw.githubusercontent.com/daniel-kun/kube-alive/master/deploy.sh | bash
Using 192.168.178.79 as the exposed IP to access kube-alive.
deployment "getip-deployment" created
service "getip" created
deployment "healthcheck-deployment" created
service "healthcheck" created
deployment "cpuhog-deployment" created
service "cpuhog" created
horizontalpodautoscaler "cpuhog-hpa" created
deployment "frontend-deployment" created
service "frontend" created
FINISHED!
You should now be able to access kube-alive at http://192.168.178.79/.
Load-Balancing
The most basic capability of Kubernetes is load balancing between multiple services of the same kind. To observe whether the request was served from the same or from different instances, I decided to let the service return it's host's IP address. In order to run a service in Kubernetes, you need to a) write the service, b) build a container hosting the service, c) push the container to a registry, d) create an object in Kubernetes that runs your container and finally e) make the service accessible from outside the cluster.
Phase A: Writing the Service
So let's dig into the code. I wrote a server in Go that serves on port 8080, parses the output of the command "ip a" and returns the container's IP address.
package main
import "fmt"
import "bufio"
import "os/exec"
import "log"
import "strings"
import "net/http"
/**
getip starts an HTTP server on 8080 that returns nothing but this container's IP address (the last one outputted by "ip a").
**/
func getIP() string {
// Left out for brevity, see
https://raw.githubusercontent.com/daniel-kun/kube-alive/master/src/getip/main.go
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, getIP())
})
fmt.Printf("'getip' server starting, listening to 8080 on all interfaces.\n")
http.ListenAndServe(":8080", nil)
}
Phase B: Building the Container
Since everything running in Kubernetes must be a container, I wrote a Dockerfile to run this service:
FROM golang
COPY main.go /go/src/getip/main.go
RUN go install getip
ENTRYPOINT /go/bin/getip
This Dockerfile is simple: It uses a golang base container that is prepared to compile and run Go code. It then copies over the only source code file, main.go and compiles and installs it to /go/bin/ using "go install".
The installed binary /go/bin/getip is set as the Entrypoint, so that when no argument is given to docker run, it executes our service.
You can build the container using:
docker build .
Note that there is a "." at the end of the command, meaning that you must have cd'ed to the getip source directory before executing docker build.
After docker build finishes, you will be able to see the new container with a new, randomly generated image id via
docker images
The container is only available locally on the machine that it has been built. Since Kubernetes will run this container on any node that it sees fit, the container must be made available on all nodes. That's where a Docker Registry steps into the game, which is basically a remote repository for Docker containers that is available from all nodes.
Phase C: Pushing the Container to a Registry
I first tried to set up a local registry, which can be done, but the setup is not
portable across clusters. That's why I decided to simply use Docker's own registry, https://hub.docker.com. To push your freshly built container, you first need to register at Docker Hub, then tag the container with the repository, desired container name and an optional tag. If no tag is given, "latest" is assumed.
docker tag <your-repository>/getip <image id> # tag the docker image with your repository name and the service name, such as "getip"
docker login # enter your username and password of http://hub.docker.com now.
docker push <your-repository>/getip # and then push your container
It is now available to be pulled (without authorization) by anyone - including your Kubernetes nodes.
Phase D: Define a Replica Set
To let Kubernetes know that it should run this container as a service, and to run multiple instances of this service, you should use a Replica Set. I wrapped the Replica Set into a Deployment to easily upgrade the service later:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: getip-deployment
labels:
app: getip
spec:
replicas: 4
selector:
matchLabels:
app: getip
template:
metadata:
labels:
app: getip
spec:
containers:
- name: getip
image: <your-repository>/getip
ports:
- containerPort: 8080
I set the number of replicas to 4, which means that Kubernetes will do everything it can to always have exactly 4 instances running at any time. However, this does not give us a single URL to connect to these instances. We will use a Service to Load Balance between these instances:
kind: Service
apiVersion: v1
metadata:
name: getip
spec:
selector:
app: getip
ports:
- protocol: TCP
port: 80
targetPort: 8080
This service provides a single, load-balanced URL to access the individual service instances. It remaps default HTTP port 80 to the service's own port 8080 in the process. The service will be available as http://getip.svc.default.cluster or, even shorter, as http://getip on any running Kubernetes Pod.
However, this service is only available from inside Kubernetes and not from "outside" the cluster.
Phase E: Publish the Service
I decided to build my own nginx container to serve the static HTML and JavaScript files that make up the frontend and publish the services from a specific IP.
events {
# empty
}
http {
server {
root /www/data;
location / {
# for the frontend SPA
}
# Forward traffic to <yourip>/getip to the getip service.
location /getip {
proxy_pass http://getip/;
}
# I have left out the other services like "cpuhog" and "healthcheck" here for brevity.
# See their code on https://github.com/daniel-kun/kube-alive/
# Allow WebSocket connections to the Kubernetes API:
location /api {
proxy_pass https://kubernetes.default/api;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Authorization "Bearer %%SERVICE_ACCOUNT_TOKEN%%";
}
}
}
So we see that nginx expects the SPA in /www/data/, which will be the target of COPY commands in our Dockerfile. The service getip is reached via Kubernetes DNS, which will automatically resolve a service's name to it's Cluster IP, which in turn load-balances requests to the service instances. The third location /api is used by the frontend to receive information about running pods. (Currently, the full API is exposed with full admin privileges, so this is highly insecure - do it in isolated environments only! I will fix this in the near future.)
Here's the Dockerfile for the frontend service:
FROM nginx
COPY nginx.conf /etc/nginx/
COPY index.html /www/data/
COPY output/main.js /www/data/output/main.js
COPY run_nginx_with_service_account.sh /kube-alive/
CMD /kube-alive/run_nginx_with_service_account.sh
The shell script run_nginx_with_service_account.sh will substitute variables in the nginx.conf to use the Kubernetes Service Account token in the authorization header to let nginx handle the authorization so that the frontend does not have to.
So now we are prepared to put the last piece of the puzzle into place: A Replica Set to run the frontend and a Service that externally publishes the frontend. Note that I wrapped the Replica Set into a Deployment again:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: frontend-deployment
labels:
app: frontend
spec:
replicas: 1
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: frontend
imagePullPolicy: Always
image: <your-repository>/frontend_amd64
ports:
- containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
name: frontend
spec:
selector:
app: frontend
externalIPs:
- <put your external IP, e.g. of your cluster's master, here>
ports:
- protocol: TCP
port: 80
targetPort: 80
That's it! You can kubectl apply this after you inserted a valid externalIP and everything should be up and running to execute your first experiment with Kubernete's load balancing.
Reaching to the IP that your "kubectl" is configured against, should give you this UI:
… followed by more experiments. These all follow the same concept as getip, you can have a look at their code and deployment yamls here:
Self-Healing
The code of the Self-Healing experiment is here:
https://github.com/daniel-kun/kube-alive/tree/master/src/healthcheck
https://github.com/daniel-kun/kube-alive/blob/master/deploy/healthcheck.yml
Rolling Updates
The code of the Rolling Updates experiment is here:
https://github.com/daniel-kun/kube-alive/tree/master/src/incver
https://github.com/daniel-kun/kube-alive/blob/master/deploy/incver.yml
Auto Scaling (cpu-based)
The code of the Auto Scaling experiment is here:
https://github.com/daniel-kun/kube-alive/tree/master/src/cpuhog
https://github.com/daniel-kun/kube-alive/blob/master/deploy/cpuhog.yml
I named the service "cpuhog" because it uses as much CPU as it can for 2 seconds for every request.
I plan to add more experiments in the future, such as an experiment for rolling updates using Deployments.
I hope that you found this blog post and the kube-alive services useful and would be thankful if you could leave feedback in the comments. Maybe one day kube-alive will be a starting point to see Kubernetes behaviour live in action for many starters and engineers that are evaluating Kubernetes for their own use.
Update from 01/25/2018: I removed the security warning, because security on the latest version of kube-alive has been tightened. Only a portion of the API is exposed (pods in the kube-alive namespace), and the frontend runs with a service account that has access only to the dedicated kube-alive namespace and only to read and list Pods. Hence, there is not much more information available via the API than is visible in the frontend anyways.
Update from 03/08/2018: Updated the gifs to the new, improved visuals and added the rolling updates experiment.
Top comments (9)
Daniel, my team and I are Deploying microservices and since you have a developer background maybe you can help. Our problem is we do not have a service discovery built out and all our calls to other microservices have to exit our cloud network get NATed and then come back in to reach other microservices. Is there a poor man's way to implement service discovery without creating a service registry. Making hundreds of outbound calls sometimes exhausts all outbound ports or causes unwanted latency. I have seen NGINX used along with something called console. We also looked at creating a DNS server for internal communication and separate outbound and inbound traffic but that would require maintaining internal DNS, adding internal and external endpoints to everything. Any other options ?
Mobeus
I recommend taking a look at this project istio.io service mesh.
From what I read on istio.io, it seems to run on top of Kubernetes? For service discovery, wouldn't raw Kubernetes be sufficient? However, this would mean to re-architect the infrastructure anyways, especially when it does not yet run on containers. Maybe there is something more lightweight if you don't want to go all the way? (Although, when there are more problems to solve, such as load balancing, failure recovery or scaling, it might be totally worth it.)
I'm not working in the web space, so I can not give a recommendation.
Kubernetes has been a free-time endeavor so far.
Sorry this is pre k8s. I am aware that K8s will solve this issue. We will be jumping on ACS soon to be AKS but need a tactical solution until that materializes. Thx...
Mobeus
Ingress in the right way, such as Traefik of nginx ingress controller.
Why build your own Nginx service, as opposed to writing an Ingress and deploying an Ingress controller?
I honestly think that the Ingress is way more complicated for beginners to understand then a service type LoadBalancer routing to a reverse proxy.
At least I had a hard time in the beginning understanding the Ingress really works.
From what I understand, Ingresses are just a way of saying "Here's a path, it corresponds to this service." However, setting up an Ingress controller is, in some cases, needlessly complicated imo. I like that minikube and GKE ship with one by default, but that didn't prepare me for having to set one up on my own, especially given if I want it to use HTTPS.
Like Michael already answered, the Service with an externalIP was a very simple solution, while an Ingress is much more complex. Actually, I wouldn't even know how to use an Ingress e.g. on a minikube or a local cluster that does not have a dedicated load balancer. It might be worth a look, but currently my priority is: 1) finish CI for multiarch images, so that you can deploy kube-alive on ARM and amd64 without changing anything in the manifests (almost done) 2) fix the security problems I mentioned in the article and 3) add more experiments, like observing live how a rolling update happens. Then, at prio 4) I might add support for GKE, AKS and EKS.