Kubernetes internals

In this blog post we are going to setup a kubernetes cluster with automated certificate generation using certbot. Will cover up some interesting concepts of kubernetes along the way, like:

• Job controllers
• Load balancer Service
• Role Based Access Control
• In-Cluster kubernetes Access

So let's get started.

---

Why do we need this?

When using a traditional VM's/instances, one has access to ssh into a fixed instance with an assigned and fixed public IP that the DNS can resolve to. Once the DNS is set to resolve your hostname to your instance, you can install certbot on it and generate the certs on that instance.

With kubernetes, things get a bit tricky. You will still have a set of instances in your cluster, but they aren't directly accessible from outside the cluster. Plus you cannot preempt on which node your nginx or other ingress pod will be scheduled to run. Hence the most straight forward way to setup is doing everything through kubernetes and docker. This will also provide us with a few advantages:

• The cert generation process will be documented as part of the kubernetes manifest files and Dockerfiles
• Our solution would work for any number of pods and services. Basically we won't be bogged down due to issues arising because of scalability like when adding more pods or even adding more nodes to our cluster. Things will work seamlessly until we don't create a new cluster.
• Our solution will be platform independent, just like docker and kubernetes. It will work on any cloud provider GCP, AWS or even Azure(hopefully).

---

I'll be using GCP for some parts of this blog post but replicating those parts on other cloud platforms is pretty straight forward.

We'll start by reserving a static IP for our cluster and then forward a DNS record to that IP so that certbot can easily resolve to our cluster.

---

---

Make sure to keep the region of the static IP and your cluster same.

Now that we have the static IP, we can move on to the kubernetes part - the fun part.

---

The LoadBalancer service

The first thing that we setup is the load balancer service, we will then use this service to resolve to the pods running our certbot client and later on our own application pods.
Below is the YAML for our LoadBalancer service. It utilizes the static IP we created in the previous step.

svc.yml

apiVersion: v1
kind: Service
metadata:
  name: certbot-lb
  labels:
    app: certbot-lb
spec:
  type: LoadBalancer
  loadBalancerIP: X.X.X.X
  ports:
    - port: 80
      name: "http"
      protocol: TCP
    - port: 443
      name: "tls"
      protocol: TCP
  selector:
    app: certbot-generator

Most of it is self explanatory, few fields of interest are:

• spec.type : LoadBalancer - This spins up a Google Cloud LoadBalancer on GCP and AWS Elastic LoadBalancer on AWS
• spec.loadBalancerIP - This assign the previously generated static IP to our loadBalancer, now all traffic coming to our IP address is funneled into this load balancer.
• ports.port - We opened 2 TCP ports, port 80 and port 443 for accepting HTTP and HTTPS traffic respectively.
• spec.selector - These are the set of labels that allow us to govern which pods can our loadBalancer resolve to. We'll later use the same set of labels in our Job and Pod templates

Let's deploy this service to our cluster.

$ kubectl apply -f svc.yml
service "certbot-lb" created

If we see the status of our service now, we should see this

$ kubectl get svc certbot-lb
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
certbot-lb   LoadBalancer   X.X.X.X         X.X.X.X       80:30271/TCP,443:32224/TCP   1m

If you're trying this on minikube, LoadBalancer service is not available. Besides, it makes no sense trying to generate a SSL cert on your local environment.

---

Next we have to think about where to run our certbot container.

Job Controllers

The Job controllers of kubernetes allows us to schedule pods which run to completion. That is, these are jobs that if finished without error, need not be run again in future. This is exactly what we want when generating SSL certs. Once the certbot process is done and has given us the certificates, we no longer need that container to be running. Also we don't want kubernetes to restart this pod when it exits. However we can ask kubernetes to automatically reschedule the pod in case is case the pod exists with a failure/error.

So lets write a Job spec file that will generate the SSL cert for us using certbot. Certbot provides an official docker container that we can just reuse in our case.

jobs.yml

apiVersion: batch/v1
kind: Job
metadata:
  name: certbot
spec:
  template:
    metadata:
      labels:
        app: certbot-generator
    spec:
      containers:
        - name: certbot
          image: certbot/certbot
          command: ["certbot"]
          args: ["certonly", "--noninteractive", "--agree-tos", "--staging", "--standalone", "-d", "staging.ishankhare.com", "-m", "[e-mail]"]
          ports:
            - containerPort: 80
            - containerPort: 443
      restartPolicy: "Never"

Few important fields here are:

• spec.template.metadata.labels - This matches with the spec.selector specified in our LoadBalancer service. This brings our pod under our loadBalancer. Now everything coming on port 80 and 443 will be funneled to our pod.
• spec.template.spec.containers[0].image - The certbot/certbot docker image. Kubernetes will do a docker pull certbot/certbot on the server for us when scheduling this pod.
• spec.template.spec.containers[0].command - The command to run in the pod.
• spec.template.spec.containers[0].args - The arguments to the above command. We're using the standalone mode of certbot to generate the certs here as it makes things straightforward. You can read more about this command in certbot docs
• spec.template.spec.containers[0].ports - We've opened port 80 and 443 for our container.

Before deploying this to our cluster, make sure that the domain you specify is actually pointing to the static IP we setup in the previous steps.

Let's deploy this to our cluster:

$ kubectl apply -f jobs.yml
job.batch "certbot" created

This Job will spin-up a single pod for us, we can get that:

$ kubectl get pods
NAME            READY     STATUS    RESTARTS   AGE
certbot-sgd4w   1/1       Running   0          5s

We can now see the STDOUT logs on this pod

$ kubectl logs -f certbot-sgd4w
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
   Your cert will expire on 2019-03-04. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

After this, the certbot process exits without error and so does our pod. If we now list our pods

$ kubectl get pods
NAME            READY     STATUS      RESTARTS   AGE
certbot-sgd4w   0/1       Completed   0          45m

We see only a single pod whose status is completed.

We can add .spec.ttlSecondsAfterFinish to our jobs.yml like so:

spec:
  ttlSecondsAfterFinish: 10

This will automatically garbage collect our certbot pod after its finished.
Only supported in kubernetes v1.12 alpha. Hence not recommended right now.

---

Where to save the generated SSL certificate?

We currently have 2 options when it comes to saving the certs:

1. Save it in a mounted volume. (Not recommended)
   • Would require a PersistentVolume and a PersistentVolumeClaim
   • The certificate might be just a few KB in size, but the minimum volume size that GKE allots is 1Gi. Hence this is highly inefficient.
2. Use kubernetes Secrets. (Recommended)
   • This would also require us to use kubernetes client's in cluster configuration, which is straightforward to use and is usually the recommended way in such cases.
   • Would require us to setup ClusterRole and ClusterRoleBinding.
   • Can be configured to allow specific access to specific people using the concept of rbac (RoleBasedAccessControl)

---

Why in-cluster access?

To understand this, we need to go a little deeper into our current architecture.
Our current setup looks roughly like this:

As we can see here, the certs generated by our certbot Job Controller Pod are already inside the cluster. We want these cert credentials to be stored on the secrets.

When we fetch/create/modify secrets in normal flow, its usually done through the kubectl client like:

$ kubectl get secrets
NAME                  TYPE                                  DATA      AGE
default-token-hks8k   kubernetes.io/service-account-token   3         1m

$ kubectl create secret
Create a secret using specified subcommand.

Available Commands:
  docker-registry Create a secret for use with a Docker registry
  generic         Create a secret from a local file, directory or literal value
  tls             Create a TLS secret

Usage:
  kubectl create secret [flags] [options]

But, in our case, we want to store these credentials as secrets from inside a Pod which is itself inside the cluster. This is exactly what I meant by in-cluster access above.

---

What all do we need for In-Cluster access?

• kube-proxy
• kubectl
• RBAC

We'll need the first 2 mentioned above in the same Pod that is trying to access the secrets.
Hence its best to extend the certbot/certbot docker image with the above dependencies added in the container image itself.

This will change our cluster architecture a bit. The image below shows the rough cluster arch with our modified setup.

Let's first create our Dockerfile for our extended container:

FROM certbot/certbot
COPY ./script.sh /script.sh
RUN wget https://storage.googleapis.com/kubernetes-release/release/v1.6.3/bin/linux/amd64/kubectl
RUN chmod +x kubectl
RUN mv kubectl /usr/local/bin
ENTRYPOINT /script.sh

We have also used an extra script.sh in our container. We'll use this script as our entrypoint instead of the default entrypoint of certbot/certbot image.
This allows us to start our auxiliary kube-proxy and use kubectl as we desire. We present the script.sh below:

#!/bin/sh

# start kube proxy
kubectl proxy &

certbot certonly --noninteractive --force-renewal --agree-tos --staging --standalone -d staging.ishankhare.com -m [e-mail]

kubectl create secret tls cert --cert=/etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem \
    --key=/etc/letsencrypt/live/staging.ishankhare.com/privkey.pem

# kill the kubectl process running in background
kill %1

• First we start our kube-proxy as a background process.
• Next we run our certbot command for generating the cert.
• The certbot command generates certs for us at /etc/letsencrypt/live/staging.ishankhare.com/, we now use these paths to create the tls type Secret using kubectl create secret tls. This command has the following syntax:

$ kubectl create secret tls -h
Create a TLS secret from the given public/private key pair.

The public/private key pair must exist before hand. The public key certificate must be .PEM encoded
and match the given private key.

Examples:
  # Create a new TLS secret named tls-secret with the given key pair:
  kubectl create secret tls tls-secret --cert=path/to/tls.cert --key=path/to/tls.key

Hence our command above will create a secret named cert

• Finally we kill our kube-proxy background process.

Update our jobs.yml

Since we are now using an updated Dockerfile with own own script as its entrypoint, we can modify the Job manifest file like below:

apiVersion: batch/v1
kind: Job
metadata:
  #labels:
  #  app: certbot-generator
  name: certbot
spec:
  template:
    metadata:
      labels:
        app: certbot-generator
    spec:
      containers:
        - name: certbot
          image: ishankhare07/certbot:0.0.6
            - containerPort: 80
            - containerPort: 443
      restartPolicy: "Never"

With this we are ready to run our job now.

First verify that our LoadBalancer service is up:

$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
certbot-lb   LoadBalancer   10.11.240.100   X.X.X.X          80:31855/TCP,443:32457/TCP   32m

# delete our older job
$ kubectl delete job certbot
job.batch "certbot" deleted

Now we deploy our new Job file

$ kubectl apply -f jobs.yml
job.batch "certbot" created

# list the pod created for the job
$ kubectl get pods
NAME            READY     STATUS    RESTARTS   AGE
certbot-5nn5h   1/1       Running   0          3s

# get STDOUT logs for this pod
$ kubectl logs -f certbot-5nn5h
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
Starting to serve on 127.0.0.1:8001IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
   Your cert will expire on 2019-03-13. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"

The certificate generation was successful. But we cannot yet write to Secrets. The last line in the above output shows us that.

Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"

Why is that? Because RBAC....

RBAC or Role Based Access Control in kubernetes consists of 2 parts:

• Role/ClusterRole
• RoleBinding/ClusterRoleBinding

We will be using ClusterRole and ClusterRoleBinding approach to provide cluster wide access to our Pod. More fine-grained, role-based access can be provided with the Role and RoleBinding approach which can be referred from the docs - https://kubernetes.io/docs/reference/access-authn-authz/rbac/

Let's create 2 files called rbac-cr.yml and rbac-crb.yml with the following contents:
rbac-cr.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: default
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create"]

This allows access to Secrets, in-particular to get, list and create Secrets. The last verb create is what concerns us.

rbac-crb.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: secret-reader
subjects:
  - kind: User
    name: <your account here>
    namespace: default
  - kind: ServiceAccount
    name: default
    namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Fields of interest here are .subjects. It is an array and we have defined two kinds in this.

1. kind: User : This refers to the current user, who is executing these commands using kubectl. This is required so as to grant the current user enough permissions to grant the .rules.resources and .rules.verbs related access that we have defined in the rbac-cr.yml i.e. our ClusterRole definition.
2. kind: ServiceAccount : This refers to the account use inside the cluster when our pod will be creating the secrets using the kube-proxy.

We push this to our kubernetes cluster in this particular order only:

$ kubectl apply -f rbac-crb.yml
clusterrolebinding.rbac.authorization.k8s.io "secret-reader" created
$ kubectl apply -f rbac-cr.yml
clusterrole.rbac.authorization.k8s.io "secret-reader" created

The order is important, because we first need to create ClusterRoleBinding (defined in rbac-crb.yml) and then using that binding on our account, we are going to apply a ClusterRole (defined in rbac-cr.yml).

Finally re-run jobs.yml again

I say re-run because since previous job still exists:

$ kubectl get jobs
NAME      DESIRED   SUCCESSFUL   AGE
certbot   1         1            1h

and because of this job, a stagnant Pod exists as well:

$ kubectl get pods
NAME            READY     STATUS      RESTARTS   AGE
certbot-5nn5h   0/1       Completed   0          1h

If we just try to apply our jobs.yml file now, kubernetes sees that nothing as changed in our yml manifest and chooses to take no action:

$ kubectl apply -f jobs.yml
job.batch "certbot" unchanged

Hence we delete and apply our job from scratch:

$ kubectl delete job certbot
job.batch "certbot" deleted

$ kubectl apply -f jobs.yml
job.batch "certbot" created

# get the pod created by the above job
$ kubectl get pods
NAME            READY     STATUS    RESTARTS   AGE
certbot-c9h6m   1/1       Running   0          2s

We try to see the STDOUT logs of this container

$ kubectl logs -f certbot-c9h6m
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
Starting to serve on 127.0.0.1:8001IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
   Your cert will expire on 2019-03-13. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
secret "cert" created

The last line says secret "cert" created

Mission accomplished!

We can now see this recently created secret:

$ kubectl get secret cert
NAME      TYPE                DATA      AGE
cert      kubernetes.io/tls   2         9m

If you actually want to get the contents of the cert you can:

$ kubectl get secret cert -o yaml > cert.yml
# or
$ kubectl get secret cert -o json > cert.json

It is always desirable to redirect the above streams to a file rather than printing them directly to the console.

With this goal achieved, I'll wrap up this post here now. I'll be back with more posts soon detailing on:

1. Options available for using these certificated once we have created them.
2. Proper use of Init Containers along with the Job Controllers used in this post and our good old Pods and Deployments to see how we can make these interdependent services wait for the depending services to complete before spawning themselves.

Do let me know what you think of this post and if you have any questions in the comments section below.

This post was originally published on my blog ishankhare.com