In this blog post we are going to setup a kubernetes cluster with automated certificate generation using certbot. Will cover up some interesting concepts of kubernetes along the way, like:
So let's get started.
Why do we need this?
When using a traditional VM's/instances
, one has access to ssh into a fixed instance with an assigned and fixed public IP that the DNS can resolve to. Once the DNS is set to resolve your hostname to your instance, you can install certbot on it and generate the certs on that instance.
With kubernetes, things get a bit tricky. You will still have a set of instances in your cluster, but they aren't directly accessible from outside the cluster. Plus you cannot preempt on which node your nginx
or other ingress pod will be scheduled to run. Hence the most straight forward way to setup is doing everything through kubernetes
and docker
. This will also provide us with a few advantages:
- The cert generation process will be documented as part of the kubernetes manifest files and Dockerfiles
- Our solution would work for any number of pods and services. Basically we won't be bogged down due to issues arising because of scalability like when adding more pods or even adding more nodes to our cluster. Things will work seamlessly until we don't create a new cluster.
- Our solution will be platform independent, just like docker and kubernetes. It will work on any cloud provider GCP, AWS or even Azure(hopefully).
I'll be using GCP for some parts of this blog post but replicating those parts on other cloud platforms is pretty straight forward.
We'll start by reserving a static IP for our cluster and then forward a DNS record to that IP so that certbot can easily resolve to our cluster.
Make sure to keep the region of the static IP and your cluster same.
Now that we have the static IP, we can move on to the kubernetes part - the fun part.
The LoadBalancer service
The first thing that we setup is the load balancer service, we will then use this service to resolve to the pods running our certbot client and later on our own application pods.
Below is the YAML for our LoadBalancer service. It utilizes the static IP we created in the previous step.
svc.yml
apiVersion: v1
kind: Service
metadata:
name: certbot-lb
labels:
app: certbot-lb
spec:
type: LoadBalancer
loadBalancerIP: X.X.X.X
ports:
- port: 80
name: "http"
protocol: TCP
- port: 443
name: "tls"
protocol: TCP
selector:
app: certbot-generator
Most of it is self explanatory, few fields of interest are:
-
spec.type
:LoadBalancer
- This spins up a Google Cloud LoadBalancer on GCP and AWS Elastic LoadBalancer on AWS -
spec.loadBalancerIP
- This assign the previously generated static IP to ourloadBalancer
, now all traffic coming to our IP address is funneled into this load balancer. -
ports.port
- We opened 2 TCP ports,port 80
andport 443
for accepting HTTP and HTTPS traffic respectively. -
spec.selector
- These are the set of labels that allow us to govern which pods can our loadBalancer resolve to. We'll later use the same set of labels in ourJob
andPod
templates
Let's deploy this service to our cluster.
$ kubectl apply -f svc.yml
service "certbot-lb" created
If we see the status of our service now, we should see this
$ kubectl get svc certbot-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
certbot-lb LoadBalancer X.X.X.X X.X.X.X 80:30271/TCP,443:32224/TCP 1m
If you're trying this on minikube, LoadBalancer service is not available. Besides, it makes no sense trying to generate a SSL cert on your local environment.
Next we have to think about where to run our certbot container.
Job Controllers
The Job controllers of kubernetes allows us to schedule pods which run to completion. That is, these are jobs that if finished without error, need not be run again in future. This is exactly what we want when generating SSL certs. Once the certbot process is done and has given us the certificates, we no longer need that container to be running. Also we don't want kubernetes to restart this pod when it exits. However we can ask kubernetes to automatically reschedule the pod in case is case the pod exists with a failure/error.
So lets write a Job spec file that will generate the SSL cert for us using certbot. Certbot provides an official docker container that we can just reuse in our case.
jobs.yml
apiVersion: batch/v1
kind: Job
metadata:
name: certbot
spec:
template:
metadata:
labels:
app: certbot-generator
spec:
containers:
- name: certbot
image: certbot/certbot
command: ["certbot"]
args: ["certonly", "--noninteractive", "--agree-tos", "--staging", "--standalone", "-d", "staging.ishankhare.com", "-m", "me@ishankhare.com"]
ports:
- containerPort: 80
- containerPort: 443
restartPolicy: "Never"
Few important fields here are:
-
spec.template.metadata.labels
- This matches with thespec.selector
specified in ourLoadBalancer
service. This brings our pod under our loadBalancer. Now everything coming on port 80 and 443 will be funneled to our pod. -
spec.template.spec.containers[0].image
- Thecertbot/certbot
docker image. Kubernetes will do adocker pull certbot/certbot
on the server for us when scheduling this pod. -
spec.template.spec.containers[0].command
- The command to run in the pod. -
spec.template.spec.containers[0].args
- The arguments to the above command. We're using the standalone mode of certbot to generate the certs here as it makes things straightforward. You can read more about this command in certbot docs -
spec.template.spec.containers[0].ports
- We've opened port 80 and 443 for our container.
Before deploying this to our cluster, make sure that the domain you specify is actually pointing to the static IP we setup in the previous steps.
Let's deploy this to our cluster:
$ kubectl apply -f jobs.yml
job.batch "certbot" created
This Job
will spin-up a single pod for us, we can get that:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
certbot-sgd4w 1/1 Running 0 5s
We can now see the STDOUT logs on this pod
$ kubectl logs -f certbot-sgd4w
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
Your cert will expire on 2019-03-04. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
After this, the certbot process exits without error and so does our pod. If we now list our pods
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
certbot-sgd4w 0/1 Completed 0 45m
We see only a single pod whose status is completed.
We can add
.spec.ttlSecondsAfterFinish
to ourjobs.yml
like so:spec: ttlSecondsAfterFinish: 10
This will automatically garbage collect our certbot pod after its finished.
Only supported in kubernetes v1.12 alpha. Hence not recommended right now.
Where to save the generated SSL certificate?
We currently have 2 options when it comes to saving the certs:
- Save it in a mounted volume. (Not recommended)
- Would require a
PersistentVolume
and aPersistentVolumeClaim
- The certificate might be just a few KB in size, but the minimum volume size that GKE allots is
1Gi
. Hence this is highly inefficient.
- Would require a
- Use
kubernetes Secrets
. (Recommended)- This would also require us to use kubernetes client's
in cluster configuration
, which is straightforward to use and is usually the recommended way in such cases. - Would require us to setup
ClusterRole
andClusterRoleBinding
. - Can be configured to allow specific access to specific people using the concept of
rbac
(RoleBasedAccessControl)
- This would also require us to use kubernetes client's
Why in-cluster access?
To understand this, we need to go a little deeper into our current architecture.
Our current setup looks roughly like this:
As we can see here, the certs generated by our certbot
Job Controller Pod are already inside the cluster. We want these cert credentials to be stored on the secrets.
When we fetch/create/modify secrets in normal flow, its usually done through the kubectl
client like:
$ kubectl get secrets
NAME TYPE DATA AGE
default-token-hks8k kubernetes.io/service-account-token 3 1m
$ kubectl create secret
Create a secret using specified subcommand.
Available Commands:
docker-registry Create a secret for use with a Docker registry
generic Create a secret from a local file, directory or literal value
tls Create a TLS secret
Usage:
kubectl create secret [flags] [options]
But, in our case, we want to store these credentials as secrets from inside a Pod
which is itself inside the cluster. This is exactly what I meant by in-cluster access above.
What all do we need for In-Cluster access?
We'll need the first 2 mentioned above in the same Pod that is trying to access the secrets.
Hence its best to extend thecertbot/certbot
docker image with the above dependencies added in the container image itself.
This will change our cluster architecture a bit. The image below shows the rough cluster arch with our modified setup.
Let's first create our Dockerfile
for our extended container:
FROM certbot/certbot
COPY ./script.sh /script.sh
RUN wget https://storage.googleapis.com/kubernetes-release/release/v1.6.3/bin/linux/amd64/kubectl
RUN chmod +x kubectl
RUN mv kubectl /usr/local/bin
ENTRYPOINT /script.sh
We have also used an extra
script.sh
in our container. We'll use this script as our entrypoint instead of the default entrypoint ofcertbot/certbot
image.
This allows us to start our auxiliarykube-proxy
and use kubectl as we desire. We present thescript.sh
below:
#!/bin/sh
# start kube proxy
kubectl proxy &
certbot certonly --noninteractive --force-renewal --agree-tos --staging --standalone -d staging.ishankhare.com -m me@ishankhare.com
kubectl create secret tls cert --cert=/etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem \
--key=/etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
# kill the kubectl process running in background
kill %1
- First we start our kube-proxy as a background process.
- Next we run our certbot command for generating the cert.
- The certbot command generates certs for us at
/etc/letsencrypt/live/staging.ishankhare.com/
, we now use these paths to create thetls
type Secret usingkubectl create secret tls
. This command has the following syntax:
$ kubectl create secret tls -h
Create a TLS secret from the given public/private key pair.
The public/private key pair must exist before hand. The public key certificate must be .PEM encoded
and match the given private key.
Examples:
# Create a new TLS secret named tls-secret with the given key pair:
kubectl create secret tls tls-secret --cert=path/to/tls.cert --key=path/to/tls.key
Hence our command above will create a secret named cert
- Finally we kill our kube-proxy background process.
Update our jobs.yml
Since we are now using an updated Dockerfile with own own script as its entrypoint, we can modify the Job manifest file like below:
apiVersion: batch/v1
kind: Job
metadata:
#labels:
# app: certbot-generator
name: certbot
spec:
template:
metadata:
labels:
app: certbot-generator
spec:
containers:
- name: certbot
image: ishankhare07/certbot:0.0.6
- containerPort: 80
- containerPort: 443
restartPolicy: "Never"
With this we are ready to run our job now.
First verify that our LoadBalancer service is up:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
certbot-lb LoadBalancer 10.11.240.100 X.X.X.X 80:31855/TCP,443:32457/TCP 32m
# delete our older job
$ kubectl delete job certbot
job.batch "certbot" deleted
Now we deploy our new Job file
$ kubectl apply -f jobs.yml
job.batch "certbot" created
# list the pod created for the job
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
certbot-5nn5h 1/1 Running 0 3s
# get STDOUT logs for this pod
$ kubectl logs -f certbot-5nn5h
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
Starting to serve on 127.0.0.1:8001IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
Your cert will expire on 2019-03-13. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"
The certificate generation was successful. But we cannot yet write to Secrets. The last line in the above output shows us that.
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:default:default" cannot create secrets in the namespace "default"
Why is that? Because RBAC....
RBAC or Role Based Access Control in kubernetes consists of 2 parts:
-
Role
/ClusterRole
-
RoleBinding
/ClusterRoleBinding
We will be using ClusterRole
and ClusterRoleBinding
approach to provide cluster wide access to our Pod. More fine-grained, role-based access can be provided with the Role
and RoleBinding
approach which can be referred from the docs - https://kubernetes.io/docs/reference/access-authn-authz/rbac/
Let's create 2 files called rbac-cr.yml
and rbac-crb.yml
with the following contents:
rbac-cr.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
namespace: default
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "create"]
This allows access to Secrets
, in-particular to get
, list
and create
Secrets. The last verb create
is what concerns us.
rbac-crb.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: secret-reader
subjects:
- kind: User
name: <your account here>
namespace: default
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
Fields of interest here are .subjects
. It is an array and we have defined two kind
s in this.
-
kind: User
: This refers to the current user, who is executing these commands using kubectl. This is required so as to grant the current user enough permissions to grant the.rules.resources
and.rules.verbs
related access that we have defined in therbac-cr.yml
i.e. ourClusterRole
definition. -
kind: ServiceAccount
: This refers to the account use inside the cluster when our pod will be creating the secrets using thekube-proxy
.
We push this to our kubernetes cluster in this particular order only:
$ kubectl apply -f rbac-crb.yml
clusterrolebinding.rbac.authorization.k8s.io "secret-reader" created
$ kubectl apply -f rbac-cr.yml
clusterrole.rbac.authorization.k8s.io "secret-reader" created
The order is important, because we first need to create ClusterRoleBinding
(defined in rbac-crb.yml) and then using that binding on our account, we are going to apply a ClusterRole
(defined in rbac-cr.yml).
Finally re-run jobs.yml again
I say re-run because since previous job still exists:
$ kubectl get jobs
NAME DESIRED SUCCESSFUL AGE
certbot 1 1 1h
and because of this job, a stagnant Pod exists as well:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
certbot-5nn5h 0/1 Completed 0 1h
If we just try to apply our jobs.yml file now, kubernetes sees that nothing as changed in our yml manifest and chooses to take no action:
$ kubectl apply -f jobs.yml
job.batch "certbot" unchanged
Hence we delete and apply our job from scratch:
$ kubectl delete job certbot
job.batch "certbot" deleted
$ kubectl apply -f jobs.yml
job.batch "certbot" created
# get the pod created by the above job
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
certbot-c9h6m 1/1 Running 0 2s
We try to see the STDOUT logs of this container
$ kubectl logs -f certbot-c9h6m
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
Starting to serve on 127.0.0.1:8001IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/staging.ishankhare.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/staging.ishankhare.com/privkey.pem
Your cert will expire on 2019-03-13. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
secret "cert" created
The last line says secret "cert" created
Mission accomplished!
We can now see this recently created secret:
$ kubectl get secret cert
NAME TYPE DATA AGE
cert kubernetes.io/tls 2 9m
If you actually want to get the contents of the cert you can:
$ kubectl get secret cert -o yaml > cert.yml
# or
$ kubectl get secret cert -o json > cert.json
It is always desirable to redirect the above streams to a file rather than printing them directly to the console.
With this goal achieved, I'll wrap up this post here now. I'll be back with more posts soon detailing on:
- Options available for using these certificated once we have created them.
- Proper use of
Init Containers
along with theJob Controllers
used in this post and our good oldPods
andDeployments
to see how we can make these interdependent services wait for the depending services to complete before spawning themselves.
Do let me know what you think of this post and if you have any questions in the comments section below.
This post was originally published on my blog ishankhare.com
Top comments (0)