Monitor Kubernetes cluster by running Boatswain as a DaemonSet

Originally posted on Boatswain Blog.

---

In a Kubernetes cluster, each node has a container runtime for running containers in pods. Kubernetes supports various container runtimes including:

• Docker
• containerd
• CRI-O

So if the Kubernetes cluster is using Docker as container runtime, we could install Boatswain on the nodes and monitoring its status as well as collecting the container logs.

NOTE: The example in this article is for experimental purpose and we do not suggest using Boatswain in any production Kubernetes cluster.

The DaemonSet in Kubernetes

In Kubernetes, we could deploy the Boatswain pod as a DaemonSet. Unlike a Deployment which we could replicates multiple pods across different nodes, a DaemonSet makes sure that a single pod runs on the selected nodes even if the node is newly added to the cluster. It is particular useful for storage daemon, log collection and monitoring on the node.

Running Boatswain as a DaemonSet

Here are the steps about setting up the Boatswain DaemonSet in a Kubernetes cluster.

1. Create the Boatswain namespace
2. Setup the Boatswain token as a secret
3. Setup the config file of the Boatswain DaemonSet
4. Start the Boatswain DaemonSet

Create the Boatswain namespace

Let's create a new namespace to separate the Boatswain setup from those already exist.

kubectl create namespace boatswain

Setup the Boatswain token as a secret

This token is required for running Boatswain. We store it as a secret called boatswain and later use it in the DaemonSet configuration.

kubectl create secret generic boatswain --from-literal=token=<TO_BE_REPLACED> -n boatswain

Setup the config file of the Boatswain DaemonSet

Here comes the DaemonSet configuration.

daemon-set.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: boatswain
  namespace: boatswain
spec:
  selector:
    matchLabels:
      name: boatswain
  template:
    metadata:
      labels:
        name: boatswain
    spec:
      containers:
      - name: boatswain
        image: boatswainio/boatswain:<latest or tag>
        env:
        - name: BOATSWAIN_TOKEN
          valueFrom:
            secretKeyRef:
              name: boatswain
              key: token
        securityContext:
          capabilities:
            add: ["NET_ADMIN"]
        volumeMounts:
          - name: dockersock
            mountPath: "/var/run/docker.sock"
      volumes:
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
      hostNetwork: true
      hostPID: true
      restartPolicy: Always
      nodeSelector:
        kubernetes.io/hostname: <hostname of the selected node>

Before saving the yaml file:

1. Replace the version of Boatswain at line 17.
2. Set the hostname of the target node which you want to install Boatswain or remove line 37 and 38 for installation on all nodes.

Start the Boatswain DaemonSet

Execute the following command to start the Boatswain DaemonSet.

kubectl create -f daemon-set.yaml

Check the status of the DaemonSet.

[ykyuen@camus ~]$ kubectl get ds -n boatswain
NAME        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                        AGE
boatswain   1         1         1       1            1           kubernetes.io/hostname=gke-boatswain-default-pool-v2-1234abcd-5678   52s

And we could read the metrics and logs on Boatswain.

Summary

The example above shows that it is possible to use Boatswain to monitor the Kuberenetes cluster. But currently it might not able to show the network metrics (confirmed on GKE cluster above). It might be related to the security settings and so far we haven't investigated further about this issue. Moreover, running Boatswain on all nodes does not work well as the usage will hit the 2GB data limit of the beta trial in a few hours. If you have any suggestions or thoughts, please let us know. 😀