Deploy Kubernetes resources automatically with nsinjector

Original post: https://blog.blakelead.com/posts/2020/10/12/nsinjector

A bit of context

I recently had the opportunity to work on the elaboration of a CI/CD pipeline based on the GitOps methodology. In this pipeline, code pushed by developers triggers the creation of namespaces for each development branch. Since several teams share the same Kubernetes cluster, RBAC rules are applied to each namespace. If the namespace is created by the front team, the back team should not have access to it. And of course, developers do not have the rights to create RBAC resources (otherwise it wouldn't be fun :D).

I then took advantage of these specific needs to learn more about the Kubernetes controller pattern in order to automate the deployment of RBAC resources when namespaces are created. nsinjector is the result of my readings.

In this article, I explain how to use nsinjector to automatically deploy Kubernetes resources (in this case a ClusterRole and a ClusterRoleBinding) when a namespace with a name beginning by dev- is created.

Prerequisite

The rest of this post assumes a base knowledge of Kubernetes. If you want to know more about Kubernetes basics take a look at this doc.

Kubernetes

We will first need a Kubernetes cluster for our tests. If you are here you probably have access to a cluster in one way or another, but if you need one, you can create it locally very quickly with minikube. And if you're on MacOS it's even simpler since Docker Desktop embeds a local Kubernetes cluster.

kubectl

In order to interact with the cluster, we will use the command line tool kubectl. Follow the official documentation for installation process.

Helm (optional)

In this article, I present the installation of nsinjector with kubectl, but if you prefer Helm, I also propose a Chart here.

Installing nsinjector

Custom Resource Definition (CRD)

I won't go into details here, but a controller is generally based on the definition of custom resources (CRD) which allow to extend the functionalities of Kubernetes beyond the native objects (Deployment, Service, Job, etc.). nsinjector therefore uses the CRD NamespaceResourcesInjector to selectively deploy resource when a namespace is created.

To install this CRD, run the following command:

> kubectl apply -f https://raw.githubusercontent.com/blakelead/nsinjector/master/deploy/k8s/crd/namespaceresourcesinjector-crd-1.16.yaml

Controller

The second step is to deploy the controller. Since the controller needs to interact with a large number of resources, we first apply the following RBAC objects:

# rbac.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: nsinjector-controller

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nsinjector-controller
  namespace: nsinjector-controller

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nsinjector-controller
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["list", "get", "watch"]
- apiGroups: ["blakelead.com"]
  resources: ["namespaceresourcesinjectors"]
  verbs: ["list", "get", "watch", "update"]
- apiGroups: ["rbac"]
  resources: ["*"]
  verbs: ["list", "get", "watch", "update"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: nsinjector-controller
subjects:
- kind: ServiceAccount
  name: nsinjector-controller
  namespace: nsinjector-controller
roleRef:
  kind: ClusterRole
  name: nsinjector-controller
  apiGroup: rbac.authorization.k8s.io

Then apply them:

> kubectl apply -f rbac.yaml

We can then deploy the controller itself:

# nsinjector-controller.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nsinjector-controller
  namespace: nsinjector-controller
  labels:
    app: nsinjector-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nsinjector-controller
  template:
    metadata:
      labels:
        app: nsinjector-controller
    spec:
      serviceAccountName: nsinjector-controller
      containers:
      - name: nsinjector-controller
        image: blakelead/nsinjector-controller:v0.1.0

Apply the file:

> kubectl apply -f nsinjector-controller.yaml

NamespaceResourcesInjector

The controller is responsible for monitoring the creation of namespaces, and applying matching NamespaceResourcesInjector objects. Here is an example of a NamespaceResourcesInjector that can be applied:

# rbac-dev-inject.yaml

kind: NamespaceResourcesInjector
apiVersion: blakelead.com/v1alpha1
metadata:
  name: nri-test
spec:
  namespaces:
  - dev-.*
  resources:
  - |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: dev-role
    rules:
      - apiGroups: [""]
        resources: ["pods", "pods/portforward", "services", "deployments", "ingresses"]
        verbs: ["list", "get"]
      - apiGroups: [""]
        resources: ["pods/portforward"]
        verbs: ["create"]
      - apiGroups: [""]
        resources: ["namespaces"]
        verbs: ["list", "get"]
  - |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: dev-rolebinding
    subjects:
    - kind: User
      name: dev
    roleRef:
      kind: Role
      name: dev-role
      apiGroup: rbac.authorization.k8s.io

Apply the file :

> kubectl apply -f rbac-dev-inject.yaml

This will instruct the controller to deploy the dev-role and dev-rolebinding objects when a namespace whose name matches the dev-.* regex is created.

Create a namespace

Now if you create a namespace, you'll see that the two resources will automatically be created:

> kubectl create namespace dev-namespace-1
namespace "dev-namespace-1" created

> kubectl get clusterroles
NAME                        CREATED AT
dev-role                    2020-10-12T16:50:53Z
...

> kubectl get clusterrolebindings
NAME                ROLE                             AGE
dev-rolebinding     ClusterRole/dev-role             1m
...

Conclusion

That's all for me. If you wan't to dive in the code of this controller, go to https://github.com/blakelead/nsinjector. And if you want to discuss about it, I'd be happy to!