Running Kafka on Kubernetes for local development with Helm

Now that I have posted about running Kafka on Kubernetes using Persistent Volumes(PV) and Persistent Volume Claims(PVC), and another post about running it using Storage Classes it's time to grab that knowledge and have a version of it running with Helm Charts.

Helm is a package manager for Kubernetes, Helm helps you manage, define, install, and upgrade Kubernetes applications. It promotes re-usability and standardization.

In this post we will use Helm to convert the Local Kafka setup using Storage Classes I explained in a previous post. So if you haven't seen that post yet I suggest that you read and play with that setup first as it will help you to understand and extend the approach explained here.

Helm has many more features than the ones covered in this simple post.

As usual you can find the source code with instructions in my Github Repository. To clone it: git clone [e-mail]:mmaia/kafka-local-kubernetes.git, have fun.

Pre-reqs, install:

• Docker
• Kubectl
• Kind
• Helm

The setup using Helm

In this next section I will just cover the relevant information covering the helm setup and the actual Kubernetes files used are basically the same as the ones covered within the Running Kafka for local development with Storage Class post, so please refer to that article for further details on the Kubernetes setup.

After you clone the repo as described above if you expand the folder helm you will find the following structure:

As seen in my previous posts the kind-config.yaml file is used to setup the local Kubernetes cluster using Kind.

kind.config.yaml file:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane
  - role: worker
    extraPortMappings:
      - containerPort: 30092 # internal kafka nodeport
        hostPort: 9092 # port exposed on "host" machine for kafka
      - containerPort: 30081 # internal schema-registry nodeport
        hostPort: 8081 # port exposed on "host" machine for schema-registry
    extraMounts:
      - hostPath: ./tmp
        containerPath: /var/local-path-provisioner
        readOnly: false
        selinuxRelabel: false
        propagation: Bidirectional

Then inside the local-kafka-dev folder you'll find the main Helm Chart file: Chart.yaml and a file to set variables: values.yaml.

Chart.yaml file:

apiVersion: v2 #helm 3
name: local-kafka-dev
appVersion: "7.0.1"
description: A Helm chart for a local, single node Confluent Kafka, Schema Registry and Zookeeper for development.
version: 1.0.0
type: application

values.yaml file:

confluent:
  version: 7.0.1
  kafka: confluentinc/cp-kafka
  schemaRegistry: confluentinc/cp-schema-registry
  zookeeper: confluentinc/cp-zookeeper

Then inside the templates folder you will find all the same Kubernetes files used in the previous post as mentioned before with same minimal changes to use a few of the Helm variables defined in the values.yaml file. The only new file in there is the NOTES.txt which is used to display a nice message after you run the Helm chart as you will be able to see next while running it.

Running it

1. Open a terminal and cd to helm folder
2. Start Kind running: kind create cluster --config=kind-config.yml
3. Run helm install ${pick_a_name} local-kafka-dev. i.e - helm install local-kafka local-kafka-dev
4. When done stop kafka setup using helm uninstall ${name_picked_step_3}, you may also stop Kind if you want: kind delete cluster

That's it, it's done, you have a functional local Kafka +
Schema Registry running on Kubernetes and managed with Helm Charts which you can reach from your application running on your developer machine or IDE.

Enjoy!

Photo by ThisisEngineering RAEng on Unsplash