DEV Community: Sunny Bhambhani

Python - Stop storing passwords in plain text! A guide to werkzeug.security

Sunny Bhambhani — Mon, 05 Jan 2026 18:27:12 +0000

Problem

Storing passwords in plain text presents numerous dangers.

What happens if your database is compromised?
What if you accidentally reveal the wrong window while demonstrating and sharing your screen?

Such mistakes can result in serious issues, including losing the trust of customers, damaging brand value, financial repercussions, legal penalties, etc. These kinds of incidents are not rare; news reports about data breaches frequently appear worldwide.

Refer to https://databreaches.net/ for more details around data breaches worldwide.

Therefore, we should consider security in each aspect of application development, starting from planning, coding, building, deployment, etc.

And in this article, we will be talking about hashing passwords, but how do we actually do that? What options are available, etc.?

There are numerous options available, out of which some of the famous ones are werkzeug.security, argon2, bcrypt, etc.
Here we will see what werkzeug.security is and its practical implementation.

Prerequisites

Working Python setup.
Werkzeug Library installed.

pip install werkzeug

Basics

How does this work?
werkzeug.security provides helper functions for secure password storage and verification, and the two most commonly used functions are:

generate_password_hash(password, method=XXX, salt_length=XXX)
check_password_hash(PASSWORD_HASH, ACTUAL_PASSWORD)

More details around this can be found in its official documentation, refer: https://werkzeug.palletsprojects.com/en/stable/utils/ for more details.

The basic principle behind it is,

generate_password_hash takes the actual plain-text password as an input.
Further generates a salt of specified length.
The password + salt is processed through the chosen algorithm.
- scrypt(default)
- pbkdf2
Finally, it generates the output as a structured, hashed string.
Then check_password_hash comes into the picture, it securely checks if the given password and the stored password hash match or not.
Once that is evaluated, it produces a Boolean result, either true or false.

HOWTO

generate_password_hash

# Import generate_password_hash
from werkzeug.security import generate_password_hash

# Plain text password, if using Flask, it can come from any web form
plain_password = "SHH_SECRET"

# Generate a hashed string using default parameters
hashed = generate_password_hash(plain_password)

# Printing a hashed string
print("Hashed String: ", hashed)

# Output
Hashed String:  scrypt:32768:8:1$A2qi9GPp8Ut3cESv$31e31ffc4d21251396b391c31f09f6637e9169d4c5541a790cf0d9fcae3d58d2ba8a2df51e13f2601ba77fabe2e68d517fe845c63b489418309791ae1e350d3b

If you observe, the hashed string contains multiple sections
scrypt:N:r:p(METHOD_and_PARAMETERS)$salt(RANDOM_SALT)$derived_hash(ACTUAL_HASHED_PASSWORD)

In the above example, we just used the default values to generate the password hash.

Method: scrypt
N: 32768
r: 8
p: 1

Quick note about the parameters: Based on my understanding, the parameters N, r, and p in scrypt directly control how much compute and memory are required to derive a single password hash. The larger the numbers, the harder and more computationally intensive it is to break the password.

If you want to use custom parameters, this is how we can use them:

generate_password_hash(plain_password, method="scrypt:32768:16:1", salt_length=32)

It will produce the output as:

scrypt:32768:16:1$0hkax1LolbhqTyEN9rMSi8MC2MoF5B8w$85a7175deab3795db17898d17f03e3164df0ae1d158081ca3ed1680f9f940ff9cac76df18a71595160fd39963f4b4811c82348354e0c4a67a328104c56900230

if you want to use another method, this is how we can use it:

generate_password_hash(plain_password, method="pbkdf2:sha256")

The execution of generate_password_hash function generates a string that needs to be stored and checked for verification.

check_password_hash

# Import generate_password_hash and check_password_hash
from werkzeug.security import generate_password_hash, check_password_hash

# Plain text password, if using Flask, it can come from any web form
plain_password = "SHH_SECRET"
# Generate a hashed string using default parameters
hashed = generate_password_hash(plain_password)

# Login attempt 1 with wrong password
# If using Flask, it can come from any web form, like a login screen
login_attempt_1 = "SHH_WRONG_SECRET"
is_valid = check_password_hash(hashed, login_attempt_1)
print("Password correct?", is_valid)

# Login attempt 2 with correct password
# If using Flask, it can come from any web form, like a login screen
login_attempt_2 = "SHH_SECRET"
is_valid = check_password_hash(hashed, login_attempt_2)
print("Password correct?", is_valid)

# Output
# This is the first attempt in which we used the wrong password, and it concluded that it was incorrect, hence it gave the output as false
Password correct? False
# This is the second attempt in which we used the correct password, and it concluded that it was correct, hence it gave the output as true
Password correct? True

Conclusion

Never store plaintext passwords; store only the hash strings.
Do not re-invent the wheel; we can easily rely on vetted libraries and generate hash strings.
Keep your dependencies up-to-date.
Don’t compare strings manually; we can easily leverage functions like check_password_hash, which handle secure comparison.

Soon, I will be adding details about other libraries and functions as well. I hope this will be helpful.

Feel free to add your thoughts and experiences. Also, since I am still learning, correct me if I am wrong or have misinterpreted anything. Happy Learning! 😄

A smarter way to use imagePullSecrets in Kubernetes Cluster using ServiceAccounts

Sunny Bhambhani — Mon, 12 May 2025 12:16:05 +0000

Kubernetes is everywhere now a days, so does the container images and fetching the images from a private registry is a norm because of N number of reasons including security, that being the topmost.

Recap

Just to give you a heads up before we dive further, let's assume I fire kubectl create deployment nginx --image nginx what do you think happens?

Using kubeconfig, kubectl first authenticates with the API server.
It then constructs a deployment object using the values which we provided like deployment name as nginx, image to be used as nginx and some default values and then sends a request to the API server.
API server then validates the request and the object definition.
This is further stored in etcd datastore.
Then the controller manager specifically deployment controller comes into the picture which creates a replica set.
Then the replica set creates a pod (this is where our nginx image will be used).
Then the schedular assigns the pod to a node, where it will run based on multiple conditions. If interested please check out my previous article on what all things happens in the background: Resource management in Kubernetes
Once the node is identified, the kubelet running on the node is responsible for the management of the pod.
Once the pod is scheduled on a specific node, the kubelet looks for the image to be used and if it is already present.

if that is present locally, it will use it.

If not, it pulls the image from the registry.

If the registry is public, there are no issues, it just pulls the image, runs the pods, and we are done.

When imagePullSecrets are used?

Assume the registry is private, this is where imagePullSecrets come into play.
Kubernetes will need the secrets to authenticate the registry, so that it can pull the image.

How it works?

Firstly, we need to create a secret which contains the credentials and some details of our docker registry like server-name, username, password, email, etc.
It can be created both imperative as well as declarative way.

$ k get secrets  -o yaml dockersec
apiVersion: v1
data:
  .dockerconfigjson: ENCODED JSON OBJECT
kind: Secret
metadata:
  name: dockersec
  namespace: default
type: kubernetes.io/dockerconfigjson

$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD

Quick note: The secret must be of a type kubernetes.io/dockerconfigjson or docker-registry based on the way you choose to create the secret.
Once the secret is created, it must be attached to a pod so that it can use it.
Just add below content in your pod/deployment/statefulset yaml files, the name of the secret for me is dockersec but you can change based on your requirements.

spec:
  imagePullSecrets:
    - name: dockersec

Before pulling the image, the kubelet checks if the image is from a private registry. If so, it looks for imagePullSecrets in the spec.
Without proper imagePullSecrets, the image pull will fail with ImagePullBackOff.

Problem

Now assume you have hundreds of YAML files and hundreds of Kubernetes Object where we have to use this imagePullSecrets.
Just for the sake of doing once, you thought to do it manually but just imagine there was a change in the name of the Kubernetes secret.
Maybe because the organization now decides to streamline the naming convention of all the Kubernetes objects based on environments like prod/preprod/staging/etc.
- Or your org is now moving to a different registry or a different provider.
- Or the application is getting re-factored based on different namespaces or functions.
- There can be N number of reasons and doing all these changes again is a tricky and tiresome task.

One way to look at it is, use of helm charts which can help us to manage all the Kubernetes objects and this kind of a change can be pretty straight forward provided the helm charts are structured correctly and using values file we can just update the name of the secret and then can upgrade the chart, and we are done.

But if in case we are not using and package manager like helm and we are just working with vanilla YAML files then for that as well there is a neat way to do it.

The NEAT way!

We can attach the imagePullSecrets with the serviceaccounts and where-ever the serviceaccount is used, it will automatically populate the imagePullSecrets in the respective object like Pods.

Below is the current status of my Kubernetes objects, no secrets, a default serviceaccount and no imagePullSecrets updated in the pod.

$ k get secrets
No resources found in default namespace.
$ k get sa
NAME      SECRETS   AGE
default   0         33m
$ k get pods
NAME                     READY   STATUS             RESTARTS   AGE
nginx-65fd88fc88-8xjc9   0/1     ImagePullBackOff   0          32m

Create the docker-registry secret

$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD
secret/dockersec created
$ k get secret
NAME        TYPE                             DATA   AGE
dockersec   kubernetes.io/dockerconfigjson   1      3s

Patch the default serviceaccount to have the imagePullSecrets. It can be any serviceaccount based on your requirements.

$ kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "dockersec"}]}'
serviceaccount/default patched
$ k get sa
NAME      SECRETS   AGE
default   0         35m

Or you can manually edit the serviceaccount as well and add below content.

$ k get sa default -o yaml
apiVersion: v1
imagePullSecrets:   # ADD
- name: dockersec   # ADD
kind: ServiceAccount
metadata:
  name: default
  namespace: default

To see this in action now, delete the pod which is in failed state.

$ k delete pod nginx-65fd88fc88-8xjc9 --force

Check the events of the pod, now it was successfully able to pull the image from the private registry.

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  65s                default-scheduler  Successfully assigned default/nginx-65fd88fc88-w8mph to minikube
  Normal   Pulling    64s                kubelet            Pulling image "sunnybhambhani/test:v1"
  Normal   Pulled     37s                kubelet            Successfully pulled image "sunnybhambhani/test:v1" in 30.288s (30.288s including waiting). Image size: 567185619 bytes.
  Normal   Created    21s (x3 over 36s)  kubelet            Created container: test
  Normal   Started    21s (x3 over 36s)  kubelet            Started container test

Interesting bit is imagePullSecrets are automatically added to the object.

$ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i serviceaccountname
serviceAccountName: default
$ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i imagepullse -A 1
  imagePullSecrets:
  - name: dockersec

Bonus Tip

In case the pod is not in running status, it can be due to multiple reasons.
Just describe the pod to see what is happening in the background.
kubectl describe pod POD_NAME | awk '/^Events:/ {found=1; next} found', it looks for the line starting with Events, once found prints the remaining lines.

$ kubectl describe pod nginx-65fd88fc88-8xjc9 | awk '/^Events:/ {found=1; next} found'
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  7m21s                   default-scheduler  Successfully assigned default/nginx-65fd88fc88-8xjc9 to minikube
  Normal   Pulling    4m17s (x5 over 7m20s)   kubelet            Pulling image "REGISTRY/test:v1"
  Warning  Failed     4m14s (x5 over 7m17s)   kubelet            Failed to pull image "REGISTRY/test:v1": Error response from daemon: pull access denied for REGISTRY/test, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
  Warning  Failed     4m14s (x5 over 7m17s)   kubelet            Error: ErrImagePull
  Warning  Failed     2m41s (x19 over 7m16s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    2m20s (x21 over 7m16s)  kubelet            Back-off pulling image "REGISTRY/test:v1"

Here the logs clearly state, requested access to the resource is denied and in-turn the status of the pod is ImagePullBackOff.

$ k get pods
NAME                     READY   STATUS             RESTARTS   AGE
nginx-65fd88fc88-8xjc9   0/1     ImagePullBackOff   0          76s

References:

Google.
Kubernetes docs (https://kubernetes.io/docs/home/).
kubectl explain.
Linux man pages.

PS: This works with any flavor of Kubernetes: minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.

I hope this will be helpful, feel free to add your thoughts and experiences, Happy Learning! 😄

Kubernetes hardening made easy: Running CIS Benchmarks with kube-bench

Sunny Bhambhani — Mon, 03 Mar 2025 04:41:17 +0000

In today's world, where security risks and breaches are growing daily, it is crucial to maintain our applications and infrastructure's compliance with security standards and that is where CIS benchmarks from CIS (Center for Internet Security) comes in. And with kube-bench, running these checks becomes straightforward, helping you strengthen your Kubernetes clusters with confidence.

CIS Benchmarks

Here is the mission statement from CIS's website.

Our mission is to make the connected world a safer place by developing, validating, and promoting timely best practice solutions that help people, businesses, and governments protect themselves against pervasive cyber threats.

Ref. https://www.cisecurity.org/ for more details.

It basically helps us to jot down:

What all best practices/guidelines to use?
Which tools are available to assist us in scanning our application or infrastructure?
Making sure the best practices are updated on timely manner.

The benchmarks are basically available for multiple platforms including:

All the public cloud providers.
Softwares
DevSecOps tools
Mobile devices
Operating Systems
and much more.

You can find their extensive list here: https://learn.cisecurity.org/benchmarks

CIS Benchmarks: Download PDF for your platform

For downloading the PDF files you just need to provide some general details about yourself.

Once done, press "Submit" and you will receive an email with the link from where you can download all kinds of benchmarks.

It looks something like this:

Upon pressing "Access PDFs" you will be routed to their extensive list of benchmarks which looks something like this:

You can choose whatever you want to, but since we are interested in Kubernetes as of now, we will scroll down to the Kubernetes section.

And the good part here is, this is not limited to vanilla Kubernetes, we can even download benchmarks for all the Kubernetes flavours like EKS from AWS, AKS from Azure, GKE from Google, etc.

Once you download a specific PDF, you will see it contains a great amount of details including recommendations, problem statement, impact, remediation, etc.

Now, if you have observed the PDF contains thousands of recommendations and going thru them and applying them one by one is a time-consuming task and just imagine you have 100s of clusters. Though I would recommend you to at-least go thru it once and get an idea of what all details it contains and how we can make use of it.

To make our lives easier there are couple of tools which can help us to automate this process and help us to identify where we are lacking. So that we can fix them quickly.

CIS Benchmarks: Tools

There are couple of tools which are managed by CIS itself like CIS-CAT Lite/CIS-CAT Pro/etc. CIS-CAT Lite is a free version and it supports a limited options excluding Kubernetes. CIS-CAT Pro is the one which supports Kubernetes but it is just available for CIS SecureSuite Members.

Ref. https://www.cisecurity.org/cybersecurity-tools for more details about the tools.

Now, let's talk about the good part, the community has given us couple of opensource tools which does the same 😉 the most used one is kube-bench (from Aqua Security).

HOWTO: kube-bench

kube-bench is a tool that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.

There are multiple ways to run kube-bench

Install it via a package manager

Download the latest package based on your OS from github releases page and install it.
Ref: https://github.com/aquasecurity/kube-bench/releases
Since, I am using Ubuntu, I will use dpkg to get this package installed.

$ wget https://github.com/aquasecurity/kube-bench/releases/download/v0.10.2/kube-bench_0.10.2_linux_amd64.deb
$ sudo dpkg -i kube-bench_0.10.2_linux_amd64.deb
Selecting previously unselected package kube-bench.
(Reading database ... 41333 files and directories currently installed.)
Preparing to unpack kube-bench_0.10.2_linux_amd64.deb ...
Unpacking kube-bench (0.10.2) ...
Setting up kube-bench (0.10.2) ...
$ kube-bench version
0.10.2

Run as a job in the Kubernetes Cluster

Get the job.yaml from the same github link and deploy it on your cluster.
Ref: https://github.com/aquasecurity/kube-bench/blob/main/job.yaml
Based on your platforms there are multiple job*.yaml files available like for EKS, AKS, GKE, etc.

$ kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
job.batch/kube-bench created

$ k get pods
NAME               READY   STATUS      RESTARTS   AGE
kube-bench-n22k9   0/1     Completed   0          96s

$ k logs kube-bench-n22k9 | head -n 20
[INFO] 1 Control Plane Security Configuration
[INFO] 1.1 Control Plane Node Configuration Files
[PASS] 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated)
[PASS] 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated)
[PASS] 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated)
[PASS] 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated)
[PASS] 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated)
[PASS] 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated)

OUTPUT TRIMMED

$ k logs kube-bench-n22k9 | tail -n 20
OUTPUT TRIMMED

5.7.3 Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a
suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker
Containers.

5.7.4 Ensure that namespaces are created to allow for appropriate segregation of Kubernetes
resources and that all new resources are created in a specific namespace.


== Summary policies ==
0 checks PASS
6 checks FAIL
29 checks WARN
0 checks INFO

== Summary total ==
57 checks PASS
19 checks FAIL
54 checks WARN
0 checks INFO

Just check the logs of the pod for all the recommendations. These logs contain multiple sections for different-different Kubernetes components like Controlplane, ETCD, Worker nodes, etc.
At the end of each section you can see statistics about the checks and at the end of the logs you will see a quick summary about the total (as mentioned above).

== Summary node ==
14 checks PASS
2 checks FAIL
8 checks WARN
0 checks INFO

Run as a docker container

docker run --rm --net host --pid host --user 0 \
  -v /etc:/etc:ro \
  -v /var:/var:ro \
  -v /usr/bin:/usr/bin:ro \
  -v /usr/lib:/usr/lib:ro \
  aquasec/kube-bench:latest --benchmark cis-1.24

You can change the version based on your requirement, 1.24 is the latest one.
More details about the versions can be found here: https://github.com/aquasecurity/kube-bench/tree/main/cfg
For this as well, just check the logs of the container you will find the same results as we described in earlier section.

This is how the remediation steps look like:

== Remediations master ==
1.1.9 Run the below command (based on the file location on your system) on the control plane node.
For example, chmod 600 <path/to/cni/files>

1.1.11 On the etcd server node, get the etcd data directory, passed as an argument --data-dir,
from the command 'ps -ef | grep etcd'.
Run the below command (based on the etcd data directory found above). For example,
chmod 700 /var/lib/etcd

1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir,
from the command 'ps -ef | grep etcd'.
Run the below command (based on the etcd data directory found above).
For example, chown etcd:etcd /var/lib/etcd

1.1.19 Run the below command (based on the file location on your system) on the control plane node.
For example,
chown -R root:root /etc/kubernetes/pki/

References:

Feel free to add your thoughts and experiences, Happy Learning! 😄

k9s - manage your Kubernetes cluster and it’s objects like a pro!

Sunny Bhambhani — Sat, 30 Nov 2024 06:56:46 +0000

Introduction
k9s is a terminal based GUI to manage any Kubernetes(k8s) cluster. Using this single utility, we can manage, traverse, watch all our Kubernetes objects.

More information around k9s can be found here: https://k9scli.io/

We will dive a bit into k9s and see how it can help us in our day-to-day life, how we can get started, etc.

Features
Before we go into the example on how it can help, what it can do for us, let's see some of its features, it has a ton of features, but we will focus on the ones which can help us in our day-to-day activities:

Shows a helicopter view of all the Kubernetes objects.
Easily we can move from one namespace to another.
Watch/observe the state of Kubernetes objects using pulses.
Get the tree kind of structure using xrays to identify objects co-relation.
Manage the objects directly from the CLI, operations like deleting, editing, restarting is just one button away.
Interestingly using a single button, you can get shell access to the pods.
Check the logs from any of the pod without actually remembering the name of the pod, just select the pod and the logs are just one button away.
Sanitize/clean up all the pods which are in completed/error state.
Works with any flavor of Kubernetes minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.
.... and a lot more.

Installation

Its installation is straight forward and is available for almost all the platforms.
Refer https://k9scli.io/topics/install/ for more details.
We will be doing this on Ubuntu.

Download the latest version of k9s binary, you can use the same URL/version if required or get the latest version released and update below URL accordingly.

For latest version, refer: https://github.com/derailed/k9s/releases

$ wget https://github.com/derailed/k9s/releases/download/v0.32.7/k9s_linux_amd64.deb

Install it using apt package manager.

$ sudo apt install ./k9s_linux_amd64.deb

Once done it will get the k9s binary installed here: /usr/bin/k9s.

HOWTO

Launch k9s

$ k9s

Once k9s is launched you will be presented with the beautiful layout with lots of options.

Press 0 to view all the pods from all the namespaces.

Navigate between Kubernetes objects
Navigating is pretty easy; it is more or less like vi or vim.

For example, if you want to view deployments then press : it will bring the cursor to a text area where you can write the object which you are interested in, for instance in current example deployments.
The short forms too are accepted i.e. deploy in case of deployment, svc in case of service, etc.
This also supports crds :)

Pods management

Press : and type in pods, it will show you all the pods in selected namespace, if you want to see the pods from all namespaces press 0.

There are times where we have to delete the pods on the fly. Just select the pod and press ctrl+d and you are done.

There are times when you have to login to a pod. Just select the pod and press s and you are done. If you want to exit, press ctrl+d or type in exit.

Want to describe the pod details, press d.

Want to get the yaml output of the Kubernetes objects, just select the object and press y.

Want to see the logs of a specific pod, select the pod and press l.

You can even do port-forward from the same UI, isn't it cool :), Just press shift+f and if you want to see any existing port-forward are present or not, press f.

Sanitize/Clean up pods: For instance in below screen dump we can see there are 2 pods which are in Completed and we want to clean them up.

Just press z, you will be asked "if you are sure", type in "Yes Please!" and the job is done.

XRAY
Xray gives you a great detail in terms of co-relation between k8s objects, it basically provides a tree like structure.

Pulses
Pulses give a great dashboard to see what exactly is happening in your cluster, what all objects are there, health of your objects, etc?

NOTE: Make sure metrics-server is installed and running in the cluster otherwise you won't see proper results. In my case it was not installed, therefore it was just stating blank results.

After installing metrics-server and launching pulses it shows an awesome dashboard.

The great part about it is, it shows the current status of your cluster.
It even shows what all objects are healthy and what all objects are unhealthy.
Legend: GREEN/HEALTHY & YELLOW/UNHEALTHY

References:

Feel free add your thoughts, Happy learning :)

Helmister: Updated the logic to allow custom path

Sunny Bhambhani — Thu, 28 Nov 2024 07:53:31 +0000

Its a follow-up post on helmister, refer here for more details: https://dev.to/aws-builders/installing-multiple-helm-charts-in-one-go-approach-3-using-simple-bash-utility-2795

Updated the logic allowing users to specify a custom path for config.yaml.
Log entry: [2024-11-27 12:30:06] [INFO] Generic/common values based on /home/sunny/config.yaml file.
-f | --file
Helps you to mention explicit/specific config.yaml. If not provided it will consider the default one which is placed in the base folder.
Example: ./helmister install -f /opt/app/config-dev.yaml

Installing multiple helm charts in one go [Approach 3 - using simple bash utility]

Sunny Bhambhani for AWS Community Builders ・ Jul 23 '24

#helm #kubernetes #devops #eks

sftp refresher - 101

Sunny Bhambhani — Tue, 29 Oct 2024 11:34:22 +0000

This is just a very basic refresher around sftp server, I know there are lots of other and on-cloud solutions available for file transfer and sftp server is rarely used.

But since I got a query yesterday about it, I thought to create a small write-up :)

sftp server

There was a time when sftp was widely used for secure file transfers, though now there are many options available, but this used to be a hero service then and was commonly asked during interviews.
The name itself expands to 'Secure File Transfer Protocol'.
This allows for a secure file transfer over SSH (On port 22).
It is quite fast and efficient.

Pre-requisites

Any linux machine with internet access (here I am using Ubuntu).

HOWTO

Update the package lists on your Ubuntu(debian based) machine.

$ apt update

Install ssh/openssh-sftp-server/vim(if not present).

$ apt install ssh openssh-sftp-server vim -y

Edit the sshd config file and append below details

$ vim /etc/ssh/sshd_config

Match group sftp
ChrootDirectory /home
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Match group sftp: This statement states below settings will be applicable to all the users who belong to sftp group.
ChrootDirectory: This basically changes the root directory to mentioned path.
X11Forwarding: This disables X11 forwarding (this is enabled in some GUI use-cases) and for sftp this is not required.
AllowTcpForwarding: Disables TCP forwarding.
ForceCommand: This forcefully mandates that the used of this group should be only allowed to use sftp and nothing else.

Restart the service, you can use this way or use systemctl based on your distribution.

$ /etc/init.d/ssh restart

Add a new group called sftp to the system.

$ addgroup sftp

Create a new user with sftp group attached to it.

$ useradd -m sunny -g sftp

Change the password of the newly created user.

$ passwd sunny

Change the permission of the directory to USER/OWNER only.

$ chmod 700 /home/sunny/ -R

Try connecting to sftp server

$ sftp sunny@HOST_IP

Enter the password and try using get and put commands.

*Feel free to add any details which I might have missed, happy learning :) *

Quick guide to install minikube, kubectl, helm and enable ingress controller on Ubuntu

Sunny Bhambhani — Mon, 21 Oct 2024 12:39:29 +0000

This is just a quick guide to install minikube and enable ingress controller on Ubuntu.

The purpose of this post is to consolidate all the commands in one place for quick reference.

Minikube is a lightweight Kubernetes cluster that enables you to run Kubernetes clusters locally. It's ideal for learning Kubernetes while developing and testing on a single machine.

More information about minikube can be found here: https://minikube.sigs.k8s.io/docs/

Install minikube (Ref: https://minikube.sigs.k8s.io/docs/start)

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64

Install docker (Ref: https://docs.docker.com/engine/install/ubuntu/)

sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Add current user to docker group

sudo usermod -aG docker $USER && newgrp docker

Install kubectl (Ref: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
sudo chmod +x kubectl
sudo mv kubectl /usr/local/bin
alias k=kubectl

Install helm (Ref: https://helm.sh/docs/intro/install/)

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Start minikube

minikube start
# Once started
k get nodes
# To get a list of underlying nodes
k get pods -A
# To get a list of all the pods from all namespaces

Get minikube's IP and enable ingress controller

minikube ip
minikube addons enable ingress

Installing multiple helm charts in one go [Approach 3 - using simple bash utility]

Sunny Bhambhani — Tue, 23 Jul 2024 10:34:55 +0000

In this article, we will be talking about Approach 3 i.e. how to get multiple helm charts installed using a simple bash utility.

If you haven't read the previous article where I discussed other approaches, feel free to read it over.

Motive

Why I thought of a simple bash utility?

In some of the air-gapped environments it is sometimes a bit difficult to use the tools/utilities available because moving things inside an air-gapped environment is a challenge.
Some of the environments are so secure that one may need to follow a whole process of getting all the security clearances and approval before using a tool/utility, which altogether is a nightmare.
I chose bash, the reason being it is pretty common among engineers and it is easily understandable.
The source code can be found here: https://github.com/sunnybhambhani/helmister
You can copy it, and tweak it based on your requirements.

PS: I chose a name for this utility as well helmister or helm-minister but you can call it whatever you want 🙂

Prerequisites

A running Kubernetes cluster with proper permissions, it can be any flavor of Kubernetes minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.
kubectl and helm installed on your machine to interact with the Kubernetes cluster.
- Ref: https://kubernetes.io/docs/tasks/tools/
- Ref: https://helm.sh/docs/intro/install/
A clone of helmister repository.
yq(required) and cowsay(optional) packages installed on your machine.

Helmister

This is a small bash utility that can help to install and uninstall multiple helm charts in one go. The idea is inspired by helmfile.
Its usage is quite simple, just run the utility followed by the option like install or uninstall i.e. ./helmister [install/uninstall].
Under the hood, it calls helm binary. Therefore it is kind of a wrapper around helm.
It consumes a config.yaml file which contains all the necessary details about the releases, and common parameters.
This supports both oci:// and https:// helm registries.
PS: In the future, I am planning to add some more options/functionalities to this.

Directory structure

.
├── config.yaml
├── helmister
├── logs
│   ├── archive
│   │   └── helmister_20240227_170201.tar.gz
│   └── helmister.log
├── README.md
└── values
    ├── argo-cd.yaml
    └── nginx-values.yaml

helmister, this is the script written in bash you can just cat and see what all things it contains.
config.yaml, this is the main configuration file or you can call the state file which contains the list of all releases you want to install in a cluster, plus it also contains some additional key:value pairs that are generic and common across all the releases. I first kept this configuration file in csv format, but later decided to convert it to yaml because it is more readable.
logs, this is a directory that holds the logs of this utility, it contains the information about the execution of the last iteration, plus any archived/past logs (if required for reference).
README.md, contains bit of a documentation about this utility, and what options are present.
values, this is the directory where all the values files are placed (it can be anywhere in your system but for simplicity, I have placed them in the same directory).
Now let's talk about config.yaml which is the main ingredient.

config.yaml

dry_run: false
create_namespace: true
wait: false
timeout: false # If true, defaults to 20 mins
charts:
  - release_name: nginx
    chart_name: nginx
    chart_repo: oci://registry-1.docker.io/bitnamicharts
    values_file: values/nginx-values.yaml
  - release_name: argocd
    chart_name: argo-cd
    chart_repo: https://argoproj.github.io/argo-helm
    values_file: values/argo-cd.yaml
    version: 6.4.0
    namespace: argo-cd

The initial key:value pairs are common across all the releases.
charts, is an array that contains a list of what all releases need to be installed in a Kubernetes cluster.
dry_run, it is a boolean [true or false], and if true, none of the Kubernetes resources will be created it will just do a dry_run.
create_namespace, it is a boolean [true or false], and if true it will automatically create a namespace for the release specified in the charts array.
wait, it is a boolean [true or false], and if true it acts similar to helm --wait wherein the shell will be kept occupied until all the Kubernetes resources are created.
timeout, it is a boolean [true or false], and if true it acts similar to helm --timeout=20m i.e. if all the resources are not created within 20 mins the execution will fail. By default, I have kept the timeout as 20 minutes which is more than enough.
charts, it is an array that contains details around individual releases. Except for version and namespace all the key:value pairs are mandatory.
- release_name, this is the name of the release.
- chart_name, this is the name of the chart that needs to be installed.
- chart_repo, this is the helm registry where the chart is located. It can be any oci:// or https:// registry.
- values_file, which contains the path of the values file for individual releases.
- version, this is the version of the chart that needs to be installed. This is optional and if not provided, It will consider the latest chart version.
- namespace, this is where the chart will be installed. This is optional as well and if not provided it will be installed in the default namespace.

Let's see this in action

I will use the same config.yaml which will install one helm release from an oci:// registry in default namespace (since I haven't specified any namespace for that release) and another one from https:// registry in argo-cd namespace. Note the version as well for nginx, there as well I haven't specified any version, it will pick the latest available one automatically.
This is my cluster's current status.

$ k get ns
NAME              STATUS   AGE
default           Active   28s
kube-node-lease   Active   28s
kube-public       Active   29s
kube-system       Active   29s

$ helm list -A
NAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION

$ ./helmister install
[2024-07-23 15:02:47] [INFO] Using file: config.yaml
 ____________________________
< Helmister, Install charts! >
 ----------------------------
   \
    \
        .--.
       |o_o |
       |:_/ |
      //   \ \
     (|     | )
    /'\_   _/`\
    \___)=(___/

[2024-07-23 15:02:47] [INFO] Generic/common values based on config.yaml file
[2024-07-23 15:02:48] [INFO] Dry Run: false
[2024-07-23 15:02:48] [INFO] Create Namespace: true
[2024-07-23 15:02:48] [INFO] Wait: false
[2024-07-23 15:02:48] [INFO] Timeout: false
[2024-07-23 15:02:48] [INFO] ****************************
[2024-07-23 15:02:48] [INFO] Chart specific values based on config.yaml file
[2024-07-23 15:02:48] [INFO] Release Name: nginx
[2024-07-23 15:02:48] [INFO] Chart Name: nginx
[2024-07-23 15:02:48] [INFO] Chart Repo: oci://registry-1.docker.io/bitnamicharts
[2024-07-23 15:02:48] [INFO] Values File: values/nginx-values.yaml
[2024-07-23 15:02:48] [INFO] Version: null
[2024-07-23 15:02:48] [INFO] Namespace: null
[2024-07-23 15:02:48] [INFO] ****************************
[2024-07-23 15:02:48] [INFO] Installing nginx with release name nginx from oci://registry-1.docker.io/bitnamicharts with version null using values file: values/nginx-values.yaml in default namespace
...
...
OUTPUT TRIMMED

Actual stdout output can be found here.
Here you will see that it provides all the minute details of what exactly it is doing.
For example: Generic values
- Here I have marked create_namespace as true because I don't already have argo-cd namespace wherein I want to install argocd release.

[2024-07-23 15:02:47] [INFO] Generic/common values based on config.yaml file
[2024-07-23 15:02:48] [INFO] Dry Run: false
[2024-07-23 15:02:48] [INFO] Create Namespace: true
[2024-07-23 15:02:48] [INFO] Wait: false
[2024-07-23 15:02:48] [INFO] Timeout: false

Next, you will see, chart/release specific values of all the items in charts array one by one:

[2024-07-23 15:03:31] [INFO] Chart specific values based on config.yaml file
[2024-07-23 15:03:31] [INFO] Release Name: argocd
[2024-07-23 15:03:31] [INFO] Chart Name: argo-cd
[2024-07-23 15:03:31] [INFO] Chart Repo: https://argoproj.github.io/argo-helm
[2024-07-23 15:03:31] [INFO] Values File: values/argo-cd.yaml
[2024-07-23 15:03:31] [INFO] Version: 6.4.0
[2024-07-23 15:03:31] [INFO] Namespace: argo-cd

Here I have explicitly added an additional check that will check if the pods are up and healthy (this is just specific to pods it won't consider any other k8s objects).
It will continuously check for next 20mins and will check every 5 secs. If the pods are still in non-running state it will terminate the process.

[2024-07-23 15:03:47] [WARN] Pods for release argocd are not yet in Running state, checking again in 5 seconds

Once all the pods are in running state and it has deployed all the releases it will show a successful message.

[2024-07-23 15:05:20] [INFO] All actions completed successfully

Once this is done, if required you can see the logs as well from the logs directory. A sample can be found here.
And here is my cluster's current status now:

$ helm list -A
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
argocd  argo-cd         1               2024-07-23 15:03:34.46715742 +0530 IST  deployed        argo-cd-6.4.0   v2.10.1
nginx   default         1               2024-07-23 15:02:53.590367521 +0530 IST deployed        nginx-18.1.5    1.27.0

$ k get ns
NAME              STATUS   AGE
argo-cd           Active   58m
default           Active   62m
kube-node-lease   Active   62m
kube-public       Active   62m
kube-system       Active   62m

$ k get pods -n argo-cd
NAME                                                READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                     1/1     Running   0          58m
argocd-applicationset-controller-68cd75bc89-d4q8x   1/1     Running   0          58m
argocd-dex-server-84b5bbdcbf-c4qv2                  1/1     Running   0          58m
argocd-notifications-controller-7bc55c495d-xz6gt    1/1     Running   0          58m
argocd-redis-5d5cdcfd54-2rx26                       1/1     Running   0          58m
argocd-repo-server-756ff4cd7d-rpvh7                 1/1     Running   0          58m
argocd-server-7587d49b7b-9h47l                      1/1     Running   0          58m

$ k get pods -n default
NAME                     READY   STATUS    RESTARTS   AGE
nginx-569b6bc698-48qqt   1/1     Running   0          59m
nginx-569b6bc698-9qzpv   1/1     Running   0          59m
nginx-569b6bc698-wflrj   1/1     Running   0          59m

If you want to uninstall it, simply do:

$ ./helmister uninstall

It will get everything cleaned for all the releases that are specified in config.yaml.
The sample log can be found here.

Feel free to use it, and tweak it based on your requirements.

I will soon add some more functionalities to it.

Happy learning!

References:

Google/Stackoverflow/Linux man pages/etc

Update multiple Kubernetes objects/configmaps in one go!

Sunny Bhambhani — Wed, 22 May 2024 12:54:25 +0000

There may be cases wherein we just need to update a Kubernetes configmap or any other Kubernetes object based on our requirements.

And let's say it's just one or two configmaps(here we will be talking about configmaps just for simplicity but its applicable for other Kubernetes objects as well) then that's fine, right? We can simply do the following:

kubectl edit -n NAMESPACE_NAME configmap CONFIGMAP_NAME
Update it based on our requirements.
Save it.

But think about a scenario, wherein we need to update +100 configmaps with the same values how to do it?

We cannot update or edit each and every object since that would be a bit tedious and will be error-prone as well. Plus there might be chances we might skip some of them.

So, the best way to do it is by using patch command in an automated way.

Refer: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_patch/ for more details about it.

Let's see this in action:

I have a very minimal configmap which contains some DATABASE details.

apiVersion: v1
data:
  DATABASE_HOST: "api01.ybdb.io"
  DATABASE_NAME: "api01"
  DATABASE_PORT: "5433"
kind: ConfigMap
metadata:
  name: api01-configmap
  namespace: default

Now, let's assume there is an update instead of port number 5433. Our database now listens on port 5444.
We can easily accomplish this using kubectl edit but let's see patch in action:

$ kubectl patch configmap api01-configmap -n default -p '{"data":{"DATABASE_PORT":"5444"}}'
configmap/api01-configmap patched
$ kubectl get cm -n default -o yaml api01-configmap
apiVersion: v1
data:
  DATABASE_HOST: api01.ybdb.io
  DATABASE_NAME: api01
  DATABASE_PORT: "5444"
kind: ConfigMap
metadata:
  creationTimestamp: "2024-05-22T10:06:39Z"
  name: api01-configmap
  namespace: default
  resourceVersion: "67651"
  uid: 2d49a0ea-6910-420c-b2bf-00e42e6c08d7

The command is pretty straight forward kubectl patch configmap api01-configmap -n default -p '{"data":{"DATABASE_PORT":"5444"}}'
-n is to specify the namespace name where the configmap resides.
-p is to specify what patch/update we want in the Kubernetes object.
Here, under data: we are updating a key called DATABASE_PORT: to 5444.
Note: If in case DATABASE_PORT: key doesn't exist, it will create that key for us.

Assume you want to see the output as well in the same command then add -o or --output flag with the type [json,name,yaml,etc].

$ kubectl patch configmap api01-configmap -n default -p '{"data":{"DATABASE_PORT":"5444"}}' -o yaml
apiVersion: v1
data:
  DATABASE_HOST: api01.ybdb.io
  DATABASE_NAME: api01
  DATABASE_PORT: "5444"
kind: ConfigMap
metadata:
  creationTimestamp: "2024-05-22T10:47:41Z"
  name: api01-configmap
  namespace: default
  resourceVersion: "69416"
  uid: a303bd0e-7f61-49c0-b671-06649ecdf999

But there is another catch, you can also not run the patch command +100 times, I mean definitely you can but again that might be error-prone too.

So, in that case we can write a simple reusable bash script that can do the job for us and can help us to update +100 configmaps in one go.

It can be in any language but I have chosen bash, since it's pretty common.

Quick rundown of the script:

Here I have three variables, NAMESPACE (where the object resides), CONFIGMAP_FILE (input file which contains a list of all the configmaps that need to be updated), DATA (a set of key:value pair which needs to be replaced/added).
Here I am creating log files as well (update_cm.log - contains generic readable information, patch.log - contains output of patch command).
As of now, I just have added action as add but we can have other actions as well like replace or remove (based on our requirements).

#!/bin/bash

NAMESPACE="default"
CONFIGMAP_FILE="configmaps"
DATA='{"data":{"DATABASE_PORT":"5444"}}'

rm patch.log update_cm.log

if [ ! -f "$CONFIGMAP_FILE" ]; then
  echo "File $CONFIGMAP_FILE not found."
  exit 1
fi

echo `date` >> update_cm.log
echo "----------------------------" >> update_cm.log

action=$1

if [ "$action" == "add" ]; then
  echo "Action: $action"
  echo "Action: $action" >> update_cm.log
  while IFS= read -r cm; do
    echo "Updating ConfigMap: $cm"
    echo "Updating ConfigMap: $cm" >> update_cm.log
    kubectl patch configmap "$cm" -n "$NAMESPACE" -p $DATA >> patch.log 2>&1
  done < "$CONFIGMAP_FILE"

else
  echo "Invalid action, please specify add".
  exit 1
fi

echo "----------------------------" >> update_cm.log

Feel free to use it, tweak it, based on your requirements. This script is for configmaps but similar logic we can write for other objects as well if required.

Happy learning :)

References:

Google.
Kubernetes docs (https://kubernetes.io/docs/home/).
kubectl explain

Disclaimer:

We should not manually update/patch the Kubernetes objects, this should be done when absolutely necessary or needed on the fly.
Ideally these all values should come from the version control system wherein the values files are maintained.

PS: This works with any flavor of Kubernetes minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.

Installing multiple helm charts in one go [Approach 2 - using helmfile]

Sunny Bhambhani — Tue, 26 Dec 2023 13:09:52 +0000

In this article we will be talking about Approach 2 i.e. how to get multiple helm charts installed using helmfile.

If you haven't read the previous article where I discussed about Approach 1, feel free to read it over.

Ref: https://dev.to/aws-builders/installing-multiple-helm-charts-in-one-go-approach-1-5d1p

Prerequisites:

A running Kubernetes cluster with proper permissions, here I have used minikube.
kubectl, helm, and helmfile installed on your machine to interact with the Kubernetes cluster.

Installation of helmfile is pretty straight forward, we just need to grab its binary and put that in /usr/local/bin and we are done (Ref: https://helmfile.readthedocs.io/en/latest/#installation for operating system other than linux).

Grab the latest one from the assets section from here: https://github.com/helmfile/helmfile/releases.

sudo wget https://github.com/helmfile/helmfile/releases/download/v0.159.0/helmfile_0.159.0_linux_amd64.tar.gz
sudo tar -xxf helmfile_0.159.0_linux_amd64.tar.gz
sudo rm helmfile_0.159.0_linux_amd64.tar.gz
sudo mv helmfile /usr/local/bin/

Once helmfile is installed, we need to initialize it so that it can download necessary helm plugins.

helmfile init

If you want to list it out what all plugins init installs, it can be found using below command:

$ helm plugin list
NAME            VERSION DESCRIPTION
diff            3.8.1   Preview helm upgrade changes as a diff
helm-git        0.12.0  Get non-packaged Charts directly from Git.
s3              0.14.0  Provides AWS S3 protocol support for charts and repos. https://github.com/hypnoglow/helm-s3
secrets         4.1.1   This plugin provides secrets values encryption for Helm charts secure storing

Note: A notice to upgrade helm may appear if the helm version is 3.10 or lower.

helm version is too low, the current version is 3.10.1+g9f88ccb, the required version is 3.12.3
use: 'https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3' [y/n]: y

Approach 2:

helmfile is pretty intresting and a very useful utility when it comes to get multiple helm charts installed. Using a single file helmfile.yaml we can manage N number of charts.

When it comes to installing mutiple helm charts, helmfile is a really helpful and interesting tool. The helmfile.yaml values file and helmfile binary is all that is needed to manage N number of charts. Internally helmfile invokes helm command to manage Kubernetes objects/charts. Also you can think of helmfile as a wrapper around helm together with some helm plugins.

Below is a quick example, wherein I have a 3-Tier application comprising of a frontend, backend, and database and for all of them I have 3 individual charts.

$ tree
.
├── backend
│   ├── Chart.yaml
│   ├── templates
│   │   ├── _helpers.tpl
│   │   ├── deployment.yaml
│   │   ├── hpa.yaml
│   │   ├── ingress.yaml
│   │   ├── service.yaml
│   │   └── serviceaccount.yaml
│   └── values.yaml
├── database
│   ├── Chart.yaml
│   ├── templates
│   │   ├── _helpers.tpl
│   │   ├── deployment.yaml
│   │   ├── hpa.yaml
│   │   ├── ingress.yaml
│   │   ├── service.yaml
│   │   └── serviceaccount.yaml
│   └── values.yaml
└── webapp
    ├── Chart.yaml
    ├── templates
    │   ├── _helpers.tpl
    │   ├── deployment.yaml
    │   ├── hpa.yaml
    │   ├── ingress.yaml
    │   ├── service.yaml
    │   └── serviceaccount.yaml
    └── values.yaml

6 directories, 24 files

Now, we will use helmfile (that too a single command to get these charts installed).

In the same directory where my charts are there, lets create a new file called helmfile.yaml. Just to add here we are referring local charts now, but we will refer charts from helm repositories in next examples.

Location: Local helm charts

$ cat helmfile.yaml
releases:
  - name: webapp
    namespace: default
    chart: ./webapp
    version: "0.1.0"
    wait: true
    installed: true
  - name: backend
    namespace: default
    chart: ./backend
    wait: true
  - name: database
    namespace: default
    chart: ./database
    wait: true

Lets break this file now:

There can be multiple sections in this YAML file, but majorly we will focus on .releases and .repositories sections.
For more details around what all things we can have in a helmfile.yaml, ref: https://helmfile.readthedocs.io/en/latest/#configuration.
.releases is an array which basically signifies what all helm releases we need to deploy in a Kubernetes cluster.
- Important KV pairs in each item are .releases[].name which refers to the name of the release.
- .releases[].namespace, it refers to the name of the namespace where we need to install the chart.
- .releases[].chart, it refers to the location of the chart from where helm needs to pull it. In this example its in my local machine and in the current directory.
- .releases[].version, it refers to the version of chart which needs to be installed.
- .releases[].installed, it is a boolean key which referes if we want to get specific chart installed or not (by default the value is true).
- .releases[].wait, it is same as --wait flag of helm, which means the operation will wait untill all the components are ready/healthy. Ref: https://helm.sh/docs/intro/using_helm/.

Now, lets see helmfile in action:

As of now nothing is installed.

$ helm list -A
NAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION
$

We will first initialize helm to gather all the dependencies.

$ helmfile init
helmfile initialization completed!
$

Lets list out what all releases are defined in our release file?

$ helmfile list
NAME            NAMESPACE       ENABLED INSTALLED       LABELS  CHART           VERSION
webapp          default         true    true                    ./webapp        0.1.0
backend         default         true    true                    ./backend
database        default         true    true                    ./database
$

All, looks good, lets apply the state file using helmfile apply.

$ helmfile apply
Building dependency release=webapp, chart=webapp
Building dependency release=backend, chart=backend
Building dependency release=database, chart=database
Comparing release=webapp, chart=webapp
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
default, webapp, Deployment (apps) has been added:
-
+ # Source: webapp/templates/deployment.yaml
+ apiVersion: apps/v1

#.........
#.........
# OUTPUT TRIMMED

UPDATED RELEASES:
NAME       CHART        VERSION   DURATION
webapp     ./webapp     0.1.0           3s
database   ./database   0.1.0           3s
backend    ./backend    0.1.0           3s

If you will see here, using a single command it instructed helm to build the dependency of all the charts specified in the state file then it used helm diff to list out actual k8s yaml files, then it installs all the releases.
If you will fire helm list or kubectl get pods now, you will see all the intended resources.

$ helm list -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
backend         default         1               2023-12-11 16:13:24.769966852 +0530 IST deployed        backend-0.1.0   1.16.0
database        default         1               2023-12-11 16:13:24.771955874 +0530 IST deployed        database-0.1.0  1.16.0
webapp          default         1               2023-12-11 16:13:24.771767015 +0530 IST deployed        webapp-0.1.0    1.16.0
$ k get pods
NAME                       READY   STATUS    RESTARTS   AGE
backend-7f458d4566-zwcz2   1/1     Running   0          3m47s
database-b4f679788-z6r84   1/1     Running   0          3m47s
webapp-7f6ffdc676-g5w7j    1/1     Running   0          3m47s
$

Get the status using helmfile status if required.

$ helmfile status
Getting status backend
Getting status webapp
Getting status database
NAME: backend
LAST DEPLOYED: Mon Dec 11 16:13:24 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

NAME: webapp
LAST DEPLOYED: Mon Dec 11 16:13:24 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

NAME: database
LAST DEPLOYED: Mon Dec 11 16:13:24 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Now, lets assume you just need to update the database release and that too just the replicaCount,
- How to update that using values file or using set key?
- What helmfile will do?
- How to check what all changes helmfile will make?

By default, similar to helm, this will also use helm chart's default values.yaml file which is a part of helm chart itself. So either you can update that and fulfill your requirement (which idelly is not a good practice, since in real world scenario that is most unlikely to happen and ideally we should to override intended keys).

To cater this need we have something called values and set which we can specify in the state file and helmfile will consider those values and will override them.

Lets use the same example, but now I have added few additional line for backend as well as for the database item in our releases array.

releases:
  - name: webapp
    namespace: default
    chart: ./webapp
    version: "0.1.0"
    wait: true
    installed: true
  - name: backend
    namespace: default
    chart: ./backend
    wait: true
    set:
      - name: replicaCount
        value: 2
  - name: database
    namespace: default
    chart: ./database
    wait: true
    values:
      - "./db-values.yaml"

$ cat db-values.yaml
replicaCount: 2

Just to cut short, values is same as --values and set is same as --set in helm.
Here if you will see, for backend I have used set and I am overwriding relicaCount to 2 and for database I have used 'values' and there I have passed a path where my values file resides which contains again the replicaCount as 2 but for database release.
Now, comes the 2nd question, how to see what all differences will helmfile make, simply fire helmfile diff.

$ helmfile diff
Building dependency release=webapp, chart=webapp
Building dependency release=backend, chart=backend
Building dependency release=database, chart=database
Comparing release=webapp, chart=webapp
Comparing release=backend, chart=backend
default, backend, Deployment (apps) has changed:
  # Source: backend/templates/deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: backend
# ......
# OUTPUT TRIMMED

-   replicas: 1
+   replicas: 2
# ......
# OUTPUT TRIMMED

Comparing release=database, chart=database
default, database, Deployment (apps) has changed:
  # Source: database/templates/deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: database
# ......
# OUTPUT TRIMMED

-   replicas: 1
+   replicas: 2
# ......
# OUTPUT TRIMMED

Here, if you will observe, firstly it compared the release webapp (wherein it didn't found any differences then it compared backend where it found that the replicaCount was updated to 2) then it compared database where it again found that the replicaCount was updated to 2.
Using helmfile apply, we will apply the changes now.

Note: helmfile apply basically calls two operations helmfile diff and helmfilesync.

Again it will show you what all changes it will bring then will apply the changes.

$ helmfile apply
Building dependency release=webapp, chart=webapp
Building dependency release=backend, chart=backend
Building dependency release=database, chart=database
Comparing release=webapp, chart=webapp
default, backend, Deployment (apps) has changed:
# ..........
# OUTPUT TRIMMED

-   replicas: 1
+   replicas: 2

default, database, Deployment (apps) has changed:
# ..........
# OUTPUT TRIMMED

-   replicas: 1
+   replicas: 2

Upgrading release=database, chart=database
Upgrading release=backend, chart=backend
Release "backend" has been upgraded. Happy Helming!
NAME: backend
LAST DEPLOYED: Mon Dec 11 16:37:27 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

Listing releases matching ^backend$
Release "database" has been upgraded. Happy Helming!
NAME: database
LAST DEPLOYED: Mon Dec 11 16:37:27 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

UPDATED RELEASES:
NAME       CHART        VERSION   DURATION
backend    ./backend    0.1.0           3s
database   ./database   0.1.0           3s

$ helm list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
backend         default         2               2023-12-11 16:37:27.366097438 +0530 IST deployed        backend-0.1.0   1.16.0
database        default         2               2023-12-11 16:37:27.366586959 +0530 IST deployed        database-0.1.0  1.16.0
webapp          default         1               2023-12-11 16:13:24.771767015 +0530 IST deployed        webapp-0.1.0    1.16.0

Revision got incremented by 1, and now I have 2 new pods running +1 for backend and +1 for database.

$ k get pods
NAME                       READY   STATUS    RESTARTS   AGE
backend-7f458d4566-6jgxk   1/1     Running   0          12s
backend-7f458d4566-zwcz2   1/1     Running   0          24m
database-b4f679788-9kxhl   1/1     Running   0          12s
database-b4f679788-z6r84   1/1     Running   0          24m
webapp-7f6ffdc676-g5w7j    1/1     Running   0          24m

In the above example we referred the helm charts which were placed locally, now lets see how to consume the helm charts which are a part of OCI or HTTPS repositories.

Location: helm charts from OCI repositories

Let's now install a helm chart from an opensource oci repository (ref: https://artifacthub.io/); Just for an example we will install nginx chart.
Again, I have a new helmfile.yaml and the only difference here is in the chart key.

releases:
  - name: nginx
    namespace: default
    chart: oci://registry-1.docker.io/bitnamicharts/nginx
    version: "15.4.4"
    wait: true
    set:
      - name: service.type
        value: ClusterIP

Here, if you will observe firstly helmfile is pulling the helmchart then will list out the difference i.e. what all things it will create and then will get the k8s resources created.

$ helmfile apply
Pulling registry-1.docker.io/bitnamicharts/nginx:15.4.4
Comparing release=nginx, chart=/tmp/helmfile976774944/default/nginx/nginx/15.4.4/nginx
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
default, nginx, Deployment (apps) has been added:
# ......
# ......
# OUTPUT TRIMMED

Location: helm charts from HTTPS repositories

Let's now install a helm chart from an opensource HTTPS repository and here I have taken an example from the official helmfile documentation (ref: https://helmfile.readthedocs.io/en/latest/#getting-started)
Here is the new helmfile.yaml and now the difference here is .repositories and .releases[].chart.

repositories:
 - name: prometheus-community
   url: https://prometheus-community.github.io/helm-charts

releases:
- name: prom-norbac-ubuntu
  namespace: prometheus
  chart: prometheus-community/prometheus
  set:
  - name: rbac.create
    value: false

Here, if you will observe in this helmfile we have a new section called .repositories which again is an array and can contain multiple repositories with a human friendly name which can be referred in the .releases section. For instance here the repo is https://prometheus-community.github.io/helm-charts and it can be referred in the releases section as prometheus-community.
And in-turn .releases[].chart now is prometheus-community/prometheus where prometheus is the chart name. When installing many charts from the same source, this eliminates duplication, and if we only need to change the repository, the update would only be needed at one place and we are done.
Another point to note here is, adding .repositories is similar to running helm repo add prometheus-communit https://prometheus-community.github.io/helm-charts this command.

$ helmfile apply
Adding repo prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

Comparing release=prom-norbac-ubuntu, chart=prometheus-community/prometheus
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
prometheus, prom-norbac-ubuntu-alertmanager, ConfigMap (v1) has been added:
# ......
# ......
# OUTPUT TRIMMED

Repositories

We already touched upon this topic but just for completeness sake similar to above example we can have OCI repositories as well defined under .repositories section.
Here is an example for the same:

repositories:
  - name: ocirepo
    url: registry-1.docker.io/bitnamicharts
    oci: true
releases:
  - name: nginx
    namespace: default
    chart: ocirepo/nginx
    version: 15.4.4
    wait: true
    set:
      - name: service.type
        value: ClusterIP

If you are referring an oci repository make sure to use repositories[].oci as true.

$ helmfile apply
Pulling registry-1.docker.io/bitnamicharts/nginx:15.4.4
Comparing release=nginx, chart=/tmp/helmfile4237214626/default/nginx/nginx/15.4.4/nginx
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
default, nginx, Deployment (apps) has been added:
# ......
# ......
# OUTPUT TRIMMED

Feel free to read it over and share your thoughts and comments.

Merry Christmas and a prosperous New Year to everyone! :)

Happy Learning!

PS: The cover image is created using canva.

References:

https://helmfile.readthedocs.io/en/latest/

Installing multiple helm charts in one go [Approach 1 - using parent/child charts]

Sunny Bhambhani — Fri, 01 Dec 2023 20:42:32 +0000

Helm is really a very powerful tool for managing Kubernetes objects and is widely adopted across various organizations. It is truly a game changer on how Kubernetes objects are being managed.

With a single command, we can install or upgrade multiple related Kubernetes entities together, we need not to worry about how the resources will be created, Helm will do all the heavy lifting for us.

$ helm install [RELEASE_NAME] [REPO]/[CHART]

But the below post is not about the basics of helm or how to install or upgrade a chart, etc. It's about some new ways of getting charts deployed.

Let's say you have a requirement for getting more than 100 charts installed on a specific Kubernetes cluster, what would you do in this scenario?

On top of my head, I can think of a couple of approaches:

Prerequisites:

A running Kubernetes cluster with proper permissions, here I have used minikube.
kubectl and helm installed on your machine to interact with the Kubernetes cluster.
- Ref: https://kubernetes.io/docs/tasks/tools/
- Ref: https://helm.sh/docs/intro/install/

Approach 1:

Club inter-related charts into one, as in getting a parent/wrapper chart created which will in turn have the related child charts as dependency. Which mean if we install the parent/wrapper chart it will get all the Kubernetes resources of all the child charts deployed (provided they are marked as enabled in Chart.yaml).

Below is a quick example, wherein I have a 3-Tier application named app01.

If you notice the output of both the commands in the parent's Chart.yaml I have all three charts listed as dependencies and in the other one, individual charts have their own Kubernetes resources.

$ cat Chart.yaml
apiVersion: v2
name: app01
description: A Helm chart for app01 which contains FE, BE, DB.
version: 0.1.0
appVersion: 1.0.0
dependencies:
  - name: webapp
    version: "0.1.0"
    repository: "file://./charts/webapp"
    enabled: true
  - name: backend
    version: "0.1.0"
    repository: "file://./charts/backend"
    enabled: true
  - name: database
    version: "0.1.0"
    repository: "file://./charts/database"
    enabled: true

$ tree
.
├── Chart.yaml
├── charts
│   ├── backend
│   │   ├── Chart.yaml
│   │   ├── charts
│   │   ├── templates
│   │   │   ├── _helpers.tpl
│   │   │   ├── deployment.yaml
│   │   │   ├── hpa.yaml
│   │   │   ├── ingress.yaml
│   │   │   ├── service.yaml
│   │   │   └── serviceaccount.yaml
│   │   └── values.yaml
│   ├── database
│   │   ├── Chart.yaml
│   │   ├── charts
│   │   ├── templates
│   │   │   ├── _helpers.tpl
│   │   │   ├── deployment.yaml
│   │   │   ├── hpa.yaml
│   │   │   ├── ingress.yaml
│   │   │   ├── service.yaml
│   │   │   └── serviceaccount.yaml
│   │   └── values.yaml
│   └── webapp
│       ├── Chart.yaml
│       ├── charts
│       ├── templates
│       │   ├── _helpers.tpl
│       │   ├── deployment.yaml
│       │   ├── hpa.yaml
│       │   ├── ingress.yaml
│       │   ├── service.yaml
│       │   └── serviceaccount.yaml
│       └── values.yaml
└── values.yaml

10 directories, 26 files

When we install the parent chart it will get all the Kubernetes resources deployed.

$ helm install app01 ./app01/
NAME: app01
LAST DEPLOYED: Sat Dec  2 00:14:50 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1

$ helm list -A
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
app01   default         1               2023-12-02 00:14:50.075444065 +0530 IST deployed        app01-0.1.0     1.0.0

$ k get all
NAME                                  READY   STATUS    RESTARTS   AGE
pod/app01-backend-965cb9b5b-q7gqv     1/1     Running   0          17m
pod/app01-database-56bdbcd49c-bh4s2   1/1     Running   0          17m
pod/app01-webapp-5b67db8bdf-m6psg     1/1     Running   0          17m

NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/app01-backend    ClusterIP   10.107.53.8      <none>        8080/TCP  17m
service/app01-database   ClusterIP   10.104.173.124   <none>        3306/TCP  17m
service/app01-webapp     ClusterIP   10.111.168.18    <none>        80/TCP    17m
service/kubernetes       ClusterIP   10.96.0.1        <none>        443/TCP   8d

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app01-backend    1/1     1            1           17m
deployment.apps/app01-database   1/1     1            1           17m
deployment.apps/app01-webapp     1/1     1            1           17m

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/app01-backend-965cb9b5b     1         1         1       17m
replicaset.apps/app01-database-56bdbcd49c   1         1         1       17m
replicaset.apps/app01-webapp-5b67db8bdf     1         1         1       17m

Another thing to note here is now we have 4 values.yaml files (one for each child chart and one for the parent helm chart).
And the good part here is using just one parent's values.yaml file we can manage all 3 of them.
Make sure in parent's values.yaml, the child chart's values are indented correctly, and they are indented under specific chart name as mentioned in Chart.yaml under .dependencies[*].name

$ cat values.yaml
webapp:
  replicaCount: 2

database:
  replicaCount: 1

backend:
  replicaCount: 3

Here we are just updating the replicaCount of all child charts and are upgrading the release.

$ helm upgrade --install app01 ./app01/
Release "app01" has been upgraded. Happy Helming!
NAME: app01
LAST DEPLOYED: Sat Dec  2 01:21:56 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

As specified in parent chart values.yaml, the replicaCount is now increased to 2,3 respectively.

$ k get deploy; k get pods
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
app01-backend    3/3     3            3           105m
app01-database   1/1     1            1           105m
app01-webapp     2/2     2            2           105m
NAME                              READY   STATUS    RESTARTS   AGE
app01-backend-965cb9b5b-dg6gq     1/1     Running   0          38m
app01-backend-965cb9b5b-q7gqv     1/1     Running   0          105m
app01-backend-965cb9b5b-z92c8     1/1     Running   0          38m
app01-database-56bdbcd49c-bh4s2   1/1     Running   0          105m
app01-webapp-5b67db8bdf-c25kc     1/1     Running   0          38m
app01-webapp-5b67db8bdf-m6psg     1/1     Running   0          105m

As you see here with a single command, we deployed multiple helm charts and using a single values.yaml file we have the ability to manage all of them.

Approach 2:

Using helmfile.

Approach 3:

Using bash or any other scripting language (I am planning to write a simple utility to accomplish this).

Further, I am in the process of writing another article around Approach 2 and 3, plus there can be multiple other approaches for getting this accomplished (feel free to share your thoughts and comments).

Happy learning :)

PS: The cover image is created using canva.

Bridging the Gap: Leveraging Secret Store CSI Drivers to Access Secrets from Google Secret Manager in GKE Cluster

Sunny Bhambhani — Thu, 08 Jun 2023 04:00:03 +0000

Today, preserving and securely storing secrets like API keys, passwords, and certificates is an essential part of creating and deploying applications.

Google Secret Manager provides a powerful and centralized solution for secret management, as well as a secure vault to store and access sensitive information.

However, getting these secrets and seamlessly integrating them into applications might be difficult. This post will detail this process and demonstrate how simple it is to consume secrets from Google Secret Manager into a GKE cluster.

Before we get started there are couple of pre-requisites:

Make sure the GKE cluster which you are creating should have WORKLOAD_IDENTITY enabled.
Make sure you have a IAM SERVICE_ACCOUNT created with proper permissions.
One which we are most interested in is "Secret Manager Secret Accessor(roles/secretmanager.secretAccessor). Refer this for more details.
gcloud command line gives a great way to quickly create and destroy resources on GCP.
In below command, replace CLUSTER_NAME with the name of the GKE cluster, PROJECT_ID with the name of the project where you want to create the GKE cluster, SERVICE_ACCOUNT with the name of the service account which contains the necessary permissions to interact with GCP resources (for example in this case it would be Google Secret Manager), MACHINE_TYPE with the size of machine to deploy your workloads since we would be installing couple of resources make sure you choose a medium machine, ZONE with the name of the zone, NUMBER_OF_NODES with an integer value stating minimum number of nodes per zone, CLUSTER_VERSION with the version of the GKE cluster you want to install.

gcloud container clusters create CLUSTER_NAME --workload-pool=PROJECT_ID.svc.id.goog --service-account=SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com --zone ZONE --machine-type MACHINE_TYPE --num-nodes=NUMBER_OF_NODES --cluster-version=CLUSTER_VERSION

gcloud container clusters create test-k8s \
    --workload-pool=proj-101.svc.id.goog \
    --service-account=test-k8s@proj-101.iam.gserviceaccount.com \
    --zone us-central1-a \
    --machine-type e2-standard-4 \
    --num-nodes=1 \
    --cluster-version=1.23.17-gke.5600

Install the Secret Store CSI Drivers, the simplest way is to get them installed via helm.
Additionally, if you want to create secret k8s resource as well make sure to enable syncSecret flag.

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

helm upgrade --install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system

Install the GCP provider plugin.

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp/main/deploy/provider-gcp-plugin.yaml

Make sure your service-account is binded with IAM GCP account
Use below command to add/check IAM policy binding.
Make sure to follow proper naming convention for service-account i.e. [NAMESPACE/SERVICE_ACCOUNT_NAME] for instance here we have [default/app-sa].

gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$proj-101.svc.id.goog[default/app-sa]" \
    test-k8s@proj-101.iam.gserviceaccount.com

Once all these things are done you are ready to use the secrets from Google Secret Manager and consume them in your deployments.

Use the same GCP IAM service account in k8s service account annotations.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: test-k8s@proj-101.iam.gserviceaccount.com
  name: app-sa

Create the k8s SecretProviderClass resource and have your path of secret appended to resourceName key (this is the one which will come from Google Secret Manager).
In this example, I already created a secret called testpassword in GSM.

---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: test-secret
spec:
  provider: gcp
  parameters:
    secrets: |
      - resourceName: "projects/proj-101/secrets/testpassword/versions/latest"
        fileName: "testpassword"
  secretObjects:
  - secretName: testpassword
    data:
    - key: testpassword
      objectName: testpassword
    type: Opaque

Create the pod/deployment and use your secrets in your application.
Here we are creating a volume item which is referring secrets-store.csi.k8s.io csi drivers and using the secretProviderClass which we created in above step.
Further, we are mounting this as a volume to be consumed.
Note: If you are planning to use the secret as an environment variable make sure the secret (k8s resource) is present.

---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  serviceAccountName: app-sa
  containers:
  - image: gcr.io/google.com/cloudsdktool/cloud-sdk:slim
    name: mypod
    resources:
      requests:
        cpu: 100m
    tty: true
    volumeMounts:
      - mountPath: "/var/secrets"
        name: vol-secret
        readOnly: true
  volumes:
  - name: vol-secret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "test-secret"

I hope you find this post useful, feel free to add your thoughts or comments.

Happy learning!

References: